GoRequest -- Simplified HTTP client ( inspired by nodejs SuperAgent )

Overview

GoRequest

GoRequest -- Simplified HTTP client ( inspired by famous SuperAgent lib in Node.js )

GopherGoRequest

"Shooting Requests like a Machine Gun" - Gopher

Sending request has never been as fun nor easier than this. It comes with lots of features:

  • Get/Post/Put/Head/Delete/Patch/Options
  • Set - simple header setting
  • JSON - made it simple with JSON string as a parameter
  • Multipart-Support - send data and files as multipart request
  • Proxy - sending request via proxy
  • Timeout - setting timeout for a request
  • TLSClientConfig - taking control over tls where at least you can disable security check for https
  • RedirectPolicy
  • Cookie - setting cookies for your request
  • CookieJar - automatic in-memory cookiejar
  • BasicAuth - setting basic authentication header
  • more to come..

Installation

$ go get github.com/parnurzeal/gorequest

Documentation

See Go Doc or Go Walker for usage and details.

Status

Drone Build Status Travis Build Status

Why should you use GoRequest?

GoRequest makes thing much more simple for you, making http client more awesome and fun like SuperAgent + golang style usage.

This is what you normally do for a simple GET without GoRequest:

resp, err := http.Get("http://example.com/")

With GoRequest:

request := gorequest.New()
resp, body, errs := request.Get("http://example.com/").End()

Or below if you don't want to reuse it for other requests.

resp, body, errs := gorequest.New().Get("http://example.com/").End()

How about getting control over HTTP client headers, redirect policy, and etc. Things can quickly get more complicated in golang. You need to create a Client, set headers in a different command, ... just to do only one GET

client := &http.Client{
  CheckRedirect: redirectPolicyFunc,
}

req, err := http.NewRequest("GET", "http://example.com", nil)

req.Header.Add("If-None-Match", `W/"wyzzy"`)
resp, err := client.Do(req)

Why make things ugly while you can just do it as follows:

request := gorequest.New()
resp, body, errs := request.Get("http://example.com").
  RedirectPolicy(redirectPolicyFunc).
  Set("If-None-Match", `W/"wyzzy"`).
  End()

DELETE, HEAD, POST, PUT, PATCH are now supported and can be used in the same way as GET:

request := gorequest.New()
resp, body, errs := request.Post("http://example.com").End()
// PUT -> request.Put("http://example.com").End()
// DELETE -> request.Delete("http://example.com").End()
// HEAD -> request.Head("http://example.com").End()
// ANYTHING -> request.CustomMethod("TRACE", "http://example.com").End()

JSON

For a JSON POST with standard libraries, you might need to marshal map data structure to json format, set headers to 'application/json' (and other headers if you need to) and declare http.Client. So, your code becomes longer and harder to maintain:

m := map[string]interface{}{
  "name": "backy",
  "species": "dog",
}
mJson, _ := json.Marshal(m)
contentReader := bytes.NewReader(mJson)
req, _ := http.NewRequest("POST", "http://example.com", contentReader)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Notes","GoRequest is coming!")
client := &http.Client{}
resp, _ := client.Do(req)

Compared to our GoRequest version, JSON is for sure a default. So, it turns out to be just one simple line!:

request := gorequest.New()
resp, body, errs := request.Post("http://example.com").
  Set("Notes","gorequst is coming!").
  Send(`{"name":"backy", "species":"dog"}`).
  End()

Moreover, it also supports struct type. So, you can have a fun Mix & Match sending the different data types for your request:

type BrowserVersionSupport struct {
  Chrome string
  Firefox string
}
ver := BrowserVersionSupport{ Chrome: "37.0.2041.6", Firefox: "30.0" }
request := gorequest.New()
resp, body, errs := request.Post("http://version.com/update").
  Send(ver).
  Send(`{"Safari":"5.1.10"}`).
  End()

Not only for Send() but Query() is also supported. Just give it a try! :)

Callback

Moreover, GoRequest also supports callback function. This gives you much more flexibility on using it. You can use it any way to match your own style! Let's see a bit of callback example:

func printStatus(resp gorequest.Response, body string, errs []error){
  fmt.Println(resp.Status)
}
gorequest.New().Get("http://example.com").End(printStatus)

Multipart/Form-Data

You can specify the content-type of the request to type multipart to send all data as multipart/form-data. This feature also allows you to send (multiple) files! Check the examples below!

gorequest.New().Post("http://example.com/").
  Type("multipart").
  Send(`{"query1":"test"}`).
  End()

The SendFile function accepts strings as path to a file, []byte slice or even a os.File! You can also combine them to send multiple files with either custom name and/or custom fieldname:

          f, _ := filepath.Abs("./file2.txt")
bytesOfFile, _ := ioutil.ReadFile(f)

gorequest.New().Post("http://example.com/").
  Type("multipart").
  SendFile("./file1.txt").
  SendFile(bytesOfFile, "file2.txt", "my_file_fieldname").
  End()

Check the docs for SendFile to get more information about the types of arguments.

Proxy

In the case when you are behind proxy, GoRequest can handle it easily with Proxy func:

request := gorequest.New().Proxy("http://proxy:999")
resp, body, errs := request.Get("http://example-proxy.com").End()
// To reuse same client with no_proxy, use empty string:
resp, body, errs = request.Proxy("").Get("http://example-no-proxy.com").End()

Basic Authentication

To add a basic authentication header:

request := gorequest.New().SetBasicAuth("username", "password")
resp, body, errs := request.Get("http://example-proxy.com").End()

Timeout

Timeout can be set in any time duration using time package:

request := gorequest.New().Timeout(2*time.Millisecond)
resp, body, errs:= request.Get("http://example.com").End()

Timeout func defines both dial + read/write timeout to the specified time parameter.

EndBytes

Thanks to @jaytaylor, we now have EndBytes to use when you want the body as bytes.

The callbacks work the same way as with End, except that a byte array is used instead of a string.

resp, bodyBytes, errs := gorequest.New().Get("http://example.com/").EndBytes()

EndStruct

We now have EndStruct to use when you want the body as struct.

The callbacks work the same way as with End, except that a struct is used instead of a string.

Supposing the URL http://example.com/ returns the body {"hey":"you"}

heyYou struct {
  Hey string `json:"hey"`
}

var heyYou heyYou

resp, _, errs := gorequest.New().Get("http://example.com/").EndStruct(&heyYou)

Retry

Supposing you need retry 3 times, with 5 seconds between each attempt when gets a BadRequest or a InternalServerError

request := gorequest.New()
resp, body, errs := request.Get("http://example.com/").
                    Retry(3, 5 * time.Second, http.StatusBadRequest, http.StatusInternalServerError).
                    End()

Handling Redirects

Redirects can be handled with RedirectPolicy which behaves similarly to net/http Client's CheckRedirect function. Simply specify a function which takes the Request about to be made and a slice of previous Requests in order of oldest first. When this function returns an error, the Request is not made.

For example to redirect only to https endpoints:

request := gorequest.New()
resp, body, errs := request.Get("http://example.com/").
                    RedirectPolicy(func(req Request, via []*Request) error {
                      if req.URL.Scheme != "https" {
                        return http.ErrUseLastResponse
                      }
                    }).
                    End()

Clone

You can reuse settings of a Request by cloning it before making any requests. This can be useful if you wish to re-use the SuperAgent across multiple requests without worrying about concurrency or having too many Transports being created.

Clones will copy the same settings (headers, query, etc..), but will only shallow copy any "Data" given to it. They will also share the same Transport and http.Client.

baseRequest := gorequest.New()
// apply anything you want to these settings. Eg:
baseRequest.Timeout(10 * time.Millisecond).
  BasicAuth("user", "password")

// then reuse the base request elsewhere, cloning before modifying or using it.
resp, body, errs := baseRequest.Clone().Get("http://exmaple.com/").End()

Debug

For debugging, GoRequest leverages httputil to dump details of every request/response. (Thanks to @dafang)

You can just use SetDebug or environment variable GOREQUEST_DEBUG=0|1 to enable/disable debug mode and SetLogger to set your own choice of logger.

Thanks to @QuentinPerez, we can see even how gorequest is compared to CURL by using SetCurlCommand.

Noted

As the underlying gorequest is based on http.Client in most use cases, gorequest.New() should be called once and reuse gorequest as much as possible.

Contributing to GoRequest:

If you find any improvement or issue you want to fix, feel free to send me a pull request with testing.

Thanks to all contributors thus far:

Contributors
https://github.com/alaingilbert
https://github.com/austinov
https://github.com/coderhaoxin
https://github.com/codegoalie
https://github.com/dafang
https://github.com/davyzhang
https://github.com/dickeyxxx
https://github.com/figlief
https://github.com/fraenky8
https://github.com/franciscocpg
https://github.com/heytitle
https://github.com/hownowstephen
https://github.com/kemadz
https://github.com/killix
https://github.com/jaytaylor
https://github.com/na-ga
https://github.com/piotrmiskiewicz
https://github.com/pencil001
https://github.com/pkopac
https://github.com/quangbuule
https://github.com/QuentinPerez
https://github.com/smallnest
https://github.com/WaveCutz
https://github.com/xild
https://github.com/yangmls
https://github.com/6david9

Also, co-maintainer is needed here. If anyone is interested, please email me (parnurzeal at gmail.com)

Credits

  • Renee French - the creator of Gopher mascot
  • Wisi Mongkhonsrisawat for providing an awesome GoRequest's Gopher image :)

License

GoRequest is MIT License.

Issues
  • nil buffer fix

    nil buffer fix

    Fixed nil reader content regression introduced with PR #129.

    • Don't allow buffers with nil contents to be passed to http.NewRequest constructor.

      Fixes errors like:

      Get https://[SOME-WEBSITE]: stream error: stream ID 1; REFUSED_STREAM

    opened by jaytaylor 13
  • gorequest install problems

    gorequest install problems

    i am using ubuntu and muanlly install go in my home dir with GOPATH all set. and got this: package golang.org/x/net/publicsuffix: unrecognized import path "golang.org/x/net/publicsuffix"

    opened by csrgxtu 13
  • "Clone" the SuperAgent for reuse

    The aim of this is to allow a base gorequest to be re-used throughout someone's code base.

    Ideally, gorequest itself would be concurrent safe, but I think that would require much larger changes (and possibly API changes). This takes the approach of Cloning the base gorequest, so it can be safely used by multiple routines at the same time.

    In my scenario I run a web server. I want to be able to easily re-use a single gorequest base, that gets configured at startup (Eg. timeout, transport, auth, etc...). Adding the Clone method lets me do this safely without running into concurrency issues.

    While refactoring a bit a few other changes had to be made:

    1. The "Timeout" function points to the base gorequest. It also looks like it may have been written after http.Client.Timeout existed. So I switched it to that. Let me know if my assumption there is wrong.
    2. Try to "safely" manage any changes to http.Client & http.Transport. Ideally the user should not be allowed to edit the Client or Transport after calling Clone. However, that would break the API. So instead, I do a "best effort" clone of the Client and Transport so a clone cannot affect the base.

    An example is provided in the readme, with this code example:

    baseRequest := gorequest.New()
    // apply anything you want to these settings. Eg:
    baseRequest.Timeout(10 * time.Millisecond).
      BasicAuth("user", "password")
    
    // then reuse the base request elsewhere, cloning before modifying or using it.
    resp, body, errs := baseRequest.Clone().Get("http://exmaple.com/").End()
    

    Should fix, or provide a workaround for #147 - Concurrent write access to a map. Clone makes new Header & Query maps. Should solve this issue. #125 - Clone allows the underlying Client and transport to be re-used. #73 - Calling Clone on the base allows for re-use #27 - Should fix most of the concurrency issues if Clone is used.

    Let me know what you think!

    opened by fromkeith 11
  • Possibility to add Query params in form of struct.

    Possibility to add Query params in form of struct.

    Don'y You mind if i'll implement such thing? Its pretty obvious to send struct as query params too, due to already existing possibility to do this with POST (sendStruct). Why I'm asking - i have a structure which represents some parameters for GET request and what am i doing right now its marshalling data and sending them to Query function in form of string. And when Query unmarshalls it again. Its absolutely unnecessary overhead. Of course i can concatenate this string in cycle - but it would be really shitty way.

    So, what would You say?

    enhancement 
    opened by a-urth 8
  • Package broken with dependency

    Package broken with dependency "http2curl"

    Hello. I'm facing the below issue using de tag v0.2.15.

    Can I fix this in my app?

    ../github.com/parnurzeal/gorequest/gorequest.go:31:2: cannot find package "github.com/moul/http2curl" in any of:
    	/usr/local/go/src/github.com/moul/http2curl (from $GOROOT)
    	/go/src/github.com/moul/http2curl (from $GOPATH)
    

    It started today, before, was running perfect.

    Thanks,

    opened by tonytcb 6
  • queryMap or queryStruct should accept more value type than string

    queryMap or queryStruct should accept more value type than string

    func (s *SuperAgent) queryStruct(content interface{}) *SuperAgent {
    	if marshalContent, err := json.Marshal(content); err != nil {
    		s.Errors = append(s.Errors, err)
    	} else {
    		var val map[string]interface{}
    		if err := json.Unmarshal(marshalContent, &val); err != nil {
    			s.Errors = append(s.Errors, err)
    		} else {
    			for k, v := range val {
    				k = strings.ToLower(k)
    				s.QueryData.Add(k, v.(string))
    			}
    		}
    	}
    	return s
    }
    

    v.(string) always assume v is a string ,which is not always true in practice, I'd like to add some more basic types such as int,float64, etc, even struct with encoded json string

    If you guys ok with this feature, I can cook up a pr.

    opened by davyzhang 6
  • Add http2curl

    Add http2curl

    Hello,

    I added a method to display a CURL command line.

    $> cat curl.go
    package main
    
    import "github.com/parnurzeal/gorequest"
    
    func main() {
        gorequest.New().Get("https://www.google.com").SetCurlCommand(true).End()
        gorequest.New().Post("https://www.google.com").SetCurlCommand(true).Send("{\"foo\": \"bar\"}").End()
    }
    $> go run curl.go
    [curl] 2015/10/16 12:22:03 CURL command line: curl -X GET https://www.google.com
    [curl] 2015/10/16 12:22:03 CURL command line: curl -X POST -d "{\"foo\":\"bar\"}" -H "Content-Type: application/json" https://www.google.com
    
    opened by QuentinPerez 6
  • Add support for unmarshaling the response body into struct

    Add support for unmarshaling the response body into struct

    How about adding support for something like this:

    type Person struct {
      Name string
    }
    
    var friend Person
    response, _, errs := request.Post("http://example.com").BindBody(&friend).End()
    
    opened by brasilikum 6
  • Can't get *http.Request from *SuperAgent

    Can't get *http.Request from *SuperAgent

    In case to do testing, I need *http.Request but I can't get *http.Request from *SuperAgent.

    https://golang.org/pkg/net/http/httptest/#example_ResponseRecorder

    opened by quangbuule 5
  • Auto decompress body

    Auto decompress body

    Hi,

    Thank you for this useful package.

    Looks like now you simply return body even if it is compressed. Maybe you can automatically decompress body when you find Content-Encoding: gzip header.

    Go net/http is doing that already.

    opened by matrixik 5
  • Memory Leak?

    Memory Leak?

    code

    request := gorequest.New().Timeout(20 * time.Second)

    for {
        select {
        case value := <-checkpool:
            if value == "" {
                continue
            }
            proxyinfo := proxy(string(value))
            resp, _, err := request.Proxy(proxyinfo).Head(checkUrl).End()
            if err == nil {
                headers := resp.Header["Server"]
                if headers != nil {
                    serverHeader := headers[0]
                    if strings.Contains(serverHeader, bws) {
                        cleanpool <- value
                        fmt.Printf("%s %s success\n", name, value)
                    }
                }
            } else {
                fmt.Printf("%s %s failed %s\n", name, value, err)
            }
        }
    }
    

    I use go request like above,and the memory of progress keep larger,did my code cause a memory leak or it's a bug of gorequest

    opened by hybridstyle 5
  • can't send body

    can't send body "{}"

    I expected to send a body data like' {}' , but it failed , seemed like gorequest directly igonre body data if length of Data is 0, any method to fix this ?

    opened by 915898891 1
  • support for go modules

    support for go modules

    At the moment, when this package is imported, it creates indirect references in my projects go.mod.

    golang.org/x/net v0.0.0-20210614182718-04defd469f4e // indirect
    

    Is there any plans to use go.mod in future?

    opened by boddumanohar 0
  • dkron stops running job for no apparen reason

    dkron stops running job for no apparen reason

    We are running a three-node dkron cluster with 3.1.8, Periodically we find the jobs stop running. I went through the logs back to the time of last run, and found this error starting and running continously for the last four days:

    Aug 20 13:55:10 nj-dcos02-cl01 dkron: time="2021-08-20T13:55:10-04:00" level=error msg="job: Error quering for running executions" error="context deadline exceeded" node=nj-dcos02-cl01 Aug 20 13:55:10 nj-dcos02-cl01 dkron: time="2021-08-20T13:55:10-04:00" level=error msg="grpc: error dialing." error="context deadline exceeded" method=GetActiveExecutions node=nj-dcos02-cl01 server_addr

    What does this mean?

    opened by seanfulton 0
  • http_proxy env support

    http_proxy env support

    I think support http_proxy will be more nice. other language or lib auto read http_proxy and auto support it , so i must set proxy for gorequest.

    now, my case is that using http_proxy to proxy k8s cluster, so i can request k8s service on my mac. In prod, i never use proxy, so i can't set proxy hard code.

    opened by wanglihui 0
  • unexpected end of JSON input in req.EndStruct(body)

    unexpected end of JSON input in req.EndStruct(body)

    The below code gives unexpected end of JSON input. The URL will retrieve 2000 documents from Cloudant database.

    req := request.New().Get(client.Path + path) req.SetBasicAuth(client.Username, client.Password) req.Header.Add("Accept-Encoding", "application/json") req.Debug = true

    body := &cloudantLib.Resp{}
    resp, _, errs := req.EndStruct(body)
    
    Is there any limit in the response body ? When we try with small document size like 100, we don't get any error and it works fine. 
    
    opened by MalarvizhiK 0
Owner
Theeraphol Wattanavekin
Triple eyes pineapple who loves to meddle with other people codes.
Theeraphol Wattanavekin
Cake is a lightweight HTTP client library for GO, inspired by Java Open-Feign.

Cake is a lightweight HTTP client library for GO, inspired by Java Open-Feign. Installation # With Go Modules, recommanded with go version > 1.16

snown 4 Mar 4, 2022
Http client call for golang http api calls

httpclient-call-go This library is used to make http calls to different API services Install Package go get

pzenteno 15 Jun 29, 2022
fhttp is a fork of net/http that provides an array of features pertaining to the fingerprint of the golang http client.

fhttp The f stands for flex. fhttp is a fork of net/http that provides an array of features pertaining to the fingerprint of the golang http client. T

Flexagon 61 Jun 10, 2022
Fast HTTP package for Go. Tuned for high performance. Zero memory allocations in hot paths. Up to 10x faster than net/http

fasthttp Fast HTTP implementation for Go. Currently fasthttp is successfully used by VertaMedia in a production serving up to 200K rps from more than

Aliaksandr Valialkin 18.2k Aug 7, 2022
Speak HTTP like a local. (the simple, intuitive HTTP console, golang version)

http-gonsole This is the Go port of the http-console. Speak HTTP like a local Talking to an HTTP server with curl can be fun, but most of the time it'

mattn 65 Jul 14, 2021
NATS HTTP Round Tripper - This is a Golang http.RoundTripper that uses NATS as a transport.

This is a Golang http.RoundTripper that uses NATS as a transport. Included is a http.RoundTripper for clients, a server that uses normal HTTP Handlers and any existing http handler mux and a Caddy Server transport.

R.I.Pienaar 76 Jul 28, 2022
Simple HTTP package that wraps net/http

Simple HTTP package that wraps net/http

Kris 0 Jan 17, 2022
Http-conection - A simple example of how to establish a HTTP connection using Golang

A simple example of how to establish a HTTP connection using Golang

Jonathan Gonzaga 0 Feb 1, 2022
Full-featured, plugin-driven, extensible HTTP client toolkit for Go

gentleman Full-featured, plugin-driven, middleware-oriented toolkit to easily create rich, versatile and composable HTTP clients in Go. gentleman embr

Tom 961 Jul 27, 2022
An enhanced http client for Golang

go-http-client An enhanced http client for Golang Documentation on go.dev ?? This package provides you a http client package for your http requests. Y

Furkan Bozdag 41 Jul 14, 2022
An enhanced HTTP client for Go

Heimdall Description Installation Usage Making a simple GET request Creating a hystrix-like circuit breaker Creating a hystrix-like circuit breaker wi

Gojek 2.3k Aug 6, 2022
Enriches the standard go http client with retry functionality.

httpRetry Enriches the standard go http client with retry functionality using a wrapper around the Roundtripper interface. The advantage of this libra

Alexander Gehres 20 May 10, 2022
http client for golang

Request HTTP client for golang, Inspired by Javascript-axios Python-request. If you have experience about axios or requests, you will love it. No 3rd

Monaco.HappyHacking 207 Jul 28, 2022
Simple HTTP and REST client library for Go

Resty Simple HTTP and REST client library for Go (inspired by Ruby rest-client) Features section describes in detail about Resty capabilities Resty Co

Go Resty 6.5k Aug 8, 2022
A nicer interface for golang stdlib HTTP client

rq A nicer interface for golang stdlib HTTP client Documents rq: here client: here jar: here Why? Because golang HTTP client is a pain in the a... Fea

Ddo 40 Jan 23, 2022
A Go HTTP client library for creating and sending API requests

Sling Sling is a Go HTTP client library for creating and sending API requests. Slings store HTTP Request properties to simplify sending requests and d

Dalton Hubble 1.5k Jul 27, 2022
a Go HTTP client with timeouts

go-httpclient requires Go 1.1+ as of v0.4.0 the API has been completely re-written for Go 1.1 (for a Go 1.0.x compatible release see 1adef50) Provides

Matt Reiferson 290 Sep 7, 2021
gout to become the Swiss Army Knife of the http client @^^@---> gout 是http client领域的瑞士军刀,小巧,强大,犀利。具体用法可看文档,如使用迷惑或者API用得不爽都可提issues

gout gout 是go写的http 客户端,为提高工作效率而开发 构架 feature 支持设置 GET/PUT/DELETE/PATH/HEAD/OPTIONS 支持设置请求 http header(可传 struct,map,array,slice 等类型) 支持设置 URL query(可

guonaihong 1k Aug 1, 2022
Retry, Race, All, Some, etc strategies for http.Client calls

reqstrategy Package reqstrategy provides functions for coordinating http.Client calls. It wraps typical call strategies like making simultaneous reque

Sergiy Yavorsky 11 Apr 30, 2021