Go (golang) http calls with retries and backoff

Related tags

HTTP Clients pester
Overview

pester

pester wraps Go's standard lib http client to provide several options to increase resiliency in your request. If you experience poor network conditions or requests could experience varied delays, you can now pester the endpoint for data.

  • Send out multiple requests and get the first back (only used for GET calls)
  • Retry on errors
  • Backoff

Simple Example

Use pester where you would use the http client calls. By default, pester will use a concurrency of 1, and retry the endpoint 3 times with the DefaultBackoff strategy of waiting 1 second between retries.

/* swap in replacement, just switch
   http.{Get|Post|PostForm|Head|Do} to
   pester.{Get|Post|PostForm|Head|Do}
*/
resp, err := pester.Get("http://sethammons.com")

Backoff Strategy

Provide your own backoff strategy, or use one of the provided built in strategies:

  • DefaultBackoff: 1 second
  • LinearBackoff: n seconds where n is the retry number
  • LinearJitterBackoff: n seconds where n is the retry number, +/- 0-33%
  • ExponentialBackoff: n seconds where n is 2^(retry number)
  • ExponentialJitterBackoff: n seconds where n is 2^(retry number), +/- 0-33%
client := pester.New()
client.Backoff = func(retry int) time.Duration {
    // set up something dynamic or use a look up table
    return time.Duration(retry) * time.Minute
}

Complete example

For a complete and working example, see the sample directory. pester allows you to use a constructor to control:

  • backoff strategy
  • retries
  • concurrency
  • keeping a log for debugging
package main

import (
    "log"
    "net/http"
    "strings"

    "github.com/sethgrid/pester"
)

func main() {
    log.Println("Starting...")

    { // drop in replacement for http.Get and other client methods
        resp, err := pester.Get("http://example.com")
        if err != nil {
            log.Println("error GETing example.com", err)
        }
        defer resp.Body.Close()
        log.Printf("example.com %s", resp.Status)
    }

    { // control the resiliency
        client := pester.New()
        client.Concurrency = 3
        client.MaxRetries = 5
        client.Backoff = pester.ExponentialBackoff
        client.KeepLog = true

        resp, err := client.Get("http://example.com")
        if err != nil {
            log.Println("error GETing example.com", client.LogString())
        }
        defer resp.Body.Close()
        log.Printf("example.com %s", resp.Status)
    }

    { // use the pester version of http.Client.Do
        req, err := http.NewRequest("POST", "http://example.com", strings.NewReader("data"))
        if err != nil {
            log.Fatal("Unable to create a new http request", err)
        }
        resp, err := pester.Do(req)
        if err != nil {
            log.Println("error POSTing example.com", err)
        }
        defer resp.Body.Close()
        log.Printf("example.com %s", resp.Status)
    }
}

Example Log

pester also allows you to control the resiliency and can optionally log the errors.

c := pester.New()
c.KeepLog = true

nonExistantURL := "http://localhost:9000/foo"
_, _ = c.Get(nonExistantURL)

fmt.Println(c.LogString())
/*
Output:

1432402837 Get [GET] http://localhost:9000/foo request-0 retry-0 error: Get http://localhost:9000/foo: dial tcp 127.0.0.1:9000: connection refused
1432402838 Get [GET] http://localhost:9000/foo request-0 retry-1 error: Get http://localhost:9000/foo: dial tcp 127.0.0.1:9000: connection refused
1432402839 Get [GET] http://localhost:9000/foo request-0 retry-2 error: Get http://localhost:9000/foo: dial tcp 127.0.0.1:9000: connection refused
*/

Tests

You can run tests in the root directory with $ go test. There is a benchmark-like test available with $ cd benchmarks; go test. You can see pester in action with $ cd sample; go run main.go.

For watching open file descriptors, you can run watch "lsof -i -P | grep main" if you started the app with go run main.go. I did this for watching for FD leaks. My method was to alter sample/main.go to only run one case (pester.Get with set backoff stategy, concurrency and retries increased) and adding a sleep after the result came back. This let me verify if FDs were getting left open when they should have closed. If you know a better way, let me know! I was able to see that FDs are now closing when they should :)

Are we there yet?

Are we there yet? Are we there yet? Are we there yet? Are we there yet? ...

Comments
  • Retry on 429 is not granular enough

    Retry on 429 is not granular enough

    I think #32 should be backed out.

    As far as I'm aware you can't configure per-status-code backoff strategies. The problem is that with a 429 further attempts to retry may actually keep counting against your rate limiter. So unlike a 503 where you may want to keep trying once a second, trying once a second on a 429 may mean that you are never able to successfully make a call.

    At the very least I think if #32 is kept in there needs to be a way to configure a separate backoff strategy for it. And if it is kept in, perhaps an additional boolean can be added to control its behavior.

    opened by jefferai 10
  • Feature Request: Can we have retry logic applied for HTTP 429 (Rate limiting)

    Feature Request: Can we have retry logic applied for HTTP 429 (Rate limiting)

    @sethgrid Please let me know your thoughts. We are planning to implement rate limiting on our vault cluster and envconsul, consul-template is one of our main tools to read secrets. As per RFC, we will return 429 in the event of rate limiting. With this feature, envconsul and consul template will retry if pester add 429 in its retry list.

    opened by pritamsarkar007 6
  • Re-use http.Request

    Re-use http.Request

    Hi! I've added small optimization and modules support. The main idea is to re-use http.Request created from params or provided directly and re-create only request.Body from originalBody for retries. This change simplifies flow. Also I've changed PostForm because it's mostly the same as Post and can be handled with input params without additional logic. Tests are passing and no breaking changes.

    opened by groovili 5
  • Reusing Pester client?

    Reusing Pester client?

    Assuming we only need 1 backoff/retry strategy, are we meant to create a new Pester client for every HTTP request?

    If so, it seems like this directly contradicts Go's net/http Client, as it also looks like every new Pester client has a fresh net/http Client by default.

    https://golang.org/pkg/net/http/#Client

    The Client's Transport typically has internal state (cached TCP connections), so Clients should be reused instead of created as needed. Clients are safe for concurrent use by multiple goroutines.

    opened by willtrking 5
  • Make retries cancelable

    Make retries cancelable

    Make retries cancelable

    When using a request.WithContext it is possible to cancel a long running request. However, when using a cancelable request with retries enabled, we want to quit the retry loop.

    This is impactful when using a conservative backoff strategy and a large number of retries.

    opened by stevenjackson 4
  • Project is missing a tagged release

    Project is missing a tagged release

    We are using pester because it does its job pretty well. But we miss a tag to pin our dependency manager to. How about releasing at least an alpha or beta tag?

    Ty for your work!

    opened by marcioAlmada 3
  • Test Improvements & LogHook Support.

    Test Improvements & LogHook Support.

    • With the way tests were split out in different package namespace. It was almost impossible to fork and send PR.

    • Use standard library methods to parse Host and Port instead of string substitutions. Example errors where the code would fail when not using standard Libaray:

    --- FAIL: TestCookiesJarPersistence (0.00s)
    	main_test.go:186: unable to start cookie server unable to determine port strconv.Atoi: parsing "0.0.0.0:36515": invalid syntax
    === RUN   TestEmbeddedClientTimeout
    --- FAIL: TestEmbeddedClientTimeout (0.00s)
    	main_test.go:216: unable to start timeout server unable to determine port strconv.Atoi: parsing "0.0.0.0:36293": invalid syntax
    === RUN   TestConcurrentRequestsNotRacyAndDontLeak_FailedRequest
    --- FAIL: TestConcurrentRequestsNotRacyAndDontLeak_FailedRequest (0.00s)
    	main_test.go:234: unable to start server unable to determine port strconv.Atoi: parsing "0.0.0.0:42805": invalid syntax
    
    • Add support for LogHooks to emit failures line-by-line
    opened by meson10 3
  • Leaking goroutines

    Leaking goroutines

    We are seeing goroutine leakage when trying to use latest pester. We suspect that this goroutine might not exit https://github.com/sethgrid/pester/blob/master/main.go#L287-L304 The latest version that does not have the goroutine leakage is ce7da0d5dbbb1872da10e76d3cd7bfd7a5e0e8b3.

    Thanks for looking into this.

    opened by mariash 3
  • Race when setting successful request numbers

    Race when setting successful request numbers

    We are trying to update to newer version of pester and seeing an issue when running our tests:

    WARNING: DATA RACE
    Write by goroutine 38:
      github.com/sethgrid/pester.(*Client).pester()
          /go/src/github.com/sethgrid/pester/main.go:309 +0xe06
      github.com/sethgrid/pester.(*Client).Do()
          /go/src/github.com/sethgrid/pester/main.go:349 +0x132
    

    It looks like setting these variable is not threadsafe:

    https://github.com/sethgrid/pester/blob/master/main.go#L308-L309

    opened by mariash 3
  • Only close body when using concurrency

    Only close body when using concurrency

    Response body are closed so users of pester will not be able to consume the Body. This PR will make it so that closing is only done when using pester with concurrency.

    opened by nlundbo 3
  • How I set headers on Post?

    How I set headers on Post?

    With net/http I can use req.Header.Set("PRIVATE-TOKEN", "myPrivateToken") to set a private token on a header. But I'm trying to do the same on this example, with no success.

    What is the right way to set a header in this example?

    opened by rdeavila 2
  • Retry Connection reset?

    Retry Connection reset?

    When I am making GET requests using Pester, I sporadically get connection reset by peer errors:

    read tcp 10.24.8.45:37286->13.49.93.125:443 read: connection reset by peer
    

    I can detect these errors using:

    if opErr, ok := err.(*net.OpError); ok {
        if syscallErr, ok := opErr.Err.(*os.SyscallError); ok {
            if syscallErr.Err == syscall.ECONNRESET {
                fmt.Println("Found a ECONNRESET")
            }
        }
    }
    

    I would like Pester to retry these requests. What do you think?

    opened by StevenACoffman 3
  • socket fd leaks if dial timeout occur

    socket fd leaks if dial timeout occur

    req, err := http.NewRequest("POST", s.url, buffer)
    if err != nil {
    	return err
    }
    if s.auth != nil {
    	req.SetBasicAuth(s.auth.Login, s.auth.Password)
    }
    
    req.Header.Add("Content-Type", "text/xml; charset=\"utf-8\"")
    req.Header.Add("SOAPAction", soapAction)
    req.Header.Set("User-Agent", "gowsdl/0.1")
    req.Close = true
    
    tr := &http.Transport{
    	TLSClientConfig: &tls.Config{
    		InsecureSkipVerify: s.tls,
    	},
    	Dial: (&net.Dialer{
    		Timeout:   60 * time.Second,
    		KeepAlive: 10 * time.Second,
    	}).Dial,
    	TLSHandshakeTimeout:   5 * time.Second,
    	ResponseHeaderTimeout: 300 * time.Second,
    	ExpectContinueTimeout: 1 * time.Second,
    }
    
    client := &http.Client{Transport: tr}
    reliableClient := pester.NewExtendedClient(client)
    res, err := reliableClient.Do(req)
    if nil != res && nil != res.Body {
    	defer res.Body.Close()
    }
    if err != nil {
    	return err
    }
    

    If dial timeout occur, opened socket fd will leak. sudo lsof -n -a -p $(pidof processname) |grep "can't identify protocol" |wc -l You will see many 'can't identify protocol' socket.

    opened by trulyliu 4
Releases(v1.2.0)
Owner
Seth Ammons
Principal Software Developer in Montana. I love making systems scale.
Seth Ammons
Http client call for golang http api calls

httpclient-call-go This library is used to make http calls to different API services Install Package go get

pzenteno 16 Oct 7, 2022
Go library that makes it easy to add automatic retries to your projects, including support for context.Context.

go-retry Go library that makes it easy to add automatic retries to your projects, including support for context.Context. Example with context.Context

Conner Douglass 6 Aug 15, 2022
Retry - Efficient for-loop retries in Go

retry Package retry implements an efficient loop-based retry mechanism that allo

Roger Peppe 16 Aug 23, 2022
Retry, Race, All, Some, etc strategies for http.Client calls

reqstrategy Package reqstrategy provides functions for coordinating http.Client calls. It wraps typical call strategies like making simultaneous reque

Sergiy Yavorsky 11 Apr 30, 2021
a simple wrapper around resty to report HTTP calls metrics to prometheus

restyprom a simple wrapper around resty to report HTTP calls metrics to prometheus If you're using resty and want to have metrics of your HTTP calls,

Felipe Volpone 2 Sep 25, 2022
Speak HTTP like a local. (the simple, intuitive HTTP console, golang version)

http-gonsole This is the Go port of the http-console. Speak HTTP like a local Talking to an HTTP server with curl can be fun, but most of the time it'

mattn 65 Jul 14, 2021
fhttp is a fork of net/http that provides an array of features pertaining to the fingerprint of the golang http client.

fhttp The f stands for flex. fhttp is a fork of net/http that provides an array of features pertaining to the fingerprint of the golang http client. T

Flexagon 62 Nov 11, 2022
NATS HTTP Round Tripper - This is a Golang http.RoundTripper that uses NATS as a transport.

This is a Golang http.RoundTripper that uses NATS as a transport. Included is a http.RoundTripper for clients, a server that uses normal HTTP Handlers and any existing http handler mux and a Caddy Server transport.

R.I.Pienaar 81 Nov 13, 2022
Http-conection - A simple example of how to establish a HTTP connection using Golang

A simple example of how to establish a HTTP connection using Golang

Jonathan Gonzaga 0 Feb 1, 2022
Fast HTTP package for Go. Tuned for high performance. Zero memory allocations in hot paths. Up to 10x faster than net/http

fasthttp Fast HTTP implementation for Go. Currently fasthttp is successfully used by VertaMedia in a production serving up to 200K rps from more than

Aliaksandr Valialkin 18.7k Nov 22, 2022
Simple HTTP package that wraps net/http

Simple HTTP package that wraps net/http

Kris 0 Jan 17, 2022
httpreq is an http request library written with Golang to make requests and handle responses easily.

httpreq is an http request library written with Golang to make requests and handle responses easily. Install go get github.com/binalyze/http

Binalyze 56 Feb 10, 2022
A golang tool which makes http requests and prints the address of the request along with the MD5 hash of the response.

Golang Tool This repository is a golang tool which makes http requests to the external server and prints the address of the request along with the MD5

null 0 Oct 17, 2021
This is a simple single-host reverse proxy that intercept and save HTTP requests and responses

HTTP Telescope Debug HTTP requests using a reverse proxy. Description This is a simple single-host reverse proxy that intercept and save HTTP requests

Enrico 3 Mar 20, 2022
Goget will send a http request, and show the request time, status, response, and save response to a file

Goget will send a http request, and show the request time, status, response, and save response to a file

LAZPbanahaker 2 Feb 9, 2022
An enhanced http client for Golang

go-http-client An enhanced http client for Golang Documentation on go.dev ?? This package provides you a http client package for your http requests. Y

Furkan Bozdag 47 Nov 9, 2022
http client for golang

Request HTTP client for golang, Inspired by Javascript-axios Python-request. If you have experience about axios or requests, you will love it. No 3rd

Monaco.HappyHacking 223 Nov 19, 2022
A nicer interface for golang stdlib HTTP client

rq A nicer interface for golang stdlib HTTP client Documents rq: here client: here jar: here Why? Because golang HTTP client is a pain in the a... Fea

Ddo 42 Sep 27, 2022
HTTP mocking for Golang

httpmock Easy mocking of http responses from external resources. Install Currently supports Go 1.7 - 1.15. v1 branch has to be used instead of master.

Jared Morse 1.5k Nov 23, 2022