An efficient and feature complete Hystrix like Go implementation of the circuit breaker pattern.

Overview

Mascot

Circuit

Build Status GoDoc Coverage Status

Circuit is an efficient and feature complete Hystrix like Go implementation of the circuit breaker pattern. Learn more about the problems Hystrix and other circuit breakers solve on the Hystrix Wiki. A short summary of advantages are:

  • A downstream service failed and all requests hang forever. Without a circuit, your service would also hang forever. Because you have a circuit, you detect this failure quickly and can return errors quickly while waiting for the downstream service to recover.
  • Circuits make great monitoring and metrics boundaries, creating common metric names for the common downstream failure types. This package goes further to formalize this in a SLO tracking pattern.
  • Circuits create a common place for downstream failure fallback logic.
  • Downstream services sometimes fail entirely when overloaded. While in a degraded state, circuits allow you to push downstream services to the edge between absolute failure and mostly working.
  • Open/Close state of a circuit is a clear early warning sign of downstream failures.
  • Circuits allow you to protect your dependencies from abnormal rushes of traffic.

There are a large number of examples on the godoc that are worth looking at. They tend to be more up to date than the README doc.

Feature set

  • No forced goroutines
  • recoverable panic()
  • Integrated with context.Context
  • Comprehensive metric tracking
  • Efficient implementation with Benchmarks
  • Low/zero memory allocation costs
  • Support for Netflix Hystrix dashboards, even with custom circuit transition logic
  • Multiple error handling features
  • Expose circuit health and configuration on expvar
  • SLO tracking
  • Customizable state transition logic, allowing complex circuit state changes
  • Live configuration changes
  • Many tests and examples
  • Good inline documentation
  • Generatable interface wrapping support with https://github.com/twitchtv/circuitgen
  • Support for Additive increase/multiplicative decrease
  • Prometheus metrics collector.

Usage

Hello world circuit

This example shows how to create a hello-world circuit from the circuit manager

// Manages all our circuits
h := circuit.Manager{}
// Create a circuit with a unique name
c := h.MustCreateCircuit("hello-world")
// Call the circuit
errResult := c.Execute(context.Background(), func(ctx context.Context) error {
  return nil
}, nil)
fmt.Println("Result of execution:", errResult)
// Output: Result of execution: <nil>

Hello world fallback

This example shows how fallbacks execute to return alternate errors or provide logic when the circuit is open.

// You can create circuits without using the manager
c := circuit.NewCircuitFromConfig("hello-world-fallback", circuit.Config{})
errResult := c.Execute(context.Background(), func(ctx context.Context) error {
	return errors.New("this will fail")
}, func(ctx context.Context, err error) error {
	fmt.Println("Circuit failed with error, but fallback returns nil")
	return nil
})
fmt.Println("Execution result:", errResult)
// Output: Circuit failed with error, but fallback returns nil
// Execution result: <nil>

Running inside a Goroutine

It is recommended to use circuit.Execute and a context aware function. If, however, you want to exit your run function early and leave it hanging (possibly forever), then you can call circuit.Go.

h := circuit.Manager{}
c := h.MustCreateCircuit("untrusting-circuit", circuit.Config{
  Execution: circuit.ExecutionConfig{
    // Time out the context after a few ms
    Timeout: time.Millisecond * 30,
  },
})

errResult := c.Go(context.Background(), func(ctx context.Context) error {
  // Sleep 30 seconds, way longer than our timeout
  time.Sleep(time.Second * 30)
  return nil
}, nil)
fmt.Printf("err=%v", errResult)
// Output: err=context deadline exceeded

Hystrix Configuration

All configuration parameters are documented in config.go. Your circuit open/close logic configuration is documented with the logic. For hystrix, this configuration is in closers/hystrix and well documented on the Hystrix wiki.

This example configures the circuit to use Hystrix open/close logic with the default Hystrix parameters

configuration := hystrix.ConfigFactory{
  // Hystrix open logic is to open the circuit after an % of errors
  ConfigureOpener: hystrix.ConfigureOpener{
    // We change the default to wait for 10 requests, not 20, before checking to close
    RequestVolumeThreshold: 10,
    // The default values match what hystrix does by default
  },
  // Hystrix close logic is to sleep then check
  ConfigureCloser: hystrix.ConfigureCloser{
    // The default values match what hystrix does by default
  },
}
h := circuit.Manager{
  // Tell the manager to use this configuration factory whenever it makes a new circuit
  DefaultCircuitProperties: []circuit.CommandPropertiesConstructor{configuration.Configure},
}
// This circuit will inherit the configuration from the example
c := h.MustCreateCircuit("hystrix-circuit")
fmt.Println("This is a hystrix configured circuit", c.Name())
// Output: This is a hystrix configured circuit hystrix-circuit

Enable dashboard metrics

Dashboard metrics can be enabled with the MetricEventStream object. This example creates an event stream handler, starts it, then later closes the handler

// metriceventstream uses rolling stats to report circuit information
sf := rolling.StatFactory{}
h := circuit.Manager{
  DefaultCircuitProperties: []circuit.CommandPropertiesConstructor{sf.CreateConfig},
}
es := metriceventstream.MetricEventStream{
  Manager: &h,
}
go func() {
  if err := es.Start(); err != nil {
    log.Fatal(err)
  }
}()
// ES is a http.Handler, so you can pass it directly to your mux
http.Handle("/hystrix.stream", &es)
// ...
if err := es.Close(); err != nil {
  log.Fatal(err)
}
// Output:

Enable expvar

If you wanted to publish hystrix information on Expvar, you can register your manager.

h := circuit.Manager{}
expvar.Publish("hystrix", h.Var())

Custom metrics

Implement interfaces CmdMetricCollector or FallbackMetricCollector to know what happens with commands or fallbacks. Then pass those implementations to configure.

config := circuit.Config{
  Metrics: circuit.MetricsCollectors{
    Run: []circuit.RunMetrics{
      // Here is where I would insert my custom metric collector
    },
  },
}
circuit.NewCircuitFromConfig("custom-metrics", config)

Panics

Code executed with Execute does not spawn a goroutine and panics naturally go up the call stack to the caller. This is also true for Go, where we attempt to recover and throw panics on the same stack that calls Go. This example will panic, and the panic can be caught up the stack.

h := circuit.Manager{}
c := h.MustCreateCircuit("panic_up")

defer func() {
 r := recover()
 if r != nil {
   fmt.Println("I recovered from a panic", r)
 }
}()
c.Execute(context.Background(), func(ctx context.Context) error {
 panic("oh no")
}, nil)
// Output: I recovered from a panic oh no

Runtime configuration changes

Most configuration properties on the Hystrix Configuration page that say they are modifyable at runtime can be changed on the Circuit in a thread safe way. Most of the ones that cannot are related to stat collection.

This example shows how to update hystrix configuration at runtime.

// Start off using the defaults
configuration := hystrix.ConfigFactory{}
h := circuit.Manager{
  // Tell the manager to use this configuration factory whenever it makes a new circuit
  DefaultCircuitProperties: []circuit.CommandPropertiesConstructor{configuration.Configure},
}
c := h.MustCreateCircuit("hystrix-circuit")
fmt.Println("The default sleep window", c.OpenToClose.(*hystrix.Closer).Config().SleepWindow)
// This configuration update function is thread safe.  We can modify this at runtime while the circuit is active
c.OpenToClose.(*hystrix.Closer).SetConfigThreadSafe(hystrix.ConfigureCloser{
  SleepWindow: time.Second * 3,
})
fmt.Println("The new sleep window", c.OpenToClose.(*hystrix.Closer).Config().SleepWindow)
// Output:
// The default sleep window 5s
// The new sleep window 3s

Not counting early terminations as failures

If the context passed into a circuit function ends, before the circuit can finish, it does not count the circuit as unhealthy. You can disable this behavior with the IgnoreInterrupts flag.

This example proves that terminating a circuit call early because the passed in context died does not, by default, count as an error on the circuit. It also demonstrates setting up internal stat collection by default for all circuits

// Inject stat collection to prove these failures don't count
f := rolling.StatFactory{}
manager := circuit.Manager{
  DefaultCircuitProperties: []circuit.CommandPropertiesConstructor{
    f.CreateConfig,
  },
}
c := manager.MustCreateCircuit("don't fail me bro")
// The passed in context times out in one millisecond
ctx, cancel := context.WithTimeout(context.Background(), time.Millisecond)
defer cancel()
errResult := c.Execute(ctx, func(ctx context.Context) error {
  select {
  case <- ctx.Done():
    // This will return early, with an error, since the parent context was canceled after 1 ms
    return ctx.Err()
  case <- time.After(time.Hour):
    panic("We never actually get this far")
  }
}, nil)
rs := f.RunStats("don't fail me bro")
fmt.Println("errResult is", errResult)
fmt.Println("The error and timeout count is", rs.ErrTimeouts.TotalSum() + rs.ErrFailures.TotalSum())
// Output: errResult is context deadline exceeded
// The error and timeout count is 0

Configuration factories

Configuration factories are supported on the root manager object. This allows you to create dynamic configuration per circuit name.

You can use DefaultCircuitProperties to set configuration dynamically for any circuit

myFactory := func(circuitName string) circuit.Config {
  timeoutsByName := map[string]time.Duration{
    "v1": time.Second,
    "v2": time.Second * 2,
  }
  customTimeout := timeoutsByName[circuitName]
  if customTimeout == 0 {
    // Just return empty if you don't want to set any config
    return circuit.Config{}
  }
  return circuit.Config{
    Execution: circuit.ExecutionConfig{
      Timeout: customTimeout,
    },
  }
}

// Hystrix manages circuits with unique names
h := circuit.Manager{
  DefaultCircuitProperties: []circuit.CommandPropertiesConstructor{myFactory},
}
h.MustCreateCircuit("v1")
fmt.Println("The timeout of v1 is", h.GetCircuit("v1").Config().Execution.Timeout)
// Output: The timeout of v1 is 1s

StatsD configuration factory

A configuration factory for statsd is provided inside ./metrics/statsdmetrics

This example shows how to inject a statsd metric collector into a circuit.

// This factory allows us to report statsd metrics from the circuit
f := statsdmetrics.CommandFactory{
  SubStatter: &statsd.NoopClient{},
}

// Wire the statsd factory into the circuit manager
h := circuit.Manager{
  DefaultCircuitProperties: []circuit.CommandPropertiesConstructor{f.CommandProperties},
}
// This created circuit will now use statsd
h.MustCreateCircuit("using-statsd")
// Output:

Service health tracking

Most services have the concept of an SLA, or service level agreement. Unfortunantly, this is usually tracked by the service owners, which creates incentives for people to inflate the health of their service.

This Circuit implementation formalizes an SLO of the template "X% of requests will return faster than Y ms". This is a value that canont be calculated just by looking at the p90 or p99 of requests in aggregate, but must be tracked per request. You can define a SLO for your service, which is a time less than the timeout time of a request, that works as a promise of health for the service. You can then report per circuit not just fail/pass but an extra "healthy" % over time that counts only requests that resopnd quickly enough.

This example creates a SLO tracker that counts failures at less than 20 ms. You will need to provide your own Collectors.

sloTrackerFactory := responsetimeslo.Factory{
  Config: responsetimeslo.Config{
    // Consider requests faster than 20 ms as passing
    MaximumHealthyTime: time.Millisecond * 20,
  },
  // Pass in your collector here: for example, statsd
  CollectorConstructors: nil,
}
h := circuit.Manager{
  DefaultCircuitProperties: []circuit.CommandPropertiesConstructor{sloTrackerFactory.CommandProperties},
}
h.CreateCircuit("circuit-with-slo")

Not counting user error as a fault

Sometimes users pass invalid functions to the input of your circuit. You want to return an error in that case, but not count the error as a failure of the circuit. Use SimpleBadRequest in this case.

This example shows how to return errors in a circuit without considering the circuit at fault. Here, even if someone tries to divide by zero, the circuit will not consider it a failure even if the function returns non nil error.

c := circuit.NewCircuitFromConfig("divider", circuit.Config{})
divideInCircuit := func(numerator, denominator int) (int, error) {
  var result int
  err := c.Run(context.Background(), func(ctx context.Context) error {
    if denominator == 0 {
      // This error type is not counted as a failure of the circuit
      return &circuit.SimpleBadRequest{
        Err: errors.New("someone tried to divide by zero"),
      }
    }
    result = numerator / denominator
    return nil
  })
  return result, err
}
_, err := divideInCircuit(10, 0)
fmt.Println("Result of 10/0 is", err)
// Output: Result of 10/0 is someone tried to divide by zero

Benchmarking

This implementation is more efficient than go-hystrix in every configuration. It has comparable efficiency to other implementations, in most faster when running with high concurrency. Run benchmarks with make bench.

I benchmark the following alternative circuit implementations. I try to be fair and if there is a better way to benchmark one of these circuits, please let me know!

> make bench
cd benchmarking && go test -v -benchmem -run=^$ -bench=. . 2> /dev/null
goos: darwin
goarch: amd64
pkg: github.com/cep21/circuit/benchmarking
BenchmarkCiruits/cep21-circuit/Hystrix/passing/1-8       	 2000000	       896 ns/op	     192 B/op	       4 allocs/op
BenchmarkCiruits/cep21-circuit/Hystrix/passing/75-8      	 3000000	       500 ns/op	     192 B/op	       4 allocs/op
BenchmarkCiruits/cep21-circuit/Hystrix/failing/1-8       	10000000	       108 ns/op	       0 B/op	       0 allocs/op
BenchmarkCiruits/cep21-circuit/Hystrix/failing/75-8      	20000000	        82.5 ns/op	       0 B/op	       0 allocs/op
BenchmarkCiruits/cep21-circuit/Minimal/passing/1-8       	10000000	       165 ns/op	       0 B/op	       0 allocs/op
BenchmarkCiruits/cep21-circuit/Minimal/passing/75-8      	20000000	        87.7 ns/op	       0 B/op	       0 allocs/op
BenchmarkCiruits/cep21-circuit/Minimal/failing/1-8       	20000000	        64.4 ns/op	       0 B/op	       0 allocs/op
BenchmarkCiruits/cep21-circuit/Minimal/failing/75-8      	100000000	        19.6 ns/op	       0 B/op	       0 allocs/op
BenchmarkCiruits/cep21-circuit/UseGo/passing/1-8         	 1000000	      1300 ns/op	     256 B/op	       5 allocs/op
BenchmarkCiruits/cep21-circuit/UseGo/passing/75-8        	 5000000	       374 ns/op	     256 B/op	       5 allocs/op
BenchmarkCiruits/cep21-circuit/UseGo/failing/1-8         	 1000000	      1348 ns/op	     256 B/op	       5 allocs/op
BenchmarkCiruits/cep21-circuit/UseGo/failing/75-8        	 5000000	       372 ns/op	     256 B/op	       5 allocs/op
BenchmarkCiruits/GoHystrix/DefaultConfig/passing/1-8     	  200000	      8146 ns/op	    1001 B/op	      18 allocs/op
BenchmarkCiruits/GoHystrix/DefaultConfig/passing/75-8    	  500000	      2498 ns/op	     990 B/op	      20 allocs/op
BenchmarkCiruits/GoHystrix/DefaultConfig/failing/1-8     	  200000	      6299 ns/op	    1020 B/op	      19 allocs/op
BenchmarkCiruits/GoHystrix/DefaultConfig/failing/75-8    	 1000000	      1582 ns/op	    1003 B/op	      20 allocs/op
BenchmarkCiruits/rubyist/Threshold-10/passing/1-8        	 1000000	      1834 ns/op	     332 B/op	       5 allocs/op
BenchmarkCiruits/rubyist/Threshold-10/passing/75-8       	 2000000	       849 ns/op	     309 B/op	       4 allocs/op
BenchmarkCiruits/rubyist/Threshold-10/failing/1-8        	20000000	       114 ns/op	       0 B/op	       0 allocs/op
BenchmarkCiruits/rubyist/Threshold-10/failing/75-8       	 5000000	       302 ns/op	       0 B/op	       0 allocs/op
BenchmarkCiruits/gobreaker/Default/passing/1-8           	10000000	       202 ns/op	       0 B/op	       0 allocs/op
BenchmarkCiruits/gobreaker/Default/passing/75-8          	 2000000	       698 ns/op	       0 B/op	       0 allocs/op
BenchmarkCiruits/gobreaker/Default/failing/1-8           	20000000	        90.6 ns/op	       0 B/op	       0 allocs/op
BenchmarkCiruits/gobreaker/Default/failing/75-8          	 5000000	       346 ns/op	       0 B/op	       0 allocs/op
BenchmarkCiruits/handy/Default/passing/1-8               	 2000000	      1075 ns/op	       0 B/op	       0 allocs/op
BenchmarkCiruits/handy/Default/passing/75-8              	 1000000	      1795 ns/op	       0 B/op	       0 allocs/op
BenchmarkCiruits/handy/Default/failing/1-8               	 1000000	      1272 ns/op	       0 B/op	       0 allocs/op
BenchmarkCiruits/handy/Default/failing/75-8              	 1000000	      1686 ns/op	       0 B/op	       0 allocs/op
BenchmarkCiruits/iand_circuit/Default/passing/1-8        	10000000	       119 ns/op	       0 B/op	       0 allocs/op
BenchmarkCiruits/iand_circuit/Default/passing/75-8       	 5000000	       349 ns/op	       0 B/op	       0 allocs/op
BenchmarkCiruits/iand_circuit/Default/failing/1-8        	100000000	        20.4 ns/op	       0 B/op	       0 allocs/op
BenchmarkCiruits/iand_circuit/Default/failing/75-8       	300000000	         5.46 ns/op	       0 B/op	       0 allocs/op
PASS
ok      github.com/cep21/circuit/benchmarking   59.518s

Limiting to just high concurrency passing circuits (the common case).

BenchmarkCiruits/cep21-circuit/Minimal/passing/75-8      	20000000	        87.7 ns/op	       0 B/op	       0 allocs/op
BenchmarkCiruits/GoHystrix/DefaultConfig/passing/75-8    	  500000	      2498 ns/op	     990 B/op	      20 allocs/op
BenchmarkCiruits/rubyist/Threshold-10/passing/75-8       	 2000000	       849 ns/op	     309 B/op	       4 allocs/op
BenchmarkCiruits/gobreaker/Default/passing/75-8          	 2000000	       698 ns/op	       0 B/op	       0 allocs/op
BenchmarkCiruits/handy/Default/passing/75-8              	 1000000	      1795 ns/op	       0 B/op	       0 allocs/op
BenchmarkCiruits/iand_circuit/Default/passing/75-8       	 5000000	       349 ns/op	       0 B/op	       0 allocs/op

Development

Make sure your tests pass with make test and your lints pass with make lint.

Example

You can run an example set of circuits inside the /example directory

make run

The output looks something like this:

< make run
go run example/main.go
2017/12/19 15:24:42 Serving on socket :8123
2017/12/19 15:24:42 To view the stream, execute:
2017/12/19 15:24:42   curl http://localhost:8123/hystrix.stream
2017/12/19 15:24:42
2017/12/19 15:24:42 To view expvar metrics, visit expvar in your browser
2017/12/19 15:24:42   http://localhost:8123/debug/vars
2017/12/19 15:24:42
2017/12/19 15:24:42 To view a dashboard, follow the instructions at https://github.com/Netflix/Hystrix/wiki/Dashboard#run-via-gradle
2017/12/19 15:24:42   git clone [email protected]:Netflix/Hystrix.git
2017/12/19 15:24:42   cd Hystrix/hystrix-dashboard
2017/12/19 15:24:42   ../gradlew jettyRun
2017/12/19 15:24:42
2017/12/19 15:24:42 Then, add the stream http://localhost:8123/hystrix.stream

If you load the Hystrix dasbhoard (following the above instructions), you should see metrics for all the example circuits.

dashboard

Comments
  • Treat deadline expired as failure

    Treat deadline expired as failure

    if the parent context is canceled before a timeout is reached, we don't consider the circuit unhealthy. But originalContext.Err() can be also context.DeadlineExceeded. Should we treat context.DeadlineExceeded as failures ?

    opened by zuzmic 10
  • Retrieve request context data through metrics callback

    Retrieve request context data through metrics callback

    Problem

    Contexts can be used to manage information about requests. Some of these request's information may require logging. It will be useful to get these additional information via the original context in the metrics callbacks.

    Or is there any other way to go about this?

    v4 
    opened by aadedamola 9
  • Closing circuit when majority of requests are bad requests

    Closing circuit when majority of requests are bad requests

    Our service has a client that does 90%+ bad requests (errors that satisfy the BadRequest interface); we wrap DynamoDB ConditionalCheckFailedException errors from aws-sdk-go in SimpleBadRequest. Occasionally it will switch from a mode of doing bad requests, to a mode of making successful requests that timeout, triggering the circuit to open, and then go back to doing bad requests. Since bad requests are not considered for closing the circuit, the circuit remains open until the hystrix closer happens to allow a non-bad request through.

    What are your thoughts on allowing bad requests to be considered for closing the circuit?

    opened by kollektiv 7
  • How to deal with Inconsistent execution timeout?

    How to deal with Inconsistent execution timeout?

    My backend service has many interface, most of interfaces timeout threshold is 2s, but some of interfaces timeout threshold should be 5s. The config of execution timeout(ExecutionConfig.Timeout) is static.

    Can execution timeout be dynamic? Or Circuit.checkErrTimeout support check whether ret from runFunc func(context.Context) error is a timeout error, so that it can be collect by Circuit.CmdMetricCollector.ErrTimeout? Or any good idea? thanks~

    question 
    opened by yangxikun 7
  • Expose `Error` interface

    Expose `Error` interface

    The documentation for Execute states:

    // Internal library errors will match the interface Error and you can use type casting to check this.

    But, the Error type is unexported so I am unable to do a type check.

    question 
    opened by Codonaut 5
  • while I set General Config Disabled=true, err_concurrency_limit_reject still occur

    while I set General Config Disabled=true, err_concurrency_limit_reject still occur

    While I set General Config Disabled=true, err_concurrency_limit_reject still occur. The code shows:

    if c.isEmptyOrNil() || c.threadSafeConfig.CircuitBreaker.Disabled.Get() {
    		return runFunc(ctx)
    	}
    

    If Disabled is true, it will not stats run and fallback. Here is my breaker create code:

    manager := &CircuitManager{
    	Manager: circuit.Manager{
    		DefaultCircuitProperties: []circuit.CommandPropertiesConstructor{
    			hystrixFactory.Configure,
    		},
    	},
    }
    breaker, err := manager.CreateCircuit(name)
    if err != nil {
    	breaker = manager.GetCircuit(name)
    }
    
    breaker.SetConfigThreadSafe(circuit.Config{
    	General: circuit.GeneralConfig{
    		Disabled: true,
    	},
    	Execution: circuit.ExecutionConfig{
    		MaxConcurrentRequests: 10,
    	},
    	Fallback: circuit.FallbackConfig{
    		MaxConcurrentRequests: 10,
    	},
    })
    

    So why this happens?

    opened by wangzz719 5
  • Timeout not working with Execute method (working with Go method)

    Timeout not working with Execute method (working with Go method)

    Hi, I am running following sample code to test out execution timeout

    h := circuit.Manager{}
    cb := h.MustCreateCircuit("redis-cb", circuit.Config{
    		Execution: circuit.ExecutionConfig{
    			// Time out the context after a few ms
    			Timeout: time.Millisecond * 30,
    		},})
    
    
    	start := time.Now()
    	errCB := cb.Execute(c, func(ctx context.Context) error {
    		time.Sleep(2*time.Second)
    		return nil
    	}, nil)
    
    	fmt.Println(time.Now().Sub(start).Seconds())
    	if errCB != nil {
    		fmt.Println(errCB)
    	}
    

    Output (when using Execute) -

    2.000189195

    If I use .Go instead of .Execute, then the execution times out after 30 ms properly. Output (when using .Go)

    0.03288965 context deadline exceeded

    question 
    opened by jarpit96 4
  • [trivial] remove benchmark dependencies from v3

    [trivial] remove benchmark dependencies from v3

    Currently, importing github.com/cep21/circuit/v3 requires transitive dependencies on every circuit-breaking library that is used in the benchmarking package, even though they are not actually used by the real library. By creating a separate go.mod/go.sum in benchmarking/, we can remove these transitive dependencies.

    Process

    cd v3/benchmarking
    go mod init
    go mod tidy
    cd ..
    go mod tidy
    
    opened by kwasnick 3
  • Option to mock the clock for Closer.reopenCircuitCheck

    Option to mock the clock for Closer.reopenCircuitCheck

    Since we're able to configure the Timekeeper for a Circuit, it would be helpful if the library also allowed mocking the TimeAfterFunc of the reopenCircuitCheck field. https://github.com/cep21/circuit/blob/700836dc9a95756136ce20df20ae5d3f3d8b59fd/v3/closers/hystrix/closer.go#L13-L23

    https://github.com/cep21/circuit/blob/700836dc9a95756136ce20df20ae5d3f3d8b59fd/v3/faststats/timedcheck.go#L20

    Great library!

    enhancement help wanted good first issue 
    opened by gavi-anchorlabs 3
  • Circuit module requires competing libraries for benchmark

    Circuit module requires competing libraries for benchmark

    Hi @cep21 -- thank you for building this awesome library!

    I was hoping we could restructure the v3 module to have the benchmark -- and its dependencies -- in a different module.

    This is important, because it means that downstream applications end up with every one of the competing libraries in our go.sum:

    $ go mod graph
    
    ...
    github.com/cep21/circuit/[email protected] github.com/afex/[email protected]
    github.com/cep21/circuit/[email protected] github.com/iand/[email protected]
    github.com/cep21/circuit/[email protected] github.com/rubyist/[email protected]+incompatible
    github.com/cep21/circuit/[email protected] github.com/sony/[email protected]
    ...
    

    It would also be nice if other modules, such as gopher-js, were optional.

    opened by benjigoldberg 3
  • circuit can't open?

    circuit can't open?

    func main() {
    	// f := rolling.StatFactory{}
    	h := circuit.Manager{
    		DefaultCircuitProperties: []circuit.CommandPropertiesConstructor{
    			// f.CreateConfig,
    			func(circuitName string) circuit.Config {
    				return circuit.Config{
    					Execution: circuit.ExecutionConfig{
    						MaxConcurrentRequests: 1000,
    					},
    					General: circuit.GeneralConfig{
    						OpenToClosedFactory: hystrix.CloserFactory(hystrix.ConfigureCloser{
    							//		// This should allow a new request every 10 milliseconds
    							SleepWindow: time.Millisecond * 10,
    						}),
    						ClosedToOpenFactory: hystrix.OpenerFactory(hystrix.ConfigureOpener{
    							RequestVolumeThreshold:   10,
    							ErrorThresholdPercentage: 50,
    						}),
    					},
    				}
    			},
    		},
    	}
    
    	c := h.MustCreateCircuit("hello-world")
    	fmt.Printf("var = %+v\n", h.Var())
    	loop := 100
    	for i := 0; i < loop; i++ {
    		time.Sleep(500 * time.Millisecond)
    		errResult := c.Execute(context.Background(), func(ctx context.Context) error {
    
    			if i > 10 {
    				return nil
    			}
    			return fmt.Errorf("err i %d", i)
    		}, nil)
    
    		fmt.Println("Result of execution:", errResult)
    	}
    
    }
    

    AKAIK, circuit should open after 10*50%=5 failed response within rolling window(default: 10s), but it execute successfully revision: 768780274348ba1e986a23a5feef9c829ab37426

    Output

    Result of execution: err i 0
    Result of execution: err i 1
    Result of execution: err i 2
    Result of execution: err i 3
    Result of execution: err i 4
    Result of execution: err i 5
    Result of execution: err i 6
    Result of execution: err i 7
    Result of execution: err i 8
    Result of execution: err i 9
    Result of execution: err i 10
    Result of execution: <nil>
    Result of execution: <nil>
    Result of execution: <nil>
    Result of execution: <nil>
    Result of execution: <nil>
    
    opened by jiacai2050 3
  • v4 library feedback thread

    v4 library feedback thread

    Hi!

    I'm thinking about what to modify for the v4 version of the library. Here is a feedback thread for all to contribute.

    Some ideas off the top of my head:

    • Remove the "/v3" root directory I needed to support projects that weren't on go modules. Use git tags instead
    • Move statsd implementation to another library (#113 )
    • Add ctx to the stats interfaces (#108 )
    • Make the "zero" value "off" for configuration, rather than the defaults (See #114 )
    • Make a common "circuit check" interface for people to add things like (#34 ) and move all the checks we currently have (concurrency, timeout, etc) into their own interface implementation
    • Move benchmarks to their own repo

    Do people have other thoughts on things they would change?

    help wanted question v4 
    opened by cep21 0
  • example config error

    example config error

    at https://github.com/cep21/circuit/blob/v3.1.0/v3/example/main.go#L81

    circuit.Config{} should be configured, like

    circuit.Config{
    General: circuit.GeneralConfig{
    	OpenToClosedFactory: hystrix.CloserFactory(hystrix.ConfigureCloser{}),
    	ClosedToOpenFactory: hystrix.OpenerFactory(hystrix.ConfigureOpener{}),
    }
    

    right?

    help wanted good first issue typo 
    opened by daliny 1
  • Allow adaptive limits

    Allow adaptive limits

    Problem

    In the basic case, we want to time out or limit the rare bad request so we can maintain a good SLA. However, when problems happen (maybe the database takes 110ms rather than 100ms for all requests because of a DB issue), we don't want to fail 100% of requests and would rather increase our timeout by a bit while requests are slow, and move it back down when things normalize.

    Idea

    Move all limits from static numbers to (min/max/rate of change). For example, you could have a timeout normally at 100ms, but allow it to increase by 10ms per some unit if requests are slower than 100ms, but not allow requests to ever be slower than 300ms. Then, when things settle down, allow requests to timeout at 100ms again.

    Solution

    Circuit open/close logic is defined inside https://github.com/cep21/circuit/blob/master/closers.go#L9 and they listen to all the events on https://github.com/cep21/circuit/blob/master/metrics.go#L164

    The function ShouldOpen is called when a circuit decides if it should open: https://github.com/cep21/circuit/blob/master/closers.go#L14

    Right now, for hystrix, we open directly on error percentage https://github.com/cep21/circuit/blob/master/closers/hystrix/opener.go#L140

    Instead of opening on some threshold, it could detect why the circuit is failing (if it is because of too many timeouts or concurrency limits). If it is, it would modify the thread safe config on the circuit https://github.com/cep21/circuit/blob/master/circuit.go#L71 to increase the timeout. On concurrent Success, we can inspect the timeouts and lower the limit if things recover.

    Similarly, on ErrConcurrencyLimitReject calls, we could increase the concurrency limits up to a point, and decrease it on Success without ErrInterrupt.

    enhancement help wanted 
    opened by cep21 1
  • Implement waiting max concurrency

    Implement waiting max concurrency

    Add a feature that uses https://github.com/golang/sync/blob/master/semaphore/semaphore.go to limit concurrency. In this implementation, rather than failing fast with an error, the function would wait according to the context if was at a limit of MaxConcurrentRequests

    enhancement help wanted good first issue 
    opened by cep21 8
Releases(v3.2.2)
Owner
Jack Lindamood
Jack Lindamood
A golang framework like koa.js

koa.go Expressive HTTP middleware framework for Golang to make web applications and APIs more enjoyable to write like Koa.js. Koa's middleware stack f

Ya Hui Liang(Ryou) 30 Sep 26, 2022
There are http server which handle transaction request(like: SET,GET,DELETE)

Loco_test_assesment There are http server which handle transaction request(like: SET,GET,DELETE) File "Backend Assessment.docx" has the proper informa

kumari tanushree 0 Jan 14, 2022
An experimental GraphQL implementation with Go.

Description An experimental GraphQL implementation with Go. This repo focuses on improve GraphQL Parse and Resolve speed t

karminski-牙医 57 Jun 16, 2022
A Golang blocking leaky-bucket rate limit implementation

Go rate limiter This package provides a Golang implementation of the leaky-bucket rate limit algorithm. This implementation refills the bucket based o

Uber Go 3.3k Sep 21, 2022
This repository contains rest api implementation with golang

?? go-rest This repository contains rest api implementation with golang ⚡ LIVE To check out the live demo of this app ABOUT ?? Building a Rest API ??

null 3 Apr 20, 2022
Go package that handles HTML, JSON, XML and etc. responses

gores http response utility library for Go this package is very small and lightweight, useful for RESTful APIs. installation go get github.com/alioygu

Ali OYGUR 100 Aug 30, 2022
Go package for easily rendering JSON, XML, binary data, and HTML templates responses.

Render Render is a package that provides functionality for easily rendering JSON, XML, text, binary data, and HTML templates. This package is based on

Cory Jacobsen 1.7k Sep 25, 2022
Simple, lightweight and faster response (JSON, JSONP, XML, YAML, HTML, File) rendering package for Go

Package renderer Simple, lightweight and faster response (JSON, JSONP, XML, YAML, HTML, File) rendering package for Go Installation Install the packag

Saddam H 243 Sep 20, 2022
Mahi is an all-in-one HTTP service for file uploading, processing, serving, and storage.

Mahi is an all-in-one HTTP service for file uploading, processing, serving, and storage. Mahi supports chunked, resumable, and concurrent uploads. Mahi uses Libvips behind the scenes making it extremely fast and memory efficient.

Rodrigo Lessa 32 May 29, 2022
A http service to verify request and bounce them according to decisions made by CrowdSec.

traefik-crowdsec-bouncer A http service to verify request and bounce them according to decisions made by CrowdSec. Description This repository aim to

Fabien Bonalair 150 Sep 23, 2022
Goya circuit is a circuit breaker mechanism implementation in microservices.

Goya-Circuit: 类似于Hystrix的熔断器实现 Goya circuit is a circuit breaker mechanism implementation in microservices. It can prevent the whole link avalanche ca

Anderson Lu 6 Mar 8, 2022
gobreaker implements the Circuit Breaker pattern in Go.

gobreaker gobreaker implements the Circuit Breaker pattern in Go.

Sony 2.2k Sep 13, 2022
gRelay is an open source project written in Go that provides the circuit break pattern with a relay idea behind.

gRELAY gRelay is an open source project written in Go that provides: Circuit Break ✔️ Circuit Break + Relay ✔️ Concurrecny Safe ✔️ Getting start Insta

null 32 Sep 1, 2022
Netflix's Hystrix latency and fault tolerance library, for Go

hystrix-go Hystrix is a great project from Netflix. Hystrix is a latency and fault tolerance library designed to isolate points of access to remote sy

keith 3.8k Sep 24, 2022
this is an example of hystrix-go usage in web dev

hystrix-go-example this is an example of hystrix-go usage in web dev Explanation this example contains 2 service: alpha as our main service, circuit b

Vembry 2 Apr 22, 2022
A feature complete and high performance multi-group Raft library in Go.

Dragonboat - A Multi-Group Raft library in Go / 中文版 News 2021-01-20 Dragonboat v3.3 has been released, please check CHANGELOG for all changes. 2020-03

lni 4.4k Sep 18, 2022
A feature complete and high performance multi-group Raft library in Go.

Dragonboat - A Multi-Group Raft library in Go / 中文版 News 2021-01-20 Dragonboat v3.3 has been released, please check CHANGELOG for all changes. 2020-03

lni 4.4k Sep 24, 2022
golang feature toggle library - a library to help make golang feature toggling clean and easy

toggle supports env_variable backed toggling. It can also be updated via a pubsub interface (tested w/ redis) 2 engines for toggle backing are include

John Calabrese 24 Mar 29, 2022
Go-ant-pattern: An ant pattern parser

go-ant-pattern - An ant pattern parser. Usage package main import ( "fmt" "github.com/cbuschka/go-ant-pattern" ) func main() { path := "

Cornelius Buschka 0 Dec 7, 2021
Leader-follower-pattern - Build leader-follower system pattern with etcd election

主备系统模式 原理 使用分布式锁实现主备节点系统。通过对分布式锁进行续期,保持长期锁, 从而使当前服务节点处于主服务节点 无法获取分布式锁的服务节点,则作为备选

null 0 Jan 24, 2022