Go Micro is a standalone framework for distributed systems development

Overview

Go Micro License Go.Dev reference

Go Micro is a framework for distributed systems development.

Overview

Go Micro provides the core requirements for distributed systems development including RPC and Event driven communication. The Micro philosophy is sane defaults with a pluggable architecture. We provide defaults to get you started quickly but everything can be easily swapped out.

Features

Go Micro abstracts away the details of distributed systems. Here are the main features.

  • Authentication - Auth is built in as a first class citizen. Authentication and authorization enable secure zero trust networking by providing every service an identity and certificates. This additionally includes rule based access control.

  • Dynamic Config - Load and hot reload dynamic config from anywhere. The config interface provides a way to load application level config from any source such as env vars, file, etcd. You can merge the sources and even define fallbacks.

  • Data Storage - A simple data store interface to read, write and delete records. It includes support for memory, file and CockroachDB by default. State and persistence becomes a core requirement beyond prototyping and Micro looks to build that into the framework.

  • Service Discovery - Automatic service registration and name resolution. Service discovery is at the core of micro service development. When service A needs to speak to service B it needs the location of that service. The default discovery mechanism is multicast DNS (mdns), a zeroconf system.

  • Load Balancing - Client side load balancing built on service discovery. Once we have the addresses of any number of instances of a service we now need a way to decide which node to route to. We use random hashed load balancing to provide even distribution across the services and retry a different node if there's a problem.

  • Message Encoding - Dynamic message encoding based on content-type. The client and server will use codecs along with content-type to seamlessly encode and decode Go types for you. Any variety of messages could be encoded and sent from different clients. The client and server handle this by default. This includes protobuf and json by default.

  • RPC Client/Server - RPC based request/response with support for bidirectional streaming. We provide an abstraction for synchronous communication. A request made to a service will be automatically resolved, load balanced, dialled and streamed.

  • Async Messaging - PubSub is built in as a first class citizen for asynchronous communication and event driven architectures. Event notifications are a core pattern in micro service development. The default messaging system is a HTTP event message broker.

  • Synchronization - Distributed systems are often built in an eventually consistent manner. Support for distributed locking and leadership are built in as a Sync interface. When using an eventually consistent database or scheduling use the Sync interface.

  • Pluggable Interfaces - Go Micro makes use of Go interfaces for each distributed system abstraction. Because of this these interfaces are pluggable and allows Go Micro to be runtime agnostic. You can plugin any underlying technology.

Getting Started

To make use of Go Micro

import "github.com/asim/go-micro/v3"

// create a new service
service := micro.NewService(
    micro.Name("helloworld"),
)

// initialise flags
service.Init()

// start the service
service.Run()

See the examples for detailed information on usage.

Code Generation

See cmd/protoc-gen-micro for protobuf code generation.

Example Usage

See examples directory for usage examples.

Plugins

See plugins directory for all the plugins.

License

Go Micro is Apache 2.0 licensed.

Comments
  • example of compose two or more services with single binary

    example of compose two or more services with single binary

    I like go-micro design, but if i want to provide single binary with all my services together. For example that implements some api with 10 different services. How can i do that with go-micro? All this services need to communicate via message bus (nats or etcd).

    opened by vtolstov 23
  • RPC stream client/server mutex fix

    RPC stream client/server mutex fix

    As described in #878 (and #532) it is not possible to send messages while waiting to receive them as well (or vice-versa), due to RPC stream client and server being locked by both Send() and Recv() functions.

    In order to solve this issue, I unlock the Mutex before calling the codec to receive data, and locking it again after codec returned. This way we still have the Mutex protection of the client/server structs, but we can send/receive at same time.

    :warning: Note that this assumes that codec does not need protection while being active.

    I did not update Send() because I assumed that it won't block. By only fixing Recv() the Send() worked as welll. Maybe depending on transport/codec it can block as well?

    I only considered these 2 components/files, as we are (currently) not having issues in other parts of the go-micro framework regarding to this. If needed I can update other parts as well to stay in sync with this change.

    I have tested this in our micro-service environment and it all seems to work fine. I did not trigger race conditions or other panics. But I did not yet perform (high) load or long running tests, which would make race conditions more likely to pop up (if there are any).

    enhancement WIP RPC 
    opened by veger 22
  • build command-line-arguments: cannot load crypto/ed25519: cannot find module providing package crypto/ed2551

    build command-line-arguments: cannot load crypto/ed25519: cannot find module providing package crypto/ed2551

    When I run the go-micro,it tips build command-line-arguments: cannot load crypto/ed25519: cannot find module providing package crypto/ed25519,but i find the module is exists in gopath.

    question more info requsted go.mod 
    opened by songdony 20
  • consul registry broken: mismatch in type due to import being expected from

    consul registry broken: mismatch in type due to import being expected from "github.com/hashicorp/consul/vendor"

    My Go version is 1.12.5.

    I'm building my package binary using the golang1.12.5-alpine3.9 Docker image as a package builder and I'm running the following commands in my Dockerfile to install all the necessary dependencies -

    # install source dependencies
    RUN go get -d -v ./...
    RUN go install -v ./...
    

    the go get command works as expected but the go install command crashes with an exit code: 2 and the following message -

    # github.com/micro/go-micro/registry/consul /go/src/github.com/micro/go-micro/registry/consul/watcher.go:48:33: cannot use cr.Client (type *"github.com/hashicorp/consul/api".Client) as type *"github.com/hashicorp/consul/vendor/github.com/hashicorp/consul/api".Client in argument to wp.RunWithClientAndLogger /go/src/github.com/micro/go-micro/registry/consul/watcher.go:213:37: cannot use cw.r.Client (type *"github.com/hashicorp/consul/api".Client) as type *"github.com/hashicorp/consul/vendor/github.com/hashicorp/consul/api".Client in argument to wp.RunWithClientAndLogger

    Just FYI, I was able to generate the Docker based package builder successfully up till 3 days ago.

    opened by desertjinn 20
  • gossip registry

    gossip registry

    I'm experiment with gossip registry and may be found some specific issue. When i'm start first service1 with specify gossip.Address("172.16.1.254:4223") and on other server service2 with registry.Addrs("172.16.1.254:4223") and gossip.Address("172.16.1.1:0")

    i see that members equal 2 on both sides, but the second service does not registered in registry. But if i stop service1 and start it with registry.Addrs("172.16.1.254:4223") and gossip.Address("172.16.1.1:xxx") where xxx is port provided by service2 all works fine. So issue appears only on first service1 when it start first without other members.

    opened by vtolstov 20
  • [BUG] Micro API fails on streaming endpoint unexpected nil error

    [BUG] Micro API fails on streaming endpoint unexpected nil error

    Describe the bug

    1. What are you trying to do? Run micro api gateway and call a streaming endpoint
    2. What did you expect to happen? Success, actual response
    3. What happens instead?
    {
        "code": 500,
        "detail": "grpc: client streaming protocol violation: get <nil>, want <EOF>",
        "id": "go.micro.client",
        "status": "Internal Server Error"
    }
    

    How to reproduce the bug: micro version 2.0.0 micro api --address :8085 --namespace some.ns --handler rpc http localhost:8085/my/streaming/endpoint

    If possible, please include a minimal code snippet here.

           ctx, cancel := context.WithCancel(ctx)
    	ch, errCh := data.GetSomeData(ctx, cancel, s.db, s.service)
    
    	for {
    		select {
    		case <-ctx.Done():
    			stream.Close()
    			return nil
    		case err := <-errCh:
    			return err
    		case msg := <-ch:
    			if msg == nil {
    				return nil
    			}
    			err := stream.Send(msg)
    			if err != nil {
    				cancel()
    				for len(ch) > 0 {
    					<-ch
    				}
    				return errors.InternalServerError(s.service.Name(), err.Error())
    			}
    		}
    	}
    

    Environment: Go Version: please paste go version output here

    go version go1.13.1 darwin/amd64
    
    opened by OGKevin 17
  • Disable timeout on RPC stream connection

    Disable timeout on RPC stream connection

    For a stream RPC connection I'd like to get rid of timeout, I see/found two solutions:

    1. Set timeout to some huge value and hope the client never stays connected long enough.
    2. Set timeout to -1 which 'somehow' seems to work. (Actually strconv.ParseUint() chokes on it and context timeout is not set (and no error is issued...))

    Both options feel hacky to me... Is there a better/recommended solution instead?

    enhancement 
    opened by veger 17
  • go get -u fails, apparently due to coreos/etcd move

    go get -u fails, apparently due to coreos/etcd move

    Go version: go1.13 Problem: when I run go get -u github.com/micro/go-micro I get the following error:

    # github.com/coreos/etcd/clientv3
    src/github.com/coreos/etcd/clientv3/auth.go:121:72: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.AuthEnable
    src/github.com/coreos/etcd/clientv3/auth.go:126:74: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.AuthDisable
    src/github.com/coreos/etcd/clientv3/auth.go:131:152: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.UserAdd
    src/github.com/coreos/etcd/clientv3/auth.go:136:144: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.UserAdd
    src/github.com/coreos/etcd/clientv3/auth.go:141:86: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.UserDelete
    src/github.com/coreos/etcd/clientv3/auth.go:146:122: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.UserChangePassword
    src/github.com/coreos/etcd/clientv3/auth.go:151:104: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.UserGrantRole
    src/github.com/coreos/etcd/clientv3/auth.go:156:80: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.UserGet
    src/github.com/coreos/etcd/clientv3/auth.go:161:72: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.UserList
    src/github.com/coreos/etcd/clientv3/auth.go:166:106: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.UserRevokeRole
    src/github.com/coreos/etcd/clientv3/auth.go:166:106: too many errors
    

    We first saw this error when our CI build began to fail, and at first couldn't reproduce it locally, but then were able to by adding the -u to the command. Alternately, here's a simple Dockerfile that demonstrates the issue when building it.

    This seems to be caused by the moving of github.com/coreos/etcd to github.com/etcd-io/etcd, as described here: https://github.com/etcd-io/etcd/pull/10044#issuecomment-417125341

    That change and the issue are from last year, so I'm not sure why this is just showing up now, but my colleagues and I (and our CI) can all reproduce the problem now.

    I will look into creating a PR that changes the imports from coreos/etcd to etcd-io/etcd.

    go.mod 
    opened by newrelic-eheinlein 17
  • Adding TLS to RPC

    Adding TLS to RPC

    I'm feeling a little bit dumb, I don't suppose you have an example of adding TLS to a RPC service?

    	security.Setup(log, security.Vault())
    	tlsConfig := &tls.Config{
    		GetCertificate: security.Vault().CertificateGetter(),
    	}
    
    	// initialise service
    	service.Init(
    		micro.Action(func(c *cli.Context) {
    			if c.Bool("debug") {
    				log.Level = logrus.DebugLevel
    				log.Debug("Debug log level enabled")
    			}
    		}),
    // transport.TLSConfig(tlsConfig),
    	)
    
    	countryHandler, err := handler.NewCountryHandler()
    	if err != nil {
    		log.Fatal(err)
    	}
    
    	country.RegisterCountriesHandler(service.Server(), countryHandler)
    
    	if err := service.Run(); err != nil {
    		log.Fatal(err)
    	}
    

    I tried adding transport options to the service.Init but it's clearly not that simple?

    opened by freman 17
  • How to add value to metadata?

    How to add value to metadata?

    Hello,

    I want to add some value to metadata from grpc method, then process in wrapper, however I found since grpc 1.40, we can't modify the context, the proposed solution is to use metadata.NewIncomingContext.

    https://github.com/grpc/grpc-go/issues/4363

    I don't know how to do it in micro?

    func (s *ServerImpl) Location(ctx context.Context, req *pb.LocationRequest, rsp *pb.City) error {
    // add value to context
    	md, _ := metadata.FromContext(ctx)
    	if md != nil {
    		md.Set("tag", "value")
    	}
    }
    
    func LogWrapper(fn server.HandlerFunc) server.HandlerFunc {
    	return func(ctx context.Context, req server.Request, rsp interface{}) error {
     		err := fn(ctx, req, rsp)
     		value, _ := md.Get("tag")  // value is empty
    }
    }
    
    help wanted 
    opened by seaguest 16
  • Documentation unavailable : not displayed due to license restrictions

    Documentation unavailable : not displayed due to license restrictions

    Hello,

    This is not a #1956 or #1953 duplicate. :)

    I tried to access : https://pkg.go.dev/github.com/micro/go-micro/v3?tab=overview But the documentation is unavailable due to license restrictions.

    Thanks,

    opened by scorsi 15
  • feat(logger): add logger option to all micro components (override DefaultLogger) closes #2556

    feat(logger): add logger option to all micro components (override DefaultLogger) closes #2556

    closes #2556

    (to get a better understanding of the reason bedind this PR, please checkout this issue #2556

    A PR to allow providing a custom logger to every micro component (client, server, store, broker,...).

    the guide lines of this implementation are :

    • do not change components API (for retro compatibility)
    • add Logger option to every component
    • fallback to DefaultLogger if no logger option provided (for retro compatibility)

    Any feedback is welcome ^^

    opened by asynxc 5
  • [FEATURE] allow passing custom logger to

    [FEATURE] allow passing custom logger to "server" component

    Is your feature request related to a problem? Please describe.

    in a context where we instantiate multiple server in same process, we may need to assign specific fields to each instance. So that it's easier to distinguish their logs.

    Describe the solution you'd like

    add new option server.WithLogger(logger log.Logger) to server options

    If this's ok with you, i can create a PR in both go-micro and plugins projects ?

    opened by asynxc 9
  • Allow passing context to store operations

    Allow passing context to store operations

    This allows passing a context with opentelemetry tracing information to remote services / store implementations.

    Regarding the pr template:

    1. Make sure this PR targets the develop branch. We follow the git-flow branching model.

    there is no develop branch ;-)

    opened by butonic 6
  • [BUG] go-micro error reported during press test

    [BUG] go-micro error reported during press test

    go-micro

    Concurrency: 5000 Connection pooling: 10

    2022/09/01 17:17:03 client.go:47: INFO : concurrency: 5000
    requests per client: 200
     
    2022/09/01 17:17:03 client.go:49: INFO : Servers: 10.130.12.14:8900
     
    2022/09/01 17:17:03 client.go:57: INFO : message size: 581 bytes
     
    2022/09/01 17:17:07 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:07 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:07 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:08 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:35 stats.go:16: INFO : 花了 31302 ms 用于 1000000 请求
    2022/09/01 17:17:35 stats.go:38: INFO : 发送请求    : 1000000
    2022/09/01 17:17:35 stats.go:39: INFO : 收到的请求    : 1000000
    2022/09/01 17:17:35 stats.go:40: INFO : 收到的成功请求 : 999963
    2022/09/01 17:17:35 stats.go:48: INFO : 吞吐量  (TPS)    : 31946
     
    2022/09/01 17:17:35 stats.go:60: INFO : 平均值: 137153203.650 ns, 中位数: 4337898.500 ns, 最大值: 4303066047.000 ns, 最小值: 243587.000 ns, p99.9: 1817774720.500 ns
    2022/09/01 17:17:35 stats.go:61: INFO : 平均值: 137.153 ms, 中位数: 4.338 ms, 最大值: 4303.066 ms, 最小值: 0.244 ms, p99.9: 1817.775 ms
    2022/09/01 17:17:35 stats.go:62: INFO : [go-micro]: TPS: 31946.84, TP99: 887.68ms, TP999: 1817.77ms (b=581 Byte, c=5000, n=1000000)
    Info: [go-micro]: TPS: 31946.84, TP99: 887.68ms, TP999: 1817.77ms (b=581 Byte, c=5000, n=1000000)
    

    grpc

    Concurrency: 5000 Connection pooling: 10

    2022/09/01 17:18:27 grpc_client.go:43: INFO : concurrency: 5000
    requests per client: 200
    
    2022/09/01 17:18:27 grpc_client.go:46: INFO : Servers: 10.130.12.14:8900
    
    2022/09/01 17:18:27 grpc_client.go:52: INFO : message size: 581 bytes
    
    
    2022/09/01 17:18:33 stats.go:16: INFO : 花了 5953 ms 用于 1000000 请求
    2022/09/01 17:18:34 stats.go:38: INFO : 发送请求    : 1000000
    2022/09/01 17:18:34 stats.go:39: INFO : 收到的请求    : 1000000
    2022/09/01 17:18:34 stats.go:40: INFO : 收到的成功请求 : 1000000
    2022/09/01 17:18:34 stats.go:48: INFO : 吞吐量  (TPS)    : 167982
    
    2022/09/01 17:18:34 stats.go:60: INFO : 平均值: 28944916.637 ns, 中位数: 25278866.000 ns, 最大值: 255435674.000 ns, 最小值: 207720.000 ns, p99.9: 137116793.500 ns
    2022/09/01 17:18:34 stats.go:61: INFO : 平均值: 28.945 ms, 中位数: 25.279 ms, 最大值: 255.436 ms, 最小值: 0.208 ms, p99.9: 137.117 ms
    2022/09/01 17:18:34 stats.go:62: INFO : [grpc]: TPS: 167982.53, TP99: 80.10ms, TP999: 137.12ms (b=581 Byte, c=5000, n=1000000)
    Info: [grpc]: TPS: 167982.53, TP99: 80.10ms, TP999: 137.12ms (b=581 Byte, c=5000, n=1000000)
    

    A grpc press test on a similarly configured machine shows no errors

    @asim

    opened by JellyZero 1
  • [BUG] go-micro registry node nil pointer

    [BUG] go-micro registry node nil pointer

    Describe the bug

    1. What are you trying to do?
    • Build a Rpc Client to request a Rpc Server
    1. What did you expect to happen?
    • when rpc client try to get rpc server addr and request, "nil pointer exception" should not happen
    1. What happens instead?
    • a panic occur which have already get a Rlock wihout unlock
    /usr/local/go17/src/runtime/panic.go:221 (0xc76946)
    /usr/local/go17/src/runtime/signal_unix.go:735 (0xc76916)
    mydir/pkg/mod/go-micro.dev/[email protected]/util/registry/util.go:65 (0x1226a8c)
    mydir/pkg/mod/go-micro.dev/[email protected]/util/registry/util.go:85 (0x1226cfc)
    mydir/pkg/mod/go-micro.dev/[email protected]/registry/cache/cache.go:126 (0x1228a78)
    mydir/pkg/mod/go-micro.dev/[email protected]/registry/cache/cache.go:436 (0x122a95d)
    mydir/pkg/mod/go-micro.dev/[email protected]/selector/default.go:52 (0x127feeb)
    mydir/pkg/mod/github.com/asim/go-micro/plugins/client/grpc/[email protected]/grpc.go:90 (0x1de1242)
    mydir/pkg/mod/github.com/asim/go-micro/plugins/client/grpc/[email protected]/grpc.go:403 (0x1de4df6)
    [email protected]/gomicro/wrap.go:595 (0x1e0c50c)
    [email protected]/gomicro/wrap.go:575 (0x1e0b99e)
    [email protected]/gomicro/wrap.go:312 (0x1e0936f)
    myproject/cmu.pb.micro.go:238 (0x1b71d4e)
    myproject/upcw_service.go:53 (0x1c0f57a)
    myproject/upcw_common.go:33 (0x1e52904)
    mydir/pkg/mod/github.com/gin-gonic/[email protected]/context.go:147 (0x1abd0ea)
    [email protected]/middleware/http/recovery.go:39 (0x1abd0d4)
    mydir/pkg/mod/github.com/gin-gonic/[email protected]/context.go:147 (0x1af3dca)
    myproject/comm/middleware/session/session.go:82 (0x1af3dad)
    mydir/pkg/mod/github.com/gin-gonic/[email protected]/context.go:147 (0x1af714d)
    myproject/comm/metadata/metadata.go:113 (0x1af7134)
    mydir/pkg/mod/github.com/gin-gonic/[email protected]/context.go:147 (0x1ab8ef3)
    [email protected]/middleware/http/gin.go:293 (0x1ab8d73)
    mydir/pkg/mod/github.com/gin-gonic/[email protected]/context.go:147 (0x116c369)
    mydir/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:403 (0x116bfb9)
    mydir/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:364 (0x116bb33)
    mydir/pkg/mod/github.com/gorilla/[email protected]/mux.go:210 (0x1cb0b6e)
    /usr/local/go17/src/net/http/server.go:2879 (0xf0ee7a)
    /usr/local/go17/src/net/http/server.go:1930 (0xf0a467)
    /usr/local/go17/src/runtime/asm_amd64.s:1581 (0xc945a0)
    
    • a goroutine panic with lock and other goroutine can't get the lock: pprof image

    • cause other goroutine can not get lock resource until I restart it.

    How to reproduce the bug: I do not know how to reproduce and why happen. I found it in product environment sometime and I have to restart the program to fix it or the program can not make rpc request.

    Environment: Go Version: please paste go version output here

    
    GO111MODULE="on"
    GOARCH="amd64"
    GOBIN=""
    GOCACHE="/home/myname/.cache/go-build"
    GOENV="/home/myname/.config/go/env"
    GOEXE=""
    GOEXPERIMENT=""
    GOFLAGS=""
    GOHOSTARCH="amd64"
    GOHOSTOS="linux"
    GOINSECURE=""
    GOMODCACHE="/home/myname/goproject17/pkg/mod"
    GONOPROXY=""
    GONOSUMDB=""
    GOOS="linux"
    GOPATH="/home/myname/goproject17"
    GOPRIVATE=""
    GOPROXY=""
    GOROOT="/usr/local/go17"
    GOSUMDB="sum.golang.org"
    GOTMPDIR=""
    GOTOOLDIR="/usr/local/go17/pkg/tool/linux_amd64"
    GOVCS=""
    GOVERSION="go1.17.6"
    GCCGO="gccgo"
    AR="ar"
    CC="gcc"
    CXX="g++"
    CGO_ENABLED="1"
    GOMOD="/dev/null"
    CGO_CFLAGS="-g -O2"
    CGO_CPPFLAGS=""
    CGO_CXXFLAGS="-g -O2"
    CGO_FFLAGS="-g -O2"
    CGO_LDFLAGS="-g -O2"
    PKG_CONFIG="pkg-config"
    GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build4281984978=/tmp/go-build -gno-record-gcc-switches"
    
    bug help wanted 
    opened by gdgrc 9
  • Remove cmd package

    Remove cmd package

    Looking at removing the cmd package. I think flag and env var parsing was initially quite useful when the surface area of go-micro was small. Over time that's gotten broader and more complex to manage. Ultimately if go-micro is pluggable, then either the cmd package needs to be totally rewritten or removed entirely.

    opened by asim 7
Releases(v4.8.1)
Owner
Asim Aslam
Working on Micro
Asim Aslam
Distributed-Services - Distributed Systems with Golang to consequently build a fully-fletched distributed service

Distributed-Services This project is essentially a result of my attempt to under

Hamza Yusuff 6 Jun 1, 2022
Distributed lock manager. Warning: very hard to use it properly. Not because it's broken, but because distributed systems are hard. If in doubt, do not use this.

What Dlock is a distributed lock manager [1]. It is designed after flock utility but for multiple machines. When client disconnects, all his locks are

Sergey Shepelev 25 Dec 24, 2019
Tarmac is a unique framework designed for the next generation of distributed systems

Framework for building distributed services with Web Assembly

Benjamin Cane 212 Sep 14, 2022
Fast, efficient, and scalable distributed map/reduce system, DAG execution, in memory or on disk, written in pure Go, runs standalone or distributedly.

Gleam Gleam is a high performance and efficient distributed execution system, and also simple, generic, flexible and easy to customize. Gleam is built

Chris Lu 3.1k Sep 21, 2022
Micro is a platform for cloud native development

Micro Overview Micro addresses the key requirements for building services in the cloud. It leverages the microservices architecture pattern and provid

Micro 11.3k Sep 23, 2022
A distributed systems library for Kubernetes deployments built on top of spindle and Cloud Spanner.

hedge A library built on top of spindle and Cloud Spanner that provides rudimentary distributed computing facilities to Kubernetes deployments. Featur

null 21 Jan 4, 2022
MIT 6.824: Distributed Systems

MIT 6.824 is a core 12-unit graduate subject with lectures, readings, programming labs, an optional project, a mid-term exam, and a final exam.

CC Chang 2 Jul 6, 2022
Distributed Systems 2021 - Miniproject 3

Distributed Systems 2021 -- Miniproject 3 Hand-in Date: 1 December 2021 (at 23:59) What to submit on learnit: a single zip-compressed file containing:

null 0 Dec 11, 2021
Labs, solutions and related materials from the MIT 6.824 Distributed Systems course.

MIT 6.824 Distributed Systems Labs, solutions and related materials from the MIT 6.824 Distributed Systems course. Overview From the official website:

Arindam Das 44 Sep 16, 2022
MIT6.824 Distributed Systems

MIT6.824-Distributed-Systems My Solutions for MIT6.824

null 0 Jan 28, 2022
Distributed reliable key-value store for the most critical data of a distributed system

etcd Note: The main branch may be in an unstable or even broken state during development. For stable versions, see releases. etcd is a distributed rel

etcd-io 41.3k Sep 24, 2022
Skynet is a framework for distributed services in Go.

##Introduction Skynet is a communication protocol for building massively distributed apps in Go. It is not constrained to Go, so it will lend itself n

null 2k Jul 25, 2022
a dynamic configuration framework used in distributed system

go-archaius This is a light weight configuration management framework which helps to manage configurations in distributed system The main objective of

null 200 Aug 31, 2022
A Distributed Content Licensing Framework (DCLF) using Hyperledger Fabric permissioned blockchain.

A Distributed Content Licensing Framework (DCLF) using Hyperledger Fabric permissioned blockchain.

Tal Derei 3 May 3, 2022
Flowgraph package for scalable asynchronous system development

flowgraph Getting Started go get -u github.com/vectaport/flowgraph go test Links Wiki Slides from Minneapolis Golang Meetup, May 22nd 2019 Overview F

Scott Johnston 52 Sep 5, 2022
distributed data sync with operational transformation/transforms

DOT The DOT project is a blend of operational transformation, CmRDT, persistent/immutable datastructures and reactive stream processing. This is an im

DOT & Chain 72 Aug 14, 2022
High performance, distributed and low latency publish-subscribe platform.

Emitter: Distributed Publish-Subscribe Platform Emitter is a distributed, scalable and fault-tolerant publish-subscribe platform built with MQTT proto

emitter 3.4k Sep 22, 2022
Simplified distributed locking implementation using Redis

redislock Simplified distributed locking implementation using Redis. For more information, please see examples. Examples import ( "fmt" "time"

Black Square Media 817 Sep 27, 2022