Go Micro is a framework for distributed systems development

Overview

Go Micro License Go.Dev reference Travis CI Go Report Card

Go Micro is a framework for distributed systems development.

Overview

Go Micro provides the core requirements for distributed systems development including RPC and Event driven communication. The Micro philosophy is sane defaults with a pluggable architecture. We provide defaults to get you started quickly but everything can be easily swapped out.

Features

Go Micro abstracts away the details of distributed systems. Here are the main features.

  • Authentication - Auth is built in as a first class citizen. Authentication and authorization enable secure zero trust networking by providing every service an identity and certificates. This additionally includes rule based access control.

  • Dynamic Config - Load and hot reload dynamic config from anywhere. The config interface provides a way to load application level config from any source such as env vars, file, etcd. You can merge the sources and even define fallbacks.

  • Data Storage - A simple data store interface to read, write and delete records. It includes support for memory, file and CockroachDB by default. State and persistence becomes a core requirement beyond prototyping and Micro looks to build that into the framework.

  • Service Discovery - Automatic service registration and name resolution. Service discovery is at the core of micro service development. When service A needs to speak to service B it needs the location of that service. The default discovery mechanism is multicast DNS (mdns), a zeroconf system.

  • Load Balancing - Client side load balancing built on service discovery. Once we have the addresses of any number of instances of a service we now need a way to decide which node to route to. We use random hashed load balancing to provide even distribution across the services and retry a different node if there's a problem.

  • Message Encoding - Dynamic message encoding based on content-type. The client and server will use codecs along with content-type to seamlessly encode and decode Go types for you. Any variety of messages could be encoded and sent from different clients. The client and server handle this by default. This includes protobuf and json by default.

  • RPC Client/Server - RPC based request/response with support for bidirectional streaming. We provide an abstraction for synchronous communication. A request made to a service will be automatically resolved, load balanced, dialled and streamed.

  • Async Messaging - PubSub is built in as a first class citizen for asynchronous communication and event driven architectures. Event notifications are a core pattern in micro service development. The default messaging system is a HTTP event message broker.

  • Synchronization - Distributed systems are often built in an eventually consistent manner. Support for distributed locking and leadership are built in as a Sync interface. When using an eventually consistent database or scheduling use the Sync interface.

  • Pluggable Interfaces - Go Micro makes use of Go interfaces for each distributed system abstraction. Because of this these interfaces are pluggable and allows Go Micro to be runtime agnostic. You can plugin any underlying technology.

Getting Started

To make use of Go Micro

import "github.com/asim/go-micro/v3"

// create a new service
service := micro.NewService(
    micro.Name("helloworld"),
)

// initialise flags
service.Init()

// start the service
service.Run()

See the docs for detailed information on the architecture, installation and use of go-micro.

Code Generation

See cmd/protoc-gen-micro for protobuf code generation.

Example Usage

See examples directory for usage examples.

Plugins

See plugins directory for all the plugins.

License

Go Micro is Apache 2.0 licensed.

Upgrade

Micro v3 consolidates Go Micro into it as a single unified platform. Please see the upgrade guide.

Comments
  • example of compose two or more services with single binary

    example of compose two or more services with single binary

    I like go-micro design, but if i want to provide single binary with all my services together. For example that implements some api with 10 different services. How can i do that with go-micro? All this services need to communicate via message bus (nats or etcd).

    opened by vtolstov 23
  • RPC stream client/server mutex fix

    RPC stream client/server mutex fix

    As described in #878 (and #532) it is not possible to send messages while waiting to receive them as well (or vice-versa), due to RPC stream client and server being locked by both Send() and Recv() functions.

    In order to solve this issue, I unlock the Mutex before calling the codec to receive data, and locking it again after codec returned. This way we still have the Mutex protection of the client/server structs, but we can send/receive at same time.

    :warning: Note that this assumes that codec does not need protection while being active.

    I did not update Send() because I assumed that it won't block. By only fixing Recv() the Send() worked as welll. Maybe depending on transport/codec it can block as well?

    I only considered these 2 components/files, as we are (currently) not having issues in other parts of the go-micro framework regarding to this. If needed I can update other parts as well to stay in sync with this change.

    I have tested this in our micro-service environment and it all seems to work fine. I did not trigger race conditions or other panics. But I did not yet perform (high) load or long running tests, which would make race conditions more likely to pop up (if there are any).

    enhancement WIP RPC 
    opened by veger 22
  • build command-line-arguments: cannot load crypto/ed25519: cannot find module providing package crypto/ed2551

    build command-line-arguments: cannot load crypto/ed25519: cannot find module providing package crypto/ed2551

    When I run the go-micro,it tips build command-line-arguments: cannot load crypto/ed25519: cannot find module providing package crypto/ed25519,but i find the module is exists in gopath.

    question more info requsted go.mod 
    opened by songdony 20
  • consul registry broken: mismatch in type due to import being expected from

    consul registry broken: mismatch in type due to import being expected from "github.com/hashicorp/consul/vendor"

    My Go version is 1.12.5.

    I'm building my package binary using the golang1.12.5-alpine3.9 Docker image as a package builder and I'm running the following commands in my Dockerfile to install all the necessary dependencies -

    # install source dependencies
    RUN go get -d -v ./...
    RUN go install -v ./...
    

    the go get command works as expected but the go install command crashes with an exit code: 2 and the following message -

    # github.com/micro/go-micro/registry/consul /go/src/github.com/micro/go-micro/registry/consul/watcher.go:48:33: cannot use cr.Client (type *"github.com/hashicorp/consul/api".Client) as type *"github.com/hashicorp/consul/vendor/github.com/hashicorp/consul/api".Client in argument to wp.RunWithClientAndLogger /go/src/github.com/micro/go-micro/registry/consul/watcher.go:213:37: cannot use cw.r.Client (type *"github.com/hashicorp/consul/api".Client) as type *"github.com/hashicorp/consul/vendor/github.com/hashicorp/consul/api".Client in argument to wp.RunWithClientAndLogger

    Just FYI, I was able to generate the Docker based package builder successfully up till 3 days ago.

    opened by desertjinn 20
  • gossip registry

    gossip registry

    I'm experiment with gossip registry and may be found some specific issue. When i'm start first service1 with specify gossip.Address("172.16.1.254:4223") and on other server service2 with registry.Addrs("172.16.1.254:4223") and gossip.Address("172.16.1.1:0")

    i see that members equal 2 on both sides, but the second service does not registered in registry. But if i stop service1 and start it with registry.Addrs("172.16.1.254:4223") and gossip.Address("172.16.1.1:xxx") where xxx is port provided by service2 all works fine. So issue appears only on first service1 when it start first without other members.

    opened by vtolstov 20
  • [BUG] Micro API fails on streaming endpoint unexpected nil error

    [BUG] Micro API fails on streaming endpoint unexpected nil error

    Describe the bug

    1. What are you trying to do? Run micro api gateway and call a streaming endpoint
    2. What did you expect to happen? Success, actual response
    3. What happens instead?
    {
        "code": 500,
        "detail": "grpc: client streaming protocol violation: get <nil>, want <EOF>",
        "id": "go.micro.client",
        "status": "Internal Server Error"
    }
    

    How to reproduce the bug: micro version 2.0.0 micro api --address :8085 --namespace some.ns --handler rpc http localhost:8085/my/streaming/endpoint

    If possible, please include a minimal code snippet here.

           ctx, cancel := context.WithCancel(ctx)
    	ch, errCh := data.GetSomeData(ctx, cancel, s.db, s.service)
    
    	for {
    		select {
    		case <-ctx.Done():
    			stream.Close()
    			return nil
    		case err := <-errCh:
    			return err
    		case msg := <-ch:
    			if msg == nil {
    				return nil
    			}
    			err := stream.Send(msg)
    			if err != nil {
    				cancel()
    				for len(ch) > 0 {
    					<-ch
    				}
    				return errors.InternalServerError(s.service.Name(), err.Error())
    			}
    		}
    	}
    

    Environment: Go Version: please paste go version output here

    go version go1.13.1 darwin/amd64
    
    opened by OGKevin 17
  • Disable timeout on RPC stream connection

    Disable timeout on RPC stream connection

    For a stream RPC connection I'd like to get rid of timeout, I see/found two solutions:

    1. Set timeout to some huge value and hope the client never stays connected long enough.
    2. Set timeout to -1 which 'somehow' seems to work. (Actually strconv.ParseUint() chokes on it and context timeout is not set (and no error is issued...))

    Both options feel hacky to me... Is there a better/recommended solution instead?

    enhancement 
    opened by veger 17
  • go get -u fails, apparently due to coreos/etcd move

    go get -u fails, apparently due to coreos/etcd move

    Go version: go1.13 Problem: when I run go get -u github.com/micro/go-micro I get the following error:

    # github.com/coreos/etcd/clientv3
    src/github.com/coreos/etcd/clientv3/auth.go:121:72: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.AuthEnable
    src/github.com/coreos/etcd/clientv3/auth.go:126:74: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.AuthDisable
    src/github.com/coreos/etcd/clientv3/auth.go:131:152: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.UserAdd
    src/github.com/coreos/etcd/clientv3/auth.go:136:144: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.UserAdd
    src/github.com/coreos/etcd/clientv3/auth.go:141:86: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.UserDelete
    src/github.com/coreos/etcd/clientv3/auth.go:146:122: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.UserChangePassword
    src/github.com/coreos/etcd/clientv3/auth.go:151:104: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.UserGrantRole
    src/github.com/coreos/etcd/clientv3/auth.go:156:80: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.UserGet
    src/github.com/coreos/etcd/clientv3/auth.go:161:72: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.UserList
    src/github.com/coreos/etcd/clientv3/auth.go:166:106: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.UserRevokeRole
    src/github.com/coreos/etcd/clientv3/auth.go:166:106: too many errors
    

    We first saw this error when our CI build began to fail, and at first couldn't reproduce it locally, but then were able to by adding the -u to the command. Alternately, here's a simple Dockerfile that demonstrates the issue when building it.

    This seems to be caused by the moving of github.com/coreos/etcd to github.com/etcd-io/etcd, as described here: https://github.com/etcd-io/etcd/pull/10044#issuecomment-417125341

    That change and the issue are from last year, so I'm not sure why this is just showing up now, but my colleagues and I (and our CI) can all reproduce the problem now.

    I will look into creating a PR that changes the imports from coreos/etcd to etcd-io/etcd.

    go.mod 
    opened by newrelic-eheinlein 17
  • Adding TLS to RPC

    Adding TLS to RPC

    I'm feeling a little bit dumb, I don't suppose you have an example of adding TLS to a RPC service?

    	security.Setup(log, security.Vault())
    	tlsConfig := &tls.Config{
    		GetCertificate: security.Vault().CertificateGetter(),
    	}
    
    	// initialise service
    	service.Init(
    		micro.Action(func(c *cli.Context) {
    			if c.Bool("debug") {
    				log.Level = logrus.DebugLevel
    				log.Debug("Debug log level enabled")
    			}
    		}),
    // transport.TLSConfig(tlsConfig),
    	)
    
    	countryHandler, err := handler.NewCountryHandler()
    	if err != nil {
    		log.Fatal(err)
    	}
    
    	country.RegisterCountriesHandler(service.Server(), countryHandler)
    
    	if err := service.Run(); err != nil {
    		log.Fatal(err)
    	}
    

    I tried adding transport options to the service.Init but it's clearly not that simple?

    opened by freman 17
  • How to add value to metadata?

    How to add value to metadata?

    Hello,

    I want to add some value to metadata from grpc method, then process in wrapper, however I found since grpc 1.40, we can't modify the context, the proposed solution is to use metadata.NewIncomingContext.

    https://github.com/grpc/grpc-go/issues/4363

    I don't know how to do it in micro?

    func (s *ServerImpl) Location(ctx context.Context, req *pb.LocationRequest, rsp *pb.City) error {
    // add value to context
    	md, _ := metadata.FromContext(ctx)
    	if md != nil {
    		md.Set("tag", "value")
    	}
    }
    
    func LogWrapper(fn server.HandlerFunc) server.HandlerFunc {
    	return func(ctx context.Context, req server.Request, rsp interface{}) error {
     		err := fn(ctx, req, rsp)
     		value, _ := md.Get("tag")  // value is empty
    }
    }
    
    help wanted 
    opened by seaguest 16
  • Documentation unavailable : not displayed due to license restrictions

    Documentation unavailable : not displayed due to license restrictions

    Hello,

    This is not a #1956 or #1953 duplicate. :)

    I tried to access : https://pkg.go.dev/github.com/micro/go-micro/v3?tab=overview But the documentation is unavailable due to license restrictions.

    Thanks,

    opened by scorsi 15
  • timestamppb.timestamp cause

    timestamppb.timestamp cause "json: cannot unmarshal object into Go value of type string" after upgrade v4.6

    Describe the bug

    1. I use golang and upgrade go micro from v4.5 to v4.6
    2. I expected that all of my integration test should pass
    3. What happens instead? My logger print errors below: {"id":"go.micro.client.codec","code":500,"detail":"json: cannot unmarshal object into Go value of type string","status":"Internal Server Error"} In general it is said go.micro.client.codec has error json: cannot unmarshal object into Go value of type string.

    How to reproduce the bug

    My message structure in proto file

    message Meeting {
        // @inject_tag: json:"start" validate:"required"
        google.protobuf.Timestamp start = 1;
    }
    

    The golang structure created by protoc

    type Meeting struct {
    	state         protoimpl.MessageState
    	sizeCache     protoimpl.SizeCache
    	unknownFields protoimpl.UnknownFields
    
    	// @inject_tag: json:"start" validate:"required"
    	Start *timestamppb.Timestamp `protobuf:"bytes,1,opt,name=start,proto3" json:"start" validate:"required"`
    }
    

    When I assign value to meeting.Start, the code like:

    	meeting.Start = timestamp.New(model.Start) // timestamp imported from "google.golang.org/protobuf/types/known/timestamppb"
    
    

    There is no such value if I continue using go micro v4.5. I am using libprotoc 3.21.6 and protoc-gen-go v1.27.1

    Go Version output:

    go version go1.17.13 darwin/amd64
    

    The go env output:

    GO111MODULE=""
    GOARCH="amd64"
    GOBIN=""
    GOCACHE="/Users/zhihong/Library/Caches/go-build"
    GOENV="/Users/zhihong/Library/Application Support/go/env"
    GOEXE=""
    GOEXPERIMENT=""
    GOFLAGS=""
    GOHOSTARCH="amd64"
    GOHOSTOS="darwin"
    GOINSECURE=""
    GOMODCACHE="/Users/zhihong/go/pkg/mod"
    GONOPROXY=""
    GONOSUMDB=""
    GOOS="darwin"
    GOPATH="/Users/zhihong/go"
    GOPRIVATE=""
    GOPROXY="https://proxy.golang.org,direct"
    GOROOT="/usr/local/go"
    GOSUMDB="sum.golang.org"
    GOTMPDIR=""
    GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
    GOVCS=""
    GOVERSION="go1.17.13"
    GCCGO="gccgo"
    AR="ar"
    CC="clang"
    CXX="clang++"
    CGO_ENABLED="1"
    GOMOD="/dev/null"
    CGO_CFLAGS="-g -O2"
    CGO_CPPFLAGS=""
    CGO_CXXFLAGS="-g -O2"
    CGO_FFLAGS="-g -O2"
    CGO_LDFLAGS="-g -O2"
    PKG_CONFIG="pkg-config"
    GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/r7/f018pp653m3_fw0xv9jx_tgh0000gn/T/go-build2635807831=/tmp/go-build -gno-record-gcc-switches -fno-common"
    

    Any suggestions welcomed, thanks,

    James

    opened by gzhhong 4
  • Broker use rabbitmq Publish is Err rpc_router.go:512 panic Call using zero Value argument

    Broker use rabbitmq Publish is Err rpc_router.go:512 panic Call using zero Value argument

    b := rabbitmq.NewBroker(  
    	broker.Addrs(rabbitmqUrl),  
    	rabbitmq.ExchangeName("user_exchange"),  
    	rabbitmq.DurableExchange(),  
    )  
    service := micro.NewService(  
    	micro.Broker(b),
    )
    event := micro.NewEvent("user_reg", service.Client())
    err = event.Publish(context.TODO(), &pb.TestReq1{Id:1340126})
    

    The pb.TestReq1 is protobuf.

    type TestReq1 struct {
         state         protoimpl.MessageState
         sizeCache     protoimpl.SizeCache
         unknownFields protoimpl.UnknownFields
    
         Id int32 `protobuf:"varint,1,opt,name=id,proto3" json:"id,omitempty"`
    }
    

    I'm using v3. The event.Publish() is Err file=server/rpc_router.go:512 panic recoverd: reflect: Call using zero Value argument.

    Then I modified the source code. It will work properly.

    vals = append(vals, reflect.ValueOf(msg.Payload())) => vals = append(vals, reflect.ValueOf(msg.Body()))

    Need help.

    opened by toegg 0
  • Allow passing context to store operations

    Allow passing context to store operations

    This allows passing a context with opentelemetry tracing information to remote services / store implementations.

    Regarding the pr template:

    1. Make sure this PR targets the develop branch. We follow the git-flow branching model.

    there is no develop branch ;-)

    opened by butonic 6
  • [BUG] go-micro error reported during press test

    [BUG] go-micro error reported during press test

    go-micro

    Concurrency: 5000 Connection pooling: 10

    2022/09/01 17:17:03 client.go:47: INFO : concurrency: 5000
    requests per client: 200
     
    2022/09/01 17:17:03 client.go:49: INFO : Servers: 10.130.12.14:8900
     
    2022/09/01 17:17:03 client.go:57: INFO : message size: 581 bytes
     
    2022/09/01 17:17:07 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:07 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:07 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:08 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
    2022/09/01 17:17:35 stats.go:16: INFO : 花了 31302 ms 用于 1000000 请求
    2022/09/01 17:17:35 stats.go:38: INFO : 发送请求    : 1000000
    2022/09/01 17:17:35 stats.go:39: INFO : 收到的请求    : 1000000
    2022/09/01 17:17:35 stats.go:40: INFO : 收到的成功请求 : 999963
    2022/09/01 17:17:35 stats.go:48: INFO : 吞吐量  (TPS)    : 31946
     
    2022/09/01 17:17:35 stats.go:60: INFO : 平均值: 137153203.650 ns, 中位数: 4337898.500 ns, 最大值: 4303066047.000 ns, 最小值: 243587.000 ns, p99.9: 1817774720.500 ns
    2022/09/01 17:17:35 stats.go:61: INFO : 平均值: 137.153 ms, 中位数: 4.338 ms, 最大值: 4303.066 ms, 最小值: 0.244 ms, p99.9: 1817.775 ms
    2022/09/01 17:17:35 stats.go:62: INFO : [go-micro]: TPS: 31946.84, TP99: 887.68ms, TP999: 1817.77ms (b=581 Byte, c=5000, n=1000000)
    Info: [go-micro]: TPS: 31946.84, TP99: 887.68ms, TP999: 1817.77ms (b=581 Byte, c=5000, n=1000000)
    

    grpc

    Concurrency: 5000 Connection pooling: 10

    2022/09/01 17:18:27 grpc_client.go:43: INFO : concurrency: 5000
    requests per client: 200
    
    2022/09/01 17:18:27 grpc_client.go:46: INFO : Servers: 10.130.12.14:8900
    
    2022/09/01 17:18:27 grpc_client.go:52: INFO : message size: 581 bytes
    
    
    2022/09/01 17:18:33 stats.go:16: INFO : 花了 5953 ms 用于 1000000 请求
    2022/09/01 17:18:34 stats.go:38: INFO : 发送请求    : 1000000
    2022/09/01 17:18:34 stats.go:39: INFO : 收到的请求    : 1000000
    2022/09/01 17:18:34 stats.go:40: INFO : 收到的成功请求 : 1000000
    2022/09/01 17:18:34 stats.go:48: INFO : 吞吐量  (TPS)    : 167982
    
    2022/09/01 17:18:34 stats.go:60: INFO : 平均值: 28944916.637 ns, 中位数: 25278866.000 ns, 最大值: 255435674.000 ns, 最小值: 207720.000 ns, p99.9: 137116793.500 ns
    2022/09/01 17:18:34 stats.go:61: INFO : 平均值: 28.945 ms, 中位数: 25.279 ms, 最大值: 255.436 ms, 最小值: 0.208 ms, p99.9: 137.117 ms
    2022/09/01 17:18:34 stats.go:62: INFO : [grpc]: TPS: 167982.53, TP99: 80.10ms, TP999: 137.12ms (b=581 Byte, c=5000, n=1000000)
    Info: [grpc]: TPS: 167982.53, TP99: 80.10ms, TP999: 137.12ms (b=581 Byte, c=5000, n=1000000)
    

    A grpc press test on a similarly configured machine shows no errors

    @asim

    opened by JellyZero 1
  • [BUG] go-micro registry node nil pointer

    [BUG] go-micro registry node nil pointer

    Describe the bug

    1. What are you trying to do?
    • Build a Rpc Client to request a Rpc Server
    1. What did you expect to happen?
    • when rpc client try to get rpc server addr and request, "nil pointer exception" should not happen
    1. What happens instead?
    • a panic occur which have already get a Rlock wihout unlock
    /usr/local/go17/src/runtime/panic.go:221 (0xc76946)
    /usr/local/go17/src/runtime/signal_unix.go:735 (0xc76916)
    mydir/pkg/mod/go-micro.dev/[email protected]/util/registry/util.go:65 (0x1226a8c)
    mydir/pkg/mod/go-micro.dev/[email protected]/util/registry/util.go:85 (0x1226cfc)
    mydir/pkg/mod/go-micro.dev/[email protected]/registry/cache/cache.go:126 (0x1228a78)
    mydir/pkg/mod/go-micro.dev/[email protected]/registry/cache/cache.go:436 (0x122a95d)
    mydir/pkg/mod/go-micro.dev/[email protected]/selector/default.go:52 (0x127feeb)
    mydir/pkg/mod/github.com/asim/go-micro/plugins/client/grpc/[email protected]/grpc.go:90 (0x1de1242)
    mydir/pkg/mod/github.com/asim/go-micro/plugins/client/grpc/[email protected]/grpc.go:403 (0x1de4df6)
    [email protected]/gomicro/wrap.go:595 (0x1e0c50c)
    [email protected]/gomicro/wrap.go:575 (0x1e0b99e)
    [email protected]/gomicro/wrap.go:312 (0x1e0936f)
    myproject/cmu.pb.micro.go:238 (0x1b71d4e)
    myproject/upcw_service.go:53 (0x1c0f57a)
    myproject/upcw_common.go:33 (0x1e52904)
    mydir/pkg/mod/github.com/gin-gonic/[email protected]/context.go:147 (0x1abd0ea)
    [email protected]/middleware/http/recovery.go:39 (0x1abd0d4)
    mydir/pkg/mod/github.com/gin-gonic/[email protected]/context.go:147 (0x1af3dca)
    myproject/comm/middleware/session/session.go:82 (0x1af3dad)
    mydir/pkg/mod/github.com/gin-gonic/[email protected]/context.go:147 (0x1af714d)
    myproject/comm/metadata/metadata.go:113 (0x1af7134)
    mydir/pkg/mod/github.com/gin-gonic/[email protected]/context.go:147 (0x1ab8ef3)
    [email protected]/middleware/http/gin.go:293 (0x1ab8d73)
    mydir/pkg/mod/github.com/gin-gonic/[email protected]/context.go:147 (0x116c369)
    mydir/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:403 (0x116bfb9)
    mydir/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:364 (0x116bb33)
    mydir/pkg/mod/github.com/gorilla/[email protected]/mux.go:210 (0x1cb0b6e)
    /usr/local/go17/src/net/http/server.go:2879 (0xf0ee7a)
    /usr/local/go17/src/net/http/server.go:1930 (0xf0a467)
    /usr/local/go17/src/runtime/asm_amd64.s:1581 (0xc945a0)
    
    • a goroutine panic with lock and other goroutine can't get the lock: pprof image

    • cause other goroutine can not get lock resource until I restart it.

    How to reproduce the bug: I do not know how to reproduce and why happen. I found it in product environment sometime and I have to restart the program to fix it or the program can not make rpc request.

    Environment: Go Version: please paste go version output here

    
    GO111MODULE="on"
    GOARCH="amd64"
    GOBIN=""
    GOCACHE="/home/myname/.cache/go-build"
    GOENV="/home/myname/.config/go/env"
    GOEXE=""
    GOEXPERIMENT=""
    GOFLAGS=""
    GOHOSTARCH="amd64"
    GOHOSTOS="linux"
    GOINSECURE=""
    GOMODCACHE="/home/myname/goproject17/pkg/mod"
    GONOPROXY=""
    GONOSUMDB=""
    GOOS="linux"
    GOPATH="/home/myname/goproject17"
    GOPRIVATE=""
    GOPROXY=""
    GOROOT="/usr/local/go17"
    GOSUMDB="sum.golang.org"
    GOTMPDIR=""
    GOTOOLDIR="/usr/local/go17/pkg/tool/linux_amd64"
    GOVCS=""
    GOVERSION="go1.17.6"
    GCCGO="gccgo"
    AR="ar"
    CC="gcc"
    CXX="g++"
    CGO_ENABLED="1"
    GOMOD="/dev/null"
    CGO_CFLAGS="-g -O2"
    CGO_CPPFLAGS=""
    CGO_CXXFLAGS="-g -O2"
    CGO_FFLAGS="-g -O2"
    CGO_LDFLAGS="-g -O2"
    PKG_CONFIG="pkg-config"
    GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build4281984978=/tmp/go-build -gno-record-gcc-switches"
    
    bug help wanted 
    opened by gdgrc 20
Releases(v4.9.0)
Owner
Asim Aslam
Working on Micro
Asim Aslam
Distributed-Services - Distributed Systems with Golang to consequently build a fully-fletched distributed service

Distributed-Services This project is essentially a result of my attempt to under

Hamza Yusuff 6 Jun 1, 2022
Distributed lock manager. Warning: very hard to use it properly. Not because it's broken, but because distributed systems are hard. If in doubt, do not use this.

What Dlock is a distributed lock manager [1]. It is designed after flock utility but for multiple machines. When client disconnects, all his locks are

Sergey Shepelev 25 Dec 24, 2019
Tarmac is a unique framework designed for the next generation of distributed systems

Framework for building distributed services with Web Assembly

Benjamin Cane 220 Nov 13, 2022
Micro is a platform for cloud native development

Micro Overview Micro addresses the key requirements for building services in the cloud. It leverages the microservices architecture pattern and provid

Micro 11.5k Nov 18, 2022
A distributed systems library for Kubernetes deployments built on top of spindle and Cloud Spanner.

hedge A library built on top of spindle and Cloud Spanner that provides rudimentary distributed computing facilities to Kubernetes deployments. Featur

null 21 Nov 9, 2022
MIT 6.824: Distributed Systems

MIT 6.824 is a core 12-unit graduate subject with lectures, readings, programming labs, an optional project, a mid-term exam, and a final exam.

CC Chang 2 Jul 6, 2022
Distributed Systems 2021 - Miniproject 3

Distributed Systems 2021 -- Miniproject 3 Hand-in Date: 1 December 2021 (at 23:59) What to submit on learnit: a single zip-compressed file containing:

null 0 Dec 11, 2021
Labs, solutions and related materials from the MIT 6.824 Distributed Systems course.

MIT 6.824 Distributed Systems Labs, solutions and related materials from the MIT 6.824 Distributed Systems course. Overview From the official website:

Arindam Das 46 Nov 5, 2022
MIT6.824 Distributed Systems

MIT6.824-Distributed-Systems My Solutions for MIT6.824

null 0 Jan 28, 2022
Distributed reliable key-value store for the most critical data of a distributed system

etcd Note: The main branch may be in an unstable or even broken state during development. For stable versions, see releases. etcd is a distributed rel

etcd-io 41.9k Nov 27, 2022
Skynet is a framework for distributed services in Go.

##Introduction Skynet is a communication protocol for building massively distributed apps in Go. It is not constrained to Go, so it will lend itself n

null 2k Nov 18, 2022
a dynamic configuration framework used in distributed system

go-archaius This is a light weight configuration management framework which helps to manage configurations in distributed system The main objective of

null 203 Nov 1, 2022
A Distributed Content Licensing Framework (DCLF) using Hyperledger Fabric permissioned blockchain.

A Distributed Content Licensing Framework (DCLF) using Hyperledger Fabric permissioned blockchain.

Tal Derei 4 Nov 4, 2022
Flowgraph package for scalable asynchronous system development

flowgraph Getting Started go get -u github.com/vectaport/flowgraph go test Links Wiki Slides from Minneapolis Golang Meetup, May 22nd 2019 Overview F

Scott Johnston 53 Sep 26, 2022
distributed data sync with operational transformation/transforms

DOT The DOT project is a blend of operational transformation, CmRDT, persistent/immutable datastructures and reactive stream processing. This is an im

DOT & Chain 74 Nov 9, 2022
High performance, distributed and low latency publish-subscribe platform.

Emitter: Distributed Publish-Subscribe Platform Emitter is a distributed, scalable and fault-tolerant publish-subscribe platform built with MQTT proto

emitter 3.4k Nov 25, 2022
Fast, efficient, and scalable distributed map/reduce system, DAG execution, in memory or on disk, written in pure Go, runs standalone or distributedly.

Gleam Gleam is a high performance and efficient distributed execution system, and also simple, generic, flexible and easy to customize. Gleam is built

Chris Lu 3.1k Nov 27, 2022
Simplified distributed locking implementation using Redis

redislock Simplified distributed locking implementation using Redis. For more information, please see examples. Examples import ( "fmt" "time"

Black Square Media 858 Nov 22, 2022