drpc is a lightweight, drop-in replacement for gRPC

Overview

DRPC

A drop-in, lightweight gRPC replacement.

Go Report Card Go Doc Beta

Please see:

DRPC is:

  • Simple, at just a few thousands lines of code!
  • Compatible. Works for many gRPC use-cases as-is!
  • Fast. DRPC has a lightning quick wire format
  • Extensible. DRPC is transport agnostic, supports middleware, and is designed around interfaces.
  • Battle Tested. Already used in production for years across tens of thousands of servers.

Compare with gRPC:

BenchmarkUnitary/GRPC-8          	   40166	     29717 ns/op	    8271 B/op	     169 allocs/op
BenchmarkUnitary/DRPC-8          	   53380	     21705 ns/op	    2882 B/op	      39 allocs/op
BenchmarkInputStream/GRPC-8      	 1308865	       836.5 ns/op	     370 B/op	      11 allocs/op
BenchmarkInputStream/DRPC-8      	  449756	      2555 ns/op	      63 B/op	       3 allocs/op
BenchmarkOutputStream/GRPC-8     	 1453718	       910.2 ns/op	     283 B/op	       9 allocs/op
BenchmarkOutputStream/DRPC-8     	  455004	      2571 ns/op	      63 B/op	       3 allocs/op
BenchmarkBidirectionalStream/GRPC-8       113724	     10273 ns/op	     873 B/op	      40 allocs/op
BenchmarkBidirectionalStream/DRPC-8       218518	      5396 ns/op	     128 B/op	       6 allocs/op

Licensing

DRPC is licensed under the MIT/expat license. See the LICENSE file for more.

Comments
  • How to get protoc-gen-go-drpc

    How to get protoc-gen-go-drpc

    Hi folks,

    I'm a bit new to the proto and trying to replace the existing grpc with drpc in my project.

    I look into the doc and it says "Place protoc-gen-go-drpc in your $PATH and have protoc generate your definitions", but find nowhere to get the binary protoc-gen-go-drpc. There is only a simple main.go and useless README under storj.io/drpc/cmd/protoc-gen-go-drpc.

    May I know how to generate the drpc go code based on the proto I already have and where to get the required tool?

    Appreciate for your help!

    opened by cindyzqtnew 23
  • improve Error package

    improve Error package

    i need to transfer error message from drpc server to drpc client, nor error type is not exportable. Do you plan to export Error type and ability to get error message ? also it will be very useful to copy codes from grpc ( https://pkg.go.dev/google.golang.org/grpc/codes?utm_source=godoc ) via const to drpc

    documentation enhancement question 
    opened by vtolstov 9
  • Allow maximum payload size to be configurable

    Allow maximum payload size to be configurable

    https://github.com/storj/drpc/blob/41e8fa5f513fe0893d3bef72310d0ccbba66cc74/drpcwire/reader.go#L119

    It appears the maximum payload size is 4MB right now and isn't configurable. Is this an intentional design choice?

    opened by kylecarbs 8
  • avoid string allocation for each request

    avoid string allocation for each request

    if you change generator and generate for each method const variable name like

    const (
      accountAccontServiceCreate = "/account.AccountService/Create"
    )
    
    func (c *drpcAccountServiceClient) Create(ctx context.Context, in *AccountCreateReq) (*Account, error) {
            out := new(Account)
            err := c.cc.Invoke(ctx, accountAccontServiceCreate, drpcEncoding_File_account_proto{}, in, out)
            if err != nil {
                    return nil, err
            }
            return out, nil
    }
    

    we completely can avoid new string allocation for each request and minimize garbage. what you think ?

    enhancement 
    opened by vtolstov 7
  • drpcconn: fully read unary RPCs

    drpcconn: fully read unary RPCs

    The previous behavior left a potential "CloseSend" frame pending on the manager's read queue. This delayed detection of a potential connection loss until the next RPC is started on the connection, which discards the old frame because of the stream ID mismatch.

    I'm sure the test can be improved; if only by simplifying testTransport.

    This change is a requirement for potential solutions to #11.

    opened by maxtruxa 7
  • What is the max message size handling approach?

    What is the max message size handling approach?

    grpc has grpc.MaxRecvMsgSize and grpc.MaxSendMsgSize which restrict the maximum amount of data which can be sent or received by a grpc client/server.

    What is the strategy that drpc uses? I did some tests with sending large amounts of data with the defaults. 1024 * 1024 bytes works fine. 1024 * 1024 * 100 bytes causes a data overflow protocol error from the server.

    Looking at the code I can see some various

    type ReaderOptions struct {
    	// MaximumBufferSize controls the maximum size of buffered
    	// packet data.
    	MaximumBufferSize int
    }
    

    and

    // Options controls configuration settings for a stream.
    type Options struct {
    	// SplitSize controls the default size we split packets into frames.
    	SplitSize int
    
    	// ManualFlush controls if the stream will automatically flush after every
    	// message send. Note that flushing is not part of the drpc.Stream
    	// interface, so if you use this you must be ready to type assert and
    	// call RawFlush dynamically.
    	ManualFlush bool
    
    	// MaximumBufferSize causes the Stream to drop any internal buffers that
    	// are larger than this amount to control maximum memory usage at the
    	// expense of more allocations. 0 is unlimited.
    	MaximumBufferSize int
    
    	// Internal contains options that are for internal use only.
    	Internal drpcopts.Stream
    }
    

    However, it isn't really clear the strategy or what I should be setting. Can anyone help here?

    question 
    opened by newhook 6
  • Document the protocol in explicit detail

    Document the protocol in explicit detail

    Some things that could use explicit documentation if one were to write a second, compatible implementation:

    • [x] The stream state machine with associated expected errors during different states. (https://github.com/storj/drpc/wiki/Docs:-Stream-state-machine)
    • [x] The wire format (https://github.com/storj/drpc/wiki/Docs:-Wire-protocol)
    • [x] Other decisions that were made around multiplexing, frame assembly, stream liveness, etc. (https://github.com/storj/drpc/wiki/Docs:-Project-goals)

    A good test would be to have someone other than those who have already worked on the code try to implement it in another language using only the documentation.

    documentation 
    opened by zeebo 6
  • Is import path supported in drpc gen?

    Is import path supported in drpc gen?

    Hi guys,

    I am trying to replace gRPC with dRPC in my testing project.

    There are a few proto files in multiple directories and under the same directory "pkg", and some basic protos like "github.com.google/protobuf" under "vendor" directory. When I try to generate the pb.go files, it keeps returning the error:

    protoc-gen-go-drpc: invalid Go import path "descriptor" for "google/protobuf/descriptor.proto"
    The import path must contain at least one period ('.') or forward slash ('/') character.
    See https://developers.google.com/protocol-buffers/docs/reference/go-generated#package for more information.
    

    Here is my gen cmd:

    for dir in ./pkg/abc/ ./pkg/efg; do \
      build/werror.sh /go_path/native/x86_64-pc-linux-gnu/protobuf/protoc
    -Ipkg:./vendor/github.com:./vendor/github.com/gogo/protobuf:./vendor/github.com/gogo/protobuf/protobuf:./vendor/go.etcd.io:./vendor/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis 
    --go-drpc_out=paths=source_relative,protolib=github.com/gogo/protobuf $dir/*.proto;
    

    There indeed a file "./vendor/github.com/gogo/protobuf/protobuf/google/protobuf/descriptor.proto" with "option go_package = descriptor". But it's pulled automatically and already exists in my local repo for a long time. I doubt the option "-I" not work with drpc-gen, so it try to locate the basic gogo proto from a wrong location?

    I've beed debugging for this issue for more than a week and look up all docs I could find but got no help.

    Appreciate for any help!

    question 
    opened by cindyzqtnew 4
  • unknown service handler option for server

    unknown service handler option for server

    https://pkg.go.dev/google.golang.org/grpc?utm_source=godoc#UnknownServiceHandler

    i'm really need this to be able to dynamically add handlers and also have ability to send custom response for unknown handler. can you implements something like this in drpc ?

    opened by vtolstov 4
  • graceful shutdown and handler timeout

    graceful shutdown and handler timeout

    i'm maintain a fork of go-micro ( github.com/unistack-org/micro ) and want to add drpc client/server support to it. grpc does not have native ability to pass context to server and also does not allow adding handler timeouts easy (via context with timeout or something like this) What you think about graceful shutdown of dprc handlers ? Do you plan to add option for server to specify handler timeout?

    question 
    opened by vtolstov 4
  • maximum number of connections

    maximum number of connections

    We've started rolling out drpc into production, and so far its been great! However, one issue we've run into is what appears to be memory leaks in the server. After looking at the memory profiles for a bit we see a bunch of memory being held by drpcwire reader ReadPacketUsing. My conjecture is that there are one of these per connection on the server, and each is holding a buffer which grows to the maximum message size that has ever been read over that connection.

    Is there a way to limit the maximum number of connections a server will allow? I see the InactivityTimeout is configurable, but no parameter for max connections.

    question 
    opened by newhook 3
  • debug logging

    debug logging

    The debug logging is pretty inflexible. It would be nice to be able to override the logging function, but at a minimum using log.Output vs creating a new standard logger would at least enable me to capture the data into our standard logging vs redirecting stdout.

    opened by newhook 0
  • logging addresses

    logging addresses

    I'd like to log address data on both the client & the server. Is that data currently accessible?

    On the server side drpc.Stream doesn't seem to expose any method to do that.

    On the client side I'm logging right now by wrapping a drpcconn.Conn, and at that point the server isn't known.

    question 
    opened by newhook 2
  • Context cancellation closes the connection

    Context cancellation closes the connection

    Hi,

    I'm looking into replacing grpc with drpc in one of my projects and initial tests show some nice performance improvements!

    Switching to drpc was fairly painless. The only snag was that context cancelling seemingly closes the connection. This behavior is different from grpc, see my tweaked drpc/grpc examples in https://github.com/tomyl/drpc/commit/acb08bd89225fe5db68ae4dd9fa82805d28b7364.

    Is this behavior intentional? I can't see that the documentation mentions it.

    Thanks, Tommy

    question 
    opened by tomyl 8
  • Reconnect client to server

    Reconnect client to server

    Hello everyone, I ran into a reconnect problem, if the server crashes in a panic and recovers, the client via tcp, of course, cannot reach it, except for creating a new client, I did not find a way to solve the problem. Maybe someone has a ready-made reconnect practice in the background?

    question 
    opened by spacecodewor 2
  • http based drpc client

    http based drpc client

    i'm try to create api gateway that support http rest, grpc and drpc on the same port

    i'm create grpc server, drpc http handler, and plain http handler and use the in http2 server as handler.

    plain http and grpc works fine, but drpc request does not goes to http2 server =( does it possible to get this worked with native drpc clients ?

    gsrv := grpc.NewServer(grpc.UnknownServiceHandler(h.ServeGRPC))
    comboHandler := newComboMux(h, gsrv, drpchttp.New(h))
    http2Server := &http2.Server{}
    hs := &http.Server{Handler: h2c.NewHandler(comboHandler, http2Server)}
    
    func (h *Handler) ServeDRPC(stream drpc.Stream, rpc string) error {
    	ctx := stream.Context()
    	logger.Infof(ctx, "drpc: %#+v", rpc)
    	return nil
    }
    
    func (h *Handler) HandleRPC(stream drpc.Stream, rpc string) error {
    	return h.ServeDRPC(stream, rpc)
    }
    
    
    func newComboMux(httph http.Handler, grpch http.Handler, drpch http.Handler) http.Handler {
    	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    		if r.ProtoMajor == 2 {
    			ct := r.Header.Get("content-type")
    			switch {
    			case strings.HasPrefix(ct, "application/grpc"):
    				grpch.ServeHTTP(w, r)
    				return
    			case strings.HasPrefix(ct, "application/drpc"):
    				drpch.ServeHTTP(w, r)
    				return
    			}
    		}
    		httph.ServeHTTP(w, r)
    	})
    }
    
    enhancement question 
    opened by vtolstov 8
Releases(v0.0.20)
Owner
Storj Labs
Storj Labs is Building the Decentralized Future
Storj Labs
Golang 微服务框架,支持 grpc/http,支持多种注册中心 etcd,consul,mdns 等

一个用于构建分布式系统的工具集或者轻框架,支持 grpc 和 http ,支持多种注册中心 consul ,etcd , mdns 等。

徐旭 26 Nov 30, 2022
Go gRPC RabbitMQ email microservice

Go, RabbitMQ and gRPC Clean Architecture microservice ?? ??‍?? Full list what has been used: GRPC - gRPC RabbitMQ - RabbitMQ sqlx - Extensions to data

Alexander 144 Dec 29, 2022
Sample cloud-native application with 10 microservices showcasing Kubernetes, Istio, gRPC and OpenCensus.

Online Boutique is a cloud-native microservices demo application. Online Boutique consists of a 10-tier microservices application. The application is

Google Cloud Platform 13.5k Dec 31, 2022
Automatic Service Mesh and RPC generation for Go micro services, it's a humble alternative to gRPC with Istio.

Mesh RPC MeshRPC provides automatic Service Mesh and RPC generation for Go micro services, it's a humble alternative to gRPC with Istio. In a nutshell

AstraNet Toolkit 69 Aug 22, 2022
Microservice Boilerplate for Golang with gRPC and RESTful API. Multiple database and client supported

Go Microservice Starter A boilerplate for flexible Go microservice. Table of contents Features Installation Todo List Folder Structures Features: Mult

Ahmad Saugi 14 Jul 28, 2022
Go microservices with REST, and gRPC using BFF pattern.

Go microservices with REST, and gRPC using BFF pattern. This repository contains backend services. Everything is dockerized and ready to

Oguzhan 152 Jan 4, 2023
A gRPC API for Warhammer Age of Sigmar

Warhammer ?? A gRPC API for Warhammer Age of Sigmar Intro ℹ️ Skip to Quick Start What is this? An API for creating, reading, deleting, and updating a

Britton Hayes 1 Oct 26, 2021
一款依赖 etcd 作为注册中心的 Golang 轻量级 GRPC 框架

Golang 微服务 GRPC 标准框架(轻量级) 特性介绍 可使用 etcd 集群或单节点作为注册中心 客户端请求服务端自带负载均衡 服务端启动后自动向 etcd 注册,默认每 10s 进行一次心跳续租 自带优雅停止 panic recover 服务端无需指定启动端口,当然你也可以通过 WithP

兰陵美酒郁金香丶❀ 2 Nov 11, 2021
Testing ground for CRUD backend using Golang, gRPC, protobufs

blog-example-service Simple example CRUD backend using Golang, gRPC, and protobufs. Using with MongoDB as the database (default) You will need a Mongo

Jordan Weiner 0 Dec 16, 2021
Just a quick demo of how you can use automatically generated protobuffer and gRPC code from buf.build

buf.build demo The purpose of this repository is to demonstrate how to use the services offered by buf.build for hosting protobuffer definitions and a

Bjørn Borud 0 Jan 4, 2022
Golang Microservice making use of protobuf and gRPC as the underlying transport protocol.

Go-Microservices Golang Microservice making use of protobuf and gRPC as the underlying transport protocol. I will be building a generic microservice,

Kathurima Kimathi 0 Jan 5, 2022
Authentication-microservice - Microservice for user authentication built with golang and gRPC

Authentication-microservice - Microservice for user authentication built with golang and gRPC

Isaque Veras 6 May 30, 2022
Article - Golang mysql rest grpc

To bring up the project run: To bring up database on docker in root folder run:

Andrew 0 Sep 26, 2022
Jaeger-s3 - Jaeger gRPC storage plugin for Amazon S3

jaeger-s3 jaeger-s3 is gRPC storage plugin for Jaeger, which uses Amazon Kinesis

Johannes Würbach 11 Dec 26, 2022
Customer-microservice - Microservice of customer built with golang and gRPC

?? Building microservices to manage customer data using Go and gRPC Command to g

Isaque Veras Labs 3 Sep 8, 2022
An production-ready microservice using Go and a few lightweight libraries

Go Micro Example This small sample project was created as a collection of the various things I've learned about best practices building microservices

Sean K Smith 102 Dec 26, 2022
Blue is a lightweight cloud-native gateway solution to handle millions of routing endpoints with a large number of connections.

Blue is a lightweight cloud-native gateway solution to handle millions of routing endpoints with a large number of connections.

null 456 Jan 19, 2022
Nano - Lightweight, facility, high performance golang based game server framework

Nano Nano is an easy to use, fast, lightweight game server networking library fo

Lonng 2.2k Jan 1, 2023
Drop-in replacement for Go's flag package, implementing POSIX/GNU-style --flags.

Description pflag is a drop-in replacement for Go's flag package, implementing POSIX/GNU-style --flags. pflag is compatible with the GNU extensions to

Steve Francia 2k Dec 30, 2022
Drop-in replacement for Go's flag package, implementing POSIX/GNU-style --flags.

Description pflag is a drop-in replacement for Go's flag package, implementing POSIX/GNU-style --flags. pflag is compatible with the GNU extensions to

Alex Ogier 526 Nov 25, 2022