Golang gRPC Middlewares: interceptor chaining, auth, logging, retries and more.

Overview

Go gRPC Middleware

Travis Build Go Report Card GoDoc SourceGraph codecov Apache 2.0 License quality: production Slack

gRPC Go Middleware: interceptors, helpers, utilities.

Middleware

gRPC Go recently acquired support for Interceptors, i.e. middleware that is executed either on the gRPC Server before the request is passed onto the user's application logic, or on the gRPC client around the user call. It is a perfect way to implement common patterns: auth, logging, message, validation, retries or monitoring.

These are generic building blocks that make it easy to build multiple microservices easily. The purpose of this repository is to act as a go-to point for such reusable functionality. It contains some of them itself, but also will link to useful external repos.

grpc_middleware itself provides support for chaining interceptors, here's an example:

import "github.com/grpc-ecosystem/go-grpc-middleware"

myServer := grpc.NewServer(
    grpc.StreamInterceptor(grpc_middleware.ChainStreamServer(
        grpc_ctxtags.StreamServerInterceptor(),
        grpc_opentracing.StreamServerInterceptor(),
        grpc_prometheus.StreamServerInterceptor,
        grpc_zap.StreamServerInterceptor(zapLogger),
        grpc_auth.StreamServerInterceptor(myAuthFunction),
        grpc_recovery.StreamServerInterceptor(),
    )),
    grpc.UnaryInterceptor(grpc_middleware.ChainUnaryServer(
        grpc_ctxtags.UnaryServerInterceptor(),
        grpc_opentracing.UnaryServerInterceptor(),
        grpc_prometheus.UnaryServerInterceptor,
        grpc_zap.UnaryServerInterceptor(zapLogger),
        grpc_auth.UnaryServerInterceptor(myAuthFunction),
        grpc_recovery.UnaryServerInterceptor(),
    )),
)

Interceptors

Please send a PR to add new interceptors or middleware to this list

Auth

  • grpc_auth - a customizable (via AuthFunc) piece of auth middleware

Logging

  • grpc_ctxtags - a library that adds a Tag map to context, with data populated from request body
  • grpc_zap - integration of zap logging library into gRPC handlers.
  • grpc_logrus - integration of logrus logging library into gRPC handlers.
  • grpc_kit - integration of go-kit logging library into gRPC handlers.

Monitoring

Client

  • grpc_retry - a generic gRPC response code retry mechanism, client-side middleware

Server

Status

This code has been running in production since May 2016 as the basis of the gRPC micro services stack at Improbable.

Additional tooling will be added, and contributions are welcome.

License

go-grpc-middleware is released under the Apache 2.0 license. See the LICENSE file for details.

Issues
  • go-grpc-middleware.v2: Action Plan.

    go-grpc-middleware.v2: Action Plan.

    Hi :wave:

    We have tons of stuff to improve in this library and there wasn't really time for this. Let's do it finally! Let's maintain this issue as a single place to track our work. :muscle: cc @domgreen @johanbrandhorst @mwitkow

    The plan is to start from scratch in the v2 branch. We will work in the background until it's done.

    Work Done

    • [x] Started v2 branch, implemented all while leaving same API. Assignee: @bwplotka

    • [x] Never again use those weird module_names :upside_down_face: Done Assignee: @bwplotka

    • [X] Multi-module architecture, allowing us to avoid dependency hell, but also retaining mono repo. Module structure: Done Assignee: @bwplotka The key part is to make sure the core is never importing the provider's code. There has to be indirection. This way the lib has minimum dependencies possible, with core literally having core only.

      If maintaining multi modules will be problematic, we can look into modularise cc @Helcaraxan

    TODO (help wanted!) for 2.0:

    Blockers:

    • [ ] Do one more review of all interfaces we expose to check if we can simplify something. Assignee @johanbrandhorst
    • [ ] Upgrade gRPC and use https://godoc.org/google.golang.org/grpc#ChainUnaryInterceptor Assignee: @bwplotka
    • [ ] Upgrade golang/protobuf to v2. Same with gogo? Assignee: @bwplotka
    • [ ] Move to GitHub actions Assignee: @bwplotka
    • [ ] MIgration docs!
    • [ ] Start using gorelease (https://godoc.org/golang.org/x/exp/cmd/gorelease)
    • [ ] Consider adding whitelist behaviour to auth interceptor: How to enable whitelist auth?
    • [ ] Add tracing interface and move opentracing to provides. Add opentelemetry providers.
    • [ ] Use the v2 in Thanos for first hand expierience in upgrading. Assignee: @bwplotka

    Can be done in 2.x

    • [x] Move https://github.com/grpc-ecosystem/go-grpc-prometheus as another provider. Deprecate repo.
    • [ ] Add failint to avoid not needed imports (e.g v1 packages)
    • [ ] Move to more maintained validators
    • [ ] Review if we can simplify retry internally
    • [ ] Stale bot
    • [ ] Dependabot

    Any other ideas?

    enhancement help wanted v2 
    opened by bwplotka 46
  • Request Logging

    Request Logging

    This PR achieves to add in a Request/Response logger in grpc_middlewares v2 branch.

    Fixes #310

    Current TODO -

    • [x] ~~Initial setup for the RequestReporter~~ Configure the request logger in the old logger itself
    • [x] Write tests for the new reporter
    • [x] Write examples for the new reporter
    • [x] Decide what to log after we can access the request/response object cc @bwplotka ref: https://github.com/grpc-ecosystem/go-grpc-middleware/pull/311/files#diff-0c98eb4ff4e54e07ef0a999e5d5bd678R137
    • [x] Naming of the reporter - Is the current naming correct, or do we want something else :stuck_out_tongue:

    If any suggestions are there, feel free to add in more :sweat_smile:

    Signed-off-by: Yash Sharma [email protected]

    v2 
    opened by yashrsharma44 22
  • Moved to GH actions; Added lint; Added issue/PR templates.

    Moved to GH actions; Added lint; Added issue/PR templates.

    Also pinned Go tools thanks to https://github.com/bwplotka/bingo

    Signed-off-by: Bartlomiej Plotka [email protected]

    opened by bwplotka 20
  • Add go-kit logging middleware

    Add go-kit logging middleware

    Greetings 👋

    I come from the Thanos project where we mostly use the go-kit logging library and we wanted some sweet gRPC logging for our users.

    I'm not a Go expert, so I mostly copy/pasted the tests from the Zap implementation and wrote the go-kit logging system.

    The tests suite is green and it works pretty swell with Thanos:

    level=debug ts=2019-08-09T12:24:01.299269Z caller=storeset.go:223 component=storeset msg="updating healthy stores" externalLabelOccurrencesInStores="map[string]int{\"\":1}"
    level=info ts=2019-08-09T12:24:01.299296Z caller=storeset.go:266 component=storeset msg="adding new store to query storeset" address=127.0.0.1:10904
    level=info ts=2019-08-09T12:24:06.300377Z caller=payload_interceptors.go:134 system=grpc span.kind=client grpc.service=thanos.Store grpc.method=Info grpc.request.content={}
    level=info ts=2019-08-09T12:24:06.302309Z caller=payload_interceptors.go:134 system=grpc span.kind=client grpc.service=thanos.Store grpc.method=Info grpc.response.content="{\"maxTime\":\"9223372036854775807\",\"storeType\":\"SIDECAR\"}"
    level=debug ts=2019-08-09T12:24:06.302375Z caller=client_interceptors.go:49 system=grpc span.kind=client grpc.service=thanos.Store grpc.method=Info msg="finished client unary call" error=null grpc.code=OK grpc.time_ms=2.145```
    
    Let me know if you have any feedback 😄 
    opened by adrien-f 17
  • v2: Client unary interceptor timeout on v2 branch

    v2: Client unary interceptor timeout on v2 branch

    👋 Hi, this PR is related to the first one on the v1 #329 It was reworked for the v2 branch

    opened by instabledesign 16
  • gRPC Zap ReplaceGrpcLoggerV2

    gRPC Zap ReplaceGrpcLoggerV2

    We at Hootsuite were noticing noisy logs coming from grpc and we noticed it's because grpc_zap implements logger v1 interface. We need the logger v2 implemented instead, so I have opened this PR.

    This is a revived and comments addressed version of #144 which was opened by @jamisonhyatt. Feel free to just merge this in after approval if it looks good!

    opened by kush-patel-hs 16
  • [Rate-limit provider]: Add token bucket implementation of rate-limiter

    [Rate-limit provider]: Add token bucket implementation of rate-limiter

    Use tockenbucket provider

    opened by MalloZup 14
  • Update Deps

    Update Deps

    We are updating golang.org/x/net deps to release-branch.go1.11 in Kubernetes but we are facing a problem as go-grpc-middleware still uses the older version of golang.org/x/net package.

    Will it be a good idea to update Gopkg.lock so that it uses the latest version of dependencies?

    on reporter 
    opened by PrasadG193 13
  • [v2] Add skip interceptor

    [v2] Add skip interceptor

    Resolve #363

    It's hard to transfer a interceptors.UnaryServerInterceptor into interceptors.Reporter, so I use the old way to implement this feature.

    opened by XSAM 13
  • CUE-based validator?

    CUE-based validator?

    CUE language for data validation has been gaining some momentum recently. It claims to have protobuf integration similar to what grpc_validator is doing.

    I'm wondering if it would make sense to bring CUE-based data validation in gRPC middleware?

    opened by sashabaranov 0
  • Add all-validator support

    Add all-validator support

    Reopen pull request #438 with new branch and target branch is v2.

    FYI, I changed PingErrorRequest to implement only validatorLegacy interface, in order to cover more possible branches and keep coverage up. All tests passed and everything looks fine.

    opened by leventeliu 0
  • v2:interceptors/logging: allow to separate request response payload logging

    v2:interceptors/logging: allow to separate request response payload logging

    Logging payloads of requests on the server-side and similarly of responses on the client-side is a pretty common and desired use-case. However, logging by default payloads of both requests as well as responses in both scenarios is IMO not a very common requirement.

    Consider the following PR to enable more granular control of what to log and when. This is achieved by using a special PayloadDecison type having the following values:

    • NoPayloadLogging - Payload logging is disabled.
    • LogRequest - Only logging of requests is enabled.
    • LogResponse - Only logging of responses is enabled.
    • LogRequestAndResponse - Logging of both requests and responses is enabled. The PayloadDecison is to be used instead of the original bool by the logging.ServerPayloadLoggingDecider and logging.ClientPayloadLoggingDecider.

    NOTE: I quickly put this together to see whether it could be considered a valid approach, thus please excuse the lack of additional unit tests. I will be glad to add those once the approach is approved.

    v2 
    opened by michaljemala 5
  • [grpc_zap] Payload error message

    [grpc_zap] Payload error message

    Hello, when i add to my GRPC server line:

    grpc_zap.PayloadUnaryServerInterceptor(logger, alwaysLoggingDeciderServer),
    

    I see not complete log,I see incoming requests, responses, but i didn't see payload

    I take

    alwaysLoggingDeciderServer
    from go-grpc-middleware tests folder.

    alwaysLoggingDeciderServer := func(ctx context.Context, fullMethodName string, servingObject interface{}) bool {
    		return true
    }
    
    {"level":"info","ts":1626187747.4904773,"caller":"zap/payload_interceptors.go:131","msg":"server response payload logged as grpc.response.content field","system":"grpc","span.kind":"server","grpc.service":"api.SimpleService","grpc.method":"DebugParse","grpc.response.content":{},"grpc.response.contentError":"json: error calling MarshalJSON for type *grpc_zap.jsonpbObjectMarshaler: jsonpb serializer failed: proto: not found"}
    

    What i need add to see bodies of requests?

    opened by koteesy 5
  • Slack link does not work anymore

    Slack link does not work anymore

    The Slack link on the README is not working.

    Is there a Slack workspace where the project community gathers? If not, we should remove the Slack badge from the README file.

    easy-fix 
    opened by olivierlemasle 1
  • logging: Restore compatibility with v1 of pb Message

    logging: Restore compatibility with v1 of pb Message

    Services generated with old versions of protoc-gen-go use the deprecated github.com/golang/protobuf/proto.Message (v1 messages) instead of google.golang.org/protobuf/proto.Message (v2 messages).

    This commit makes the payload interceptor work with both versions.

    opened by olivierlemasle 1
  • v2: logging provider: klog/v2

    v2: logging provider: klog/v2

    As most of k8s related tools are using klog as a standarad logging tools, it's great if we consider adding klog/v2 provider in v2 natively.

    enhancement help wanted v2 
    opened by m-yosefpor 2
  • go get-ing v2 version requires commit hash.

    go get-ing v2 version requires commit hash.

    Steps to reproduce

    1. go get -u github.com/grpc-ecosystem/go-grpc-middleware/providers/zap/v2

    Outcome

    go get: module github.com/grpc-ecosystem/[email protected] found (v1.3.0), but does not contain package github.com/grpc-ecosystem/go-grpc-middleware/providers/zap/v2

    Expected Behavior

    Module is installed

    Notes

    • I was able to install all other v2 modules including auth
    • https://pkg.go.dev/github.com/grpc-ecosystem/go-grpc-middleware/providers notes that v2 modules are avaiable apart from zap
    question 
    opened by cayden-uw 4
  • Exemplars support: Add ability to include trace ID

    Exemplars support: Add ability to include trace ID

    Hello 👋

    Is there any way currently (I haven't been able to find it yet) to include the Trace ID as part of the interceptor to be able to make use of Prometheus Exemplars? If not, are there plans to add this?

    Thank you and sorry if I missed something obvious! @bwplotka @brancz

    enhancement help wanted 
    opened by lilic 7
  • v2: Not all tests are executed on macOS

    v2: Not all tests are executed on macOS

    https://github.com/grpc-ecosystem/go-grpc-middleware/runs/2203038390?check_suite_focus=true#step:5:6

    There is an issue in the makefile.

    bug 
    opened by ash2k 3
Releases(v1.3.0)
Owner
gRPC Ecosystem
gRPC Ecosystem that complements gRPC
gRPC Ecosystem
A suite of gRPC debugging tools. Like Fiddler/Charles but for gRPC.

grpc-tools A suite of tools for gRPC debugging and development. Like Fiddler/Charles but for gRPC! The main tool is grpc-dump which transparently inte

Bradley Kemp 938 Jul 23, 2021
Use Consul to do service discovery, use gRPC +kafka to do message produce and consume. Use redis to store result.

目录 gRPC/consul/kafka简介 gRPC+kafka的Demo gRPC+kafka整体示意图 限流器 基于redis计数器生成唯一ID kafka生产消费 kafka生产消费示意图 本文kafka生产消费过程 基于pprof的性能分析Demo 使用pprof统计CPU/HEAP数据的

null 38 Jan 31, 2021
The Go language implementation of gRPC. HTTP/2 based RPC

gRPC-Go The Go implementation of gRPC: A high performance, open source, general RPC framework that puts mobile and HTTP/2 first. For more information

grpc 14.1k Jul 23, 2021
grpc-http1: A gRPC via HTTP/1 Enabling Library for Go

grpc-http1: A gRPC via HTTP/1 Enabling Library for Go This library enables using all the functionality of a gRPC server even if it is exposed behind a

StackRox 53 Jun 3, 2021
protoc-gen-grpc-gateway-ts is a Typescript client generator for the grpc-gateway project. It generates idiomatic Typescript clients that connect the web frontend and golang backend fronted by grpc-gateway.

protoc-gen-grpc-gateway-ts protoc-gen-grpc-gateway-ts is a Typescript client generator for the grpc-gateway project. It generates idiomatic Typescript

gRPC Ecosystem 25 Jul 21, 2021
gRPC over WebRTC

gRPC over WebRTC Just a proof of concept, please be kind. How to Start all the things Client, create-react-app + grpc-web signaling + webrtc extension

Jean-Sebastien Mouret 227 Jul 10, 2021
A Protocol Buffers compiler that generates optimized marshaling & unmarshaling Go code for ProtoBuf APIv2

vtprotobuf, the Vitess Protocol Buffers compiler This repository provides the protoc-gen-go-vtproto plug-in for protoc, which is used by Vitess to gen

PlanetScale 262 Jul 23, 2021
In memory Key/Value store in go using gRPC.

In memory cache, using gRPC Contents About Running Server Local Docker Kubernetes Example Helm Terraform API Add Get GetByPrefix GetAllItems DeleteKey

Kautilya Tripathi 136 Jul 23, 2021
gRPC/REST proxy for Kafka

Kafka-Pixy (gRPC/REST Proxy for Kafka) Kafka-Pixy is a dual API (gRPC and REST) proxy for Kafka with automatic consumer group control. It is designed

Mailgun Team 646 Jul 16, 2021
Fast time-series data storage server accessible over gRPC

tstorage-server Persistent fast time-series data storage server accessible over gRPC. tstorage-server is lightweight local on-disk storage engine serv

Bartlomiej Mika 3 Jul 17, 2021
Create a gRPC Server from Database

xo-grpc Create a gRPC Server from the generated code by the xo project. Requirements Go 1.16 or superior protoc xo, protoc-gen-go and protoc-gen-go-gr

Walter Wanderley 17 Jun 23, 2021
Protobuf files manager

Prot - protobuf files manager. It application can help your manage protobuf files and generate code based on him. !!! Before use Prot you must install

Maksim Pavlov 5 Dec 21, 2020
gproxy is a tiny service/library for creating lets-encrypt/acme secured gRPC and http reverse proxies

gproxy is a reverse proxy service AND library for creating flexible, expression-based, lets-encrypt/acme secured gRPC/http reverse proxies GProxy as a

null 13 Feb 23, 2021