A simple RPC framework with protobuf service definitions

Related tags

go protobuf rpc twirp
Overview

Twirp Logo Build Status Go Report Card GoDoc


Twirp is a framework for service-to-service communication emphasizing simplicity and minimalism. It generates routing and serialization from API definition files and lets you focus on your application's logic instead of thinking about folderol like HTTP methods and paths and JSON.

Define your service in a Protobuf file and then Twirp autogenerates Go code with a server interface and fully functional clients. It's similar to gRPC, but without the custom HTTP server and transport implementations: it runs on the standard library's extremely-well-tested-and-high-performance net/http Server. It can run on HTTP 1.1, not just http/2, and supports JSON clients for easy integrations across languages

Twirp handles routing and serialization for you in a well-tested, standardized, thoughtful way so you don't have to. Serialization and deserialization code is error-prone and tricky, and you shouldn't be wasting your time deciding whether it should be "POST /friends/:id/new" or "POST /:id/friend" or whatever. Just get to the real work of building services!

Along the way, you get an autogenerated client and a simple, smart framework for passing error messages. Nice!

For more on the motivation behind Twirp (and a comparison to REST APIs and gRPC), the announcement blog post is a good read.

Installation

Use go get to install the Go client-and-server generator:

go get github.com/twitchtv/twirp/protoc-gen-twirp

You will also need:

Documentation

On the website: https://twitchtv.github.io/twirp/docs/intro.html

Source for this documentation is in the docs subdirectory. The website is generated from that folder using Docusaurus.

Implementations in other languages

This repo only has the official generators, which write out Go and Python (partial, see #185 and #220) code.

For other languages, there are third-party generators available:

Language Clients Servers Repository
Python3 github.com/verloop/twirpy
Java github.com/fajran/protoc-gen-twirp_java_jaxrs
Java https://github.com/devork/flit
JavaScript github.com/thechriswalker/protoc-gen-twirp_js
JavaScript github.com/Xe/twirp-codegens/cmd/protoc-gen-twirp_jsbrowser
Typescript github.com/larrymyers/protoc-gen-twirp_typescript
Ruby github.com/twitchtv/twirp-ruby
Rust github.com/cretz/prost-twirp
Scala github.com/soundcloud/twinagle
Swagger github.com/elliots/protoc-gen-twirp_swagger
PHP github.com/twirphp/twirp
Dart github.com/apptreesoftware/protoc-gen-twirp_dart
Elixir github.com/keathley/twirp-elixir

This list isn't an endorsement, it's just a convenience to help you find stuff for your language.

Support and Community

We have a channel on the Gophers slack, #twirp, which is the best place to get quick answers to your questions. You can join the Gopher slack here.

Releases

Twirp follows semantic versioning through git tags, and uses Github Releases for release notes and upgrade guides: Twirp Releases

Contributing

Check out CONTRIBUTING.md for notes on making contributions.

License

This library is licensed under the Apache 2.0 License.

Issues
  • Proposed new routing format: /package/service/method

    Proposed new routing format: /package/service/method

    I think we could improve Twirp's routing scheme. It has two problems right now: the /twirp/ prefix, and the package.service scheme (instead of package/service).

    Some people have mentioned to me that they're hesitant to use Twirp because the routes are prefixed with /twirp, which is concerning to them for several reasons:

    • Trademarks: some very large organizations don't want to take any legal risks and are concerned that "twirp" could become trademarked.
    • Feels like advertising: This was mentioned in #48: putting "twirp" in all your routes feels like it's just supposed to pump Twirp's brand.
    • Homophonous with "twerp": In some Very Serious settings (like government websites), it's not okay that "twirp" sounds like "twerp", which means something like "insignificant pest."

    We currently have the /twirp prefix to avoid colliding with gRPC's routes, which live at /package.service/method, and to reserve a space for future APIs (like a reflection server).

    Slashes-based routes (instead of dot-based) would be preferable because lots of existing technology expects slash-based routing. Load balancers and proxies use slashes for namespaces. If Twirp used a slash after the package name, users could leverage this much more easily. This is often possible with dots, just annoying, as you have to write a (possibly pretty complex) regex.

    I think we can fix both of these in one go. This would be a breaking change at the protocol level, so it's something to take very seriously, but I think we can do it without much pain. It would just take a version bump to v6.


    Proposed Implementation:

    I propose we use a new routing scheme: /<package>/<service>/<method>.

    This does not collide with gRPC's routes, which is good. I think it gives us flexibility to put future routes in by putting them under a Twirp package. We could provide a reflection service like this one, which lists available services:

    syntax = "proto3";
    
    package twirp.meta
    
    import "github.com/golang/protobuf/protoc-gen-go/descriptor/descriptor.proto"
    
    service Reflector {
      rpc ListServices(ListServiceRequest) returns (ListServiceResponse);
    }
    
    message ListServiceRequest {}
    message ListServiceResponse {
      repeated protobuf.ServiceDescriptorProto services = 1;
    }
    

    It would live at /twirp.meta/Reflector, which seems good to me.

    Regarding compatibility: generated clients would always use the new routing scheme. Generated servers would be able to use both, in case your users have old generated clients still around. We would accomplish this by exporting two path prefixes:

    const (
      HaberdasherPathPrefix = "/notreal.example/Haberdasher/"
      HaberdasherLegacyPathPrefix = "/twirp/notreal.example.Haberdasher/"
    )
    

    Users can mount both on a mux if they want to support old clients. If not, they can just use the new one.

    If we do this, there are other consequences.

    • Documentation would need to be updated everywhere.
    • Server implementations in other languages would need to be updated ASAP, as new clients won't work with old servers.
    • If people have log parsing or metrics gathering code that operates on routes, they'd need to change them.

    The pain of those other consequences is a good reason that we should try to get this out promptly, if we do it.

    feature request proposal 
    opened by spenczar 36
  • Spec doesn't mention CamelCasing the URL

    Spec doesn't mention CamelCasing the URL

    The way URLs are defined makes one believe the exact name of service is accepted. https://github.com/twitchtv/twirp/blob/03ac4c/docs/spec_v5.md#urls

    In practice however both service and method names are changed. https://github.com/twitchtv/twirp/blob/03ac4c/protoc-gen-twirp/generator.go#L956

    Should I add this to the spec for clarity and such that others do not get blind-sided by it while implementing?

    @khevse discovered it in twirpy I believe at least the rust, swagger, TypeScript, Dart and php implementations suffer from this.

    Aside: The method itself is marked internal API. It might be easier for implementations if it were public https://github.com/twitchtv/twirp/blob/03ac4c/internal/gen/stringutils/stringutils.go#L71

    opened by ofpiyush 31
  • Proposal: Go streaming API

    Proposal: Go streaming API

    #3 lays out some ideas for how Twirp could support streaming RPCs on the wire. Equally important is how we support them in generated code.

    This issue is for designing the generated Go code. When we've landed on a proposal that seems pretty good, I'll edit this top text with what we've got, and then we can look at implementing it.

    Let's use this service to discuss what things would look like:

    syntax = "proto3";
    
    service Streamer {
      rpc Transaction(Req) returns (Resp);
      rpc Upload(stream Req) returns (Resp);
      rpc Download(Req) returns (stream Resp);
      rpc Bidirectional(stream Req) returns (stream Resp);
    }
    
    message Req {}
    message Resp {}
    
    help wanted proposal 
    opened by spenczar 28
  • Proposal: Client-Side Hooks in Go

    Proposal: Client-Side Hooks in Go

    Similar to how the server-side hooks area thing, it'd be beneficial to have client-side hooks as well for automating things such as client-side logging, retries, etc.

    Here's a rough draft of the API that I've thought up.

    type ClientHooks struct {
      // RequestPrepared is called as soon as a request has been created and before it has been sent 
     // to the Twirp server.
      RequestPrepared func(context.Context, *http.Request) (context.Context, error)
    
     // RequestFinished (alternatively could be named ResponseReceived and take in some response from 
     // the server as well) is called after a request has finished sending. Since this is terminal, the context is 
     // not returned. 
     RequestFinished func(context.Context)
    
     // Error hook is called whenever an error occurs during the sending of a request. The Error is passed
     // as an argument to the hook.
      Error func(context.Context, twirp.Error) context.Context 
    }
    

    Looking forward to hearing y'alls thoughts!

    opened by iheanyi 26
  • Add support for TooManyRequests (429) error code

    Add support for TooManyRequests (429) error code

    Add support for a "TooManyRequests" Twirp error code, which will get translated to the equivalent 429 HTTP error code.

    Why call this TooManyRequests instead of RateLimitExceeded?

    I'm not super particular with the exact naming here and would be happy to move to something like RateLimitExceeded if others think this is better, but on initial glance exactly matching the corresponding code from the HTTP spec seemed valuable.

    Why not use the existing ResourceExhausted error code to indicate the client should slow down

    Technically, exceeding a rate limit could be considered "exhausting a per-user quota" and be represented by the existing ResourceExhausted error. However, on first glance, my interpretation of receiving a ResourceExhausted error code as a client caller is that something is problematic or damaged on the server, not that my request pattern as a client is faulty. In fact, the Twirp service may be completely healthy, but choose to reject a client's traffic because it breaks some negotiated rate limit, even if it could handle the traffic just fine without exhausting its resources. In addition, there are many potential situations where there are no user-level limits and yet rate limiting is still expected. For example, a Twirp service may place global rate limit across all callers on a very expensive API operation to ensure a downstream remains healthy.

    Secondly, breaking this out into its own error code allows the very simple creation of rate-limiting Twirp middleware via ServerHooks. If we chose to reuse the existing ResourceExhausted error code, any middleware would have to inspect both the error code as well as actual error message to ensure that the ResourceExhausted error received was due to rate limiting instead of some other generic resource exhaustion. Checking the error message directly is more brittle to changes.

    In my mind, having a error code that either strictly or loosely correlates with the 429 TooManyRequests error code from the HTTP spec is valuable for both Twirp clients, service developers and the Twirp ecosystem as a whole.

    By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

    opened by thesilentg 16
  • Making logging to stdout optional

    Making logging to stdout optional

    The generated twirp servers include import log "log"

    They're used in error cases like this:

    	if _, err = resp.Write(buf.Bytes()); err != nil {
    		log.Printf("errored while writing response to client, but already sent response status code to 200: %s", err)
    	}
    

    It would be great if I could provide my own logger of some kind.

    One proposed API would be to expose an optional function I could type assert for.

    func (s *myServer) SetLogger(logger *log.Logger) {
      s.logger = logger
    }
    

    The client code could then call

    x := NewMyServer(...)
    x.(twirp.Logthingie).SetLogger(myOwnLogger)
    

    If you're ambitious, you could have a specific twirp type logger for more strongly typed information

    type TwirpLogger interface {
      ErrorWritingReponse(ctx context.Context, writeErr err)
    }
    

    My implementation could then extract the method name from the ctx.

    feature request 
    opened by cep21 15
  • Allow generation of Twirp in a different Golang Package

    Allow generation of Twirp in a different Golang Package

    I'll keep this pretty short for now (I also know we discussed this very briefly @spenczar) but just to get the conversation going since this is on my near-term radar:

    • It would be nice to be able to have twitchtv/twirp generated code in a different package than the golang/protobuf generated code. This in theory would allow you to have separation of your Protobuf definitions and the specific RPC protocol you use. For example, I'd love to have:

    • foo/barpb - Protobuf messages for package foo.bar

    • foo/twitchbarpb - Twirp client/servers for package foo.bar

    • foo/grpcbarpb - gRPC client/servers for package foo.bar - I'm not sure if this is will be allowed with the new protoc-gen-go-grpc via https://github.com/protocolbuffers/protobuf-go/blob/master/compiler/protogen/protogen.go but we can leave this out of it

    • foo/magicbarpb - My custom magical RPC implementation (not in reality but as an example)

    What this would require is protoc-gen-twirp recognizing when it was generating code in a different package than the package of the golang/protobuf generated code, and then generating imports, ie:

    // all shorthand, not even importing context
    import barpb "github.com/alice/bob/foo/barpb"
    
    type EchoService interface {
      Echo(context.Context, *barpb.EchoRequest) (*barpb.EchoResponse, error)
    }
    

    This might be able to be implemented with some combination of the Mkey=value (which would be annoying), import_path, and import_prefix options, but honestly it would be so much easier if there was just a definition_path=github.com/alice/bob/foo/barpb option or something similar to just denote this easily. There are Golang Protobuf plugin libraries that handle this kind of behavior, so there's a lot of work to pattern off of.

    Thoughts?

    no-issue-activity 
    opened by bufdev 15
  • Add support for custom route prefixes

    Add support for custom route prefixes

    Hello,

    I was really excited to read about twirp and want to undertake a project with it. Unfortunately, twirp seems to make some pretty opinionated decisions about my API's pathing. I've noticed that the generator prepends the literal "twirp" to the url, and seems to require the services internal name be exposed. I'm not familiar with writing proto generators but I assume it is possible to provide some sort of generator option to disable or customize this functionality.

    For the hard coded /twirp, I'm wondering if this is a twitch specific thing or if you're open to removing it. Considering I can use my own mux, I'm capable of prepending paths already.

    For the service name, it seems a bit less straight forward. It is possible to expose the handlers so they can be plugged directly into the mux. I guess the apprehension is keeping twirp from becoming a complex mux in and of itself, but I think it can be made possible without convoluting the code base.

    feature request 
    opened by lordnynex 15
  • Handle the standard M= and import_prefix= golang plugin flags

    Handle the standard M= and import_prefix= golang plugin flags

    Example: https://github.com/yarpc/yarpc-go/blob/dev/internal/protoplugin/runner.go#L50

    Will be needed when request/response types are outside of the package of the service, or if generating the Twirp handlers to a different golang package.

    Nice work by the way!

    feature request waiting for info 
    opened by bufdev 15
  • Proposal: add context.SetHTTPCookie

    Proposal: add context.SetHTTPCookie

    Hi,

    I'm trying to add 'Set-Cookie' header in Response but need some value computed inside RPC. Setting context value with a middleware doesn't seem to work since twirp doesn't allow rpc to modify context value (correct me if I'm wrong but only request hooks and middlewares can do that).

    Wondering if anyone had a similar requirement and what would be the recommended solution.

    Thank you!

    opened by Bankq 13
  • Go package imported twice if proto package split into multiple files

    Go package imported twice if proto package split into multiple files

    If this is intended behavior, my bad. I tried searching through issues and I couldn't find much.

    I have the following proto file (where I have just services defined):

    // file services.proto
    syntax = "proto3";
    
    package rpc;
    option go_package = "github.com/lrstanley/spectrograph/internal/rpc";
    
    import "google/protobuf/empty.proto";
    import "internal/models/protos/servers.proto";
    import "internal/models/protos/worker.proto";
    
    service Worker {
        rpc Health(google.protobuf.Empty) returns (models.WorkerHealth); // models.WorkerHealth being from worker.proto
        rpc UpdateServer(models.ServerDiscordData) returns (google.protobuf.Empty); // models.ServerDiscordData being from servers.proto
        rpc UpdateServerStatus(models.ServerStatus) returns (google.protobuf.Empty); // models.ServerStatus being from servers.proto
    }
    

    It has two imports:

    import "internal/models/protos/servers.proto";
    import "internal/models/protos/worker.proto";
    

    Both of these imports are within the same directory of eachother, but not in the directory of the services/twirp files, and both actually specify the same go_package and package line, as they're only split into multiple files given the size and specific-scope of the messages:

    // file: servers.proto
    syntax = "proto3";
    
    package models;
    option go_package = "github.com/lrstanley/spectrograph/internal/models";
    
    [... server-specific messages defined below...]
    
    // file: worker.proto
    syntax = "proto3";
    
    package models;
    option go_package = "github.com/lrstanley/spectrograph/internal/models";
    
    [... worker-specific messages defined below...]
    

    I am able to import both and use them successfully. However, I noticed the generated *.twirp.go files have:

    import models "github.com/lrstanley/spectrograph/internal/models"
    import models1 "github.com/lrstanley/spectrograph/internal/models"
    

    And reference the same go_package as multiple imports throughout the file, which can be rather confusing.

    This works successfully (it's supported behavior by protobuf, and from my understanding, not discouraged), however I'm wondering if we can add some logic that ensures if there are multiple imports with the same exact go package name and path, that only one line is imported?

    The more serious issue I suspect is: if the current behavior could cause issues for those that use init() functions in their Go packages, where the *.pb.go files are generated, as I suspect these would get executed twice. I.e. if variables or similar are initialized, this could cause weird state issues for users, where two sets of initialized state are used.

    opened by lrstanley 2
  • Import types/funcs from github.com/twitchtv/twirp/... instead of generating them.

    Import types/funcs from github.com/twitchtv/twirp/... instead of generating them.

    I have multiple proto files and I want all of them to be generated into a single directory. As a result, I have "redeclared in this package" error caused by redeclaring types like HTTPClient, TwirpServer and functions like newServerOpts, WriteError, writeError, and others. Maybe it would be better just to import them instead of generating them in every .twirp.go file.

    opened by f0mster 4
  • Support Go Modules

    Support Go Modules

    Twirp should be able to support Go Modules for a few reasons:

    1. Pinning dependencies correctly when GO111MODULE will be set to on by default in the next release of Go 1.13

    2. It will make the onboarding experience to twirp a little bit easier because you'll be able to get binaries such as protoc-gen-twirp at an exact version without having to use retool. For example: go get github.com/twitchtv/twirp/[email protected]

    3. There will be no need to vendor dependencies on both the twirp side as well as the user's side.

    I noticed that there was already an issue about adding go.mod/go.sum and the reason for closing it was this comment: https://github.com/twitchtv/twirp/issues/140#issuecomment-452401108

    In particular:

    the transition will involve quite a bit more than just adding go.mod and go.sum files as import paths will need to change

    For that reason, I created a tool to automatically upgrade import paths.

    I have a draft PR to demonstrate that all import paths are upgraded and that the tests past (at least locally for me).

    The PR is in draft because A. I'm not familiar with Twirp's roadmap and B. I imagine a lot of documentation needs to be updated to show the new flow of getting started.

    Please feel free to keep this open until Twirp officially supports module, and also feel free to add all the TODOs that need to be accomplished before rendering this issue resolved.

    pending 
    opened by marwan-at-work 19
  • Proposal: Support streaming in the Twirp spec

    Proposal: Support streaming in the Twirp spec

    I think this is a great addition for RPC for Go (as well as other languages!). The main thing that I expect will limit adoption is the lack of streaming support. There are mechanisms to do streaming on http/1.x (a great example is the grpc-gateway stuff) where they effectively just use chunked encoding with blobs to do the streaming.

    feature request proposal pending 
    opened by jacksontj 50
Releases(v8.1.0)
  • v8.1.0(Jun 16, 2021)

    Changes:

    • PR #323: Match service errors with errors.As instead of a type cast. Service implementations can now use their own error types if they implement the twirp.Error interface, but also if they wrap another Twirp error, or if theyimplement the method As(interface{}) bool. See docs on Twirp Erros for more details.
    • PR #299: ServerOption twirp.WithServerJSONCamelCaseNames(true) configures JSON serialization to use the proto3 standard serialization method instead of the Twirp default to use the original proto names. When enabled, the JSON field names will be lowerCamelCase by default. If the json_name field option is used in the proto file, the specified value will be used as the key instead.
    • PR #318: Removed example Python client generator. This repo contains the Go implementation only now.
    • PR #319 refactors ServerOptions and ClientOptions to allow new options in the future without forcing version updates.
    • PR #321 and #313 improves and simplifies the CI build process, running clientcompat tests and removing faulty coverage checks.
    • PR #312 sorts generated code imports so the generator always produces the same output.

    Update instructions:

    • Required Go version 1.13+ is needed for the errors package Is/As/Unwrap functions.
    • Runtime library is backwards compatible, can be safely updated to v8.1.0.
    • Generated clients are backwards compatible, but will require the importer to update their Twirp runtime library.
    • The PR #323 introduces a potential breaking change for services. Read below for safe update instructions.

    Previous Twirp services (v8.0.0 and older) check if returned errors are Twirp errors with a type-cast:

    twerr, ok := err.(twirp.Error)
    

    Newer Twirp services (v8.1.0) will check if returned errors are Twirp errors by matching with errors.As:

    var twerr twirp.Error
    ok := errors.As(err, &twerr)
    

    Matching with errors.As is similar to the type-cast, but it can also unwrap errors of the same type. Before you update to version v8.1.0, please double check if any non-twirp errors returned by your service are not accidentally wrapping another Twirp error that should not be returned on the API. In particular, look for instances of fmt.Errorf("foo: %w", twerr), using %w for wrapping other Twirp errors, for example:

    // Calling another Twirp service from a Twirp method handler
    resp, clientErr := anotherService.MyMethod(ctx, req)
    if clientErr != nil {
        // Wrapping another Twirp error will expose the inner error on v8.1.0
        err := fmt.Errorf("otherService: %w", clientErr)
        return nil, err
    }
    

    If you believe your service may be mistakenly wrapping other Twirp errors, you can explicitly wrap the error as an internal:

        return nil, twirp.InternalErrorWith(err)
    

    You could also use this error hook to identify possible discrepancies before updating to the new version. In the other hand, if your service was already using idiomatic Go errors, the new implementation will give you more power to your own Twirp error implementations.

    Source code(tar.gz)
    Source code(zip)
  • v8.0.0(May 3, 2021)

    PR #304: Twirp v8 generates code that depends on Protobuf APIv2 (instead of APIv1).

    Relevant links

    • Protobuf APIv1 repo: https://github.com/golang/protobuf
    • Protobuf APIv2 repo: https://github.com/protocolbuffers/protobuf-go
    • Protobuf APIv2 release blog post: https://blog.golang.org/protobuf-apiv2
    • Twirp version compatibility matrix: https://twitchtv.github.io/twirp/docs/version_matrix.html

    Update Instructions

    The runtime library github.com/twitchtv/twirp does not have any changes. The new generated code works with both old and new versions of the runtime library.

    Re-generate code with Twirp v8 and Protobuf APIv2:

    • Install the new protoc-gen-go plugin: go install google.golang.org/protobuf/cmd/[email protected].
    • Install the new protoc-gen-twirp plugin: go install github.com/twitchtv/twirp/[email protected].
    • Re-generate code. For example: protoc --go_out=paths=source_relative:. --twirp_out=paths=source_relative:. service.proto

    See Install docs for more information.

    Generated clients and services will use the new imports: google.golang.org/protobuf (APIv2). Projects that make use of the generated clients/servers will have to update their import paths.

    The new google.golang.org/protobuf (APIv2) is mostly backwards compatible, but not completely. You may have to make additional changes to work with the new protobuf library:

    • In the proto file, the option go_package is mandatory and must include a "/" (it is supposed to be a full import path). If you have to add a full path to the go_package, you may want to generate with the options paths=source_relative. For example: protoc --go_out=. --go_opt=paths=source_relative --twirp_out=. --twirp_opt=paths=source_relative myfile.proto
    • Generated message models (structs) now contain a mutex, your linter may complain if the models are copied by value. The solution is to pass pointers instead. Using reflect.DeepEqual will not be able to compare protobuf, you can use proto.Equal instead.
    • Check protobuf go compatibility and their releases for more details.
    Source code(tar.gz)
    Source code(zip)
  • v7.2.0(Apr 9, 2021)

    • PR #294: handling and returning valid error code for context.Canceled and context.DeadlineExceeded during body read error, and returning meaningful error 4XX instead of a 5XX error.
    Source code(tar.gz)
    Source code(zip)
  • v7.1.1(Feb 12, 2021)

  • v7.1.0(Sep 25, 2020)

    • PR #276 implements interceptors, an elegant way to plugin new functionality to clients and services with a simple but powerful interface.
    • PR #277, fixing #257 on the client side, allowing Go clients to talk to services implemented in different languages (e.g. Python) when the proto definition contains Service and Method names that are not CamelCased. Note that only Go services generated with v7 can handle literal routes, previous versions only support CamelCased routes.
    Source code(tar.gz)
    Source code(zip)
  • v7.0.0(Sep 16, 2020)

    Twirp v7 is a major release that includes a protocol spec update. The jump from v5 to v7 is to avoid confusion with the archived protocol spec v6 that was not released.

    Changes in the protocol spec from v5 to v7:

    • Twirp routes can have any prefix: <prefix>/<package>.<Service>/<Method>. In v5 the prefix was mandatory, always/twirp.
    • Error code ResourceExhausted maps to HTTP code 429. In v5 it mapped to 403.

    Changes in the Go Twirp v7.0.0 release:

    • #264 Optional Twirp Prefix. Implements the proposal #263 with the optional prefix, using an option to specify a different prefix than the default value "/twirp". The default value ensures backwards compatibility when updating the service. Using different prefixes, it is now possible to mount the same Twirp service in multiple routes, which may help migrating existing services from the "/twirp" prefix to a new one.
    • #264 also introduces server options on the server constructor, that can be used to setup hooks and new options like the path prefix.
    • #270 ResourceExhausted error code changed to 429 HTTP status code. This may affect middleware if it depends on a specific status code being used.
    • #271 Server JSON responses using EmitDefaults: responses include all fields, even if they have zero-values. This can be reverted to the previous behavior (skip zero-values) with the server option: twirp.WithServerJSONSkipDefaults(true).
    • #257 Go services handle literal routes. Fixing part of the issue #244, affecting cross-language communications when the proto definitions are not using the recommended Protobuf Style Guide.
    • #268 Bufgfix: Allow to update context from server Error hooks, so they can communicate with the ResponseSent hook if needed.

    Go Upgrade instructions

    For the most part, the upgrade to the new version should work out of the box. But there are a few areas to pay attention, depending if your service is using some specific features or not.

    Twirp has two components, the twirp package and the code generator plugin (protoc-gen-twirp). One thing is to upgrade the package to the latest version, and another thing is to re-generate the server/client code with the latest version. You should upgrade the package first, because it works with older generated clients.

    Upgrade the twirp package to version v7.0.0 in your project.

    • If you are explicitly using twirp.ServerHTTPStatusFromErrorCode(twirp.ResourceExhausted), the returned value will change from 403 to 429.
    • If your service is returning errors with the code twirp.ResourceExhausted and you have middleware and/or monitoring tools that depend on the HTTP status code, they will now see a change from 403 to 429.

    Generate code with version v7+ of the plugin protoc-gen-twirp.

    • If your service is serving JSON requests under heavy traffic requirements, make sure to skip defaults with the option twirp.WithServerJSONSkipDefaults(true). Pass this option as an extra argument to the New<Service>Server constructor. See more details on #271

    Importing Twirp clients from other services:

    • If your service is importing Twirp clients from other services that were generated in older versions, don't worry, they still work.
    • The new v7 package works for clients that were generated with the older v5 generator.
    • But the new v7 clients require the new v7 package to compile.
    • If you have multiple services that depend on each other, update the twirp package on all services first. Don't re-generate the code yet. Once all services have the v7 package, you can update the protoc-gen-twirp to v7 and re-generate all clients.
    Source code(tar.gz)
    Source code(zip)
  • v5.12.1(Jul 24, 2020)

    This release fixes an issue in the generated Go client code where the request context is not properly passed to the ResponseReceived client hook.

    Source code(tar.gz)
    Source code(zip)
  • v5.12.0(Jun 11, 2020)

    • PR #229 adds the function twirp.WrapError(twerr Error, err error) twirp.Error, that returns the same Twirp error but wrapping the original error, that can can be unwrapped later (for example in middleware) using errors.Is or (github.com/pkg/errors).Unwrap.
    Source code(tar.gz)
    Source code(zip)
  • v5.11.0(May 24, 2020)

    • PR #235 adds the ability to use Go 13+ standard library methods errors.Is and errors.As to unwrap errors from Twirp's internal errors. This works for service errors, usually unwrapped on middleware, and also client errors if they fail on the client, for example a context cancelled error. NOTE: client errors can only be unwrapped if the client was generated with Twirp v5.11.0 or above (even if your service is using a different Twirp version). For older versions, you can still use (github.com/pkg/errors).Unwrap.
    • PR #227 adds the helper function AddHTTPResponseHeader, that can be used to set multiple values to the response header in the same key.
    Source code(tar.gz)
    Source code(zip)
    protoc-gen-twirp_python-Darwin-x86_64(4.09 MB)
    protoc-gen-twirp_python-Darwin-x86_64.sha256sum(112 bytes)
    protoc-gen-twirp_python-Linux-x86_64(4.13 MB)
    protoc-gen-twirp_python-Linux-x86_64.sha256sum(111 bytes)
    protoc-gen-twirp-Darwin-x86_64(5.04 MB)
    protoc-gen-twirp-Darwin-x86_64.sha256sum(105 bytes)
    protoc-gen-twirp-Linux-x86_64(5.10 MB)
    protoc-gen-twirp-Linux-x86_64.sha256sum(104 bytes)
  • v5.10.2(May 11, 2020)

    • Fixes the issue in #196, where importing a .proto file with the same Go package as the file being generated would not be handled correctly.
    • Includes the response body in the error metadata when the response body's error JSON has an invalid Twirp error code to assist with debugging.
    Source code(tar.gz)
    Source code(zip)
  • v5.10.1(Jan 22, 2020)

  • v5.10.0(Dec 16, 2019)

    This release includes two PRs:

    #198 adds ClientHooks. These are a structure that can be passed to client constructors to hook off of client request lifecycle events. This can be used to build standard tools for logging, monitoring, or authorization steps in all outbound calls.

    #201 fixes the Twirp library WriteError function to return an error value instead of swallowing the error.

    Source code(tar.gz)
    Source code(zip)
  • v5.9.0(Nov 6, 2019)

    This release includes three changes.

    • #192 added a new github.com/twitchtv/twirp.WriteError function to the main Twirp library. This should help users write general code that plays well in a broader Twirp ecosystem.
    • #193 fixes Twirp client behavior when a server responds with an error that is JSON-encoded, but not a valid Twirp error. Now, the client should provide a more meaningful error message in this case.
    • #191 fixes a documentation comment in github.com/twitchtv/twirp/hooks/statsd.
    Source code(tar.gz)
    Source code(zip)
  • v5.8.0(Jul 26, 2019)

    This release adds a new error code, malformed, which the server returns if a client's message is not deserializable without errors. This error gets a 400 HTTP Status Code. This should make it easier to filter out bad hand-written clients from server error monitoring.

    This was #182.

    Source code(tar.gz)
    Source code(zip)
  • v5.7.0(Apr 16, 2019)

    This release includes two changes to make panics easier to handle in Twirp servers, as well as a minor bug fix to correct #171.

    First, panics are now propagated to Error hooks, allowing application owners to add instrumentation around them. Panics will be wrapped with as twirp.Errors, but will support a Cause() error method, so you can use github.com/pkg/errors to retrieve the root cause of errors in hooks. This was implemented in #160.

    Second, Twirp servers now flush error responses when panicking, so that clients always get the message, and don't see a broken pipe if the server crashes due to the panic. In addition, Twirp servers now set Content-Length on all responses, so clients can be sure they have received the complete message. This was implemented in #168.

    Finally, protoc-gen-twirp now generated correct code for service definitions with no methods. This was implemented in #171.

    Source code(tar.gz)
    Source code(zip)
    protoc-gen-twirp_python-Darwin-x86_64(4.09 MB)
    protoc-gen-twirp_python-Darwin-x86_64.sha256sum(112 bytes)
    protoc-gen-twirp-Darwin-x86_64(5.03 MB)
    protoc-gen-twirp_python-Linux-x86_64.sha256sum(111 bytes)
    protoc-gen-twirp-Linux-x86_64.sha256sum(104 bytes)
    protoc-gen-twirp_python-Linux-x86_64(4.13 MB)
    protoc-gen-twirp-Darwin-x86_64.sha256sum(105 bytes)
    protoc-gen-twirp-Linux-x86_64(5.09 MB)
  • v5.6.0(Apr 1, 2019)

    This release adds a new method to generated servers. They now support a PathPrefix() string method which returns the path that all of their routes are prefixed with. This should let users of Twirp build better libraries around registration of Twirp services.

    Source code(tar.gz)
    Source code(zip)
  • v5.5.2(Feb 15, 2019)

    This release includes a fix to the generator to support Protobuf messages that are not defined in camel case. For example, messages such as Hat_v1 would have the Go type name HatV1 in the protoc-gen-go generated code, but the Twirp generated code would expect Hat_v1.

    Source code(tar.gz)
    Source code(zip)
  • v5.5.1(Dec 13, 2018)

    A security vulnerability was discovered on the requirements.txt file that is used to test Python-generated code, and could be used as sub-dependency checks for Python projects using Twirp.

    Fix submitted in this PR: https://github.com/twitchtv/twirp/pull/137

    Source code(tar.gz)
    Source code(zip)
  • v5.5.0(Sep 24, 2018)

    This release adds support for a new generator option, paths=source_relative, to match the behavior of the official protoc-gen-go generator. That behavior is documented at https://github.com/golang/protobuf/pull/533.

    Now, protoc-gen-twirp can work outside of GOPATH to generate code. Files are output relative to where the proto source was found.

    The change to make this possible is #126.

    Source code(tar.gz)
    Source code(zip)
  • v5.4.2(Aug 10, 2018)

    This is a bug fix release. It fixes #123.

    Previously, if a service had a method of the same name, the generated Go code would not compile. For example, generated code for this service would not compile:

    service Echo {
      rpc Echo(Msg) returns (Msg);
    }
    

    This was fixed in #124.

    Source code(tar.gz)
    Source code(zip)
  • v5.4.1(May 22, 2018)

    This release includes two bug fixes.

    First, there was a bug in import statement generation when multiple .proto files were imported which shared an identical protobuf package, but which had different output go packages. This would most commonly occur when importing multiple "well-known type" definitions, like google/protobuf/empty.proto and google/protobuf/wrappers.proto. This was reported in #104, and fixed in #107.

    Second, generated Go clients would previously return non-nil response types even when returning non-nil errors. This led to confusing behavior in some obscure cases, as users usually expect non-nil error to imply a nil response. This was fixed in #105.

    Source code(tar.gz)
    Source code(zip)
  • v5.4.0(May 15, 2018)

    This release includes #102, which adds support for flags for protoc-gen-twirp to map .proto files to particular Go import paths. This was requested in #13.

    Documentation is available in https://github.com/twitchtv/twirp/blob/2d98ac4d4ccb29e08ea745c758e6d103f971eba5/docs/command_line.md, which is (as of this release) on the doc website at https://twitchtv.github.io/twirp/docs/command_line.html.

    Source code(tar.gz)
    Source code(zip)
  • v5.3.0(Feb 20, 2018)

    This release includes the following changes:

    • Generated code no longer logs with the standard library's log package under any circumstances. Previously, it would log in rare cases like when failing to write a complete error body back to clients or when failing to close request bodies. Now, those circumstances trigger an Error hook if it hasn't already been triggered once during handling of the request. (dfbd75ae3c12b7f8c9a355f582b635cf77c4645a)
    • Generated Go clients now set an Accept header in requests. The JSON client sets application/json, and the Protobuf client sets application/protobuf. This should make generated clients work more smoothly with API gateways. Servers don't do any particular handling with this new header and it is not required by the Twirp spec, but it's recommended that clients set it in other language implementations as well. (f91c5c69fd3b0f2bc8fd263b1dae3a67073fd9f0)
    • Generated Python clients now work with Python 3. (dbf7a0df8630b787da395fee3facdfb7ef5c38ce)
    Source code(tar.gz)
    Source code(zip)
  • v5.2.0(Jan 31, 2018)

    This release includes the following changes:

    • Context values describing the requested service, package, and method are now available to HTTPClients passed to generated Twirp clients. (a22cc722f7feaeb12b7768699bf5e202464fa4b0)

    • Generated clients no longer emit logs to the standard library logger under any circumstances. Instead, they will return any error encountered closing an HTTP response body if they found no other errors during the request. If they hit other errors, then HTTP response body close errors are ignored. (c0c435531e9a1787cec6159bd6262bcf5aa48bea)

    • Handling of Content-Type headers with name=value pairs after the MIME type is now correct. (2067a95fa595ef8d484d02bdab04c922479a2225)

    • The vendor directory is now committed. This should make installation of protoc-gen-twirp more straightforward. (2642ca800080894c36e3457b9a1603e4ee752eba)

    Source code(tar.gz)
    Source code(zip)
  • v5.1.0(Jan 24, 2018)

    Release v5.1.0 improves the flexibility of client constructors. It also fixes some typos, fixes a bug in the generator, and makes minor performance improvements.

    Generated client constructors now accept an interface, type HTTPClient interface { Do(*http.Request) (*http.Response, error) } instead of the concrete type *http.Client. This lets users provide clients that smartly retry or use custom service discovery systems. It should be backwards compatible, as *http.Client fulfills that interface and can still be plugged in without changes. This is #39, and was proposed and designed in #12.

    Generated servers will avoid an unnecessary string concatenation when handling requests; the string is now precomputed. This is #53.

    Code generation now works correctly if a proto file imports another proto file found in the same directory. This was #15, fixed in #47.

    #36 and #44 fixed typos in documenting comments and error messages.

    All associated PRs bundled into this release can be tracked found here: https://github.com/twitchtv/twirp/pulls?q=is%3Apr+is%3Aclosed+milestone%3Av5.1.0

    Source code(tar.gz)
    Source code(zip)
  • v5.0.0(Jan 17, 2018)

Protocol Buffers - Google's data interchange format

Protocol Buffers - Google's data interchange format Copyright 2008 Google Inc. https://developers.google.com/protocol-buffers/ Overview Protocol Buffe

Protocol Buffers 51.2k Oct 14, 2021
A pluggable backend API that enforces the Event Sourcing Pattern for persisting & broadcasting application state changes

A pluggable "Application State Gateway" that enforces the Event Sourcing Pattern for securely persisting & broadcasting application state changes

null 25 Aug 26, 2021
Fast time-series data storage server accessible over gRPC

tstorage-server Persistent fast time-series data storage server accessible over gRPC. tstorage-server is lightweight local on-disk storage engine serv

Bartlomiej Mika 5 Oct 12, 2021
RPC over libp2p pubsub with error handling

go-libp2p-pubsub-rpc RPC over libp2p pubsub with error handling Table of Contents Background Install Usage Contributing Changelog License Background g

textile.io 4 Aug 21, 2021
🎉 An awesome version control tool for protoc and its related plugins.

❤️ PowerProto is actively maintained! Any questions in use can be directly raised issue, I will respond to you as fast as possible. If you think the p

storyicon 109 Oct 15, 2021
Simple, fast and scalable golang rpc library for high load

gorpc Simple, fast and scalable golang RPC library for high load and microservices. Gorpc provides the following features useful for highly loaded pro

Aliaksandr Valialkin 643 Oct 14, 2021
A Protocol Buffers compiler that generates optimized marshaling & unmarshaling Go code for ProtoBuf APIv2

vtprotobuf, the Vitess Protocol Buffers compiler This repository provides the protoc-gen-go-vtproto plug-in for protoc, which is used by Vitess to gen

PlanetScale 343 Oct 21, 2021
A protoc-gen-go wrapper including an RPC stub generator

// Copyright 2013 Google. All rights reserved. // // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE fi

Kyle Lemons 35 May 13, 2020
A simple RPC framework with protobuf service definitions

Twirp is a framework for service-to-service communication emphasizing simplicity and minimalism. It generates routing and serialization from API defin

Twitch 5.2k Oct 22, 2021
A simple server which can be used as an RPC endpoint in popular Ethereum wallets.

RPC Endpoint This repository contains code for a simple server which can be used as an RPC endpoint in popular Ethereum wallets. The endpoint is https

Flashbots 19 Oct 22, 2021
ETH <-> XMR atomic swap prototype

ETH-XMR Atomic Swaps This is a prototype of ETH<->XMR atomic swaps, which was worked on during ETHLisbon. Instructions Start ganache-cli with determin

null 3 Oct 24, 2021
📦 Command line peer-to-peer data transfer tool based on libp2p.

pcp - Peer Copy Command line peer-to-peer data transfer tool based on libp2p. Table of Contents Motivation Project Status How does it work? Usage Inst

Dennis Trautwein 794 Oct 18, 2021
webrpc is a schema-driven approach to writing backend services for modern Web apps and networks

webrpc is a schema-driven approach to writing backend servers for the Web. Write your server's api interface in a schema format of RIDL or JSON, and t

null 398 Oct 21, 2021
Protobuf files manager

Prot - protobuf files manager. It application can help your manage protobuf files and generate code based on him. !!! Before use Prot you must install

Maksim Pavlov 5 Dec 21, 2020
Fast and Scalable RPC Framework

Rony (Fast and Scalable RPC Framework) About Rony lets you create a clustered aware service easily. Checkout Wiki Performance Rony is very fast and wi

Ronak Software Group 37 Oct 20, 2021
🚀Gev is a lightweight, fast non-blocking TCP network library based on Reactor mode. Support custom protocols to quickly and easily build high-performance servers.

gev 中文 | English gev is a lightweight, fast non-blocking TCP network library based on Reactor mode. Support custom protocols to quickly and easily bui

徐旭 1.3k Oct 19, 2021
Netpoll is a high-performance non-blocking I/O networking framework, which focused on RPC scenarios, developed by ByteDance.

Netpoll is a high-performance non-blocking I/O networking framework, which focused on RPC scenarios, developed by ByteDance. RPC is usually heavy on processing logic and therefore cannot handle I/O serially. But Go's standard library net designed blocking I/O API, so that the RPC framework can only follow the One Conn One Goroutine design.

CloudWeGo 1.9k Oct 20, 2021
TGameServer

TGameServer TGame服务端 使用Go + TCP + UDP +My doc: https://github.com/ALEXTANGXIAO/TGameServer 客户端地址: https://github.com/ALEXTANGXIAO/TGameUnity Tested on

ALEXTANG 21 Sep 12, 2021