A Protocol Buffers compiler that generates optimized marshaling & unmarshaling Go code for ProtoBuf APIv2

Related tags

go protobuf grpc codegen vitess
Overview

vtprotobuf, the Vitess Protocol Buffers compiler

This repository provides the protoc-gen-go-vtproto plug-in for protoc, which is used by Vitess to generate optimized marshall & unmarshal code.

The code generated by this compiler is based on the optimized code generated by gogo/protobuf, although this package is not a fork of the original gogo compiler, as it has been implemented to support the new ProtoBuf APIv2 packages.

Available features

vtprotobuf is implemented as a helper plug-in that must be run alongside the upstream protoc-gen-go generator, as it generates fully-compatible auxiliary code to speed up (de)serialization of Protocol Buffer messages.

The following features can be generated:

  • size: generates a func (p *YourProto) SizeVT() int helper that behaves identically to calling proto.Size(p) on the message, except the size calculation is fully unrolled and does not use reflection. This helper function can be used directly, and it'll also be used by the marshal codegen to ensure the destination buffer is properly sized before ProtoBuf objects are marshalled to it.

  • marshal: generates the following helper methods

    • func (p *YourProto) MarshalVT() ([]byte, error): this function behaves identically to calling proto.Marshal(p), except the actual marshalling has been fully unrolled and does not use reflection or allocate memory. This function simply allocates a properly sized buffer by calling SizeVT on the message and then uses MarshalToSizedBufferVT to marshal to it.

    • func (p *YourProto) MarshalToVT(data []byte) (int, error): this function can be used to marshal a message to an existing buffer. The buffer must be large enough to hold the marshalled message, otherwise this function will panic. It returns the number of bytes marshalled. This function is useful e.g. when using memory pooling to re-use serialization buffers.

    • func (p *YourProto) MarshalToSizedBufferVT(data []byte) (int, error): this function behaves like MarshalTo but expects that the input buffer has the exact size required to hold the message, otherwise it will panic.

  • unmarshal: generates a func (p *YourProto) UnmarshalVT(data []byte) that behaves similarly to calling proto.Unmarshal(data, p) on the message, except the unmarshalling is performed by unrolled codegen without using reflection and allocating as little memory as possible. If the receiver p is not fully zeroed-out, the unmarshal call will actually behave like proto.Merge(data, p). This is because the proto.Unmarshal in the ProtoBuf API is implemented by resetting the destionation message and then calling proto.Merge on it. To ensure proper Unmarshal semantics, ensure you've called proto.Reset on your message before calling UnmarshalVT, or that your message has been newly allocated.

  • pool: generates the following helper methods

    • func (p *YourProto) ResetVT(): this function behaves similarly to proto.Reset(p), except it keeps as much memory as possible available on the message, so that further calls to UnmarshalVT on the same message will need to allocate less memory. This an API meant to be used with memory pools and does not need to be used directly.

    • func (p *YourProto) ReturnToVTPool(): this function returns message p to a local memory pool so it can be reused later. It clears the object properly with ResetVT before storing it on the pool. This method should only be used on messages that were obtained from a memory pool by calling YourProtoFromVTPool. Using p after calling this method will lead to undefined behavior.

    • func YourProtoFromVTPool() *YourProto: this function returns a YourProto message from a local memory pool, or allocates a new one if the pool is currently empty. The returned message is always empty and ready to be used (e.g. by calling UnmarshalVT on it). Once the message has been processed, it must be returned to the memory pool by calling ReturnToVTPool() on it. Returning the message to the pool is not mandatory (it does not leak memory), but if you don't return it, that defeats the whole point of memory pooling.

Usage

  1. Install protoc-gen-go-vtproto:
go install github.com/planetscale/vtprotobuf/cmd/protoc-gen-go-vtproto
  1. Ensure your project is already using the ProtoBuf v2 API (i.e. google.golang.org/protobuf). The vtprotobuf compiler is not compatible with APIv1 generated code.

  2. Update your protoc generator to use the new plug-in. Example from Vitess:

for name in $(PROTO_SRC_NAMES); do \
    $(VTROOT)/bin/protoc \
    --go_out=. --plugin protoc-gen-go="${GOBIN}/protoc-gen-go" \
    --go-grpc_out=. --plugin protoc-gen-go-grpc="${GOBIN}/protoc-gen-go-grpc" \
    --go-vtproto_out=. --plugin protoc-gen-go-vtproto="${GOBIN}/protoc-gen-go-vtproto" \
    --go-vtproto_opt=features=marshal+unmarshal+size \
    proto/$${name}.proto; \
done

Note that the vtproto compiler runs like an auxiliary plug-in to the protoc-gen-go in APIv2, just like the new GRPC compiler plug-in, protoc-gen-go-grpc. You need to run it alongside the upstream generator, not as a replacement.

  1. (Optional) Pass the features that you want to generate as --go-vtproto_opt. If no features are given, all the codegen steps will be performed.

  2. Compile the .proto files in your project. You should see _vtproto.pb.go files next to the .pb.go and _grpc.pb.go files that were already being generated.

  3. (Optional) Switch your RPC framework to use the optimized helpers (see following sections)

Using the optimized code with RPC frameworks

The protoc-gen-go-vtproto compiler does not overwrite any of the default marshalling or unmarshalling code for your ProtoBuf objects. Instead, it generates helper methods that can be called explicitly to opt-in to faster (de)serialization.

vtprotobuf with GRPC

To use vtprotobuf with the new versions of GRPC, you need to register the codec provided by the github.com/planetscale/vtprotobuf/codec/grpc package.

package servenv

import (
    "github.com/planetscale/vtprotobuf/codec/grpc"
	"google.golang.org/grpc/encoding"
	_ "google.golang.org/grpc/encoding/proto"
)

func init() {
	encoding.RegisterCodec(grpc.Codec{})
}

Note that we perform a blank import _ "google.golang.org/grpc/encoding/proto" of the default proto coded that ships with GRPC to ensure it's being replaced by us afterwards. The provided Codec will serialize & deserialize all ProtoBuf messages using the optimized codegen.

Twirp

I actually have no idea of how to switch encoders in Twirp. Maybe it's not even possible.

DRPC

To use vtprotobuf as a DRPC encoding, simply pass github.com/planetscale/vtprotobuf/codec/drpc as the protolib flag in your protoc-gen-go-drpc invocation.

Example:

protoc --go_out=. --go-vtproto_out=. --go-drpc_out=. --go-drpc_opt=protolib=github.com/planetscale/vtprotobuf/codec/drpc
Issues
  • Error compiling generated code with third party imports

    Error compiling generated code with third party imports

    Hello,

    I was looking to experiment with this and ran into the following error:

    type *"google.golang.org/genproto/googleapis/rpc/status".Status has no field or method MarshalToSizedBufferVT
    

    The protobuf files that are complaining import files like:

    import "google/rpc/status.proto";
    

    with messages that look like:

    message TestMessage {
       google.rpc.Status status = 2;
    }
    

    The google/rpc/status.proto file is copied locally for the code generation, but the generated code is importing the Go module from google.golang.org/genproto/googleapis/rpc/status so it's not part of the vtproto generation steps.

    Is this an issue that you've had to resolve or any suggestions on how to approach this?

    opened by gmather 4
  • Proto3 optional field support

    Proto3 optional field support

    Hello, i have just tried running this generator on my proto3 files and it failed with:

    myproto.proto is a proto3 file that contains optional fields, but code generator protoc-gen-go-vtproto hasn't been updated to support optional fields in proto3. Please ask the owner of this code generator to support proto3 optional.--go-vtproto_out:

    Is optional field supported by this generator ?

    Thanks

    opened by jbub 2
  • Discussion on the use of codec

    Discussion on the use of codec

    Does vtprotobuf supports the use of vtprotobuf and standard proto.Message in the same project (we can rely on the Pb. Go file generated by other projects); Our real scene is that the internal proto file uses vtprotobu to improve performance, and some other functions rely on the third-party pb.go file; The following error occurred during processing

    stream is closed with error: xds: stream.Recv() failed: rpc error: code = Internal desc = grpc: error while marshaling: failed to marshal, message is *envoy_service_discovery_v3.DiscoveryRequest (missing vtprotobuf helpers)" func=goexit 
    
    opened by davygeek 2
  • Add support for non nullable objects - similar to gogoproto.nullable

    Add support for non nullable objects - similar to gogoproto.nullable

    Thanks for putting this library together!

    Would it be possible to add support for non nullable objects - similar to gogo annotation gogoproto.nullable?

    opened by pavolloffay 1
  • generator: support foreign dependencies

    generator: support foreign dependencies

    Fixes https://github.com/planetscale/vtprotobuf/issues/6

    The fix suggested by @alexism (https://github.com/planetscale/vtprotobuf/issues/6#issuecomment-860878143) seems sensible, although it could be a tad expensive, so I'm skipping the special handling in all cases where we know for sure we've already generated a fast Marshal implementation for the code.

    opened by vmg 1
  • feature: add support for optional fields in proto3

    feature: add support for optional fields in proto3

    Fixes https://github.com/planetscale/vtprotobuf/issues/2

    This is a very recent ProtoBuf feature that (interestingly!) was not supported in Gogo ProtoBuf. It requires a new version of protoc -- older versions only supported optional fields in proto3 with an experimental flag enabled.

    I've written the basic amount of tests (i.e. ensuring that codegen is clean and compiles). People who actually use this feature in prod will need to test this more carefully, since Vitess does not use it.

    Signed-off-by: Vicent Marti [email protected]

    opened by vmg 0
  • Better support for maps

    Better support for maps

    Hello, While testing a bit this library in our code base, I noted that map support with pool is not perfect.

    • map themselves are not pooled
    • ResetVT does not return to pool values from a map<key, MessagePooled>
    • UnmarshalVT allocates a new message instead of using the pool for map<key, MessagePooled>

    Doing these changes manually in the .pb.go reduces a lot allocations and speeds up unmarshalling.

    I can provide a sample proto file and modifications made if needed.

    Pre changes: BenchmarkUnmarshalStdProto-12 202058 4971 ns/op 3840 B/op 54 allocs/op BenchmarkUnmarshalVTProto-12 228591 5238 ns/op 3429 B/op 47 allocs/op BenchmarkUnmarshalVTProtoWithPool-12 238689 4967 ns/op 2605 B/op 44 allocs/op

    Post changes: BenchmarkUnmarshalStdProto-12 203602 5240 ns/op 3840 B/op 54 allocs/op BenchmarkUnmarshalVTProto-12 199917 5864 ns/op 3433 B/op 47 allocs/op BenchmarkUnmarshalVTProtoWithPool-12 601562 2009 ns/op 302 B/op 5 allocs/op

    opened by ericrenard 0
  • Proto2 Support

    Proto2 Support

    Is it expected that the vtproto files aren't currently being generated for proto2 files or do I have something misconfigured?

    opened by gmather 3
  • grpc.Codec: Implement fallback to proto.Message

    grpc.Codec: Implement fallback to proto.Message

    Implement a fallback to proto.Message, this is required if you're using the grpc-gateway.

    opened by terwey 0
  • ReturnToVTPool() recursive?

    ReturnToVTPool() recursive?

    When func (p *YourProto) ReturnToVTPool() is called, children of YourProto that implement method ReturnToVTPool() should also be returned to the pool.

    opened by jacobmarble 3
  • Make proto.Marshal use MarshalVT under the hood

    Make proto.Marshal use MarshalVT under the hood

    This fast-marshaling is cool, but I would like to avoid having the VT suffix in my codebase, and would like to simply continue using proto.Marshal(...).

    I read that protoreflect's Message supports an optional ProtoMethods method, and quoting the docs:

    ProtoMethods returns optional fast-path implementions of various operations.

    Is there a way vtprotobuf's fast-marshaling could be added to those methods, instead of new (Un)MarshalVT methods?

    opened by amaurym 4
Owner
PlanetScale
We help operationalize Vitess in the cloud and on premise.
PlanetScale
A new way of working with Protocol Buffers.

Buf All documentation is hosted at https://buf.build. Please head over there for more details. Goal Buf’s long-term goal is to enable schema-driven de

null 2.4k Jun 14, 2021
🚀Gev is a lightweight, fast non-blocking TCP network library based on Reactor mode. Support custom protocols to quickly and easily build high-performance servers.

gev 中文 | English gev is a lightweight, fast non-blocking TCP network library based on Reactor mode. Support custom protocols to quickly and easily bui

徐旭 1.2k Jun 13, 2021
Easily generate gRPC services in Go ⚡️

Lile is a application generator (think create-react-app, rails new or django startproject) for gRPC services in Go and a set of tools/libraries. The p

Lile 1.3k Jun 10, 2021
Protobuf files manager

Prot - protobuf files manager. It application can help your manage protobuf files and generate code based on him. !!! Before use Prot you must install

Maksim Pavlov 5 Dec 21, 2020
A pluggable backend API that enforces the Event Sourcing Pattern for persisting & broadcasting application state changes

A pluggable "Application State Gateway" that enforces the Event Sourcing Pattern for securely persisting & broadcasting application state changes

null 24 Apr 15, 2021
Fast HTTP package for Go. Tuned for high performance. Zero memory allocations in hot paths. Up to 10x faster than net/http

fasthttp Fast HTTP implementation for Go. Currently fasthttp is successfully used by VertaMedia in a production serving up to 200K rps from more than

Aliaksandr Valialkin 15.3k Jun 13, 2021
A simple RPC framework with protobuf service definitions

Twirp is a framework for service-to-service communication emphasizing simplicity and minimalism. It generates routing and serialization from API defin

Twitch 4.8k Jun 16, 2021
Seesaw v2 is a Linux Virtual Server (LVS) based load balancing platform.

Seesaw v2 Note: This is not an official Google product. About Seesaw v2 is a Linux Virtual Server (LVS) based load balancing platform. It is capable o

Google 5.3k Jun 13, 2021
Package ethernet implements marshaling and unmarshaling of IEEE 802.3 Ethernet II frames and IEEE 802.1Q VLAN tags. MIT Licensed.

ethernet Package ethernet implements marshaling and unmarshaling of IEEE 802.3 Ethernet II frames and IEEE 802.1Q VLAN tags. MIT Licensed. For more in

Matt Layher 223 Jun 6, 2021
gRPC/REST proxy for Kafka

Kafka-Pixy (gRPC/REST Proxy for Kafka) Kafka-Pixy is a dual API (gRPC and REST) proxy for Kafka with automatic consumer group control. It is designed

Mailgun Team 641 Jun 13, 2021
A fast reverse proxy to help you expose a local server behind a NAT or firewall to the internet.

frp README | 中文文档 What is frp? frp is a fast reverse proxy to help you expose a local server behind a NAT or firewall to the Internet. As of now, it s

null 46k Jun 12, 2021
📦 Command line peer-to-peer data transfer tool based on libp2p.

pcp - Peer Copy Command line peer-to-peer data transfer tool based on libp2p. Table of Contents Motivation Project Status How does it work? Usage Inst

Dennis Trautwein 757 Jun 20, 2021
Server for Pragyan's Dalal Street

Server for Dalal Street Prerequisites Go 1.13 Download link Protocol buffers Download link MySQL Check prerequisites Check the go version installed. g

Delta Force 56 Apr 8, 2021