A Protocol Buffers compiler that generates optimized marshaling & unmarshaling Go code for ProtoBuf APIv2

Overview

vtprotobuf, the Vitess Protocol Buffers compiler

This repository provides the protoc-gen-go-vtproto plug-in for protoc, which is used by Vitess to generate optimized marshall & unmarshal code.

The code generated by this compiler is based on the optimized code generated by gogo/protobuf, although this package is not a fork of the original gogo compiler, as it has been implemented to support the new ProtoBuf APIv2 packages.

Available features

vtprotobuf is implemented as a helper plug-in that must be run alongside the upstream protoc-gen-go generator, as it generates fully-compatible auxiliary code to speed up (de)serialization of Protocol Buffer messages.

The following features can be generated:

  • size: generates a func (p *YourProto) SizeVT() int helper that behaves identically to calling proto.Size(p) on the message, except the size calculation is fully unrolled and does not use reflection. This helper function can be used directly, and it'll also be used by the marshal codegen to ensure the destination buffer is properly sized before ProtoBuf objects are marshalled to it.

  • marshal: generates the following helper methods

    • func (p *YourProto) MarshalVT() ([]byte, error): this function behaves identically to calling proto.Marshal(p), except the actual marshalling has been fully unrolled and does not use reflection or allocate memory. This function simply allocates a properly sized buffer by calling SizeVT on the message and then uses MarshalToSizedBufferVT to marshal to it.

    • func (p *YourProto) MarshalToVT(data []byte) (int, error): this function can be used to marshal a message to an existing buffer. The buffer must be large enough to hold the marshalled message, otherwise this function will panic. It returns the number of bytes marshalled. This function is useful e.g. when using memory pooling to re-use serialization buffers.

    • func (p *YourProto) MarshalToSizedBufferVT(data []byte) (int, error): this function behaves like MarshalTo but expects that the input buffer has the exact size required to hold the message, otherwise it will panic.

  • unmarshal: generates a func (p *YourProto) UnmarshalVT(data []byte) that behaves similarly to calling proto.Unmarshal(data, p) on the message, except the unmarshalling is performed by unrolled codegen without using reflection and allocating as little memory as possible. If the receiver p is not fully zeroed-out, the unmarshal call will actually behave like proto.Merge(data, p). This is because the proto.Unmarshal in the ProtoBuf API is implemented by resetting the destionation message and then calling proto.Merge on it. To ensure proper Unmarshal semantics, ensure you've called proto.Reset on your message before calling UnmarshalVT, or that your message has been newly allocated.

  • pool: generates the following helper methods

    • func (p *YourProto) ResetVT(): this function behaves similarly to proto.Reset(p), except it keeps as much memory as possible available on the message, so that further calls to UnmarshalVT on the same message will need to allocate less memory. This an API meant to be used with memory pools and does not need to be used directly.

    • func (p *YourProto) ReturnToVTPool(): this function returns message p to a local memory pool so it can be reused later. It clears the object properly with ResetVT before storing it on the pool. This method should only be used on messages that were obtained from a memory pool by calling YourProtoFromVTPool. Using p after calling this method will lead to undefined behavior.

    • func YourProtoFromVTPool() *YourProto: this function returns a YourProto message from a local memory pool, or allocates a new one if the pool is currently empty. The returned message is always empty and ready to be used (e.g. by calling UnmarshalVT on it). Once the message has been processed, it must be returned to the memory pool by calling ReturnToVTPool() on it. Returning the message to the pool is not mandatory (it does not leak memory), but if you don't return it, that defeats the whole point of memory pooling.

Usage

  1. Install protoc-gen-go-vtproto:
go install github.com/planetscale/vtprotobuf/cmd/protoc-gen-go-vtproto
  1. Ensure your project is already using the ProtoBuf v2 API (i.e. google.golang.org/protobuf). The vtprotobuf compiler is not compatible with APIv1 generated code.

  2. Update your protoc generator to use the new plug-in. Example from Vitess:

for name in $(PROTO_SRC_NAMES); do \
    $(VTROOT)/bin/protoc \
    --go_out=. --plugin protoc-gen-go="${GOBIN}/protoc-gen-go" \
    --go-grpc_out=. --plugin protoc-gen-go-grpc="${GOBIN}/protoc-gen-go-grpc" \
    --go-vtproto_out=. --plugin protoc-gen-go-vtproto="${GOBIN}/protoc-gen-go-vtproto" \
    --go-vtproto_opt=features=marshal+unmarshal+size \
    proto/$${name}.proto; \
done

Note that the vtproto compiler runs like an auxiliary plug-in to the protoc-gen-go in APIv2, just like the new GRPC compiler plug-in, protoc-gen-go-grpc. You need to run it alongside the upstream generator, not as a replacement.

  1. (Optional) Pass the features that you want to generate as --go-vtproto_opt. If no features are given, all the codegen steps will be performed.

  2. Compile the .proto files in your project. You should see _vtproto.pb.go files next to the .pb.go and _grpc.pb.go files that were already being generated.

  3. (Optional) Switch your RPC framework to use the optimized helpers (see following sections)

Using the optimized code with RPC frameworks

The protoc-gen-go-vtproto compiler does not overwrite any of the default marshalling or unmarshalling code for your ProtoBuf objects. Instead, it generates helper methods that can be called explicitly to opt-in to faster (de)serialization.

vtprotobuf with GRPC

To use vtprotobuf with the new versions of GRPC, you need to register the codec provided by the github.com/planetscale/vtprotobuf/codec/grpc package.

package servenv

import (
    "github.com/planetscale/vtprotobuf/codec/grpc"
	"google.golang.org/grpc/encoding"
	_ "google.golang.org/grpc/encoding/proto"
)

func init() {
	encoding.RegisterCodec(grpc.Codec{})
}

Note that we perform a blank import _ "google.golang.org/grpc/encoding/proto" of the default proto coded that ships with GRPC to ensure it's being replaced by us afterwards. The provided Codec will serialize & deserialize all ProtoBuf messages using the optimized codegen.

Twirp

I actually have no idea of how to switch encoders in Twirp. Maybe it's not even possible.

DRPC

To use vtprotobuf as a DRPC encoding, simply pass github.com/planetscale/vtprotobuf/codec/drpc as the protolib flag in your protoc-gen-go-drpc invocation.

Example:

protoc --go_out=. --go-vtproto_out=. --go-drpc_out=. --go-drpc_opt=protolib=github.com/planetscale/vtprotobuf/codec/drpc
Issues
  • bug fix in reset slice

    bug fix in reset slice

    Hi! We are using your plugin and its working really great, but recently we have discovered one little bug about memory pooling, connected with this issue https://github.com/planetscale/vtprotobuf/issues/8 . When the message contains repeated field you just reduse size to zero when ResetVT not to overload GC and its cool. The problem is that if repeated element contains pointer field or other repeated field, firstly when you unmarshall data to it and these fields are present in request, fields will be filled in. Secondly, when another request comes without pointer field or slice in element, you take element from pool with this fields set, unmarshal data to it, and then you have a kind of merged data from previous request. Thats why we thought that it may be good to reduse size to zero not only at top level slice, but on lower levels too, and we came with pr as one of the options to solve this issue without overhead of complexity. I hope that i could explain everything clearly. Waiting very much for your feedback

    opened by zcolleen 5
  • buf.build support?

    buf.build support?

    Unfortunately when using buf.build, #19 pops up again, except since buf is doing the work there's no easy way just run a single protoc.

    I'm not sure if this is something vtprotobuf can work around, or if this requires a feature in buf but it's a shame that these two tools don't play nicely together.

    opened by nvx 5
  • Build and test in CI

    Build and test in CI

    I had to build protobuf as this is the only way I found to obtain conformance-test-runner. This build is cached but first runs (for example any new pull request) will take a 30min-ish hit. I can remove this time to zero by caching somewhere less temporary such as some Docker image in this repo's registry. What do you think?

    While setting this up I noticed some generated files where out of sync with git. Should I add their diff to this PR?

    Details
    diff --git a/conformance/internal/conformance/conformance_vtproto.pb.go b/conformance/internal/conformance/conformance_vtproto.pb.go
    index 980bbd5..8d6971f 100644
    --- a/conformance/internal/conformance/conformance_vtproto.pb.go
    +++ b/conformance/internal/conformance/conformance_vtproto.pb.go
    @@ -89,19 +89,9 @@ func (m *ConformanceRequest) MarshalToSizedBufferVT(dAtA []byte) (int, error) {
     		i -= len(m.unknownFields)
     		copy(dAtA[i:], m.unknownFields)
     	}
    -	if m.PrintUnknownFields {
    -		i--
    -		if m.PrintUnknownFields {
    -			dAtA[i] = 1
    -		} else {
    -			dAtA[i] = 0
    -		}
    -		i--
    -		dAtA[i] = 0x48
    -	}
     	if vtmsg, ok := m.Payload.(interface {
    -		MarshalToVT([]byte) (int, error)
    -		SizeVT() int
    +		MarshalTo([]byte) (int, error)
    +		Size() int
     	}); ok {
     		{
     			size := vtmsg.SizeVT()
    @@ -[11](https://github.com/fenollp/vtprotobuf/runs/5318361134?check_suite_focus=true#step:7:11)1,6 +101,16 @@ func (m *ConformanceRequest) MarshalToSizedBufferVT(dAtA []byte) (int, error) {
     			}
     		}
     	}
    +	if m.PrintUnknownFields {
    +		i--
    +		if m.PrintUnknownFields {
    +			dAtA[i] = 1
    +		} else {
    +			dAtA[i] = 0
    +		}
    +		i--
    +		dAtA[i] = 0x48
    +	}
     	if m.JspbEncodingOptions != nil {
     		size, err := m.JspbEncodingOptions.MarshalToSizedBufferVT(dAtA[:i])
     		if err != nil {
    @@ -228,8 +228,8 @@ func (m *ConformanceResponse) MarshalToSizedBufferVT(dAtA []byte) (int, error) {
     		copy(dAtA[i:], m.unknownFields)
     	}
     	if vtmsg, ok := m.Result.(interface {
    -		MarshalToVT([]byte) (int, error)
    -		SizeVT() int
    +		MarshalTo([]byte) (int, error)
    +		Size() int
     	}); ok {
     		{
     			size := vtmsg.SizeVT()
    diff --git a/conformance/internal/conformance/test_messages_proto2_vtproto.pb.go b/conformance/internal/conformance/test_messages_proto2_vtproto.pb.go
    index 23972ed..119b4be 100644
    --- a/conformance/internal/conformance/test_messages_proto2_vtproto.pb.go
    +++ b/conformance/internal/conformance/test_messages_proto2_vtproto.pb.go
    @@ -259,6 +259,18 @@ func (m *TestAllTypesProto2) MarshalToSizedBufferVT(dAtA []byte) (int, error) {
     		i -= len(m.unknownFields)
     		copy(dAtA[i:], m.unknownFields)
     	}
    +	if vtmsg, ok := m.OneofField.(interface {
    +		MarshalTo([]byte) (int, error)
    +		Size() int
    +	}); ok {
    +		{
    +			size := vtmsg.SizeVT()
    +			i -= size
    +			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    +				return 0, err
    +			}
    +		}
    +	}
     	if m.FieldName18__ != nil {
     		i = encodeVarint(dAtA, i, uint64(*m.FieldName18__))
     		i--
    @@ -400,18 +4[12](https://github.com/fenollp/vtprotobuf/runs/5318361134?check_suite_focus=true#step:7:12),6 @@ func (m *TestAllTypesProto2) MarshalToSizedBufferVT(dAtA []byte) (int, error) {
     		i--
     		dAtA[i] = 0xcb
     	}
    -	if vtmsg, ok := m.OneofField.(interface {
    -		MarshalToVT([]byte) (int, error)
    -		SizeVT() int
    -	}); ok {
    -		{
    -			size := vtmsg.SizeVT()
    -			i -= size
    -			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    -				return 0, err
    -			}
    -		}
    -	}
     	if len(m.UnpackedNestedEnum) > 0 {
     		for iNdEx := len(m.UnpackedNestedEnum) - 1; iNdEx >= 0; iNdEx-- {
     			i = encodeVarint(dAtA, i, uint64(m.UnpackedNestedEnum[iNdEx]))
    diff --git a/conformance/internal/conformance/test_messages_proto3_vtproto.pb.go b/conformance/internal/conformance/test_messages_proto3_vtproto.pb.go
    index 76f05a8..d5fecd2 100644
    --- a/conformance/internal/conformance/test_messages_proto3_vtproto.pb.go
    +++ b/conformance/internal/conformance/test_messages_proto3_vtproto.pb.go
    @@ -104,6 +104,18 @@ func (m *TestAllTypesProto3) MarshalToSizedBufferVT(dAtA []byte) (int, error) {
     		i -= len(m.unknownFields)
     		copy(dAtA[i:], m.unknownFields)
     	}
    +	if vtmsg, ok := m.OneofField.(interface {
    +		MarshalTo([]byte) (int, error)
    +		Size() int
    +	}); ok {
    +		{
    +			size := vtmsg.SizeVT()
    +			i -= size
    +			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    +				return 0, err
    +			}
    +		}
    +	}
     	if m.FieldName18__ != 0 {
     		i = encodeVarint(dAtA, i, uint64(m.FieldName18__))
     		i--
    @@ -10[13](https://github.com/fenollp/vtprotobuf/runs/5318361134?check_suite_focus=true#step:7:13),18 +1025,6 @@ func (m *TestAllTypesProto3) MarshalToSizedBufferVT(dAtA []byte) (int, error) {
     		i--
     		dAtA[i] = 0xca
     	}
    -	if vtmsg, ok := m.OneofField.(interface {
    -		MarshalToVT([]byte) (int, error)
    -		SizeVT() int
    -	}); ok {
    -		{
    -			size := vtmsg.SizeVT()
    -			i -= size
    -			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    -				return 0, err
    -			}
    -		}
    -	}
     	if len(m.UnpackedNestedEnum) > 0 {
     		for iNdEx := len(m.UnpackedNestedEnum) - 1; iNdEx >= 0; iNdEx-- {
     			i = encodeVarint(dAtA, i, uint64(m.UnpackedNestedEnum[iNdEx]))
    diff --git a/testproto/proto3opt/opt_vtproto.pb.go b/testproto/proto3opt/opt_vtproto.pb.go
    index 726b[15](https://github.com/fenollp/vtprotobuf/runs/5318361134?check_suite_focus=true#step:7:15)c..[16](https://github.com/fenollp/vtprotobuf/runs/5318361134?check_suite_focus=true#step:7:16)a5e8b 100644
    --- a/testproto/proto3opt/opt_vtproto.pb.go
    +++ b/testproto/proto3opt/opt_vtproto.pb.go
    @@ -50,6 +50,[19](https://github.com/fenollp/vtprotobuf/runs/5318361134?check_suite_focus=true#step:7:19)8 @@ func (m *OptionalFieldInProto3) MarshalToSizedBufferVT(dAtA []byte) (int, error)
     		i -= len(m.unknownFields)
     		copy(dAtA[i:], m.unknownFields)
     	}
    +	if vtmsg, ok := m.XOptionalEnum.(interface {
    +		MarshalTo([]byte) (int, error)
    +		Size() int
    +	}); ok {
    +		{
    +			size := vtmsg.SizeVT()
    +			i -= size
    +			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    +				return 0, err
    +			}
    +		}
    +	}
    +	if vtmsg, ok := m.XOptionalBytes.(interface {
    +		MarshalTo([]byte) (int, error)
    +		Size() int
    +	}); ok {
    +		{
    +			size := vtmsg.SizeVT()
    +			i -= size
    +			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    +				return 0, err
    +			}
    +		}
    +	}
    +	if vtmsg, ok := m.XOptionalString.(interface {
    +		MarshalTo([]byte) (int, error)
    +		Size() int
    +	}); ok {
    +		{
    +			size := vtmsg.SizeVT()
    +			i -= size
    +			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    +				return 0, err
    +			}
    +		}
    +	}
    +	if vtmsg, ok := m.XOptionalBool.(interface {
    +		MarshalTo([]byte) (int, error)
    +		Size() int
    +	}); ok {
    +		{
    +			size := vtmsg.SizeVT()
    +			i -= size
    +			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    +				return 0, err
    +			}
    +		}
    +	}
    +	if vtmsg, ok := m.XOptionalDouble.(interface {
    +		MarshalTo([]byte) (int, error)
    +		Size() int
    +	}); ok {
    +		{
    +			size := vtmsg.SizeVT()
    +			i -= size
    +			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    +				return 0, err
    +			}
    +		}
    +	}
    +	if vtmsg, ok := m.XOptionalFloat.(interface {
    +		MarshalTo([]byte) (int, error)
    +		Size() int
    +	}); ok {
    +		{
    +			size := vtmsg.SizeVT()
    +			i -= size
    +			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    +				return 0, err
    +			}
    +		}
    +	}
    +	if vtmsg, ok := m.XOptionalSfixed64.(interface {
    +		MarshalTo([]byte) (int, error)
    +		Size() int
    +	}); ok {
    +		{
    +			size := vtmsg.SizeVT()
    +			i -= size
    +			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    +				return 0, err
    +			}
    +		}
    +	}
    +	if vtmsg, ok := m.XOptionalSfixed[32](https://github.com/fenollp/vtprotobuf/runs/5318361134?check_suite_focus=true#step:7:32).(interface {
    +		MarshalTo([]byte) (int, error)
    +		Size() int
    +	}); ok {
    +		{
    +			size := vtmsg.SizeVT()
    +			i -= size
    +			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    +				return 0, err
    +			}
    +		}
    +	}
    +	if vtmsg, ok := m.XOptionalFixed[64](https://github.com/fenollp/vtprotobuf/runs/5318361134?check_suite_focus=true#step:7:64).(interface {
    +		MarshalTo([]byte) (int, error)
    +		Size() int
    +	}); ok {
    +		{
    +			size := vtmsg.SizeVT()
    +			i -= size
    +			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    +				return 0, err
    +			}
    +		}
    +	}
    +	if vtmsg, ok := m.XOptionalFixed32.(interface {
    +		MarshalTo([]byte) (int, error)
    +		Size() int
    +	}); ok {
    +		{
    +			size := vtmsg.SizeVT()
    +			i -= size
    +			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    +				return 0, err
    +			}
    +		}
    +	}
    +	if vtmsg, ok := m.XOptionalSint64.(interface {
    +		MarshalTo([]byte) (int, error)
    +		Size() int
    +	}); ok {
    +		{
    +			size := vtmsg.SizeVT()
    +			i -= size
    +			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    +				return 0, err
    +			}
    +		}
    +	}
    +	if vtmsg, ok := m.XOptionalSint32.(interface {
    +		MarshalTo([]byte) (int, error)
    +		Size() int
    +	}); ok {
    +		{
    +			size := vtmsg.SizeVT()
    +			i -= size
    +			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    +				return 0, err
    +			}
    +		}
    +	}
    +	if vtmsg, ok := m.XOptionalUint64.(interface {
    +		MarshalTo([]byte) (int, error)
    +		Size() int
    +	}); ok {
    +		{
    +			size := vtmsg.SizeVT()
    +			i -= size
    +			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    +				return 0, err
    +			}
    +		}
    +	}
    +	if vtmsg, ok := m.XOptionalUint32.(interface {
    +		MarshalTo([]byte) (int, error)
    +		Size() int
    +	}); ok {
    +		{
    +			size := vtmsg.SizeVT()
    +			i -= size
    +			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    +				return 0, err
    +			}
    +		}
    +	}
    +	if vtmsg, ok := m.XOptionalInt64.(interface {
    +		MarshalTo([]byte) (int, error)
    +		Size() int
    +	}); ok {
    +		{
    +			size := vtmsg.SizeVT()
    +			i -= size
    +			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    +				return 0, err
    +			}
    +		}
    +	}
    +	if vtmsg, ok := m.XOptionalInt32.(interface {
    +		MarshalTo([]byte) (int, error)
    +		Size() int
    +	}); ok {
    +		{
    +			size := vtmsg.SizeVT()
    +			i -= size
    +			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    +				return 0, err
    +			}
    +		}
    +	}
     	if m.OptionalEnum != nil {
     		i = encodeVarint(dAtA, i, uint64(*m.OptionalEnum))
     		i--
    
    opened by fenollp 4
  • ReturnToVTPool() recursive?

    ReturnToVTPool() recursive?

    When func (p *YourProto) ReturnToVTPool() is called, children of YourProto that implement method ReturnToVTPool() should also be returned to the pool.

    opened by jacobmarble 4
  • Error compiling generated code with third party imports

    Error compiling generated code with third party imports

    Hello,

    I was looking to experiment with this and ran into the following error:

    type *"google.golang.org/genproto/googleapis/rpc/status".Status has no field or method MarshalToSizedBufferVT
    

    The protobuf files that are complaining import files like:

    import "google/rpc/status.proto";
    

    with messages that look like:

    message TestMessage {
       google.rpc.Status status = 2;
    }
    

    The google/rpc/status.proto file is copied locally for the code generation, but the generated code is importing the Go module from google.golang.org/genproto/googleapis/rpc/status so it's not part of the vtproto generation steps.

    Is this an issue that you've had to resolve or any suggestions on how to approach this?

    opened by gmather 4
  • features: equal

    features: equal

    This begins the work of porting [a variant of] gogo's equal plugin

    An unrolled non-reflective implementation of proto.Equal is about twice faster in our testing.

    Note: this is WIP. It is hard to test right now (I am working on a CI PR that should help here) mainly because this command cannot be used with forks:

    go install github.com/planetscale/vtprotobuf/cmd/[email protected]
    

    Any help around this is very welcomed!

    opened by fenollp 2
  • Duplicate Functions and Variables in Generated Package

    Duplicate Functions and Variables in Generated Package

    First, just want to say thank you for working on and releasing vtprotobuf. We're working on transitioning out of gogo/protobuf and read your great blog post announcing this alternative.

    We've found a slight issue with our use case. We have a few proto packages that have multiple files in them. The generated _vtproto.pb.go files redeclare some utility functions/variables (e.g. sov, skip, ErrInvalidLength, etc.). As an example:

    hellopb/service.proto:

    syntax = "proto3";
    
    package hellopb;
    
    ...
    
    message HelloRequest {
      string q = 1;
    }
    
    message HelloResponse {
      string response = 2;
    }
    
    service HelloService {
      rpc Hello(HelloRequest) returns (HelloResponse) {}
    }
    

    hellopb/db.proto:

    syntax = "proto3";
    
    package hellopb;
    
    ...
    
    message MessageEntry {
      string q = 1;
      ...
    }
    

    We then run:

    protoc --proto_path=. --proto_path=../../../  \
      --go_out=../../../ --plugin protoc-gen-go=/go/bin/protoc-gen-go \
      --go-grpc_out=../../../ --plugin protoc-gen-go-grpc=/go/bin/protoc-gen-go-grpc \
      --grpc-gateway_out=../../../ \
      --go-vtproto_out=../../../ --plugin protoc-gen-go-vtproto=/go/bin/protoc-gen-go-vtproto \
      --go-vtproto_opt=features=marshal+unmarshal+size hellopb/service.proto
    

    and

    protoc --proto_path=. --proto_path=../../../  \
      --go_out=../../../ --plugin protoc-gen-go=/go/bin/protoc-gen-go \
      --go-grpc_out=../../../ --plugin protoc-gen-go-grpc=/go/bin/protoc-gen-go-grpc \
      --grpc-gateway_out=../../../ \
      --go-vtproto_out=../../../ --plugin protoc-gen-go-vtproto=/go/bin/protoc-gen-go-vtproto \
      --go-vtproto_opt=features=marshal+unmarshal+size hellopb/db.proto
    

    If we run go vet we get:

    service_vtproto.pb.go:214:6: encodeVarint redeclared in this block
        db_vtproto.pb.go:125:54: previous declaration
    service_vtproto.pb.go:308:6: sov redeclared in this block
        db_vtproto.pb.go:188:23: previous declaration
    service_vtproto.pb.go:311:6: soz redeclared in this block
        db_vtproto.pb.go:191:23: previous declaration
    service_vtproto.pb.go:694:6: skip redeclared in this block
        db_vtproto.pb.go:545:36: previous declaration
    service_vtproto.pb.go:774:2: ErrInvalidLength redeclared in this block
        db_vtproto.pb.go:625:2: previous declaration
    service_vtproto.pb.go:775:2: ErrIntOverflow redeclared in this block
        db_vtproto.pb.go:626:2: previous declaration
    service_vtproto.pb.go:776:2: ErrUnexpectedEndOfGroup redeclared in this block
        db_vtproto.pb.go:627:2: previous declaration
    

    Is there anyway to avoid this? Post protoc cleanup on this is pretty tough. Is there anyway those functions/variables could just be imported from vtprotobuf?

    opened by michael-james-holloway 2
  • Question: grpc

    Question: grpc

    One of the features (which is on by default if features are not specified) is grpc. A quick look at the code and I can't seem to find any difference between the regular grpc plugin and the code generated vtprotobuf. Is there a plan to extends this to use say use pool for object creation in the future?

    opened by srikrsna 2
  • Twirp Integration

    Twirp Integration

    In response to "I actually have no idea of how to switch encoders in Twirp. Maybe it's not even possible.", this can't be done out of the box with the code generation, but can be achieved (server-side) with a simple find and replace.

    The changes are below:

    proto.Marshal(respContent) >> respContent.MarshalVT()
    proto.Unmarshal(buf, reqContent) >> reqContent.UnmarshalVT(buf)
    

    I'm using make to control all code gen, so I added the below as a final step:

    for twirp in $${dir}/*.twirp.go; \
    do \
      echo 'Updating' $${twirp}; \
      sed -i '' -e 's/respBytes, err := proto.Marshal(respContent)/respBytes, err := respContent.MarshalVT()/g' $${twirp}; \
      sed -i '' -e 's/if err = proto.Unmarshal(buf, reqContent); err != nil {/if err = reqContent.UnmarshalVT(buf); err != nil {/g' $${twirp}; \
    done; \
    
    opened by T-J-L 2
  • Discussion on the use of codec

    Discussion on the use of codec

    Does vtprotobuf supports the use of vtprotobuf and standard proto.Message in the same project (we can rely on the Pb. Go file generated by other projects); Our real scene is that the internal proto file uses vtprotobu to improve performance, and some other functions rely on the third-party pb.go file; The following error occurred during processing

    stream is closed with error: xds: stream.Recv() failed: rpc error: code = Internal desc = grpc: error while marshaling: failed to marshal, message is *envoy_service_discovery_v3.DiscoveryRequest (missing vtprotobuf helpers)" func=goexit 
    
    opened by davygeek 2
  • Proto3 optional field support

    Proto3 optional field support

    Hello, i have just tried running this generator on my proto3 files and it failed with:

    myproto.proto is a proto3 file that contains optional fields, but code generator protoc-gen-go-vtproto hasn't been updated to support optional fields in proto3. Please ask the owner of this code generator to support proto3 optional.--go-vtproto_out:

    Is optional field supported by this generator ?

    Thanks

    opened by jbub 2
  • Unable to get pooling methods to gen

    Unable to get pooling methods to gen

    I am currently attempting to put together a POC for using vtprotobuf in an application I work on. However the most promising feature (pooling) does not seem to exist in the generated code.

    My command line:

    protoc \
      -Ipkg/.patched-proto \
      --go_out=paths=source_relative:./pkg/tempopb/ \
      --go-grpc_out=paths=source_relative:./pkg/tempopb/ \
      --go-vtproto_out=paths=source_relative:./pkg/tempopb/ \
      --go-vtproto_opt=features=marshal+unmarshal+size+pool \
      pkg/.patched-proto/trace/v1/trace.proto
    

    The output files seem to be generated correctly and there are no errors:

    image

    But I'm not seeing ResetVT, ReturnVTToPool, or *FromVTPool generated. I have tried with 0.2.0 as well as tip of main. I have also tried not specifying --go-vtproto_opt without luck.

    I am seeing MarshalVT, MarshalToVT, SizeVT, UnmarshalVT, ... generated.

    Thanks for your time!

    opened by joe-elliott 7
  • TinyGo support

    TinyGo support

    First off, wanted to say thanks for such a great project. While the issue title may sound like a feature request, the code generated by vtprotobuf already works with TinyGo. You can see an example of that here: https://github.com/kyleconroy/go-wasm-plugins

    What I'd like to ask about is making TinyGo support explicit via a CI job. Is this something you'd be interested in supporting? If so I can take a first pass.

    opened by kyleconroy 1
  • gRPC and pooling

    gRPC and pooling

    I have a couple of questions regarding pooling and gRPC that I could not fully understand from existing issues or the readme. (happy to do a PR to clarify in the readme after understanding for the next person)

    In https://github.com/planetscale/vtprotobuf/issues/16#issuecomment-878939364 it was mentioned, that memory-pooled objects are automatically unmarshaled using objects from a pool, which I could get to work. How is this intended to work for the other way around though? Does it happen automatically somewhere like during marshaling (if so I can't find the code that does this)? If not, is there a recommended or suggested place where this could be done (maybe in the codec?)

    My use case is I have a lot of objects that I read from a key-value store that eventually end up in a gRPC response, but after marshaling the response I would like to return the objects to the pool.

    opened by brancz 1
  • Add support for pools for oneof fields

    Add support for pools for oneof fields

    I have implemented the minimal changes necessary for pools to work with oneof fields because the generated code currently does not compile. If needed, I can work on improving the code, add some tests and so on but I'd be interested first in knowing whether you'd be willing to merge something like this.

    opened by maticz 5
  • Provide clarification on reusing serialization buffers

    Provide clarification on reusing serialization buffers

    Documentation says:

    MarshalToVT() ... This function is useful e.g. when using memory pooling to re-use serialization buffers.

    Could you please provide clarification on how to use it? I'm asking because protobufs are typically used with gRPC, and grpc-go's SendMsg() returns before the buffer gets put on the wire. Hence, you cannot reuse it. Here is a relevant issue: https://github.com/grpc/grpc-go/issues/2159. Someone has even attempted this http://www.golangdevops.com/2019/12/31/autopool/ but it's not a solution and the finalizers have poor performance. You could find even more details here https://github.com/thanos-io/thanos/pull/4609.

    Are there any examples of the usage of this function? I couldn't find anything.

    If I am correct then the recommendation in the README seems dangerous.

    opened by GiedriusS 1
Owner
PlanetScale
We help operationalize Vitess in the cloud and on premise.
PlanetScale
A new way of working with Protocol Buffers.

Buf All documentation is hosted at https://buf.build. Please head over there for more details. Goal Buf’s long-term goal is to enable schema-driven de

null 4.3k May 12, 2022
Protocol Buffers - Google's data interchange format

Protocol Buffers - Google's data interchange format Copyright 2008 Google Inc. https://developers.google.com/protocol-buffers/ Overview Protocol Buffe

Protocol Buffers 54.4k May 10, 2022
A plugin of protoc that for using a service of Protocol Buffers as http.Handler definition

protoc-gen-gohttp protoc-gen-gohttp is a plugin of protoc that for using a service of Protocol Buffers as http.Handler definition. The generated inter

John_Suu 1 Dec 9, 2021
Go support for Protocol Buffers - Google's data interchange format

Go support for Protocol Buffers - Google's data interchange format Google's data interchange format. Copyright 2010 The Go Authors. https://github.com

Tinker Board - Android 0 Dec 15, 2021
Estudos com Golang, GRPC e Protocol Buffers

Golang, GRPC e Protocol Buffers Estudos com Golang, GRPC e Protocol Buffers Projeto feito para fins de estudos. Para rodar basta seguir os passos abai

Marcos Daniel 1 Feb 10, 2022
This is a golang C2 + Implant that communicates via Protocol Buffers (aka. protobufs).

Br4vo6ix DISCLAIMER: This tool is for educational, competition, and training purposes only. I am in no way responsible for any abuse of this tool This

RITSEC Redteam 4 Feb 18, 2022
Compiler as a Service is a compiler that is available over http/https and gRPC

BlakBoks(CaaS) Elasticsearch but for compiling untrusted code Compiler as a Service is a compiler that is available over http/2 and gRPC. Setup First

Nathan Kimutai 0 Nov 24, 2021
Control your Flipper Zero over Protobuf RPC protocol.

go-flipper Control your Flipper Zero over Protobuf RPC protocol. This library is designed to be transport agnostic, though I've tested it with RPC ove

Flipper Devices 28 Apr 25, 2022
Assembly-optimized MD4 hash algorithm in Go

md4 MD4 hash algorithm in Go. Assembly-optimized for amd64 platforms. MD4 is cryptographically broken and should should only be used where compatibili

Michael McLoughlin 15 Apr 14, 2022
Schema-free, document-oriented streaming database that optimized for monitoring network traffic in real-time

Basenine Schema-free, document-oriented streaming database that optimized for monitoring network traffic in real-time. Featured Aspects Has the fastes

UP9 34 May 5, 2022
GoScan is a port-scanner made entirely in Go-lang. The purpose of the tool is to be fast, dynamic and simple so that a professional in the CyberSecurity area can make an optimized list of ports

?? GoScan GoScan is a port-scanner made entirely in Go-lang. The purpose of the tool is to be fast, dynamic and simple so that a professional in the C

Ottoni 5 Feb 20, 2022
wire protocol for multiplexing connections or streams into a single connection, based on a subset of the SSH Connection Protocol

qmux qmux is a wire protocol for multiplexing connections or streams into a single connection. It is based on the SSH Connection Protocol, which is th

Jeff Lindsay 184 May 9, 2022
A simple tool to convert socket5 proxy protocol to http proxy protocol

Socket5 to HTTP 这是一个超简单的 Socket5 代理转换成 HTTP 代理的小工具。 如何安装? Golang 用户 # Required Go 1.17+ go install github.com/mritd/[email protected] Docker 用户 docker pull m

mritd 5 Apr 8, 2022
Protobuf files manager

Prot - protobuf files manager. It application can help your manage protobuf files and generate code based on him. !!! Before use Prot you must install

Maksim Pavlov 5 Dec 21, 2020
Generate types and service clients from protobuf definitions annotated with http rules.

protoc-gen-typescript-http Generates Typescript types and service clients from protobuf definitions annotated with http rules. The generated types fol

Einride 17 Apr 16, 2022
A simple RPC framework with protobuf service definitions

Twirp is a framework for service-to-service communication emphasizing simplicity and minimalism. It generates routing and serialization from API defin

Twitch 5.7k May 9, 2022
protobuf ではなく JSON でやり取りするファイルを出力する protoc プラグイン

protoc-gen-jsonif proto ファイルから、JSON フォーマットでやりとりする型定義ファイルを出力する protoc プラグインです。 proto ファイルで言語を越えて型定義が出来るのはとても良い しかし protobuf ライブラリを入れるのが面倒 今のプロジェクトには既に

melpon 3 Feb 28, 2022
WIP protobuf support for Gleam ✨

gleam_pb WIP protobuf support for Gleam ✨ Progress Gleam Type generation custom functions that better handle default values stop including unnecessary

Benjamin Wireman 12 Feb 26, 2022
Experiments with pulsar go client and protobuf

How to use proto schema with pulsar go client library Define a .proto file Generate Code using below command: protoc -I=. -I=$GOPATH/src -I=$GOPATH/sr

Deepak Sah 1 Nov 15, 2021