I am hopeful that there is a simple solution to get around this issue :)

A few hours ago, this commit introduced proto.ProtoPackageIsVersion3

and set const generatedCodeVersion = 3

Our CI system (and I suspect many other peoples) do a:

go get github.com/golang/protobuf/protoc-gen-go

as part of setting up the system, and also do a:

go generate ./...

and check to make sure that regenerated files match the files in the commit (basically, checking for regen on updating the .proto files).

This has been going swimmingly until a few hours ago. In trying to debug this problem, I came to realize that the fact this worked at all is a testament to how good you guys are at backward compatibility 👍

What I discovered digging in is that we have been vendoring github.com/golang/protobuf/ v1.2.0 (because of other dependencies we have), but of course go get pulls the latest from master.

v1.2.0 lacks proto.ProtoPackageIsVersion3, so our generated code suddenly doesn't compile.

So I fell back to using:

go install ./vendor/github.com/golang/protobuf/protoc-gen-go/

in our CI. Unfortunately, this seems to result in some generation differences. If I run it locally, push, and then our CI runs it, the generated code doesn't quite match.

I get diffs like:

-var fileDescriptor_networkservice_d4de7ff5f5769d53 = []byte{
+var fileDescriptor_networkservice_5b085879d0d87bdf = []byte{

I don't precisely consider this a bug on your end, you guys are super good about backward compatibility, and at some point you do have to make the change.

Do you have any ideas about how I might navigate all of this? I suspect lots and lots of folks will shortly have similar issues...

Hello,

Sharing a same .proto filename with different proto package names results in these errors:

2020/05/10 15:03:12 WARNING: proto: file "common.proto" is already registered previously from: "git.example.com/proto/common" currently from: "git.example.com/proto/a" A future release will panic on registration conflicts. See: https://developers.google.com/protocol-buffers/docs/reference/go/faq#namespace-conflict

2020/05/10 15:03:12 WARNING: proto: file "common.proto" is already registered previously from: "git.example.com/proto/common" currently from: "git.example.com/proto/b" A future release will panic on registration conflicts. See: https://developers.google.com/protocol-buffers/docs/reference/go/faq#namespace-conflict

Here is my layout:

proto/
  common/
    common.proto (package common)
  a/
    common.proto (package a and importing common/common.proto)
    foo.proto (importing common.proto)
    ...
  b/
    common.proto (package b and importing common/common.proto)
    bar.proto  (importing common.proto)
    ...

I don't think my case is similar to both quoted in documentation https://developers.google.com/protocol-buffers/docs/reference/go/faq#namespace-conflict. I don't generate multiple times a same proto file (here I generate one time common/common.proto and I import from others files) and I'm using different package names. It would be better if I rename package common to something else, but I don't think it's my issue here.

What's the best practice to solve this issue ? Is it normal to get this warning in my case ? I mean, it doesn't look uncommon to have the same filename in different projects.

Thanks

We’re in the process of migrating from github.com/golang/protobuf to the v2 API (google.golang.org/protobuf) and hit a panic when using proto.Equal to compare protos.

We traced the issue to a use of (reflect.Value).IsNil() here: https://github.com/protocolbuffers/protobuf-go/blob/1f5b6fe64530cac2061a3d315b7e44966b1a200b/internal/impl/message_reflect_field.go#L386-L392

(reflect.Value).IsNil() panics when the underlying value is not a pointer type (e.g. struct pointer, channel, slice, map, etc.).

It looks like v2’s internal API assumes that all non-scalar values are pointers, which in this case is not true—we’re using GoGo Protobuf’s feature of non-nullable fields in most of our messages, primarily time.Time values and occasional struct value where we don’t want the additional GC load.

We forked this and replaced the IsNil call with IsZero, which in this use case works correctly. Suggestion: swap out calls to IsNil with calls to IsZero to prevent this runtime panic and work correctly with legacy v1 protos.

Additionally, it looks like this package reimplements IsZero to some extent here: https://github.com/protocolbuffers/protobuf-go/blob/1f5b6fe64530cac2061a3d315b7e44966b1a200b/internal/descfmt/stringer.go#L224-L243

Thanks!

Filing on behalf of @alandonovan.

The proto.Message interface is unsatisfactory. A behavioral interface is an abstraction over the underlying data types that exposes just the operations necessary for their correct use. By contrast, proto.Message exposes essentially no useful functionality and serves only as a marker. But a marker of what? If one could assume all its implementations were protoc-generated struct types with field tags, then at least it would be possible to write reflection-based algorithms that do useful things. However, there are many concrete types satisfying Message that are not of this form.

It's not only the set of implementations that is unbounded; the set of useful operations is also large and growing. The two most important, Marshal and Unmarshal, are handled quite cleanly since there are separate behavioral interfaces for Marshaler and Unmarshaler that allow each concrete type to implement these operations. But there are many functions in the proto API, for which no interface exists: proto.Merge, proto.Clone, the extensions API, and so on.

The cross-product of concrete implementations and operations is growing, but the fraction of these combinations that actually work is diminishing.

I think we should assess what it would take to change the proto.Message interface, and all its implementations, so that it is a true behavioral interface. This would require at a minimum that the interface include a new method that provides complete control over the abstract state of a message: accessing and updating its fields, inspecting any extensions or unrecognized fields, and so on, without revealing the concrete representation. It should be possible to implement all the major functions in the proto API, as well as most users' ad hoc functions, in terms of this interface so that they work with any concrete implementation. If an optimized version of a crucial operation is available, the generic implementation should dispatch to it, as it does today for Marshal and Unmarshal.

We can't add methods to proto.Message without breaking backwards compatibility. One approach we can take is to define proto.MessageV2 that is a much more semantically complete interface that provides a form of "protobuf reflection". In Marshal, Unmarshal, Merge, Clone, and so on, we can type assert if it implements proto.MessageV2 and use that interface to implement generic versions of those functions. If proto.Message doesn't satisfy proto.MessageV2, then Merge can just fail (it already does on most third-party implementations of proto.Message).

This is a meta-issue for tracking any breaking changes that occur before the final v2 release. Since v2 is not yet released, we reserve the right to make breaking changes to improve the API to be as good as it can be.

The CL (changelist) that introduces a breaking change will be posted here as a comment. Subscribe to this issue to be notified of breakages. We try to batch multiple breaking changes close together in time to reduce user churn.

The exported XXX_ fields on generated message structs are awkward. They make "obvious" reflection code over proto messages easy to get wrong (by not skipping the XXX_ fields), and they pollute the documentation of the generated packages with irrelevant implementation details.

I believe it is possible to unexport them. They appear to need to be exported due to the restriction on reflect.Value.Interface of unexported fields, but we can observe that every nested proto.Message is reached through only exported fields or XXX_ fields.

One technique we could use to access the fields after unexporting them is double-embedding. Start with an exported type in the proto package with an unexported (pointer) accessor method. Add one (unexported) struct type per generated proto package which embeds the exported struct. Embed this unexported struct in each message which needs the corresponding field.

The unexported struct field is itself unexported, but since that embeds a struct with an unexported method defined in the proto package it is accessible within proto via type-assertion to an interface containing the method.

A rough sketch: https://play.golang.org/p/yDjZZtGmB6

Hi, I correctly installed and protobuf in my Ubuntu 18.04.01 Server Edition following the steps here outlined: https://github.com/protocolbuffers/protobuf/blob/master/src/README.md

and without any problems compiled the example found here: https://developers.google.com/protocol-buffers/docs/cpptutorial

protoc --cpp_out=. ./addressbook.proto
-rw-rw-r-- 1 marco marco  49K Jan 30 11:26 addressbook.pb.cc
-rw-rw-r-- 1 marco marco  27K Jan 30 11:26 addressbook.pb.h
-rw-rw-r-- 1 marco marco  410 Jan 30 11:25 addressbook.proto

But when trying to do the same in golang:

protoc --go_out=. addressbook.proto
protoc-gen-go: program not found or is not executable
--go_out: protoc-gen-go: Plugin failed with status code 1.

Other information:

in my .bashrc I added the following lines:

export GOPATH=${HOME}/go:$PATH
export GOBIN=$HOME/go/bin:$PATH

go version:

go version go1.10.4 linux/amd64

How to make protoc-gen-go functioning? Looking forward to your kind help. Marco

Right now, a .proto file can have a line like

option go_package = "foo";

which means that the generated .pb.go file should have a package foo statement.

I am proposing to overload this option with two new possible syntaxes:

option go_package = "github.com/example/foo";

This means that generated code should be dropped into the github.com/example/foo directory (relative to whatever is passed to protoc's --go_out flag), overriding the default behaviour of matching the path to the .proto file. It also means the .pb.go file should have a package foo statement.

option go_package = "github.com/example/foo;bar";

This means the same as the previous syntax, but additionally means that the .pb.go file should have a package bar statement, for the rare cases where people want the package name to differ from the final path component of the import path.

This would hopefully mean that far fewer M parameters need to be passed through the --go_out flag. It would subsume #137 and maybe #63.

Thoughts? Opinions?

@robpike @zellyn @mwitkow @awalterschulze @peter-edge @tamird

Passing multiple packages' worth of *.proto files currently fails when using (protoc-gen-go)[https://github.com/golang/protobuf/blob/master/protoc-gen-go/generator/generator.go#L612:L614].

The same functionality is supported by other generators (at least the C++ generator shipped with protoc). protoc-gen-go should support this as well.

I am recently try to build a project which I had build it multiple times successfully. But recently I get this error while building grpcurl as part of build process, is there any imcompatible changes made to your repo? /# Build grpcurl go get github.com/fullstorydev/grpcurl /# google.golang.org/protobuf/proto /root/go/src/google.golang.org/protobuf/proto/proto_methods.go:18:23: cannot use m.ProtoMethods() (type *protoreflect.Message) as type *struct ...

Background

Identifiers in proto are not directly translatable into Go for several reasons:

1. By convention, naming styles are different:
   • Protos uses CamelCase for messages, groups, enums, services, and methods
   • Protos uses snake_case for fields and oneof fields
   • Protos uses SCREAMING_SNAKE_CASE for enum values
   • Go uses CamelCase for everything
2. As a language rule, Go uses an uppercase letter to indicate visibility. Protobuf has no such visibility rules and everything is assumed to be publicly visible to everything else.
3. Proto allows you to declare an enum or message within a message. Thus, a Child message defined in the scope of a Parent message is named Parent.Child. However, Go has no language equivalent for this (see golang/go#20467).

Status quo

When protoc-gen-go generates Go identifiers, it goes through some lexicographical transformation to convert proto naming conventions to Go naming conventions (see generator.CamelCase).

This lexicographical re-write works well most of the time, but is sub-par in the following cases:

• It cannot handle abbreviations well: enable_gps => EnableGps
• It can result in name conflicts:
  • foo and Foo are distinct identifiers in proto, but both are named Foo in Go to work around visibility rules.
  • Parent_Child and Parent.Child are distinct identifiers proto, but both are named Parent_Child in Go to work around Go's lack of nested declarations (see my concern #513).

Proposal

I propose that we add support for a go_name option. This is an option that all Go generators should be aware of and respect, with adoption starting with protoc-gen-go.

Example:

syntax = "proto3";

package example.goname;

message FooMessage {
    string enable_gps = 1 [go_name = "EnableGPS"]
}

The option can be applied to following: MessageOptions, FieldOptions, OneOfOptions, EnumOptions, and EnumValueOptions.

• The go_name option for MessageOptions and EnumOptions must be supported by all generators since a proto file may specify a message or enum from another proto file by name. The generator must be able to reliably reference the target type.
  • Note that this bakes in the assumption that generated code will always have a top-level Go type for each message and enum proto type.
• It is optional whether a generator respects the option for the other types, although protoc-gen-go will.
  • EnumValueOptions controls the enum variable name
  • FieldOptions and OneOfOptions controls the struct field name
    • For a hypothetical generator that emit messages with no fields, but only getters and setters, it up to the implementation what they want to do. Using SetX and GetX is a reasonable approach.

Optionally supported for: ServiceOptions and MethodOptions

• protoc-gen-go does not directly support services. However, it is reasonable that grpc also respect these options.

They do not apply to: FileOptions and ExtensionRangeOptions.

• Do not apply because there is no sensible meaning for go_name here. go_package is the file-level equivalent that controls naming and import path information for the Go package.

Secondly, I propose that we formalize the rules for how Go names are generated when the go_name option is lacking. Essentially, document the functionality of generator.CamelCase. This is one step in the direction of having a sensible ecosystem of various generators and/or plugins (#547).

Use-cases

• Provide an opt-out when making breaking changes to the generator. For example, suppose we dropped the enum type prefix to fix #513, we would need to provide a way for users to continue using the old name.
• For situations where automatic CamelCasing is sub-par, provides the user the ability to provide an idiomatic Go name (e.g., EnableGPS)
  • This supersede the customname, enum_customname and enumvalue_customname options in gogo/protobuf.
• Satisfy the desire to generate unexported message and type names (see #501)
  • Intentionally choosing an unexported name means you are certain that nobody else imports your proto file.
• Provides a workaround for rare situations of name conflicts

Design specifics

What is valid syntax for go_name? The string must be a valid Go identifier per the language specification. If it is invalid, then the generator reports an error. It is the user's responsibility to ensure that Go identifiers do not conflict. A generator implementation may choose to report an error at generation time if such a conflict exists.

How does this work with nested messages and enums? Suppose you have a proto file with message Parent { message Child { option go_name = "Foo"; } }. Is the type name of the inner message going to be Foo or Parent_Foo. I argue that the name is "Foo" as that is strictly more expressive than simply prefixing the parent name.

What happens when field names are unexported? The current implementation of the proto package will crash under this situation. However, I argue that this behavior is just an implementation detail of the current package.

There are ways to workaround this. For example, you could generate a Reference method that returns a pointer to any field of a given unexported name.

func (m *FooMessage) Reference(name string) interface{} {
    switch name {
    case "field1": return &m.field1
    case "field2": return &m.field2
    case "field3": return &m.field3
    default:       panic(fmt.Sprintf("%T has no field named: %s", *m, name))
    }
}

The protoc package could type assert for this method and use it.

However, I would prefer not polluting the public API of generated messages any further. Thus, for the initial support for go_name in protoc-gen-go, we can intentionally report an error until some future time we decide to fully support this. However, other generators like protoc-gen-gogo may support this. I'm comfortable with the fact that it is at least possible.

I have several ideas for #276 that will be useful here.

What does it mean for field extensions? TBD.

What does it mean for enum value defaults? TBD.

What does it mean for fields belonging to a oneof? TBD.

\cc @neild @rogpeppe @awalterschulze @htuch

So this library now recommends to switch to google's implementation, which is ok with me. But the problem is that this library has

type JSONPBMarshaler interface {
	MarshalJSONPB(*Marshaler) ([]byte, error)
}

type JSONPBUnmarshaler interface {
	UnmarshalJSONPB(*Unmarshaler, []byte) error
}

but the Google library does not. So what is the migration path in this case? The note // Deprecated: Custom types should implement protobuf reflection instead. is quite useless because it does not say anything useful.

What version of protobuf and what language are you using? protoc-gen-go v1.28.1 protoc v3.21.9

What did you do? I'm marshaling a protobuf message using protojson

The message User has an EmployeeNumber field:

// (optional) employee number
EmployeeNumber *string `protobuf:"bytes,2,opt,name=EmployeeNumber,proto3,oneof" json:"EmployeeNumber,omitempty"`

I'm using the following go code to create such a User and marshal it:

        number := "number"
	user := User{
		EmployeeNumber: &number,
	}
	options := protojson.MarshalOptions{
		EmitUnpopulated: true,
		UseProtoNames:   false,
		UseEnumNumbers:  true,
	}
	json_bytes, err := options.Marshal(&user)
	println(string(json_bytes))

What did you expect to see? Some json containing "employeeNumber":"number". That is, using lowerCamelCase, given I opted for UseProtoNames: false The description is (from https://pkg.go.dev/google.golang.org/protobuf/encoding/protojson#MarshalOptions):

// UseProtoNames uses proto field name instead of lowerCamelCase name in JSON
// field names.
UseProtoNames [bool]

since I'm using false I expect lowerCamelCase.

What did you see instead? Some json containing "EmployeeNumber":"number". That is, not using lowerCamelCase because I used false for UseProtoNames and the description for that field (as I understand it) indicates lowerCamelCase name in JSON field names will be used.

I have a message M with a Oneof field O, one of the values O can take Ox has a field F.

I want to ignore F in a comparison.

I'm unable to address the field without either getting a panic or a compilation error using either protocmp.IgnoreFields or protocmp.IgnoreOneofs.

How can I ignore F in a cmp.Equal comparison?

Currently, when passing an invalid enum value, protobuf returns error saying invalid value for enum type: \"<value>\": Request:

{
    "enumField": "invalid_value"
}

Response:

{
    "code": 3,
    "message": "proto: (line X:Y): invalid value for enum type: \"invalid_value\"",
    "details": []
}

I expect it to return the name of the enum instead of the value. I check the source code of the decoder: https://github.com/protocolbuffers/protobuf-go/blob/f0e23c7a8f8d55910f877091fdad71d3abb26d10/encoding/protojson/decode.go#L331-L340

And I think it should have the commented line:

case protoreflect.EnumKind:
	if v, ok := unmarshalEnum(tok, fd); ok {
		return v, nil
	}
	// return protoreflect.Value{}, d.newError(tok.Pos(), "invalid value %v for enum type: %v", tok.RawString(), fd.Enum().Name())
default:
	panic(fmt.Sprintf("unmarshalScalar: invalid scalar kind %v", kind))
}
return protoreflect.Value{}, d.newError(tok.Pos(), "invalid value for %v type: %v", kind, tok.RawString())

so the error message would look like this:

"message": "proto: (line X:Y): invalid value \"invalid_value\" for enum type: enum_name"`

What version of protobuf and what language are you using? go version go1.19 windows/amd64 libprotoc 3.19.1 google.golang.org/genproto v0.0.0-20220930163606-c98284e70a91 google.golang.org/protobuf v1.28.1

What did you do? I have added oneof field into proto definition

What did you expect to see? Oneof method to have x receiver.

What did you see instead? Oneof method has m receiver.

I have a client/server app that uses gRPC to communicate. Profiler shows that up to 12.6% of memory is allocated in consumeBytesNoZero().

The code is unmarshaling a message that has a bytes (i.e. []byte) field a few levels deep. Then that data is processed and then the message object/struct could be reused to read the next message into if it was possible, but I don't see how. I mean, I can reuse it but looking at the code, a new byte slice will be allocated in any case. The bytes field is always up to N KB in size (client ensures that) so perfect for reuse on the server side. Is there any way to reuse the allocated memory that I don't see?

I see two ways:

• Add support for something like Unmarshal() to the message so that I could add custom unmarshaling logic or something like that. Ideally not a fixed method, actually, but a way to dynamically inject custom logic for a particular type.
• Add an option to UnmarshalOptions specially for that case.

Perhaps combining the two is the best way forward - ability to inject dynamic unmarshaling code for types via UnmarshalOptions?