Encode and decode Go (golang) struct types via protocol buffers.

Overview

protostructure Godoc

protostructure is a Go library for encoding and decoding a struct type over the wire.

This library is useful when you want to send arbitrary structures over protocol buffers for behavior such as configuration decoding (encoding/json, etc.), validation (using packages that use tags), etc. This works because we can reconstruct the struct type dynamically using reflect including any field tags.

This library only sends the structure of the struct, not the value. If you want to send the value, you should build your protocol buffer message in such a way that it encodes that somehow using something such as JSON.

Installation

Standard go get:

$ go get github.com/mitchellh/protostructure

Usage & Example

For usage and examples see the Godoc.

A quick code example is shown below using both the imaginary proto file and the Go code that uses it.

syntax = "proto3";
package myapp;
import "protostructure.proto";

// Response is an example response structure for an RPC endpoint.
message Response {
	protostructure.Struct config = 1;
}
type Config struct {
	Name string            `json:"name"`
	Meta map[string]string `json:"metadata"`
	Port []*Port           `json:"ports"`
}

type Port struct {
	Number uint `json:"number"`
	Desc   string `json:"desc"`
}

// You can encode the structure on one side:
message, err := protostructure.Encode(Config{})

// And you can use the structure on the other side. Imagine resp
// is populated using some protobuf RPC such as gRPC.
val, err := protostructure.New(resp.Config)
json.Unmarshal([]byte(`{
	"name": "example",
	"meta": { "env": "prod" },
	"ports": [
		{ "number": 8080 },
		{ "number": 8100, desc: "backup" },
	]
}`, val)

// val now holds the same structure dynamically. You can pair with other
// libraries such as https://github.com/go-playground/validator to also
// send validation using this library.

Limitations

There are several limitations on the structures that can be encoded. Some of these limitations are fixable but the effort hasn't been put in while others are fundamental due to the limitations of Go currently:

  • Circular references are not allowed between any struct types.
  • Embedded structs are not supported
  • Methods are not preserved, and therefore interface implementation is not known. This is also an important detail because custom callbacks such as UnmarshalJSON may not work properly.
  • Field types cannot be: interfaces, channels, functions
  • Certain stdlib types such as time.Time currently do not encode well.

But... why?

The real world use case that led to the creation of this library was to facilitate decoding and validating configuration for plugins via go-plugin, a plugin system for Go that communicates using gRPC.

The plugins for this particular program have dynamic configuration structures that were decoded using an encoding/json-like interface (struct tags) and validated using go-playground/validator which also uses struct tags. Using protostructure, we can send the configuration structure across the wire, decode and validate the configuration in the host process, and report more rich errors that way.

Another reason we wanted to ship the config structure vs. ship the config is because the actual language we are using for configuration is HCL which supports things like function calls, logic, and more and shipping that runtime across is much, much more difficult.

This was extracted into a separate library because the ability to encode a Go structure (particulary to include tags) seemed more generally useful, although rare.

Issues
  • Support *time.Time type?

    Support *time.Time type?

    HI @mitchellh , I found this library quite useful for some edge case I have.

    One question I have is regarding support for the built-in type *time.TIme or time.Time, currently it's encoded as struct (or pointer to struct) but I thought it might be useful to take-care of that as well because it's a built-in type. Does it make sense? If yes, I'll be happy to open a Pull Request.

    opened by yevgenypats 1
Owner
Mitchell Hashimoto
Mitchell Hashimoto
Protocol Buffers for Go with Gadgets

GoGo Protobuf looking for new ownership Protocol Buffers for Go with Gadgets gogoprotobuf is a fork of golang/protobuf with extra code generation feat

gogoprotobuf 5.4k Aug 12, 2022
Go support for Google's protocol buffers

Go support for Protocol Buffers This module (github.com/golang/protobuf) contains Go bindings for protocol buffers. It has been superseded by the goog

Go 8.6k Aug 11, 2022
Sqlyog-password-decoder - Simple decode passwords from .sycs file (SQLyog export connections file)

Decode password: ./sqlyog-password-decoder -str password -action decode Encode p

Maxim Yanin 0 Nov 21, 2021
Some Golang types based on builtin. Implements interfaces Value / Scan and MarshalJSON / UnmarshalJSON for simple working with database NULL-values and Base64 encoding / decoding.

gotypes Some simple types based on builtin Golang types that implement interfaces for working with DB (Scan / Value) and JSON (Marshal / Unmarshal). N

null 0 Feb 12, 2022
Converts go types no matter what

elastic Converts go types no matter what elastic is a simple library that converts any type to another the best way possible. This is useful when the

epic labs 16 Jan 24, 2022
Helper functions for the manipulation of slices of all types in Go

go-slices Unlike many other programming languages, Go doesn't provide helper functions for slices in it's core. I felt like this was quite an essentia

Ondřej Merkun 12 Sep 17, 2021
golang struct 或其他对象向 []byte 的序列化或反序列化

bytecodec 字节流编解码 这个库实现 struct 或其他对象向 []byte 的序列化或反序列化 可以帮助你在编写 tcp 服务,或者需要操作字节流时,简化数据的组包、解包 这个库的组织逻辑 copy 借鉴了标准库 encoding/json ?? 安装 使用 go get 安装最新版本

null 7 Jun 30, 2022
csvutil provides fast and idiomatic mapping between CSV and Go (golang) values.

csvutil Package csvutil provides fast and idiomatic mapping between CSV and Go (golang) values. This package does not provide a CSV parser itself, it

Jacek Szwec 738 Aug 14, 2022
Cap'n Proto library and parser for go. This is go-capnproto-1.0, and does not have rpc. See https://github.com/zombiezen/go-capnproto2 for 2.0 which has rpc and capabilities.

Version 1.0 vs 2.0 Update 2015 Sept 20: Big news! Version 2.0 of the go-bindings, authored by Ross Light, is now released and newly available! It feat

Jason E. Aten, Ph.D. 282 Aug 6, 2022
Asn.1 BER and DER encoding library for golang.

WARNING This repo has been archived! NO further developement will be made in the foreseen future. asn1 -- import "github.com/PromonLogicalis/asn1" Pac

Logicalis 51 Aug 3, 2022
auto-generate capnproto schema from your golang source files. Depends on go-capnproto-1.0 at https://github.com/glycerine/go-capnproto

bambam: auto-generate capnproto schema from your golang source files. Adding capnproto serialization to an existing Go project used to mean writing a

Jason E. Aten, Ph.D. 64 Jan 29, 2022
Golang binary decoder for mapping data into the structure

binstruct Golang binary decoder to structure Install go get -u github.com/ghostiam/binstruct Examples ZIP decoder PNG decoder Use For struct From file

Vladislav Fursov 56 Jul 18, 2022
CBOR RFC 7049 (Go/Golang) - safe & fast with standard API + toarray & keyasint, CBOR tags, float64/32/16, fuzz tested.

CBOR library in Go fxamacker/cbor is a CBOR encoder & decoder in Go. It has a standard API, CBOR tags, options for duplicate map keys, float64→32→16,

Faye Amacker 469 Aug 3, 2022
Fixed width file parser (encoder/decoder) in GO (golang)

Fixed width file parser (encoder/decoder) for GO (golang) This library is using to parse fixed-width table data like: Name Address

Oleg Lobanov 20 Aug 5, 2022
Fast implementation of base58 encoding on golang.

Fast Implementation of Base58 encoding Fast implementation of base58 encoding in Go. Base algorithm is adapted from https://github.com/trezor/trezor-c

Denis Subbotin 124 Aug 1, 2022
msgpack.org[Go] MessagePack encoding for Golang

MessagePack encoding for Golang ❤️ Uptrace.dev - All-in-one tool to optimize performance and monitor errors & logs Join Discord to ask questions. Docu

Vladimir Mihailenco 1.8k Aug 12, 2022
Dynamically Generates Ysoserial's Payload by Golang

Gososerial 介绍 ysoserial是java反序列化安全方面著名的工具 无需java环境,无需下载ysoserial.jar文件 输入命令直接获得payload,方便编写安全工具 目前已支持CC1-CC7,K1-K4和CB1链 Introduce Ysoserial is a well-

4ra1n 43 Jul 10, 2022
A k-mer serialization package for Golang

.uniq v5 This package provides k-mer serialization methods for the package kmers, TaxIds of k-mers are optionally saved, while there's no frequency in

Wei Shen 6 Jun 16, 2022
gogoprotobuf is a fork of golang/protobuf with extra code generation features.

GoGo Protobuf looking for new ownership Protocol Buffers for Go with Gadgets gogoprotobuf is a fork of golang/protobuf with extra code generation feat

null 0 Nov 26, 2021