auto-generate capnproto schema from your golang source files. Depends on go-capnproto-1.0 at

Related tags

Serialization bambam

bambam: auto-generate capnproto schema from your golang source files.

Adding capnproto serialization to an existing Go project used to mean writing a lot of boilerplate.

Not anymore.

Given a set of golang (Go) source files, bambam will generate a capnproto schema. Even better: bambam will also generate translation functions to readily convert between your golang structs and the new capnproto structs.


You'll need a recent (up-to-date) version of go-capnproto. If you installed go-capnproto before, you'll want to update it [>= f9f239fc7f5ad9611cf4e88b10080a4b47c3951d / 16 Nov 2014].

Capnproto and go-capnproto should both be installed and on your PATH.

to install: run make. This lets us record the git commit in LASTGITCOMMITHASH to provide accurate version info. Otherwise you'll get an 'undefined: LASTGITCOMMITHASH' failure.

# be sure go-capnproto and capnpc are installed first.

$ go get -t  # the -t pulls in the test dependencies.

# ignore the initial compile error about 'undefined: LASTGITCOMMITHASH'. `make` will fix that.
$ cd $GOPATH/src/
$ make  # runs tests, build if all successful
$ go install


use: bambam -o outdir -p package myGoSourceFile.go myGoSourceFile2.go ...
     # Bambam makes it easy to use Capnproto serialization[1] from Go.
     # Bambam reads .go files and writes a .capnp schema and Go bindings.
     # options:
     #   -o="odir" specifies the directory to write to (created if need be).
     #   -p="main" specifies the package header to write (e.g. main, mypkg).
     #   -X exports private fields of Go structs. Default only maps public fields.
     #   -version   shows build version with git commit hash
     #   -OVERWRITE modify .go files in-place, adding capid tags (write to -o dir by default).
     # required: at least one .go source file for struct definitions. Must be last, after options.
     # [1] 


See rw.go.txt. To see all the files compiled together in one project: (a) comment out the defer in the rw_test.go file; (b) run go test; (c) then cd testdir_* and look at the sample project files there. (d). run go build in the testdir_ to rebuild the binary. Notice that you will need all three .go files to successfully build. The two .capnp files should be kept so you can read your data from any capnp-supported language. Here's what is what in that example directory:

rw.go             # your original go source file (in this test)
translateCapn.go  # generated by bambam after reading rw.go
schema.capnp      # generated by bambam after reading rw.go
schema.capnp.go   # generated by `capnpc -ogo schema.capnp` <- you have to do this yourself or in your Makefile.
go.capnp          # always necessary boilerplate to let capnpc work, just copy it from bambam/go.capnp to your build dir.


jaten@c03:~/go/src/$ cd testdir_884497362/
jaten@c03:~/go/src/$ go build
jaten@c03:~/go/src/$ ls
go.capnp  rw.go  rw.go.txt  schema.capnp  schema.capnp.go  testdir_884497362  translateCapn.go
jaten@c03:~/go/src/$ ./testdir_884497362
Load() data matched Saved() data.
jaten@c03:~/go/src/$ # run was successful

Here is what it looks like to use the Save()/Load() methods. You end up with a Save() and Load() function for each of your structs. Simple.

package main

import (

// By default bambam will add the `capid` tags
// to a copy of your source in the output directory.
// Use bambam -OVERWRITE to modify files directly in-place.
// The capid tags control the @0, @1, field numbering 
// in the generated capnproto schema. If you change
// your go structs, the capid tags let your schema
// stay backwards compatible with prior serializations.
type MyStruct struct {
	Hello    []string  `capid:"0"`
	World    []int     `capid:"1"`

func main() {

	rw := MyStruct{
		Hello:    []string{"one", "two", "three"},
		World:    []int{1, 2, 3},

    // any io.ReadWriter will work here (os.File, etc)
	var o bytes.Buffer

    // now we have saved!

    rw2 := &MyStruct{}
    // now we have restored!


what Go types does bambam recognize?

Supported: structs, slices, and primitive/scalar types are supported. Structs that contain structs are supported. You have both slices of scalars (e.g. []int) and slices of structs (e.g. []MyStruct) available.

We handle [][]T, but not [][][]T, where T is a struct or primitive type. The need for triply nested slices is expected to be rare. Interpose a struct after two slices if you need to go deeper.

Currently unsupported (pull requests welcome): Go maps.

Also: pointers to structs to be serialized work, but pointers in the inner-most struct do not. This is not a big limitation, as it is rarely meaningful to pass a pointer value to a different process.

capid tags on go structs

When you run bambam, it will generate a modified copy of your go source files in the output directory.

These new versions include capid tags on all public fields of structs. You should inspect the copy of the source file in the output directory, and then replace your original source with the tagged version. You can also manually add capid tags to fields, if you need to manually specify a field number (e.g. you are matching an pre-existing capnproto definition).

If you are feeling especially bold, bambam -OVERWRITE my.go will replace my.go with the capid tagged version. For safety, only do this on backed-up and version controlled source files.

By default only public fields (with a Capital first letter in their name) are tagged. The -X flag ignores the public/private distinction, and tags all fields.

The capid tags allow the capnproto schema evolution to function properly as you add new fields to structs. If you don't include the capid tags, your serialization code won't be backwards compatible as you change your structs.

Deleting fields from your go structs isn't (currently) particularly well-supported. We could potentially allow fields to be // commented out in the go source and yet still parse the comments and use that parse to keep the schema correct, but that's not a trivial bit of work.

example of capid annotion use

type Job struct { 
   C int `capid:"2"`  // we added C later, thus it is numbered higher.
   A int `capid:"0"`
   B int `capid:"1"` 

other tags

Also available tags: capid:"skip" or capid:"-1" (any negative number): this field will be skipped and not serialized or written to the schema.

// capname:"Counter"
type number struct {
   A int

The above struct will be mapped into capnproto as:

struct Counter {
  a @0: Int64;

Without the // capname:"Counter" comment, you would get:

struct NumberCapn {
  a @0: Int64;

Explanation: Using a // capname:"newName" comment on the line right before a struct definition will cause bambam to use 'newName' as the name for the corresponding struct in the capnproto schema. Otherwise the corresponding struct will simply uppercase the first letter of the orignal Go struct, and append "Capn". For example: a Go struct called number would induce a parallel generated capnp struct called NumberCapn.

windows build script

see build.cmd. Thanks to Klaus Post ( for contributing this.

Copyright (c) 2015, Jason E. Aten, Ph.D.

  • embedded structs don't get serialized

    embedded structs don't get serialized


    the readme states that Structs that contain structs are supported. but it does not seem to work for me. i just started using go-capnproto/bambam so i think i am just doing something wrong...

    i have a main that just Save()s and Load()s a struct instance. the struct is define in a separate package like this:

    type Dude struct {
        Name    string   `capid:"0"`
        Age     int      `capid:"1"`
        Address *Address `capid:"2"`
    type Address struct {
        Street string `capid:"0"`
        Zip    string `capid:"1"`

    the translateCapn.go Load and Save methods dont handle the Address - (Pointer or not, also tried annonymous embedded struct).

    i invoced bambam liked this

    bambam -o capout/ -p dude dude.go

    in the subdrectory. the Load() method will actually have

    z := dude.ReadRootAddressCapn(capMsg)

    which does not work - because i am in the Package... removing the package name it works, compiles, but i have the above described problem. this part here makes me think i am just doing something else wrong...

    i can make an example repo if that helps.


    opened by fvbock 8
  • Getting demo from README to run

    Getting demo from README to run

    Hi I'm trying to get the example from the README to run. Please let me know if you can help me get further along with this.

    I followed the instructions and commented out defer before running the tests:

    glycerine/bambam/testdir_259570601(master!) $ ls
    total 56
    drwx------   7 harlow  staff   238 Dec 13 22:06 .
    drwxr-xr-x  46 harlow  staff  1564 Dec 13 22:06 ..
    -rw-r--r--   1 harlow  staff   252 Dec 13 21:09 go.capnp
    -rw-r--r--   1 harlow  staff  1510 Dec 13 22:06 rw.go
    -rw-r--r--   1 harlow  staff  1654 Dec 13 22:06 rw.go.txt
    -rw-r--r--   1 harlow  staff   726 Dec 13 22:06 schema.capnp
    -rw-r--r--   1 harlow  staff  9320 Dec 13 22:06 translateCapn.go

    But when I try to run the go program it doesn't fine the Save and Load functions

    glycerine/bambam/testdir_259570601(master!) $ go run rw.go
    # command-line-arguments
    ./rw.go:59: rw.Save undefined (type RWTest has no field or method Save)
    ./rw.go:62: rw2.Load undefined (type *RWTest has no field or method Load)
    glycerine/bambam/testdir_259570601(master!) $

    Any help would be appreciated. I'll get the README updated with some more detailed instructions once I get this working locally.

    opened by harlow 7
  • go get giving errors about unknown gists

    go get giving errors about unknown gists

    I am getting errors trying to go get this package. These come from glycerine/go-goon but I can't open issues against that...

    go get
    package cannot find package "" in any of:
        /usr/local/go/src/ (from $GOROOT)
        /Users/justin/go/src/ (from $GOPATH)
    package cannot find package "" in any of:
        /usr/local/go/src/ (from $GOROOT)
        /Users/justin/go/src/ (from $GOPATH)
    opened by justincormack 4
  • Support for uint8, uint16, and uint32 fields in structs

    Support for uint8, uint16, and uint32 fields in structs

    Hello, great project! I've just started experimenting and have found that the uint types mentioned in the subject don't seem to be supported fully. If I have a struct like so that I generate capnp from:

    type Envelope struct {
        Ver   uint8  `capid:"0"`
        SvcId string `capid:"1"`

    then the test below fails on the last 2 "if" statements. It appears as if the bytes are serializing / deserializing correctly but the "Ver" field is being set to its go default value rather than the value from the deserialized bytes:

    func TestSerializationBytes(t *testing.T) {
        source := Envelope{
            Ver:   1,
            SvcId: "Svc1",
        var o bytes.Buffer
        oBytes := o.Bytes()
        sink := &Envelope{}
        var o2 bytes.Buffer
        o2Bytes := o2.Bytes()
        if bytes.Compare(oBytes, o2Bytes) != 0 {
            t.Errorf("Source and Sink Envelope bytes do not match: %v -- %v", source, sink)
        if source.SvcId != sink.SvcId {
            t.Errorf("source.SvcId: %v != sink.SvcId", source.SvcId, sink.SvcId)
        if source.Ver != sink.Ver {
            t.Errorf("source.Ver: %v != sink.Ver: %v", source.Ver, sink.Ver)
        if !reflect.DeepEqual(&source, sink) {
            t.Errorf("Structures do not match: %+v vs %+v", source, sink)

    Do you plan to support these uint types?

    opened by dmccurley 3
  • Save/Load methods are not returning error

    Save/Load methods are not returning error

    bambam is generating two methods for structures - Save(io.Writer) and Load(io.Reader). Code generated is hiding errors instead of returning them:

    • Save is writing to buffer, but it ignores error returned by Segment.WriteTo
    • Load is reading from reader, but istead of returning an error it's panicking.

    My proposal is to change both methods to return error interface. I'm happy to implement the change and provide pull request, I just want to know if my suggestion is good.

    Here's example generated code:

    func (s *Foo) Save(w io.Writer) {
        seg := capn.NewBuffer(nil)
        FooGoToCapn(seg, s)
    func (s *Foo) Load(r io.Reader) {
        capMsg, err := capn.ReadFromStream(r, nil)
        if err != nil {
            panic(fmt.Errorf("capn.ReadFromStream error: %s", err))
        z := ReadRootFooCapn(capMsg)
          FooCapnToGo(z, s)
    opened by husio 3
  • SettersToCapn, SettersToGo: Add uint64 support to template

    SettersToCapn, SettersToGo: Add uint64 support to template

    Ran into a situation where uint64s were not being written or retrieved. Generated setters were missing/not-implemented-yet.

    Sane changes?

    "bambam is so awesome! It has saved me hours!" – :turtle:

    opened by captainpete 3
  • Overwrite assumes '/bin/cp' is available

    Overwrite assumes '/bin/cp' is available

    Here it is assumed that /bin/cp is available.

    On Windows that is not the case, and I assume it is also a bit unsafe on other platforms.

    Here is what seems like a good file copy function in golang:

    opened by klauspost 2
  • go generate question

    go generate question

    Would it be possible to add the ability to use go 1.4's new generate tool: //go:generate bambam $GOFILE

    So that you can create files using go generate?

    opened by jgoodall 2
  • Error installing

    Error installing

    Attempting to install on Mac 10.9.5, Installed pre-requisites: brew install capnp go get

    Tried to install bambam:

    $ go version
    go version go1.4.1 darwin/amd64
    $ go get -t
    ../../../../ undefined: LASTGITCOMMITHASH```
    opened by jgoodall 2
  • bambam -OVERWRITE does not work

    bambam -OVERWRITE does not work

    Hi, calling

    bambam -OVERWRITE file.go

    does not work (anymore)

    output ==>

    panic: open /: is a directory

    goroutine 1 [running]: main.(*Extractor).CopySourceFilesAddCapidTag(0x208362000, 0x0, 0x0) /go/src/ +0x7ef main.MainArgs(0x208310000, 0x3, 0x3) /go/src/ +0x12e4 main.main() /go/src/ +0x42

    I assume that this is related to this change here:

    Btw.: I'm working on a mac

    opened by BjFrDev 1
  • Bambam is not generating all of the files

    Bambam is not generating all of the files

    I assume I am doing something incorrectly, but I cant figure out what it is....

    In the current directory I have one file in a separate package that describes a struct. I run:

    $ bambam -o=../eventcapnp -p=eventcapnp event.go
    [bambam -o=../eventcapnp -p=eventcapnp event.go]
    generated files in '../eventcapnp'
    $ ls ../eventcapnp
    event.go          schema.capnp      translateCapn.go

    From looking at the documentation, I am missing schema.capnp.go and go.capnp. When I build the project, it fails:

    $ go build
    eventcapnp/translateCapn.go:26: undefined: eventcapnp
    eventcapnp/translateCapn.go:33: undefined: ContextCapn
    eventcapnp/translateCapn.go:44: undefined: ContextCapn
    eventcapnp/translateCapn.go:45: undefined: AutoNewContextCapn
    eventcapnp/translateCapn.go:68: undefined: eventcapnp
    eventcapnp/translateCapn.go:75: undefined: EventCapn
    eventcapnp/translateCapn.go:106: undefined: EventCapn
    eventcapnp/translateCapn.go:107: undefined: AutoNewEventCapn
    eventcapnp/translateCapn.go:137: undefined: ContextCapn_List
    eventcapnp/translateCapn.go:147: undefined: ContextCapn_List
    eventcapnp/translateCapn.go:107: too many errors
    opened by jgoodall 1
  • managing capn ids

    managing capn ids

    Every time I regenerate the schema, it gets a new capn id. Is there some way to control this, other than modifying the file after generation? I am also providing non Go interfaces.

    opened by justincormack 2
  • wrong numbering with already numbered fields

    wrong numbering with already numbered fields

    bambam translates this struct…:

    type Member struct {
        id             int64
        Username       string    `capid:"0"`
        UpdatedAt      time.Time `capid:"3" json:"LastUpdated"`
        Name           *string   `capid:"1" json:"DisplayName,omitempty"`
        IsAdmin        bool      `capid:"2"`
        PasswordHash   *string   `capid:"4" json:",omitempty"`
        PasswordSalt   *string   `capid:"5" json:",omitempty"`
        PrimaryGroupId *int      `capid:"6" json:",omitempty"`
        UrlToAvatar    *string   `json:",omitempty"`

    To the following, assigning capid=3 twice:

    type Member struct {
        id             int64
        Username       string    `capid:"0"`
        UpdatedAt      time.Time `capid:"3" json:"LastUpdated"`
        Name           *string   `capid:"1" json:"DisplayName,omitempty"`
        IsAdmin        bool      `capid:"2"`
        PasswordHash   *string   `capid:"4" json:",omitempty"`
        PasswordSalt   *string   `capid:"5" json:",omitempty"`
        PrimaryGroupId *int      `capid:"6" json:",omitempty"`
        UrlToAvatar    *string   `json:",omitempty" capid:"3"`

    I have expected that it continued numbering, and assigned capid=7 to the last field.

    Annotating the field with capid=7 will result in this error and no usable file output:

    $ bambam -o capn.model -p model member.go
    panic: problem in capid tag '' on field 'UrlToAvatar' in struct 'Member': number '7' is beyond the count of fields we have, largest available is 6
    # …
        UrlToAvatar    *string   `capid:"7" json:",omitempty"`

    Although not required in this example, the order of fields in a struct is important: If every member of a struct is comparable, then the struct becomes comparable. In that case you don't want to reorder fields to suit bambam.

    opened by wmark 4
  • enum support

    enum support

    Hi Björn (@HyperDrummer),

    I'm surfacing your email question to a ticket so it is public; other people may have the same question.

    I haven't implemented enum support. If you would like to assist with this, the first step would be to write down several examples of the starting and ending text that involve transformation of enums, and then to put those into a test file. The bambam project is entirely test driven (BDD to the extent possible), and this is an especially valuable approach for compilers (like bambam). Once you have a failing test in place, we can look at how to implement it. I would suggest starting a new branch so that the master doesn't have the failing test on it.

    Best regards,


    You asked:

    Imagine these structs:

    type MyStruct struct {
        Hello []string `capid:"0"`
        World []int    `capid:"1"`
    type People struct {
        Name     string `capid:"0"`
        LastName string `capid:"1"`
    type MType int
    const (
        PeopleType MType = iota
    type MsgType struct {
        Msg     MType `capid:"0"`
        MsgSize int   `capid:"1"`

    For these go structures I want to create a schema with bambam including the code generation. Ok, now I call bambam and what I end up with ist the following schema:

    using Go = import "go.capnp";
    struct MsgTypeCapn { 
       msg      @0:   Int64; 
       msgSize  @1:   Int64; 
    struct MyStructCapn { 
       hello  @0:   List(Text); 
       world  @1:   List(Int64); 
    struct PeopleCapn { 
       name      @0:   Text; 
       lastName  @1:   Text; 

    In the generated code:

    func MsgTypeCapnToGo(src MsgTypeCapn, dest *MsgType) *MsgType {
      if dest == nil {
        dest = &MsgType{}
      dest.Msg = *MTypeCapnToGo(src.Msg(), nil)
      dest.MsgSize = int(src.MsgSize())
      return dest

    which leads to build errors:

    ./translateCapn.go:37: undefined: MTypeCapnToGo ./translateCapn.go:47: undefined: MTypeGoToCapn

    The undefined data types never got generated.

    Do you have an idea how to use enums with code generation? Thanks in advance


    opened by glycerine 3
Jason E. Aten, Ph.D.
Jason E. Aten, Ph.D.
Generate TypeScript interfaces from Go structs/interfaces - useful for JSON RPC

bel Generate TypeScript interfaces from Go structs/interfaces - useful for JSON RPC bel is used in production in Getting started be

Christian Weichel 31 Oct 23, 2022
Easily and dynamically generate maps from Go static structures

structomap This package helps you to transform your struct into map easily. It provides a structomap.Serializer interface implemented by the structoma

Daniel Perez 139 Dec 9, 2022
Asn.1 BER and DER encoding library for golang.

WARNING This repo has been archived! NO further developement will be made in the foreseen future. asn1 -- import "" Pac

Logicalis 53 Nov 14, 2022
Golang binary decoder for mapping data into the structure

binstruct Golang binary decoder to structure Install go get -u Examples ZIP decoder PNG decoder Use For struct From file

Vladislav Fursov 64 Dec 17, 2022
CBOR RFC 7049 (Go/Golang) - safe & fast with standard API + toarray & keyasint, CBOR tags, float64/32/16, fuzz tested.

CBOR library in Go fxamacker/cbor is a CBOR encoder & decoder in Go. It has a standard API, CBOR tags, options for duplicate map keys, float64→32→16,

Faye Amacker 518 Jan 6, 2023
csvutil provides fast and idiomatic mapping between CSV and Go (golang) values.

csvutil Package csvutil provides fast and idiomatic mapping between CSV and Go (golang) values. This package does not provide a CSV parser itself, it

Jacek Szwec 775 Jan 6, 2023
Fixed width file parser (encoder/decoder) in GO (golang)

Fixed width file parser (encoder/decoder) for GO (golang) This library is using to parse fixed-width table data like: Name Address

Oleg Lobanov 22 Sep 27, 2022
Fast implementation of base58 encoding on golang.

Fast Implementation of Base58 encoding Fast implementation of base58 encoding in Go. Base algorithm is adapted from

Denis Subbotin 128 Dec 9, 2022[Go] MessagePack encoding for Golang

MessagePack encoding for Golang ❤️ - All-in-one tool to optimize performance and monitor errors & logs Join Discord to ask questions. Docu

Vladimir Mihailenco 1.9k Dec 28, 2022
Encode and decode Go (golang) struct types via protocol buffers.

protostructure protostructure is a Go library for encoding and decoding a struct type over the wire. This library is useful when you want to send arbi

Mitchell Hashimoto 171 Nov 15, 2022
golang struct 或其他对象向 []byte 的序列化或反序列化

bytecodec 字节流编解码 这个库实现 struct 或其他对象向 []byte 的序列化或反序列化 可以帮助你在编写 tcp 服务,或者需要操作字节流时,简化数据的组包、解包 这个库的组织逻辑 copy 借鉴了标准库 encoding/json ?? 安装 使用 go get 安装最新版本

null 7 Jun 30, 2022
Dynamically Generates Ysoserial's Payload by Golang

Gososerial 介绍 ysoserial是java反序列化安全方面著名的工具 无需java环境,无需下载ysoserial.jar文件 输入命令直接获得payload,方便编写安全工具 目前已支持CC1-CC7,K1-K4和CB1链 Introduce Ysoserial is a well-

4ra1n 43 Jul 10, 2022
A k-mer serialization package for Golang

.uniq v5 This package provides k-mer serialization methods for the package kmers, TaxIds of k-mers are optionally saved, while there's no frequency in

Wei Shen 7 Aug 19, 2022
Some Golang types based on builtin. Implements interfaces Value / Scan and MarshalJSON / UnmarshalJSON for simple working with database NULL-values and Base64 encoding / decoding.

gotypes Some simple types based on builtin Golang types that implement interfaces for working with DB (Scan / Value) and JSON (Marshal / Unmarshal). N

null 0 Feb 12, 2022
gogoprotobuf is a fork of golang/protobuf with extra code generation features.

GoGo Protobuf looking for new ownership Protocol Buffers for Go with Gadgets gogoprotobuf is a fork of golang/protobuf with extra code generation feat

null 0 Nov 26, 2021
generic sort for slices in golang

slices generic sort for slices in golang basic API func BinarySearch[E constraints.Ordered](list []E, x E) int func IsSorted[E constraints.Ordered](li

阮坤良 17 Nov 3, 2022
Bitbank-trezor - Bitbank trezor with golang

Bitbank - Trezor (c) 2022 Bernd Fix [email protected] >Y< bitbank-trezor is fre

Bernd Fix 0 Jan 27, 2022
Cap'n Proto library and parser for go. This is go-capnproto-1.0, and does not have rpc. See for 2.0 which has rpc and capabilities.

Version 1.0 vs 2.0 Update 2015 Sept 20: Big news! Version 2.0 of the go-bindings, authored by Ross Light, is now released and newly available! It feat

Jason E. Aten, Ph.D. 286 Nov 29, 2022
Git-auto-push - Auto commit and push to github repositories

Auto commit and push to github repositories. How to use git clone https://github

Z.Q.K 4 Dec 19, 2022
A tool to compare if terraform provider migration schema snapshot is equal to schema defined in resource code

migration schema comparer for Terraform When develop Terraform provider sometimes we need do some state migration(not schema migration) via StateUpgra

null 0 Nov 18, 2021