Cap'n Proto library and parser for go. This is go-capnproto-1.0, and does not have rpc. See https://github.com/zombiezen/go-capnproto2 for 2.0 which has rpc and capabilities.

Overview

Version 1.0 vs 2.0

Update 2015 Sept 20:

Big news! Version 2.0 of the go-bindings, authored by Ross Light, is now released and newly available! It features capnproto RPC and capabilities support. See https://github.com/zombiezen/go-capnproto2 for the v2 code and docs.

This repository (https://github.com/glycerine/go-capnproto) is now being called version 1.0 of the go bindings for capnproto. It does not have RPC (it was created before the RPC protocol was defined). Version 1 has schema generating tools such as https://github.com/glycerine/bambam. Personally I have many projects that use v1 with mangos for network transport; https://github.com/gdamore/mangos. Here is an example of using them together: https://github.com/glycerine/goq. Nonetheless, for new projects, especially once v2 has been hardened and tested, v2 should be preferred.

Version 1 will be maintained for applications that currently use it. However new users, new features and new code contributions should be directed to the version 2 code base to take advantage of the RPC and capabilities.

License

MIT - see LICENSE file

Documentation

In godoc see http://godoc.org/github.com/glycerine/go-capnproto

News

5 April 2014: James McKaskill, the author of go-capnproto (https://github.com/jmckaskill/go-capnproto), has been super busy of late, so I agreed to take over as maintainer. This branch (https://github.com/glycerine/go-capnproto) includes my recent work to fix bugs in the creation (originating) of structs for Go, and an implementation of the packing/unpacking capnp specification. Thanks to Albert Strasheim (https://github.com/alberts/go-capnproto) of CloudFlare for a great set of packing tests. - Jason

Getting started

New! Visit the sibling project to this one, bambam, to automagically generate a capnproto schema from the struct definitions in your go source files. Bambam makes it easy to get starting with go-capnproto.

pre-requisite: Due to the use of the customtype annotation feature, you will need a relatively recent capnproto installation. At or after 1 July 2014 (at or after b2d752beac5436bada2712f1a23185b78063e6fa) is known to work.

# first: be sure you have your GOPATH env variable setup.
$ go get -u -t github.com/glycerine/go-capnproto
$ cd $GOPATH/src/github.com/glycerine/go-capnproto
$ make # will install capnpc-go and compile the test schema aircraftlib/aircraft.capnp, which is used in the tests.
$ diff ./capnpc-go/capnpc-go `which capnpc-go` # you should verify that you are using the capnpc-go binary you just built. There should be no diff. Adjust your PATH if necessary to include the binary capnpc-go that you just built/installed from ./capnpc-go/capnpc-go.
$ go test -v  # confirm all tests are green

What is Cap'n Proto?

The best cerealization...

http://kentonv.github.io/capnproto/

Comments
  • Question on copy and paste between msgs.

    Question on copy and paste between msgs.

    Say I receive a msg and I decode it. I then save a ptr to some part of it:

    seg, _, err := capn.ReadFromMemoryZeroCopy(data)
    if err != nil {
        return nil, err
    }
    varCap := ReadRootVarCap(seg)
    
    idCap := varCap.Id()
    positions := mt.Positions(idCap.Positions())
    return &positions
    

    Then, at some later point, I want to include positions in some output msg:

    func AddToSegAutoRoot(seg *capn.Segment, pos *Positions) VarCap {
        vCap := AutoNewVarCap(seg)
        idCap := NewVarIdPosCap(seg)
        idCap.SetPositions(capn.UInt8List(*pos))
        vCap.SetId(idCap)
    

    The question is: does the entire segment from which the data was drawn (in the incoming msg), get copied/included with the output segment, or is only the exact data needed used? Whilst the docs say things about "pointer semantics", and I've read some bits of the spec for encoding, far pointers, that sort of thing, and I've also looked through the code, I can't quite come across a definitive answer.

    Basically, I'm trying to avoid the work of decoding things that I only need to relay through. But I don't want the whole of the incoming segment included in the output.

    Obviously, I can write a test to check what the code is doing right now, but I'm more curious as to what the design intention is for this.

    opened by msackman 8
  • Fix buffer overlap on multi-segmented message

    Fix buffer overlap on multi-segmented message

    When reading/writing multi-segmented message, message are sometimes broken. I found this bug while parsing multi-segmented message serialized by c++ capnp library.

    How to reproduce

    1. Generate multi-segmented message somehow (ex. capnp encode --segment-size=xx)
    2. Parse given message using go-capnproto
    3. Modify a text field which is not in a last segment
    4. Then some random field is gone, or overall structure is broken, i.e, WriteTo() does not generate valid capnproto message.

    Root Cause When parsing multiple segments in ReadFromStream(), m.Segments[*] is filled by slicing one big buffer. So they share underlying memories. Thus, When Segment.Data[] grows, it overlaps with next segment, then some random memory (list or pointer maybe) are overwritten.

    Workaround My workaround is to generate a new []byte for each segment. But I'm not sure this is the best, since this will allocate more memories.

    opened by hodduc 6
  • Fix index out of bounds bug

    Fix index out of bounds bug

    PutUint64 requires the slice to already have sufficient len - i.e. it writes to the slice, not appends. Previously, in this branch, the slice was not being extended, so PutUint64 would panic.

    opened by msackman 6
  • Generated *_Which consts only adds the type for the first value

    Generated *_Which consts only adds the type for the first value

    For example:

    struct Wrapper {
      union {
        foo @0 :Void;
        bar @1 :Void;
      }
    }
    

    results in:

    type Wrapper_Which uint16
    
    const (
      WRAPPER_FOO Wrapper_Which = 0
      WRAPPER_BAR               = 1
    )
    

    This makes WRAPPER_BAR an untyped constant, which is undesirable.

    opened by zombiezen 6
  • Unable to serialize / deserialize List(Bool) values

    Unable to serialize / deserialize List(Bool) values

    Been testing serialization w/ Heka's message schema, have nearly everything working but have hit a snag w/ multi-value boolean fields. Apologies for the complex error case, but I've created a gist w/ the relevant files: https://gist.github.com/rafrombrc/fb2d7b363def17589cb2

    In that gist you can see the capnp schema I'm using, some helper code that I've written to simplify populating one of the message objects (adding behavior to the capnp generated struct), and a simple demo tool that either creates a single message and serializes it to disk, or reads a single message in and deserializes. You'll notice that the demo code dealing w/ the boolean fields is commented out. With that code removed, everything works as expected. When I uncomment that code, however, the serialization fails entirely.

    I'm trying to use the BitList type to handle the List(Bool) values, b/c that seems to be the only list type that works w/ boolean primitives. Should I be using a different type?

    opened by rafrombrc 6
  • experimental json output broken on List(List(struct))

    experimental json output broken on List(List(struct))

    Daniel (edit: not Alberts) added some experimental JSON output. This is a great idea. Unfortunately the List(List(struct)) code generation is faulty, and needs some love.

    repro steps:

    1. turn on json support by setting JSON_enabled = true in json.go. To allow everyone to use go-capnproto without encountering this bug, the flag is currently set to false by default. Set it to true to start debugging and fixing this issue.
    
    [email protected]:~/cap:master$ cat json.go 
    package capn
    
    // JSON support for List(List(something)), e.g ZVECVEC in the aircraft.capnp, isn't finished (currently broken).
    //  We disable JSON generation to get back to a working state.
    const JSON_enabled = false
    
    [email protected]:~/cap:master$ perl -pi -e 's/false/true/' json.go 
    [email protected]:~/cap:master$ cat json.go 
    package capn
    
    // JSON support for List(List(something)), e.g ZVECVEC in the aircraft.capnp, isn't finished (currently broken).
    //  We disable JSON generation to get back to a working state.
    const JSON_enabled = true
    [email protected]:~/cap:master$
    
    
    1. rebuild and reinstall the capnpc-go compiler plugin. The 'make prepare' target in the Makefile should make this step easy.
    [email protected]:~/dev/cnet/go/src/github.com/glycerine/go-capnproto:master$ make prepare
    cd capnpc-go; go build
    go install ./capnpc-go
    cd aircraftlib; make
    make[1]: Entering directory `/home/jaten/dev/cnet/go/src/github.com/glycerine/go-capnproto/aircraftlib'
    capnp compile -ogo aircraft.capnp
    make[1]: Leaving directory `/home/jaten/dev/cnet/go/src/github.com/glycerine/go-capnproto/aircraftlib'
    which capnpc-go
    /home/jaten/dev/cnet/go/bin/capnpc-go
    diff `which capnpc-go` ./capnpc-go/capnpc-go
    # if there was a diff above, adjust your PATH to use the most-recently build capnpc-go
    [email protected]:~/dev/cnet/go/src/github.com/glycerine/go-capnproto:master$ 
    
    1. "go test -v" or "make testbuild" should now show a failing test during attempted compilation of aircraftlib/aircraft.capnp.go
    [email protected]:~/dev/cnet/go/src/github.com/glycerine/go-capnproto:master$ go test -v
    # github.com/glycerine/go-capnproto/aircraftlib
    aircraftlib/aircraft.capnp.go:1723: cannot range over s.ToArray() (type *[]capn.Object)
    FAIL    github.com/glycerine/go-capnproto [build failed]
    [email protected]:~/dev/cnet/go/src/github.com/glycerine/go-capnproto:master$ 
    
    opened by glycerine 6
  • Order of root creation matters

    Order of root creation matters

    Hi,

    So I'm getting some bugs where if I don't create a root structure before other items in the segment, the other items are overwritten / put in an invalid state.

    opened by c00w 5
  • Test failure on version_test.go

    Test failure on version_test.go

    Just installed capnproto 0.5.1 (from release tarball; compile from src), and then glycerine/go-capnproto today (c082c595166badf95e5a2e973f5dd4ca9d4c0cdb). One test fails:

    Failures:
    
      * /home/matthew/src/Go_external/src/github.com/glycerine/go-capnproto/version_test.go 
      Line 30:
      Expected: '[]byte{0x0, 0x0, 0x0, 0x0, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}'
      Actual:   '[]byte{0x0, 0x0, 0x0, 0x0, 0x2, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}'
      (Should resemble)!
    
    opened by msackman 4
  • Fix void field slot

    Fix void field slot

    This shouldn't cause broken generated code, but it did.

    --- a/aircraftlib/aircraft.capnp +++ b/aircraftlib/aircraft.capnp @@ -68,7 +68,7 @@ struct Aircraft {

    certain places.

    union {

    • void @0: Void; # @0 will be the default, so always make @0 a Void.
    • foo @0: Void; # @0 will be the default, so always make @0 a Void. b737 @1: B737; a320 @2: A320; f16 @3: F16;
    opened by slachowsky 4
  • Change convenience names for binary.LittleEndian

    Change convenience names for binary.LittleEndian

    The little32, putLittle64, etc. names prevented the compiler from inlining calls to these functions.

    The names are now less abbreviated, but they're abbreviated in a way that doesn't prevent inlining.

    opened by edsrzf 4
  • Documentation improvement

    Documentation improvement

    Hi,

    It would be very useful if the docs for all the API functions which take optional buffers made it clear whether or not the buffer may still be used once the function returns.

    For example, from my reading of ReadFromStream, the buf that you can provide to it will contain data that is reachable from the segments returned. So if you have a tight loop reading segments off a socket and pushing those segments down a channel (of capacity > 0) then it's not safe to reuse the buf - the next call to ReadFromStream will start and reset the buf before the segment has been processed at the other end of the channel.

    Is my reading of this code correct, and could you possibly add suitable notes to the docs? Many thanks.

    opened by msackman 3
  • Update install doc

    Update install doc

    Hello.

    With the new versions of go (1.12.x), capnp can not be installed. If mod-vendor is enabled, go get does not work. Also, when trying to go-build go-install manually, it seems that the go compiler can not find main module: go: cannot find main module; see 'go help modules'

    opened by iulianpascalau 2
Owner
Jason E. Aten, Ph.D.
Jason E. Aten, Ph.D.
A standard way to wrap a proto message

Welcome to Pletter ?? A standard way to wrap a proto message Pletter was born with a single mission: To standardize wrapping protocol buffer messages.

Lykon 18 May 17, 2022
Fixed width file parser (encoder/decoder) in GO (golang)

Fixed width file parser (encoder/decoder) for GO (golang) This library is using to parse fixed-width table data like: Name Address

Oleg Lobanov 21 Sep 20, 2022
idiomatic codec and rpc lib for msgpack, cbor, json, etc. msgpack.org[Go]

go-codec This repository contains the go-codec library, the codecgen tool and benchmarks for comparing against other libraries. This is a High Perform

Ugorji Nwoke 1.7k Sep 12, 2022
Generate TypeScript interfaces from Go structs/interfaces - useful for JSON RPC

bel Generate TypeScript interfaces from Go structs/interfaces - useful for JSON RPC bel is used in production in https://gitpod.io. Getting started be

Christian Weichel 27 Sep 3, 2022
Asn.1 BER and DER encoding library for golang.

WARNING This repo has been archived! NO further developement will be made in the foreseen future. asn1 -- import "github.com/PromonLogicalis/asn1" Pac

Logicalis 51 Aug 3, 2022
Go library for decoding generic map values into native Go structures and vice versa.

mapstructure mapstructure is a Go library for decoding generic map values to structures and vice versa, while providing helpful error handling. This l

Mitchell Hashimoto 6.1k Sep 19, 2022
A go library that facilitates the implementation of decorator pattern.

go-decorator go-decorator is a library that facilitates the implementation of decorator pattern. Installation To install go-decorator, use go get: go

Garen Chan 1 Nov 25, 2021
GED - Global-purpose Encoding / Decoding library

GED - Global-purpose Encoding / Decoding library This library lets you use common encoding/decoding schemes and allows you to define custom ones. Use

NoƩmien Kocher 0 Nov 28, 2021
A library that provides dynamic features of Go language.

go-dynamic go-dynamic is a library that provides dynamic features of Go language. Installation To install go-dynamic, use go get: go get -u github.com

Garen Chan 1 Dec 8, 2021
csvutil provides fast and idiomatic mapping between CSV and Go (golang) values.

csvutil Package csvutil provides fast and idiomatic mapping between CSV and Go (golang) values. This package does not provide a CSV parser itself, it

Jacek Szwec 748 Sep 17, 2022
Encode and decode binary message and file formats in Go

Encode and Decode Binary Formats in Go This module wraps the package encoding/binary of the Go standard library and provides the missing Marshal() and

Joel Ling 7 Aug 31, 2022
Some Golang types based on builtin. Implements interfaces Value / Scan and MarshalJSON / UnmarshalJSON for simple working with database NULL-values and Base64 encoding / decoding.

gotypes Some simple types based on builtin Golang types that implement interfaces for working with DB (Scan / Value) and JSON (Marshal / Unmarshal). N

null 0 Feb 12, 2022
Easily and dynamically generate maps from Go static structures

structomap This package helps you to transform your struct into map easily. It provides a structomap.Serializer interface implemented by the structoma

Daniel Perez 137 Sep 8, 2022
Go package for dealing with maps, slices, JSON and other data.

Objx Objx - Go package for dealing with maps, slices, JSON and other data. Get started: Install Objx with one line of code, or update it with another

Stretchr, Inc. 545 Sep 23, 2022
Encode and decode Go (golang) struct types via protocol buffers.

protostructure protostructure is a Go library for encoding and decoding a struct type over the wire. This library is useful when you want to send arbi

Mitchell Hashimoto 172 Sep 9, 2022
Simple, specialised, and efficient binary marshaling

?? surge Documentation A library for fast binary (un)marshaling. Designed to be used in Byzantine networks, ?? surge never explicitly panics, protects

Ren 38 Mar 3, 2022
An optimal, byte-aligned, LZ+RLE hybrid encoder, designed to maximize decoding speed on NMOS 6502 and derived CPUs

TSCrunch TSCrunch is an optimal, byte-aligned, LZ+RLE hybrid encoder, designed to maximize decoding speed on NMOS 6502 and derived CPUs, while keeping

null 26 Jun 10, 2022
auto-generate capnproto schema from your golang source files. Depends on go-capnproto-1.0 at https://github.com/glycerine/go-capnproto

bambam: auto-generate capnproto schema from your golang source files. Adding capnproto serialization to an existing Go project used to mean writing a

Jason E. Aten, Ph.D. 64 Jan 29, 2022
fuzzer for a single http parameter which checks if the response does/does not contain a certain given string

single http parameter fuzzer DISCLAIMER: ONLY USE THIS PROGRAM ON TARGETS YOU HAVE PERMISSION TO FUZZ! Initially used as a "poor man's" http fuzzer fo

null 0 Dec 19, 2021
One of the fastest alternative JSON parser for Go that does not require schema

Alternative JSON parser for Go (10x times faster standard library) It does not require you to know the structure of the payload (eg. create structs),

Leonid Bugaev 4.7k Sep 24, 2022