bebop is a bebop parser written in Go, for generating Go code.

Related tags

Utilities go serialization bebop
Overview

bebop

GoDoc Go Reference

bebop is a bebop parser written in Go, for generating Go code.

bebop can read .bop files and output .go files representing them:

package main

import (
    "github.com/200sc/bebop"
)

func main() {
    f, _ := os.Open("mybebop.bop")
    defer f.Close()
    bopf, _ := bebop.ReadFile(f)
    out, _ := os.Create("mybebop.go")
    defer out.Close()
    settings := bebop.GenerateSettings{
        PackageName: "mybebop",
    }
    bopf.Generate(out, settings)
}

These files can then be used to encode and decode their represented records:

package server

import (
    "github.com/my/project/mybebop"
    "net"
)

func sendFoo(cn net.Conn) (success bool) {
    fooReq := mybebop.FooRequest{
        Bar: "buzz",
    }
    if err := fooReq.EncodeBebop(cn); err != nil {
        // ...
    }
    fooResp := mybebop.FooResponse{}
    if err := fooResp.DecodeBebop(cn); err != nil {
        // ...
    }
    return fooResp.Success
}

bebobc-go

At main/bebobc-go there is a cli utility to take .bop files and produce .go files from them, what is effectively a little more sophisticated version of the first example in this document.

bebopfmt

At main/bebopfmt there is a cli utility to format and rewrite bop files. It takes in a -w argument to rewrite files in place, and accepts both directories and individual filepaths.

Known limitations

The following is a list of known issues with the current version of the project, ordered by approximate priority for addressing them.

Original bebop does not support one .bop file importing type definitions from another .bop file, and so neither does this, yet.

  • This is nontrivial, and requires a lot of design toward the importing / packaging ecosystem.

We should be able to generate server code with handlers given all valid messages have op codes.

Tokenization and parsing could attempt corrective fixes to be able to report more than one error.

Credit

Original bebop compiler (C#, Typescript, ...): https://github.com/RainwayApp/bebop

Valid testdata largely from above project.

Comments
  • Support optionally writing comments as struct field tags in bebobc-go

    Support optionally writing comments as struct field tags in bebobc-go

    Can you please add an option to bebobc-go to write out comments as struct field tags?

    I did this with my local copy by modifying writeFieldDefinition in gen.go; commenting out line 539 and adding fd.Comment to line 552 with the appropriate format string change.

    I can submit a pull request, if needed. Thanks!

    opened by acuozzo 10
  • Import support

    Import support

    I note in the README the statement about imports being not supported. It looks like upstream they were added a few months ago. I'm just wodering whether you're already planning on supporting this, or whether it's stalled for now? You mention the need for careful design and thought on this feature - I quite understand. The main thing to me seems the need to figure out the module import path from relative imports (assuming that you essentially go down the route of one bop file == one go file). I'm keen to discuss your thoughts around this - I may very well be keen to contribute shortly.

    opened by msackman 9
  • Feature/multiple array suffix

    Feature/multiple array suffix

    This changeset:

    • Adds missing support for field types with multiple [] suffixes e.g. byte[][][] bytes
    • Updates internal compatibility tests with rainway to 2.4.9.

    Closes #31

    opened by 200sc 5
  • Add flag to disable datatype exports?

    Add flag to disable datatype exports?

    When compiling into an existing package it might be desirable to not export any functions or datatypes. This would be usefull in cases where the functions only get used inside the package or if I want to write my own wrapper functions to convert between types or similar.

    opened by mjmar01 4
  • Messages containing dates produce incorrect GO syntax

    Messages containing dates produce incorrect GO syntax 

    message MyObj {
    1 -> date Start;
    2 -> date End;
}

    Results in code such as:

    iohelp.WriteInt64Bytes(buf[at:], (*bbp.Start.UnixNano()/100)) iohelp.WriteInt64Bytes(buf[at:], (*bbp.End.UnixNano()/100))

    golangci-lint run my_obj.go

    level=error msg="[linters context] typechecking error: my_obj.go:87:27: invalid operation: cannot indirect bbp.Start.UnixNano() (value of type int64)"
level=error msg="[linters context] typechecking error: my_obj.go:95:27: invalid operation: cannot indirect bbp.End.UnixNano() (value of type int64)"
    opened by ghost 4
  • Nested arrays result in

    Nested arrays result in "expected (Ident) got Open Square"

    This might be a little tough to solve/the generated code could change a lot if an infinite amount of nesting is allowed.

    The TypeScript version of bebop-tools compiles byte[][][] in my .bop to Array<Array<Uint8Array>>.

    And thank you for maintaining this lib!! 😄

    opened by merlindru 2
  • Hotfix/ascii opcode endianness

    Hotfix/ascii opcode endianness

    This PR fixes two compatibility issues with rainway's compiler; it:

    • Corrects the endianness of ASCII string encoded opcodes
    • Enforces that opcodes are all unique

    In addition we have more testdata and internally represent opcodes as uint32s instead of int32s.

    Closes #29

    opened by 200sc 2
  • Support bitflags in enums

    Support bitflags in enums

    This addresses issue #21.

    I'm not inclined to release this until rainway releases their implementation, with this sort of sub-language parsing there's a high chance what is implemented in 200sc/bebop is not an exact match of the behavior in rainway, with some particular questions noted in the code as TODOs currently.

    opened by 200sc 2
  • Mismatching opcodes

    Mismatching opcodes

    The opcodes generated using rainway's JS bebop-tools are in little-endian, the ones generated using this lib are in big-endian (which might make more sense?)

    Either way, this means that they mismatch by default. What's a good solution for this problem? Do we even need* a solution?!

    *currently, I'm shifting them around manually so they're either all little-endian or all big-endian

    opened by merlindru 1
  • Overhaul tokenization

    Overhaul tokenization

    Must follow bitflags PR (#22).

    This refactoring enables the tokenization component of this library to return multiple errors by making trivial guesses to complete half-written tokens.

    opened by 200sc 1
  • Add bitflags support

    Add bitflags support

    Rainway is introducing the ability to define enums with bitflag operations following a simple expression language: https://github.com/RainwayApp/bebop/issues/158

    opened by 200sc 1
  • Switch generated methods to pointer receivers

    Switch generated methods to pointer receivers

    Changed for Encoding, Marshalling, Getters and Size methods.

    Rationale is that on non-pointer receivers will cause the objects to be copied as part of method invocation. That causes additional allocs and pressure on the GC. For applications which do a lot of serialisation, this may well be noticable - a lot of performance tuning in Go focuses on optimising memory usage.

    All tests still pass.

    opened by msackman 1
  • Use pointer receiver when Marshalling and Encoding

    Use pointer receiver when Marshalling and Encoding

    Currently, the generator creates:

    func (bbp Foo) MarshalBebopTo(buf []byte) int {...}
func (bbp Foo) EncodeBebop(iow io.Writer) (err error) {...}

    By having a non-pointer receiver, it creates a copy of the struct, using memory, which I think is unnecessary. Is there any reason why this was chosen? Looking at some generated functions, it doesn't look like it modifies anything in those structs... If these were changed to:

    func (bbp *Foo) MarshalBebopTo(buf []byte) int {...}
func (bbp *Foo) EncodeBebop(iow io.Writer) (err error) {...}

    then it would avoid unnecessary allocs.

    opened by msackman 3
  • Include binaries in releases?

    Include binaries in releases?

    It would take a little bit of automation work to build them but would be theoretically trivial to build binaries for any platform supported by Go, if there's a desire to prebuild them. With Go dependencies one would usually already be coding in go and go install the tool directly, however.

    question 
    opened by 200sc 1
  • Write stress tests

    Write stress tests

    Write a test harness which will repeatedly:

    1. Generate a random, arbitrarily large bebop.File structure
    2. Write this structure out to a bop file (with formatting tooling) 2a. Randomly adjust this file in ways that maintain it as a valid bebop file (newlines and other whitespace, comments, etc)
    3. Read this structure back in
    4. Confirm that read structure matches the original structure
    5. Generate a go file from this new structure
    6. Confirm this go file compiles
    7. Confirm this go file contains a type definition for every type in the bebop definition

    A harness like this would have detected #18 sooner.

    opened by 200sc 0
  • Support returning multiple errors from compilation

    Support returning multiple errors from compilation

    Code generation is currently split into three components, and all of them will need adjustments to report multiple errors:

    Tokenization:

    • EOF errors can remain as they are
    • non-EOF errors will need to attempt to guess what the token should have been?

    Parsing:

    • EOF errors can remain as they are
    • 'unexpected token' errors will need to ignore bad tokens or populate a best guess token instead (potentially looking ahead to see if the next token is the one we want)
    • There may be a book or guide for how to restructure this section to support these corrective fixes

    Generation:

    • Should be the easiest, just have the existing Validate method build up errors instead of returning the first one it sees.
    opened by 200sc 1
Releases(v0.3.3)

  • v0.3.3(Oct 29, 2022)

    This release:

    • Adds missing support for field types with multiple [] suffixes e.g. byte[][][] bytes
    • Updates internal compatibility tests with rainway to 2.4.9.
    Source code(tar.gz)
    Source code(zip)

  • v0.3.2(Oct 24, 2022)

    This release:

    • Corrects the endianness of ASCII string encoded opcodes
    • Enforces that opcodes are all unique

    In addition we have more testdata and internally represent opcodes as uint32s instead of int32s.

    Full Changelog: https://github.com/200sc/bebop/compare/v0.3.1...v0.3.2

    Source code(tar.gz)
    Source code(zip)

  • v0.3.1(Apr 25, 2022)

    Changelog:

    • Adds a parameter to generate unexported message and constant definitions. This is not customizable by type, is either completely on or completely off. (https://github.com/200sc/bebop/pull/28)
    Source code(tar.gz)
    Source code(zip)

  • v0.3.0(Feb 11, 2022)

    Changelog:

    • Adds bitflag constant definition support (https://github.com/RainwayApp/bebop/wiki/Writing-Bops:-The-Bebop-Schema-Language#flags-enum)
    • An internal rewrite of tokenization
    Source code(tar.gz)
    Source code(zip)

  • v0.2.4(Dec 4, 2021)

  • v0.2.3(Nov 17, 2021)

  • v0.2.2(Sep 28, 2021)

  • v0.2.1(Sep 12, 2021)

    Changelog:

    Several changes for compatibility with upstream, specifically comparing against TypeScript generated code:

    • Deprecated message fields are no longer written. They can still be parsed. (https://github.com/RainwayApp/bebop/wiki/Writing-Bops:-The-Bebop-Schema-Language#the-deprecated-attribute)
    • Union sizes previously included the discriminator byte. They no longer do. (https://github.com/RainwayApp/bebop/wiki/Wire-format#unions)
    • Generated code no longer writes opcodes, they only offer them as values for users to write if they would like to write them. (from misleading graphic: https://github.com/RainwayApp/bebop/wiki/Writing-Bops:-The-Bebop-Schema-Language#opcodes)

    There is a remaining failure in compatibility with typescript that is a known problem with the upstream: https://github.com/RainwayApp/bebop/pull/148

    Other Changes:

    • When a const exceeds the limits of what can be stored in a variable in Go, instead of erroring out we now offer a warning.
    Source code(tar.gz)
    Source code(zip)

  • v0.2.0(Sep 6, 2021)

    This adds import support in two operating modes-

    1. By default, generate each .bop file to a single .go file, with a const go_package hint for where those files should be found
    2. Optionally, instead generate a single file containing all definitions in the entire import tree. This is what https://github.com/RainwayApp/bebop does, and may be necessary if working with a file you can't change to add a package hint.

    We offer (1) instead because (2) is infeasible for a complex system. If you had a large app built on bebop structures you would either need a single place that stored every definition used in the app, which would eventually fail as file sizes increased and would have a bad developer experience, or you'd need to have partial definitions throughout the codebase, which would get confusing and an additional import added somewhere could cause a cascading file size bloat.

    In addition,

    • This fixes a bug where unions could not be fields of Structs or Messages
    • Duplicate const names will now cause an error in the compiler.
    • iohelp.ErrTooShort has been superceded by io.ErrUnexpectedEOF
    • Unions and Messages will no longer read their full bodies up front when reading from a io.Reader.
    • iohelp.ErrorReader.Read will now wait until the complete payload it expects is available to read.
    • Code and template duplication in generation has been significantly reduced, fixing some inconsistencies.
    Source code(tar.gz)
    Source code(zip)

  • v0.1.4(Aug 21, 2021)

    Changelog:

    • Bumped library to Go 1.17
    • Adds const declaration support
    • Modifies test suites to document compatibility with https://github.com/RainwayApp/bebop -- These tests cover compilation compatibility, not wire format compatibility, yet.
    • Empty enum definitions will no longer generate an empty const block.
    • Negative field numbers for messages and unions will be caught at compile time as errors.
    • Fixed a bug with line and column numbers output by the compiler in error strings.
    • Enums can now use hex integers.
    Source code(tar.gz)
    Source code(zip)

  • v0.1.3(Aug 7, 2021)

    This release

    • Drops nested union support (see https://github.com/RainwayApp/bebop/pull/137/).
    • Fixes some comment generation and formatting bugs for unions and messages.
    • Ensures that union fields are generated in a consistent order.
    Source code(tar.gz)
    Source code(zip)

  • v0.1.2(Apr 27, 2021)

    This release

    • Adds Size() int to bebop.Record
    • Changes MarshalBebopTo([]byte) to return int of bytes written
    • Adds SharedMemoryStrings generation option to unmarshal functions, removing string allocations but sharing memory with the unmarshalling byte slice.
    Source code(tar.gz)
    Source code(zip)

  • v0.1.1(Apr 26, 2021)

    This release

    • Fixes the generated size of unions
    • Fixes UnmarshalBebop functions on messages and unions
    • Enforces in some places that unions have one single populated field
    • Adds additional test cases for the above
    Source code(tar.gz)
    Source code(zip)

  • v0.1.0(Apr 5, 2021)

  • v0.0.9(Feb 20, 2021)

  • v0.0.8(Dec 22, 2020)

    This release

    • Fixes a bug where streamed messages had incorrect size bytes written.
    • Fixes a bug where streamed messages did not respect opcodes.
    • Updates formatting to include spacing ahead of opcode declarations.
    • Removes support for readonly messages, per the bebop spec.
    • Adds constructors for readonly structs.
    • Fixes parsing bugs that allowed readonly to precede enums and comments.
    Source code(tar.gz)
    Source code(zip)

  • v0.0.7(Dec 22, 2020)

    This release:

    • Adds support for optional semicolons, controlled via a currently private variable in the codebase. This is mostly a theoretical demonstration that it would be easy to remove the need for semicolons from the original
    • Adds additional validation checks at Go code generation time that detect files with overlapping field names or indices.
    • Adds line and column numbers to tokenization and parsing errors.
    • Adds fuzz tests (which revealed no bugs (so far)).
    Source code(tar.gz)
    Source code(zip)

  • v0.0.6(Dec 21, 2020)

    This release:

    • Introduces bebopfmt, a binary formatter tool that can rewrite valid bebop files with standardized formatting.
    • Corrects a bug in message generation where written lengths included opcode and size bytes.
    Source code(tar.gz)
    Source code(zip)

  • v0.0.5(Dec 19, 2020)

    • bebop.Marshal and bebop.Unmarshal have been replaced with bebop.Record.MarshalBebop and bebop.Record.UnmarshalBebop.
    • These new methods, in limited benchmarks, outperform protobuf by ~2.5x.
    • bebop.Record.UnmarshalBebop includes regular size checks and can return an error. In benchmarks, this does not change performance over removing these checks. If, however, you would prefer to panic on an out of bounds error, you may pass GenerateUnsafeMethods to the generator and use the resulting MustUnmarshalBebop method.
    Source code(tar.gz)
    Source code(zip)

  • v0.0.4(Dec 16, 2020)

  • v0.0.3(Dec 14, 2020)

  • v0.0.2(Dec 13, 2020)

    • Adds a License file
    • Adds support for escaped quotes in deprecated strings
    • Adds support for transferring multi line header comments on records to .go files
    • Adds error checking to encoding/decoding read/write operations
    • Adds invalid recursive type detection
    • Corrects GUID wire format byte order
    Source code(tar.gz)
    Source code(zip)

  • v0.0.1(Dec 13, 2020)

    This is an initial release, with significant but incomplete test coverage for success paths in parsing and generating.

    Currently suitable for use in non-critical environments.

    Source code(tar.gz)
    Source code(zip)
Owner
Patrick Stephen
Go. Gamedev. Evolutionary Computation.
Patrick Stephen
GitHub
randstr is a module that contains functions for generating random strings.

randstr is a module that contains functions for generating random strings. The functions in this module uses the crypto/rand package. Installa

Henrik Christensen 3 Nov 13, 2021
Template repo for generating your own workflow executor for Azure Orkestra

Orkestra Workflow Executor Template Repo This is a template repository for a new workflow executor Usage Generate your repository from this template b

Microsoft Azure 1 Oct 22, 2021
Generating sitemap using goLang

This Go Module will generate the sitemap for any given URL. We will travese the all pages of the website and creates the sitemap We will use the 'net/

Venkatesh Macha 1 Dec 9, 2021
Clean-Swift source and test code auto-generator. It can save you time typing 500-600 lines of code.

Clean-Swift source & test code auto generator Overview Run Output Basic Usage make config.yaml target_project_name: Miro // target project name copyri

David Ha 20 Apr 13, 2022
Code generator that generates boilerplate code for a go http server

http-bootstrapper This is a code generator that uses go templates to generate a bootstrap code for a go http server. Usage Generate go http server cod

Jijo Thomas John 1 Nov 20, 2021
A port of the parser from graphql-js into golang

gqlparser This is a parser for graphql, written to mirror the graphql-js reference implementation as closely while remaining idiomatic and easy to use

Adam Scarr 402 Dec 27, 2022
A parser for Ontario's baby name data

obnp What? A parser for Ontario's baby name data Why? I wanted to see if a specific name existed in both the male and female datasets. This tool is mo

liv 1 Mar 15, 2022
A data parser lib for Go with pythonic grammar sugar and as concern as possible for high performance

mapinterface - A data parser lib for Go with pythonic grammar sugar and as concern as possible for high performance mapinterface 旨在消灭对map/list解析而产生的层层

Knownothing 1 Nov 10, 2021
Golang flags parser with zero dependency

flags Golang flags parser with zero dependency. Usage See simple.go for basic usage. Concept flags gives a simple way to get flag's value from argumen

Vincent Boutour 0 Jan 16, 2022
Golemon - A Go port of the lemon parser generator

Go lemon port A port of the Lemon Parser to Go. State This work was done entirel

null 11 Nov 3, 2022
efaceconv - Code generation tool for high performance conversion from interface{} to immutable type without allocations.

efaceconv High performance conversion from interface{} to immutable types without additional allocations This is tool for go generate and common lib (

Ivan 50 May 14, 2022
Type-driven code generation for Go

What’s this? gen is a code-generation tool for Go. It’s intended to offer generics-like functionality on your types. Out of the box, it offers offers

Matt Sherman 1.4k Jan 4, 2023
Versatile Go code generator.

Generis Versatile Go code generator. Description Generis is a lightweight code preprocessor adding the following features to the Go language : Generic

SenseLogic 37 Nov 30, 2022
Code Generation for Functional Programming, Concurrency and Generics in Golang

goderive goderive derives mundane golang functions that you do not want to maintain and keeps them up to date. It does this by parsing your go code fo

Walter Schulze 1.1k Dec 25, 2022
Golang source code parsing, usage like reflect package

gotype Golang source code parsing, usage like reflect package English 简体中文 Usage API Documentation Examples License Pouch is licensed under the MIT Li

null 52 Dec 9, 2022
Code generation tools for Go.

interfaces Code generation tools for Go's interfaces. Tools available in this repository: cmd/interfacer cmd/structer cmd/interfacer Generates an inte

Rafal Jeczalik 362 Dec 23, 2022
Jennifer is a code generator for Go

Jennifer Jennifer is a code generator for Go. package main import ( "fmt" . "github.com/dave/jennifer/jen" ) func main() { f := NewFile("m

Dave Brophy 2.7k Jan 4, 2023
Reload Go code in a running process at function/method level granularity

got reload? Function/method-level stateful hot reloading for Go! Status Very much work in progress.

null 34 Nov 9, 2022
Go library that provides fuzzy string matching optimized for filenames and code symbols in the style of Sublime Text, VSCode, IntelliJ IDEA et al.

Go library that provides fuzzy string matching optimized for filenames and code symbols in the style of Sublime Text, VSCode, IntelliJ IDEA et al. This library is external dependency-free. It only depends on the Go standard library.

Sahil Muthoo 1.2k Dec 27, 2022