Musgo is a Go code generator for binary MUS format with validation support.

Related tags

Serialization musgo
Overview

Musgo

Musgo is a Go code generator for binary MUS format with validation support. Generated code converts data to and from MUS format. More info about it and about the format you can find at "https://github.com/ymz-ncnk/musgen".

How to use

First, you should download and install Go, version 1.4 or later.

Create in your home directory foo folder with the following structure:

foo/
 |‒‒‒make/
 |    |‒‒‒musable.go
 |‒‒‒validators/
 |    |‒‒‒validators.go
 |‒‒‒foo.go

foo.go

//go:generate go run make/musable.go
package foo

type Foo struct {
  boo int `mus:"validators.BiggerThanTen"` // private fields are supported
  // too, while unmarshalling will be checked with BiggerThanTen validator
  zoo []int `mus:",,validators.BiggerThanTen"` // every element will be checked
  // with BiggerThanTen validator
  Bar MyString // alias types are supported too
  Car bool     `mus:"-"` // this field will be skiped
}

type MyString string

validators/validators.go

10 { return ErrBiggerThanTen } return nil } ">
package validators

import "errors"

var ErrBiggerThanTen error = errors.New("bigger then ten")

func BiggerThanTen(n int) error {
  if n > 10 {
    return ErrBiggerThanTen
  }
  return nil
}

make/musable.go

// +build ignore

package main

import (
  "foo"
  "reflect"

  "github.com/ymz-ncnk/musgo"
)

func main() {
  musGo, err := musgo.New()
  if err != nil {
    panic(err)
  }
  // You should "Generate" for all involved custom types.
  unsafe := false // to generate safe code
  var myStr foo.MyString
  // Alias types don't support tags, so to set up validators we use
  // GenerateAlias method.
  maxLength := 5 // restricts length of MyString values to 5 characters
  err = musGo.GenerateAlias(reflect.TypeOf(myStr), unsafe, "", maxLength, "",
    "")
  if err != nil {
    panic(err)
  }
  // reflect.Type could be created without any variable.
  err = musGo.Generate(reflect.TypeOf((*foo.Foo)(nil)).Elem(), unsafe)
  if err != nil {
    panic(err)
  }
}

Run from the comamnd line:

$ cd ~/foo
$ go mod init foo
$ go get github.com/ymz-ncnk/musgo
$ go generate

Now you can see Foo.musgen.go and MyString.musgen.go files in the foo folder. Pay attention to the location of the generated files. The data type and the code generated for it must be in the same package. Let's write some tests. Create foo_test.go file.

foo/
 |‒‒‒...
 |‒‒‒foo_test.go

foo_test.go

package foo

import (
  "foo/validators"
  "reflect"
  "testing"

  "github.com/ymz-ncnk/musgo/errs"
)

func TestFooSerialization(t *testing.T) {
  foo := Foo{
    zoo: []int{4, 2},
    boo: 5,
    Bar: MyString("hello"),
    Car: true,
  }
  buf := make([]byte, foo.SizeMUS())
  foo.MarshalMUS(buf)

  afoo := Foo{}
  _, err := afoo.UnmarshalMUS(buf)
  if err != nil {
    t.Error(err)
  }
  foo.Car = false
  if !reflect.DeepEqual(foo, afoo) {
    t.Error("something went wrong")
  }
}

func TestFooValidation(t *testing.T) {
  // test simple validator
  {
    foo := Foo{
      boo: 11,
      zoo: []int{1, 2},
      Bar: "hello",
    }
    buf := make([]byte, foo.SizeMUS())
    foo.MarshalMUS(buf)

    afoo := Foo{}
    _, err := afoo.UnmarshalMUS(buf)
    if err == nil {
      t.Error("validation doesn't work")
    }
    fieldErr, ok := err.(errs.FieldError)
    if !ok {
      t.Error("wrong field error")
    }
    if fieldErr.FieldName() != "boo" {
      t.Error("wrong field error fieldName")
    }
    if fieldErr.Cause() != validators.ErrBiggerThanTen {
      t.Error("wrong error")
    }
  }
  // test element validator
  {
    foo := Foo{
      boo: 3,
      zoo: []int{1, 12, 2},
      Bar: "hello",
    }
    buf := make([]byte, foo.SizeMUS())
    foo.MarshalMUS(buf)

    afoo := Foo{}
    _, err := afoo.UnmarshalMUS(buf)
    if err == nil {
      t.Error("validation doesn't work")
    }
    fieldErr, ok := err.(errs.FieldError)
    if !ok {
      t.Error("wrong field error")
    }
    if fieldErr.FieldName() != "zoo" {
      t.Error("wrong field error fieldName")
    }
    sliceErr, ok := fieldErr.Cause().(errs.SliceError)
    if !ok {
      t.Error("wrong slice error")
    }
    if sliceErr.Index() != 1 {
      t.Error("wrong slice error index")
    }
    if sliceErr.Cause() != validators.ErrBiggerThanTen {
      t.Error("wrong error")
    }
  }
  // test max length
  {
    foo := Foo{
      boo: 8,
      zoo: []int{1, 2},
      Bar: "hello world",
    }
    buf := make([]byte, foo.SizeMUS())
    foo.MarshalMUS(buf)

    afoo := Foo{}
    _, err := afoo.UnmarshalMUS(buf)
    if err == nil {
      t.Error("validation doesn't work")
    }
    fieldErr, ok := err.(errs.FieldError)
    if !ok {
      t.Error("wrong field error")
    }
    if fieldErr.FieldName() != "Bar" {
      t.Error("wrong field error fieldName")
    }
    if fieldErr.Cause() != errs.ErrMaxLengthExceeded {
      t.Error("wrong error")
    }
  }
}

More advanced usage you can find at https://github.com/ymz-ncnk/musgotest.

When encoding multiple values, it is impractical to create a new buffer each time. It takes too long. Instead, you can use the same buffer for each Marshal:

...
buf := make([]byte, FixedLength)
for foo := range foos {
  if foo.Size() > len(buf) {
    return errors.New("buf is too small")
  }
  i := foo.MarshalMUS(buf)
  err = handle(buf[:i])
  ...
}

To gain more performance, recover() function can be used:

...
defer func() {
  if r := recover(); r != nil {
    return errors.New("buf is too small")
  }
}()
buf := make([]byte, FixedLength)
for foo := range foos {
  i := foo.MarshalMUS(buf)
  err = handle(buf[:i])
  ...
}

It will intercept every panic, so use it with careful.

Supported Types

Supports following types:

  • uint64
  • uint32
  • uint18
  • uint8
  • uint
  • int64
  • int32
  • int18
  • int8
  • int
  • bool
  • byte
  • string
  • array
  • slice
  • map
  • struct
  • alias

Pointers are supported as well. But aliases to pointer types are not, Go doesn't allow methods for such types.

Private fields

You could encode and decode private fields too.

Unsafe code

You could generate fast unsafe code. Read more about it at "https://github.com/ymz-ncnk/musgen".

Validation

For every structure field you can set up validators using the mus:"Validator,MaxLength,ElemValidator,KeyValidator" tag , where:

  • Validator - it's a name of the function which will validate current field.
  • MaxLength - if field has a string, array, slice or map type, MaxLength will restrict its length. Should be positive number.
  • ElemValidator - it's a name of the function which will validate field's elements, if field type is an array, slice of map.
  • KeyValidator - it's a name of the function which will validate field's keys, if field type is a map.

All tag items, except MaxLength, should have the "package.FunctionName" or "FunctionName" format.

Decoding(and encoding) is performed in order, from the first field to the last one. That's why, it will stop with validation error on the first not valid field.

For alias type, you can set up validators with help of MusGo.GenerateAlias() method.

Validators

Validator is a function with the following signature func (value Type) error, where Type is a type of the value to which the validator is applied.

A few examples,

// Validator for the field.
type Foo struct {
  Field string `mus:"StrValidator"`
}

func StrValidator(str string) errorr {...}

// ElemValidator for the slice field.
type Bar struct {
  Field []string `mus:",,StrValidator"`
}

// KeyValidator for the map field.
type Zoo struct {
  Field map[int]string `mus:",,,StrValidator"`
}

// Validator for the field of a custom pointer type.
type Far struct {
  Field *Foo `mus:FooValidator`
}

func FooValidator(foo *Foo) error {...}

// Validator for the alias field.
type Ror []string

type Pac struct {
  Field Ror `mus:RorValidator`  // you can't set MaxLength or 
  // ElemValidator here, they should be applied for the Ror type.
}

func RorValidator(ror Ror) error {...}

Errors

Often validation errors are wrapped by one of the predefined error (from errs package):

  • FieldError - happens when a field validation failed. Contains field name and cause.
  • SliceError - happens when a validation of a slice element failed. Contains element index and cause.
  • ArrayError - happens when a validation of an array element failed. Contains element index and cause.
  • MapKeyError - happens when a validation of a map key failed. Contains key and cause.
  • MapValueError - happens when validation of a map value failed. Contains key, value and cause.

Benchmarks

github.com/alecthomas/go_serialization_benchmarks

Releases(v0.1.9)
Golang binary decoder for mapping data into the structure

binstruct Golang binary decoder to structure Install go get -u github.com/ghostiam/binstruct Examples ZIP decoder PNG decoder Use For struct From file

Vladislav Fursov 56 Jul 18, 2022
Simple, specialised, and efficient binary marshaling

?? surge Documentation A library for fast binary (un)marshaling. Designed to be used in Byzantine networks, ?? surge never explicitly panics, protects

Ren 38 Mar 3, 2022
Encode and decode binary message and file formats in Go

Encode and Decode Binary Formats in Go This module wraps the package encoding/binary of the Go standard library and provides the missing Marshal() and

Joel Ling 7 Jun 3, 2022
The Snappy compression format in the Go programming language.

The Snappy compression format in the Go programming language. To download and install from source: $ go get github.com/golang/snappy Unless otherwis

Go 1.3k Aug 11, 2022
Go support for Google's protocol buffers

Go support for Protocol Buffers This module (github.com/golang/protobuf) contains Go bindings for protocol buffers. It has been superseded by the goog

Go 8.6k Aug 11, 2022
gogoprotobuf is a fork of golang/protobuf with extra code generation features.

GoGo Protobuf looking for new ownership Protocol Buffers for Go with Gadgets gogoprotobuf is a fork of golang/protobuf with extra code generation feat

null 0 Nov 26, 2021
An idiomatic Go (golang) validation package. Supports configurable and extensible validation rules (validators) using normal language constructs instead of error-prone struct tags.

ozzo-validation Description ozzo-validation is a Go package that provides configurable and extensible data validation capabilities. It has the followi

Ozzo Framework 2.8k Aug 2, 2022
Golang parameter validation, which can replace go-playground/validator, includes ncluding Cross Field, Map, Slice and Array diving, provides readable,flexible, configurable validation.

Checker 中文版本 Checker is a parameter validation package, can be use in struct/non-struct validation, including cross field validation in struct, elemen

Liang Yaopei 72 Jul 28, 2022
Gookit 759 Aug 9, 2022
searchHIBP is a golang tool that implements binary search over a hash ordered binary file.

searchHIBP is a golang tool that implements binary search over a hash ordered binary file.

fblz 0 Nov 9, 2021
binary serialization format

Colfer Colfer is a binary serialization format optimized for speed and size. The project's compiler colf(1) generates source code from schema definiti

Pascal S. de Kloe 672 Aug 2, 2022
binary serialization format

Colfer Colfer is a binary serialization format optimized for speed and size. The project's compiler colf(1) generates source code from schema definiti

Pascal S. de Kloe 672 Aug 2, 2022
archy is an static binary to determine current kernel and machine architecture, with backwards compatible flags to uname, and offers alternative output format of Go runtime (i.e. GOOS, GOARCH).

archy archy is an simple binary to determine current kernel and machine architecture, which wraps uname and alternatively can read from Go runtime std

xargs-dev 3 Mar 18, 2022
A YANG-centric Go toolkit - Go/Protobuf Code Generation; Validation; Marshaling/Unmarshaling

Introduction ygot (YANG Go Tools) is a collection of Go utilities that can be used to: Generate a set of Go structures and enumerated values for a set

OpenConfig 210 Jul 22, 2022
Read metrics from a Message Queue in Json format and expose them in a Prometheus compatible format

mq2prom Read metrics from a Message Queue in Json format and expose them in a Prometheus compatible format. Currently only works for MQTT compatible M

International Business Machines 1 Jan 24, 2022
Using NFP (Number Format Parser) you can get an Abstract Syntax Tree (AST) from Excel number format expression

NFP (Number Format Parser) Using NFP (Number Format Parser) you can get an Abstract Syntax Tree (AST) from Excel number format expression. Installatio

fossabot 0 Feb 4, 2022
A slice backed binary heap with support for generic type parameters.

go-binaryheap A slice backed binary heap where the order can be customized by a comparison function. The main branch now requires go 1.18 because the

null 3 Jun 13, 2022
VS Code code snippet generator

VS Code code snippets generator.

Randy 61 May 9, 2022
Clean-Swift source and test code auto-generator. It can save you time typing 500-600 lines of code.

Clean-Swift source & test code auto generator Overview Run Output Basic Usage make config.yaml target_project_name: Miro // target project name copyri

David Ha 20 Apr 13, 2022
Code generator that generates boilerplate code for a go http server

http-bootstrapper This is a code generator that uses go templates to generate a bootstrap code for a go http server. Usage Generate go http server cod

Jijo Thomas John 1 Nov 20, 2021
Language Server Indexing Format (LSIF) generator for Go

Go LSIF indexer Visit https://lsif.dev/ to learn about LSIF. Installation Binary downloads are available on the releases tab. Installation: Linux curl

Sourcegraph 94 Jul 19, 2022
Snowflake - Simple twitter's snowflake uniquely identifiable descriptors (IDs) format generator for Go

Snowflake Dead simple and fast Twitter's snowflake id generator in Go. Installation go get github.com/HotPotatoC/snowflake Usage Generating a snowflak

Juan Christian 3 Jan 8, 2022
Go support for Protocol Buffers - Google's data interchange format

Go support for Protocol Buffers - Google's data interchange format Google's data interchange format. Copyright 2010 The Go Authors. https://github.com

Tinker Board - Android 0 Dec 15, 2021
Simple code just to try out and Binary Tree on Golang.

Character counter | ▮▮▮▮▮▮▮▮ Simple code just to try out and Binary Tree on Golang. Count characters to train openning a file and reading it, as well

Arthur Abeilice 0 May 17, 2022
Govalid is a data validation library that can validate most data types supported by golang

Govalid is a data validation library that can validate most data types supported by golang. Custom validators can be used where the supplied ones are not enough.

null 61 Apr 22, 2022
Go package for dealing with EU VAT. Does VAT number validation & rates retrieval.

Package vat Package for validating VAT numbers & retrieving VAT rates in Go. Installation Use go get. go get github.com/dannyvankooten/vat Then impor

Danny van Kooten 94 Jul 9, 2022
:key: Idiotproof golang password validation library inspired by Python's passlib

passlib for go Python's passlib is quite an amazing library. I'm not sure there's a password library in existence with more thought put into it, or wi

Hugo Landau 268 Jul 24, 2022
Struct validation using tags

Govalid Use Govalid to validate structs. Documentation For full documentation see pkg.go.dev. Example package main import ( "fmt" "log" "strings"

Travis Harmon 28 Jul 14, 2022
Validate Golang request data with simple rules. Highly inspired by Laravel's request validation.

Validate golang request data with simple rules. Highly inspired by Laravel's request validation. Installation Install the package using $ go get githu

Saddam H 1.1k Aug 6, 2022