Go structure annotations that supports encoding and decoding; similar to C-style bitfields. Supports bitfield packing, self-describing layout parameters, and alignment.

Overview

STRUCTure EXtensions

structex provides annotation rules that extend Go structures for implementation of encoding and decoding of byte backed data frames.

structex aims to make it easier to represent industry standard data formats that can be directly embedded in code.

Example

Given the first three bytes T10 SCSI Primary Commands definition of a SCSI INQUIRY data

A structex annotated struct type declaration might be

package main

import (
    "fmt"

    "github.com/HewlettPackard/structex"
)

type SCSI_Standard_Inquiry struct {
    PeripheralDeviceType uint8 `bitfield:"5"`          // Byte 0
    PeripheralQualifier  uint8 `bitfield:"3"`
    reserved0            uint8 `bitfield:"6,reserved"` // Byte 1
    LU_Cong              uint8 `bitfield:"1"`
    RMB                  uint8 `bitfield:"1"`
    Version              uint8                         // Byte 2
}

func main() {
    var inquiry = new(SCSI_Standard_Inquiry)

    // Perform IOCTL on device, returns byte reader to byte response

    if err := structex.Decode(byteReader, inquiry); err != nil {
        panic("a problem")
    }

    fmt.Printf("SCSI Peripheral Device Type: %d\n", inquiry.PeripheralDeviceType)
    fmt.Printf("SCSI Peripheral Qualifier: %d\n", inquiry.PeripheralQualifier)
    fmt.Printf("Inquiry Version: %#02x\n", inquiry.Version)
    // etc
}

Annotation Format

Bitfield

Bitfields define a structure field with an explicit size in bits. They are analogous to bit fields in the C specification.

bitfield:"[size][,reserved]"

`size`:     Specifies the size, in bits, of the field.

`reserved`: Optional modifier that specifies the field contains reserved
            bits and should be encoded as zeros.

Self-Described Layout

Many industry standards support dynamically sized return fields where the data layout is self described by other fields. To support such formats two annotations are provided.

sizeof:"[name][,relative]"

name        Specifies that the field describes the size of `name` within the
            structure. Should be used with slices.

            During decoding, if field is non-zero, the field's value is
            used to limit the number elements in the array or slice of
            name `name`.

relative    Optional modifier that specifies the value describing the size of
            `name` is relative to the field offset within the structure.
            This is often used in T10.org documentation

countof:"[name]"

name        Specifies that the value describes the count of elements in
            the `name` field.

            During decoding, if field is non-zero, the field's value is
            used to limit the number elements in the array or slice of
            name `name`.

Truncation

Structex expects sufficient data for decoding the desired structure. When data structures are used to define a maximum size of the response buffer, you can use the truncate tag on an arry or slice to permit structex to truncate the returned data with what is provided by the source buffer. If truncate is not specified and the provided buffer is of smaller size than the data structure, an io.EOF error is expected.

truncate:""

Alignment (Not Yet Implemented)

Annotations can specified the byte-alignment requirement for structure fields. Analogous to the alignas specifier in C. Can only be applied to non-bitfield structure fields.

align:"[value]"

value       An integer value specifying the byte alignment of the field.
            Invalid non-zero alignments panic.

Full Tags

The tags documented above are abbreviated for ease of use; if desired, the full tag format is supported. This provides clarity that tags are part of the structex package, but it means typing more and using every sort of quote and backtic at your disposal.

Examples

`structex:"bitfield='3'"`
`structex:"bitfield='3,reserved'"`
`structex:"countOf='D'"`
`structex:"sizeOf='E'"`
`structex:"sizeOf='F,relative'"`
`structex:"align='8'"`
`structex:"truncate"`

Performance

structex places code and structure definition readability ahead of any pack/unpack performance. In brief, performance is abysmal.

For those looking for more performant code, consider code generation or a cache based solutions

Comments
  • Dealing with endianness?

    Dealing with endianness?

    When I'm using binary/encoding this library processes endianness issues. As in this exemple.

    Is it planned to implement such mechanism for [u]int16, [u]int32, [u]int64?

    documentation enhancement 
    opened by yifu 3
  • panic: Field of type uint8 cannot be set. Make sure it is exported.

    panic: Field of type uint8 cannot be set. Make sure it is exported.

    
    import (
    	"bytes"
    	"fmt"
    
    	"github.com/HewlettPackard/structex"
    )
    
    type SCSI_Standard_Inquiry struct {
    	PeripheralDeviceType uint8 `bitfield:"5"` // Byte 0
    	PeripheralQualifier  uint8 `bitfield:"3"`
    	reserved0            uint8 `bitfield:"6,reserved"` // Byte 1
    	LU_Cong              uint8 `bitfield:"1"`
    	RMB                  uint8 `bitfield:"1"`
    	Version              uint8 // Byte 2
    }
    
    func main() {
    	var inquiry = new(SCSI_Standard_Inquiry)
    
    	// Perform IOCTL on device, returns byte reader to byte response
    	var tmp = []byte{0xff, 0xff, 0xff}
    	if err := structex.DecodeByteBuffer(bytes.NewBuffer([]byte(tmp)), inquiry); err != nil {
    		panic(err)
    	}
    
    	fmt.Printf("SCSI Peripheral Device Type: %d\n", inquiry.PeripheralDeviceType)
    	fmt.Printf("SCSI Peripheral Qualifier: %d\n", inquiry.PeripheralQualifier)
    	fmt.Printf("Inquiry Version: %#02x\n", inquiry.Version)
    	// etc
    }
    
    opened by SKlayer 1
  • Add support for signed values?

    Add support for signed values?

    
    func TestBitfieldSimple(t *testing.T) {
    
    	value := int32(1)
    	type Chunk struct {
    		Property1 int32 `bitfield:"10"`
    		Property2 int32 `bitfield:"19"`
    		Property3 int32 `bitfield:"3"`
    	}
    
    	tests := []struct {
    		value int32
    	}{
    		{1},
    		{-1},
    	}
    
    	for i, test := range tests {
    		chunk := &Chunk{Property2: value}
    		data, err := structex.EncodeByteBuffer(chunk)
    		if err != nil {
    			t.Fatalf("%d encode: %s", i, err)
    		}
    
    		fmt.Printf("%d in value: %d 0x%x encoded 0x%x\n", i, test.value, test.value, data)
    
    		err = structex.DecodeByteBuffer(bytes.NewBuffer(data), chunk)
    		if err != nil {
    			t.Fatalf("initial decode: %s", err)
    		}
    
    		fmt.Printf("%d out value: %d 0x%x\n", i, chunk.Property2, chunk.Property2)
    		if test.value != chunk.Property2 {
    			t.Fatalf("%d failed: got %d, want %d", i, chunk.Property2, test.value)
    		}
    	}
    }
    

    results in:

    0 in value: 1 0x1 encoded 0x00040000
    0 out value: 1 0x1
    1 in value: -1 0x-1 encoded 0x00040000
    1 out value: 1 0x1
    --- FAIL: TestBitfieldSimple (0.00s)
      simple_test.go:48: 1 failed: got 1, want -1
    
    opened by xackery 0
  • Support for global endian format with X_STRUCTEX_DEFAULT_ENDIANNESS environmental variable

    Support for global endian format with X_STRUCTEX_DEFAULT_ENDIANNESS environmental variable

    Passes: $ X_STRUCTEX_DEFAULT_ENDIANNESS=little go test ./...

    Fails (expected, default little-endian fields are all byte reversed) $ X_STRUCTEX_DEFAULT_ENDIANNESS=big go test ./...

    Signed-off-by: Nate Roiger [email protected]

    opened by NateThornton 0
  • big endian support for array

    big endian support for array

    It looks like, that the big="" is not recognized on multi-byte arrays, they coded always little endian. I tried to fix it, see pull request, see the tests how I want to use it. A one bit flag with countof="1" tag to include/exclude a 16 bit big endian value inside a [1]uint16{} array. Could you add a configuration flag to use always big endian encoding for network packet handling? Thanks.

    opened by kwarnke70 0
Releases(v1.0.4)
Owner
Hewlett Packard Enterprise
Hewlett Packard Enterprise on github; please contact [email protected] for access
Hewlett Packard Enterprise
Download, build, cache and run a Go app easily.

demand -- An easy way to install apps demand will download, build, cache and run a Go app. You can use it as an interpreter. Create a file bytes2human

Tv 12 Sep 10, 2017
A program to build, run, and restart a Go program on code change

devrun A program to build, run, and restart a Go program on code change. It also supports watching all your Go imports too. So if you change the code

Kevin Darlington 19 Apr 4, 2022
Yet another Go REPL that works nicely. Featured with line editing, code completion, and more.

gore Yet another Go REPL that works nicely. Featured with line editing, code completion, and more. (Screencast taken with cho45/KeyCast) Usage gore Af

Hironao OTSUBO 4.7k Nov 20, 2022
cmd tool for automatic storage and comparison of benchmarks results

prettybenchcmp prettybenchcmp is cmd tool for storage and comparison of benchmarks results. There is a standard tool benchcmp, but I don't think that

Petr 18 Apr 6, 2021
Will autobuild and kill/relaunch the target when you update the code.

Use like rerun github.com/skelterjohn/go.uik/uiktest Usage: rerun [--test] [--build] [--race] [--no-run] <import path> [arg]* For any go executable in

John Asmuth 258 Jul 23, 2022
golang feature toggle library - a library to help make golang feature toggling clean and easy

toggle supports env_variable backed toggling. It can also be updated via a pubsub interface (tested w/ redis) 2 engines for toggle backing are include

John Calabrese 24 Mar 29, 2022
Aegis is a library that allows you detect if your software is being debugged or not on Linux, FreeBSD, NetBSD, OpenBSD and Windows

Aegis is a library that allows you detect if your software is being debugged or not on Linux, FreeBSD, NetBSD, OpenBSD and Windows. You can use it natively from C or use the Go bind.

Rafael Santiago 28 Aug 29, 2022
Docker For Go Application Packaging And Pushing To Docker Hub

DOCKER-FOR-GO-APPLICATION-PACKAGING-AND-PUSHING-TO-DOCKER-HUB ##DOCKER COMMANDS: (Don't forget to navigate to the directory that contains the app and

ANDRÉS R. BUCHELI (Andrés F. Regalado Bucheli) 1 Jan 22, 2022
Template project to get started with a simple API skeleton in Go and Docker

A template project to create a Docker image for a Go application. The example application exposes an HTTP endpoint through a REST API packaged as a static binary.

Harsh Mishra 11 Apr 4, 2022
A simple application to show how to use dd-trace-go's tracer and profiler.

dd-trace-go-demo A simple application to show how to use dd-trace-go's tracer and profiler. Usage To run this demo application, simply clone this repo

Felix Geisendörfer 4 Oct 8, 2022
Memory-Alignment: a tool to help analyze layout of fields in struct in memory

Memory Alignment Memory-Alignment is a tool to help analyze layout of fields in struct in memory. Usage go get github.com/vearne/mem-aligin Example p

vearne 27 Oct 26, 2022
go-eexcel implements encoding and decoding of XLSX like encoding/json

go-eexcel go-eexcel implements encoding and decoding of XLSX like encoding/json Usage func ExampleMarshal() { type st struct { Name string `eexce

sago35 0 Dec 9, 2021
Bitfield - A collection of code samples, listings, and solutions to challenges from the book For the Love of Go

For the Love of Go - code listings This is a collection of code samples, listing

Zack Proser 1 Jan 10, 2022
🔑A high performance Key/Value store written in Go with a predictable read/write performance and high throughput. Uses a Bitcask on-disk layout (LSM+WAL) similar to Riak.

bitcask A high performance Key/Value store written in Go with a predictable read/write performance and high throughput. Uses a Bitcask on-disk layout

James Mills 10 Sep 26, 2022
gup aka Get All Urls parameters to create wordlists for brute forcing parameters.

Description GUP is a tool to create wrodlists from the urls. Purpose The purpose of this tool is to create wordlists for brute forcing parameters. Ins

Chan Nyein Wai 14 Feb 25, 2022
A simple, semantic and developer-friendly golang package for encoding&decoding and encryption&decryption

A simple, semantic and developer-friendly golang package for encoding&decoding and encryption&decryption

null 319 Nov 21, 2022
Some Golang types based on builtin. Implements interfaces Value / Scan and MarshalJSON / UnmarshalJSON for simple working with database NULL-values and Base64 encoding / decoding.

gotypes Some simple types based on builtin Golang types that implement interfaces for working with DB (Scan / Value) and JSON (Marshal / Unmarshal). N

null 0 Feb 12, 2022
Go-ipfs-cmds - Cmds offers tools for describing and calling commands both locally and remotely

Go-ipfs-cmds - Cmds offers tools for describing and calling commands both locally and remotely

y 0 Jan 18, 2022
? ID3 decoding and encoding library for Go

id3v2 Supported ID3 versions: 2.3, 2.4 Installation go get -u github.com/bogem/id3v2 Usage example package main import ( "fmt" "log" "github.com

Albert Nigmatzianov 271 Nov 18, 2022
Encoding and decoding GeoJSON <-> Go

go.geojson Go.geojson is a package for encoding and decoding GeoJSON into Go structs. Supports both the json.Marshaler and json.Unmarshaler interfaces

Paul Mach 222 Nov 24, 2022