peanut is a Go package to write tagged data structs to disk in a variety of formats.

Overview

peanut

BSD3 Build Status codecov Go Report Card Used By Godoc

peanut is a Go package to write tagged data structs to disk in a variety of formats.

Its primary purpose is to provide a single consistent interface for easy, ceremony-free persistence of record-based struct data.

Each distinct struct type is written to an individual file (or table), automatically created, each named according to the name of the struct. Field/column names in each file/table are derived from struct tags. All writers use the same tags.

Currently supported formats are CSV, TSV, Excel (.xlsx), JSON Lines (JSONL), and SQLite. Additional writers are also provided to assist with testing and debugging. Mutiple writers can be combined using MultiWriter.

All writers perform atomic file operations, writing data to a temporary location and moving it to the final output location when Close is called.

About

When building an app or tool that needs to output data consisting of multiple different record types to disk, perhaps with requirements that change over time (whether during development or after initial deployment), perhaps requiring multiple output formats (during development/testing, or as final output) — is where peanut might be 'the right tool for the job'.

Ideal for use as an output solution for, e.g. data conversion tools, part of an ETL pipeline, data-acquistion or extraction tools/apps, web-scrapers, structured logging, persistence of captured data/metadata/events, job reporting, etc. Whether building an ad-hoc tool as a quick hack, or as part of a bigger, more serious project.

peanut initially evolved as part of a larger closed-source project, is tried and tested, and production-ready.

Quickstart

Installation

Get the package:

go get github.com/jimsmart/peanut

Use the package within your code:

import "github.com/jimsmart/peanut"

API

All peanut writers implement this interface:

type Writer interface {
    Write(r interface{}) error
    Close() error
    Cancel() error
}

Usage

  1. Tag some structs.
  2. Initialise a peanut.Writer to use.
  3. Collect and assign data into tagged structs.
  4. Call Write() to write records, repeating until done.
  5. Call Close() to finish.

Example Code

See GoDocs.

Documentation

GoDocs https://godoc.org/github.com/jimsmart/peanut

Testing

To run the tests execute go test inside the project folder.

For a full coverage report, try:

go test -coverprofile=coverage.out && go tool cover -html=coverage.out

License

Package peanut is copyright 2020-2021 by Jim Smart and released under the BSD 3-Clause License.

History

  • v1.0.3 (2021-04-19) Relax semantics of Close/Cancel. Improved error handling.
  • v1.0.2 (2021-04-19) Fixup handling of uints.
  • v1.0.1 (2021-04-19) Repository made public.
Owner
Jim Smart
🌀 Software engineer & solutions architect. Back-end, front-end, middle bits — and the fiddly bits around the edge. From 6502 assembly through to Go.
Jim Smart
Convert arbitrary formats to Go Struct (including json, toml, yaml, etc.)

go2struct Convert arbitrary formats to Go Struct (including json, toml, yaml, etc.) Installation Run the following command under your project: go get

Afeyer 34 Jul 6, 2022
A parser generator where rules defined as go structs and code generation is optional

A parser generator where rules defined as go structs and code generation is optional. The concepts are introduced in the simple example below.

Arnaud Delobelle 6 Jul 1, 2022
A codec for Go structs with support for chainable encoding/decoding hooks.

structool A codec for Go structs with support for chainable encoding/decoding hooks. Features Provide a uniform codec by combining mapstructure and st

Luo Peng 11 Jan 15, 2022
Enforce default values on structs in Go

Defaults Enforce default values on struct fields. type User struct { Name string `default:"Goku"` Power float64 `default:"9000.01"` } var u

Christian Rocha 24 Aug 3, 2022
An encoder for Go structs to HTML

GOHTML An encoder for a Go struct to HTML Using the "reflect" package and recursion this package is able to convert a complex go struct into HTML Feat

Dominic Cooper-Wootton 25 Jul 19, 2022
Structscanner is a simple library to make going from database queries to structs easier

structscanner is a simple library to make going from database queries to structs easier, while retaining the flexibility of joins and mapping using struct tags.

Folktale 3 May 25, 2022
Graphoscope: a solution to access multiple independent data sources from a common UI and show data relations as a graph

Graphoscope A solution to access multiple independent data sources from a common UI and show data relations as a graph: Contains a list of by default

CERT.LV 29 May 26, 2022
Package set is a small wrapper around the official reflect package that facilitates loose type conversion and assignment into native Go types.

Package set is a small wrapper around the official reflect package that facilitates loose type conversion and assignment into native Go types. Read th

null 37 Jun 15, 2022
A tree like tool help you to explore data structures in your redis server

Redis-view is a tree like tool help you explore data structures in your redis server

dreamersdw 19 Mar 17, 2022
Bitset data structure

Your basic bit Set data structure for positive numbers A bit array, or bit set, is an efficient set data structure. It consists of an array that compa

Algorithms to Go 127 Jul 29, 2022
Probabilistic set data structure

Your basic Bloom filter Golang probabilistic set data structure A Bloom filter is a fast and space-efficient probabilistic data structure used to test

Algorithms to Go 73 Jul 6, 2022
Probabilistic data structures for processing continuous, unbounded streams.

Boom Filters Boom Filters are probabilistic data structures for processing continuous, unbounded streams. This includes Stable Bloom Filters, Scalable

Tyler Treat 1.5k Aug 1, 2022
Data structure and algorithm library for go, designed to provide functions similar to C++ STL

GoSTL English | 简体中文 Introduction GoSTL is a data structure and algorithm library for go, designed to provide functions similar to C++ STL, but more p

stirlingx 692 Aug 8, 2022
Gota: DataFrames and data wrangling in Go (Golang)

Gota: DataFrames, Series and Data Wrangling for Go This is an implementation of DataFrames, Series and data wrangling methods for the Go programming l

null 2.3k Aug 9, 2022
A simple Set data structure implementation in Go (Golang) using LinkedHashMap.

Set Set is a simple Set data structure implementation in Go (Golang) using LinkedHashMap. This library allow you to get a set of int64 or string witho

Studio Sol Comunicação Digital Ltda 21 Jul 22, 2022
Data structure and relevant algorithms for extremely fast prefix/fuzzy string searching.

Trie Data structure and relevant algorithms for extremely fast prefix/fuzzy string searching. Usage Create a Trie with: t := trie.New() Add Keys with:

Derek Parker 597 Jul 31, 2022
Graph algorithms and data structures

Your basic graph Golang library of basic graph algorithms Topological ordering, image by David Eppstein, CC0 1.0. This library offers efficient and we

Algorithms to Go 565 Jul 29, 2022
BTree provides a simple, ordered, in-memory data structure for Go programs.

BTree implementation for Go This package provides an in-memory B-Tree implementation for Go, useful as an ordered, mutable data structure. The API is

Google 2.9k Aug 10, 2022