Access and modify property values in deeply nested maps, using dot-separated paths

Overview

Dig lets you access and modify property values in deeply nested, unstructured maps, using dot-separated paths:

source := make(map[string]interface{})
data := []byte(`{"a": "b", "c": {"a": "b"}}`)
json.Unmarshal(data, &source)


d := dig.NewMap(source)
log.Printf("%+v", s.Source())
// map[a:b c:map[a:b]]

s.SetValue("c.a", "42")
s.SetValue("a", "1000")
log.Printf("%+v", s.Source())
// map[a:1000 c:map[a:42]]

b, err := s.GetValue("c.a")
log.Println(b.(string))
// 42

NOTE: Still in development and mainly for educational purposes. Use with caution!

Why unstructured?

Most programming languages use (some sort of) structs for structured and maps for unstructured data. While structs are a great choice when one knows the layout of the incoming data, this may often not be the case. This is where generic maps hit the stage.

Motivation

One of the projects I worked on, was a data transformation pipeline that allowed users of the pipeline to send raw data to it, along with series of tiny rule mappings. Each rule mapping was a key-value pair, where both the key and the value would be tuples of the kind:

(nested.dotted.path.propertyToFilterUpon: filterValue), (nested.dotted.path.propertyToSetOrReplaceValueOf: newValue)

The first item of the tuple would be a dotted string, representing a nested path inside the data, e.g.:

location.address.postalCode

The value in each would be used to filter incoming data upon, or set the property under the given path, respectively. Key requirements for the pipeline were:

  • agnostic to the data structure being passed to it
  • resilient to change
  • easy to configure (as in the example above), even by non-programmers

Advantages of dig

Upon creating a dig map, an index gets created under the hood, traversing each key until it reaches value, which cannot be traversed further (no maps, or slices). An index may look like this:

a -> val ptr,
a.b -> val.ptr
a.b.c -> val.ptr
a.b.d -> val.ptr
a.f -> val.ptr
// etc ...

Successing retrieval or value replacement with a nested path, is made using the index alone.

Creating the index is a slight overhead at the beginning, but it dramatically speeds up the look up of deeply nested paths, as it can be seen on the benchmark resutls below.

Performance Benchmarks

Listed below is the (after-index) performance of dig, when compared to manually traversing the path in a regular Go map. Lastly, GJSON - a popular JSON de/serialization library with similar capabilities gets thrown into the mix. As it can be seen, dig's reading speed is significantly higher.

Test                                            ns. per operation
---
BenchmarkPerf/dig-8                             29.1 ns/op
BenchmarkPerf/go-map-8                          64.6 ns/op
BenchmarkPerf/gjson-8                           201 ns/op
Owner
Preslav Rachev
A software engineer turning writer. Check out my first book: Generative Art in Go (https://preslav.me/generative-art-in-golang/)
Preslav Rachev
Rhythm - Euclidean Rhythm generator written in Go with nested circular lists 🤹

rhythm Euclidean Rhythm generator written in Go with nested circular lists ?? Us

Justin Casali 0 Jan 31, 2022
Go-path - A helper package that provides utilities for parsing and using ipfs paths

go-path is a helper package that provides utilities for parsing and using ipfs paths

y 0 Jan 18, 2022
Go tool to modify struct field tags

Go tool to modify/update field tags in structs. gomodifytags makes it easy to update, add or delete the tags in a struct field. You can easily add new tags, update existing tags (such as appending a new key, i.e: db, xml, etc..) or remove existing tags

Fatih Arslan 1.7k May 9, 2022
Create deep copies (clones) of your maps and slices without using reflection.

DeepCopy DeepCopy helps you create deep copies (clones) of your maps and slices. Create deep copies (clones) of your objects The package is based on t

null 3 Apr 12, 2022
A protoc plugin that generates fieldmask paths as static type properties for proto messages

protoc-gen-fieldmask A protoc plugin that generates fieldmask paths as static ty

null 6 May 11, 2022
Golang: unify nil and empty slices and maps

unifynil, unify nil and empty slices and maps in Golang Empty slices and maps can be nil or not nil in Go. It may become a nightmare in tests and JSON

Boris Nagaev 0 Jan 16, 2022
A Go package for checking conditions for slices and maps.

check Go package The check package of Go helps one to check various conditions for slices: []int []float64 []string []bool maps: map[string]int map[st

null 4 Feb 3, 2022
Automatically creates & tiles .tmx format maps from a world map interface

Autotile Create tiled maps for an arbitrarily large world space from a simple interface, then add larger objects randomly with simple rules (eg. place

null 0 Jan 9, 2022
Hex dump and read values of files quickly and swiftly with Go-Hex a program designed to dump any file in a hexadecimal format

Go-Hex Hex dump and read values of files quickly and swiftly with Go-Hex a program designed to dump any file in a hexadecimal format Dump Hashes ----

RE43P3R 0 Oct 10, 2021
Go library for decoding generic map values into native Go structures and vice versa.

mapstructure mapstructure is a Go library for decoding generic map values to structures and vice versa, while providing helpful error handling. This l

Mitchell Hashimoto 5.7k May 12, 2022
Vex is a variable-length, lexicographically-sortable hex format for uint64 values

Vex is a variable-length, lexicographically-sortable hex format for uint64 values. It can be used instead of fmt.Sprintf("%016x", v) for shorter s

Ben Johnson 13 Mar 3, 2022
reasonable handling of nullable values

null import "gopkg.in/guregu/null.v4" null is a library with reasonable options for dealing with nullable SQL and JSON values There are two packages:

Greg 1.4k May 17, 2022
DO NOT use `named return values` in Go

namedreturn DO NOT use named return values in Go namedreturn finds a named return value, you must remove it.

Tomochika Hara 3 Dec 17, 2021
Identify containers at runtime and observe them. No container runtime required. Read only access to the kernel.

Linux Telemetry The Double Slit Experiment Taken from an interesting physics anomaly where the behavior of a physical system mutates simply by being o

Kris Nóva 13 Mar 30, 2022
Package macho implements access to and creation of Mach-O object files.

go-macho [WIP] ?? Package macho implements access to and creation of Mach-O object files. Why ?? This package goes beyond the Go's debug/macho to: Cov

Bogdan Donchenko 6 Dec 22, 2021
sigbypass4xx is a utility to automate well-know techniques used to bypass access control restrictions.

sigbypass4xx sigbypass4xx is a utility to automate well-know techniques used to bypass access control restrictions. Resources Usage Installation From

Signed Security 10 Mar 3, 2022
A tool and library for using structural regular expressions.

Structural Regular Expressions sregx is a package and tool for using structural regular expressions as described by Rob Pike (link).

Zachary Yedidia 35 May 9, 2022
This project provides some working examples using Go and Hotwire Turbo.

hotwire-golang-website This project provides some working examples using Go the hotwire/turbo library published by basecamp.

Mark Wolfe 128 May 14, 2022
Bitwise AND on two byte-slices using SIMD instructions

This package provides a vectorised function which performs bitwise AND operation on all pairs of elements in two byte-slices. It detects CPU instruction set and chooses the available best one (AVX512, AVX2, SSE2).

Wei Shen 5 Dec 10, 2021