Vectorized SQL for JSON at scale: fast, simple, schemaless

Sneller is a high-performance vectorized SQL engine for JSON that runs directly on object storage. Sneller is optimized to handle large TB-sized JSON (and more generally, semi-structured data including deeply nested structures/fields) without needing a schema to be specified upfront or dedicated ETL/ELT/indexing steps. It is particularly well suited for the rapidly growing world of event data such as data from Security, Observability, Ops, Product Analytics and Sensor/IoT data pipelines. Under the hood, Sneller operates on ion, a structure-preserving, compact binary representation of the original JSON data.

Sneller's query performance derives from pervasive use of SIMD, specifically AVX-512 assembly in its 250+ core primitives. The main engine is capable of processing many lanes in parallel per core for very high processing throughput. This eliminates the need to pre-process JSON data into an alternate representation - such as search indices (Elasticsearch and variants) or columnar formats like parquet (as commonly done with SQL-based tools). Combined with the fact that Sneller's main 'API' is SQL (with JSON as the primary output format), this greatly simplifies processing pipelines built around JSON data.

Sneller extends standard SQL syntax via PartiQL by supporting path expressions to reference nested fields/structures in JSON. For example, the . operator dereferences fields within structures. In combination with normal SQL functions/operators, this makes for a far more ergonomic way to query deeply nested JSON than non-standard SQL extensions. Additionally, Sneller implements a large (and growing!) number of built-in functions from other SQL implementations.

Unlike traditional data stores, Sneller completely separates storage from compute, as it is foundationally built to use object storage such as S3, GCS, Azure Blob or Minio as its primary storage layer. There are no other dependencies, such as meta-databases or key/value stores, to install, manage and maintain. This means no complex redundancy-based architecture (HA) is needed to avoid data loss. It also means that scaling Sneller up or down is as simple as adding or removing compute-only nodes.

Here is a 50000 ft overview of what is essentially the complete Sneller pipeline for JSON -- you can also read our more detailed blog post Introducing sneller.

Build from source

Make sure you have Golang 1.18 installed, and build as follows:

$ git clone https://github.com/SnellerInc/sneller
$ cd sneller
$ go build ./...

AVX-512 support

Please make sure that your CPU has AVX-512 support. Also note that AVX-512 is widely available on all major cloud providers: for AWS we recommend c6i (Ice Lake) or r5 (Skylake), for GCP we recommend N2, M2, or C2 instance types, or either Dv4 or Ev4 families on Azure.

Quick test drive

The easiest way to try out sneller is via the (standalone) sneller executable. (Note: this is more of a development tool, for application use see either the Docker or Kubernetes section below.)

We've made some sample data available in the sneller-samples bucket, based on the (excellent) GitHub archive. Here are some queries that illustrate what you can do with Sneller on fairly complex JSON event structures containing 100+ fields.

simple count

$ go install github.com/SnellerInc/sneller/cmd/sneller@latest
$ aws s3 cp s3://sneller-samples/gharchive-1day.ion.zst .
$ du -h gharchive-1day.ion.zst
1.3G
$ sneller -j "select count(*) from 'gharchive-1day.ion.zst'"
{"count": 3259519}

search/filter (notice SQL string operations such as LIKE/ILIKE on a nested field `repo.name`)

$ # all repos containing 'orvalds' (case-insensitive)
$ sneller -j "SELECT DISTINCT repo.name FROM 'gharchive-1day.ion.zst' WHERE repo.name ILIKE '%orvalds%'"
{"name": "torvalds/linux"}
{"name": "jordy-torvalds/dailystack"}
{"name": "torvalds/subsurface-for-dirk"}

standard SQL aggregations/grouping

$ # number of events per type
$ sneller -j "SELECT type, COUNT(*) FROM 'gharchive-1day.ion.zst' GROUP BY type ORDER BY COUNT(*) DESC"
{"type": "PushEvent", "count": 1686536}
...
{"type": "GollumEvent", "count": 7443}

query custom payloads (see `payload.pull_request.created_at` only for `type = 'PullRequestEvent'` rows)

$ # number of pull requests that took more than 180 days
$ sneller -j "SELECT COUNT(*) FROM 'gharchive-1day.ion.zst' WHERE type = 'PullRequestEvent' AND DATE_DIFF(DAY, payload.pull_request.created_at, created_at) >= 180"
{"count": 3161}

specialized operators like `TIME_BUCKET`

$ # number of events per type per hour (date histogram)
$ sneller -j "SELECT TIME_BUCKET(created_at, 3600) AS time, type, COUNT(*) FROM 'gharchive-1day.ion.zst' GROUP BY TIME_BUCKET(created_at, 3600), type"
{"time": 1641254400, "type": "PushEvent", "count": 58756}
...
{"time": 1641326400, "type": "MemberEvent", "count": 316}

combine multiple queries

# fire off multiple queries simultaneously as a single (outer) select
$ sneller -j "SELECT (SELECT COUNT(*) FROM 'gharchive-1day.ion.zst') AS query0, (SELECT DISTINCT repo.name FROM 'gharchive-1day.ion.zst' WHERE repo.name ILIKE '%orvalds%') as query1" | jq
{
  "query0": 3259519,
  "query1": [
    { "name": "torvalds/linux" },
    { "name": "jordy-torvalds/dailystack" },
    { "name": "torvalds/subsurface-for-dirk" }
  ]
}

If you're a bit more adventurous, you can grab the 1month object (contains 80M rows at 29GB compressed), here as tested on a c6i.32xlarge:

$ aws s3 cp s3://sneller-samples/gharchive-1month.ion.zst .
$ du -h gharchive-1month.ion.zst 
29G
$ time sneller -j "select count(*) from 'gharchive-1month.ion.zst'"
{"count": 79565989}
real    0m4.892s
user    6m41.630s
sys     0m48.016s
$ 
$ time sneller -j "SELECT DISTINCT repo.name FROM 'gharchive-1month.ion.zst' WHERE repo.name ILIKE '%orvalds%'"
{"name": "torvalds/linux"}
{"name": "jordy-torvalds/dailystack"}
...
{"name": "IHorvalds/AstralEye"}
real    0m4.940s
user    7m11.080s
sys     0m28.268s

Performance

Depending on the type of query, sneller is capable of processing GB/s of data per second per core, as shown in these benchmarks (measured on a c6i.12xlarge instance on AWS with an Ice Lake CPU):

$ cd vm
$ # S I N G L E   C O R E
$ GOMAXPROCS=1 go test -bench=HashAggregate
cpu: Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
BenchmarkHashAggregate/case-0                  6814            170163 ns/op          6160.59 MB/s
BenchmarkHashAggregate/case-1                  5361            217318 ns/op          4823.83 MB/s
BenchmarkHashAggregate/case-2                  5019            232081 ns/op          4516.98 MB/s
BenchmarkHashAggregate/case-3                  4232            278055 ns/op          3770.13 MB/s
PASS
ok      github.com/SnellerInc/sneller/vm        6.119s
$
$ # A L L   C O R E S
$ go test -bench=HashAggregate
cpu: Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
BenchmarkHashAggregate/case-0-48             155818              6969 ns/op        150424.92 MB/s
BenchmarkHashAggregate/case-1-48             129116              8764 ns/op        119612.84 MB/s
BenchmarkHashAggregate/case-2-48             121840              9379 ns/op        111768.43 MB/s
BenchmarkHashAggregate/case-3-48             119640              9578 ns/op        109444.06 MB/s
PASS
ok      github.com/SnellerInc/sneller/vm        5.576s

The following chart shows the performance for a varying numbers of cores:

Sneller is capable of scaling beyond a single server and for instance a medium-sized r6i.12xlarge cluster in AWS can achieve 1TB/s in scanning performance, even running non-trivial queries.

Spin up stack locally

It is easiest to spin up a local stack, comprising of just Sneller as the query engine and Minio as the S3 storage layer, by using Docker. Detailed instructions can be found here using sample data from the GitHub archive (but swapping this out for your own data is trivial). Note that this setup is a single node install and therefore no-HA.

Once you have followed the instructions, you can interact with Sneller on port localhost:9180 via curl, eg. as per:

$ curl -G -H "Authorization: Bearer $SNELLER_TOKEN" --data-urlencode "database=gha" \
    --data-urlencode 'json' --data-urlencode 'query=SELECT COUNT(*) FROM gharchive' \
    'http://localhost:9180/executeQuery'
{"count": 2141038}
$ curl -G -H "Authorization: Bearer $SNELLER_TOKEN" --data-urlencode "database=gha" \
    --data-urlencode 'json' --data-urlencode 'query=SELECT type, COUNT(*) FROM gharchive GROUP BY type ORDER BY COUNT(*) DESC' \
    'http://localhost:9180/executeQuery'
{"type": "PushEvent", "count": 1303922}
{"type": "CreateEvent", "count": 261401}
...
{"type": "GollumEvent", "count": 4035}
{"type": "MemberEvent", "count": 2644}

Spin up sneller stack in the cloud

It is also possible to use Kubernetes to spin up a sneller stack in the cloud. You can either do this on AWS using S3 for storage or in another (hybrid) cloud that supports Kubernetes and potentially using an object storage such as Minio.

See the Sneller on Kubernetes instructions for more details and an example of how to spin this up.

Documentation

See the docs directory for more information (technical nature).

Explore further

See docs.sneller.io for further information:

Development

See docs/DEVELOPMENT.

Contribute

Sneller is released under the AGPL-3.0 license. See the LICENSE file for more information.

On Windows 11 amd64 32GB mem

fatal error: fault
[signal 0xc000001d code=0x0 addr=0x0 pc=0xb56ba8]

goroutine 9 [running]:
runtime.throw({0xd8ff5f?, 0x0?})
	C:/dev/go/src/runtime/panic.go:1047 +0x65 fp=0xc000277a38 sp=0xc000277a08 pc=0x998265
runtime.sigpanic()
	C:/dev/go/src/runtime/signal_windows.go:273 +0x1af fp=0xc000277a80 sp=0xc000277a38 pc=0x9ab66f
github.com/SnellerInc/sneller/vm.evalfilterbc(0xc00014c098, {0xc001f09000, 0x200, 0x200})
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/vm/evalfilter_amd64.s:30 +0x28 fp=0xc000277a98 sp=0xc000277a80 pc=0xb56ba8
github.com/SnellerInc/sneller/vm.(*wherebc).writeRows(0xc00014c000, {0xc001f09000?, 0xff6bb?, 0x200})
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/vm/filter.go:115 +0x59 fp=0xc000277ae8 sp=0xc000277a98 pc=0xb0cdd9
github.com/SnellerInc/sneller/vm.(*rowSplitter).writeVM(0xc0001761b0, {0x1f9feed0945?, 0xc001f0e83c?, 0x4?}, {0xc001f09000?, 0x200, 0x200})
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/vm/sfw.go:116 +0x12c fp=0xc000277b60 sp=0xc000277ae8 pc=0xb149cc
github.com/SnellerInc/sneller/vm.(*rowSplitter).Write(0xc0001761b0, {0x1f9feed0000, 0x100000, 0x100000})
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/vm/sfw.go:226 +0x391 fp=0xc000277c38 sp=0xc000277b60 pc=0xb154d1
github.com/SnellerInc/sneller/plan.(*writeTracker).Write(0xc000008150, {0x1f9feed0000?, 0x100000, 0xc000124038?})
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/plan/stats.go:100 +0x33 fp=0xc000277c68 sp=0xc000277c38 pc=0xb8c653
main.(*byteTracker).Write(0xe3a440?, {0x1f9feed0000?, 0xc001e80000?, 0x2b0e5?})
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/cmd/sneller/table.go:44 +0x38 fp=0xc000277c98 sp=0xc000277c68 pc=0xcf7b38
github.com/SnellerInc/sneller/ion/blockfmt.(*Decoder).Copy(0xc000124000, {0xe3a4a0, 0xc00017c000}, {0xe3a440, 0xc00012dad0})
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/ion/blockfmt/compression.go:544 +0x579 fp=0xc000277e38 sp=0xc000277c98 pc=0xcf0a59
main.(*readerTable).write(0xc000072050, {0x1fa7f0f89d8?, 0xc000008150})
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/cmd/sneller/table.go:85 +0x42c fp=0xc000277f10 sp=0xc000277e38 pc=0xcf7fac
main.(*readerTable).write-fm({0x1fa7f0f89d8?, 0xc000008150?})
	<autogenerated>:1 +0x39 fp=0xc000277f38 sp=0xc000277f10 pc=0xcf8679
github.com/SnellerInc/sneller/vm.SplitInput.func1(0x0)
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/vm/table.go:125 +0xba fp=0xc000277fc8 sp=0xc000277f38 pc=0xb3d39a
github.com/SnellerInc/sneller/vm.SplitInput.func2()
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/vm/table.go:134 +0x2a fp=0xc000277fe0 sp=0xc000277fc8 pc=0xb3d2aa
runtime.goexit()
	C:/dev/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000277fe8 sp=0xc000277fe0 pc=0x9c5101
created by github.com/SnellerInc/sneller/vm.SplitInput
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/vm/table.go:123 +0x185

goroutine 1 [semacquire]:
runtime.gopark(0x96d7e7?, 0x90?, 0x0?, 0x0?, 0x96d49f?)
	C:/dev/go/src/runtime/proc.go:363 +0xd6 fp=0xc00027b9b0 sp=0xc00027b990 pc=0x99ad36
runtime.goparkunlock(...)
	C:/dev/go/src/runtime/proc.go:369
runtime.semacquire1(0xc00001e5e8, 0x60?, 0x1, 0x0)
	C:/dev/go/src/runtime/sema.go:144 +0x20f fp=0xc00027ba18 sp=0xc00027b9b0 pc=0x9aa0af
sync.runtime_Semacquire(0xc000008348?)
	C:/dev/go/src/runtime/sema.go:56 +0x25 fp=0xc00027ba48 sp=0xc00027ba18 pc=0x9c0a05
sync.(*WaitGroup).Wait(0xc0000302a0?)
	C:/dev/go/src/sync/waitgroup.go:136 +0x52 fp=0xc00027ba70 sp=0xc00027ba48 pc=0x9dd3d2
github.com/SnellerInc/sneller/vm.SplitInput({0xe3bb48, 0xc000008138}, 0x8, 0xc000062000)
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/vm/table.go:142 +0x331 fp=0xc00027bb00 sp=0xc00027ba70 pc=0xb3d1f1
main.(*readerTable).WriteChunks(0xc000072050, {0xe3bb48, 0xc000008138}, 0x18?)
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/cmd/sneller/table.go:101 +0xa5 fp=0xc00027bb68 sp=0xc00027bb00 pc=0xcf8125
github.com/SnellerInc/sneller/plan.(*Leaf).exec(0xe3dfa8?, {0xe3be40?, 0xc000008240}, 0xc00016c1c0?, 0xc000020690, 0xc00016c1e8?)
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/plan/plan.go:275 +0x109 fp=0xc00027bbd8 sp=0xc00027bb68 pc=0xb89069
github.com/SnellerInc/sneller/plan.(*Filter).exec(0xc000064e20, {0xe3be18, 0xc00016c1c0}, 0xe3bee0?, 0xc00012dd70?, 0x96d7e7?)
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/plan/filter.go:42 +0xa3 fp=0xc00027bc30 sp=0xc00027bbd8 pc=0xb846c3
github.com/SnellerInc/sneller/plan.(*Distinct).exec(0xc00012dd10, {0xe3bee0?, 0xc00012dd70?}, 0x9bcb79?, 0xc000064d60?, 0x30120?)
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/plan/plan.go:871 +0xac fp=0xc00027bc88 sp=0xc00027bc30 pc=0xb8ad0c
github.com/SnellerInc/sneller/plan.(*Project).exec(0xc00012dd40, {0xe3bf58?, 0xc000008210}, 0x0?, 0x2?, 0x2?)
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/plan/project.go:40 +0xfb fp=0xc00027bd00 sp=0xc00027bc88 pc=0xb8bb1b
github.com/SnellerInc/sneller/plan.(*Tree).exec(0x0?, {0xe3bf58?, 0xc000008210?}, 0xc000020690?, 0x0?, 0x1fa7eef59b8?)
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/plan/tree.go:111 +0x158 fp=0xc00027bdb0 sp=0xc00027bd00 pc=0xb90b18
github.com/SnellerInc/sneller/plan.(*LocalTransport).Exec(0xc00011de48?, 0xe3ad00?, 0xdc7ca8?, {0x1fa7eef59b8?, 0xc00000a038?}, 0x4000?)
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/plan/root.go:58 +0xfa fp=0xc00027be18 sp=0xc00027bdb0 pc=0xb8c41a
github.com/SnellerInc/sneller/plan.Exec(...)
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/plan/root.go:28
main.do({0xc0000300c0?, 0x0?})
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/cmd/sneller/main.go:190 +0xbe fp=0xc00027be80 sp=0xc00027be18 pc=0xcf6c7e
main.main()
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/cmd/sneller/main.go:255 +0x4f4 fp=0xc00027bf80 sp=0xc00027be80 pc=0xcf7294
runtime.main()
	C:/dev/go/src/runtime/proc.go:250 +0x1fe fp=0xc00027bfe0 sp=0xc00027bf80 pc=0x99a99e
runtime.goexit()
	C:/dev/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00027bfe8 sp=0xc00027bfe0 pc=0x9c5101

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	C:/dev/go/src/runtime/proc.go:363 +0xd6 fp=0xc000059fb0 sp=0xc000059f90 pc=0x99ad36
runtime.goparkunlock(...)
	C:/dev/go/src/runtime/proc.go:369
runtime.forcegchelper()
	C:/dev/go/src/runtime/proc.go:302 +0xb1 fp=0xc000059fe0 sp=0xc000059fb0 pc=0x99abd1
runtime.goexit()
	C:/dev/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000059fe8 sp=0xc000059fe0 pc=0x9c5101
created by runtime.init.6
	C:/dev/go/src/runtime/proc.go:290 +0x25

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
	C:/dev/go/src/runtime/proc.go:363 +0xd6 fp=0xc00005bf90 sp=0xc00005bf70 pc=0x99ad36
runtime.goparkunlock(...)
	C:/dev/go/src/runtime/proc.go:369
runtime.bgsweep(0x0?)
	C:/dev/go/src/runtime/mgcsweep.go:297 +0xd7 fp=0xc00005bfc8 sp=0xc00005bf90 pc=0x9848d7
runtime.gcenable.func1()
	C:/dev/go/src/runtime/mgc.go:178 +0x26 fp=0xc00005bfe0 sp=0xc00005bfc8 pc=0x979ac6
runtime.goexit()
	C:/dev/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00005bfe8 sp=0xc00005bfe0 pc=0x9c5101
created by runtime.gcenable
	C:/dev/go/src/runtime/mgc.go:178 +0x6b

goroutine 4 [GC scavenge wait]:
runtime.gopark(0xc000066000?, 0xe37ef8?, 0x0?, 0x0?, 0x0?)
	C:/dev/go/src/runtime/proc.go:363 +0xd6 fp=0xc00006df70 sp=0xc00006df50 pc=0x99ad36
runtime.goparkunlock(...)
	C:/dev/go/src/runtime/proc.go:369
runtime.(*scavengerState).park(0x1090ee0)
	C:/dev/go/src/runtime/mgcscavenge.go:389 +0x53 fp=0xc00006dfa0 sp=0xc00006df70 pc=0x982953
runtime.bgscavenge(0x0?)
	C:/dev/go/src/runtime/mgcscavenge.go:622 +0x65 fp=0xc00006dfc8 sp=0xc00006dfa0 pc=0x982f65
runtime.gcenable.func2()
	C:/dev/go/src/runtime/mgc.go:179 +0x26 fp=0xc00006dfe0 sp=0xc00006dfc8 pc=0x979a66
runtime.goexit()
	C:/dev/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00006dfe8 sp=0xc00006dfe0 pc=0x9c5101
created by runtime.gcenable
	C:/dev/go/src/runtime/mgc.go:179 +0xaa

goroutine 5 [finalizer wait]:
runtime.gopark(0x99b0d7?, 0x49?, 0x0?, 0x0?, 0xc00005df70?)
	C:/dev/go/src/runtime/proc.go:363 +0xd6 fp=0xc00005de28 sp=0xc00005de08 pc=0x99ad36
runtime.goparkunlock(...)
	C:/dev/go/src/runtime/proc.go:369
runtime.runfinq()
	C:/dev/go/src/runtime/mfinal.go:180 +0x10f fp=0xc00005dfe0 sp=0xc00005de28 pc=0x978bcf
runtime.goexit()
	C:/dev/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00005dfe8 sp=0xc00005dfe0 pc=0x9c5101
created by runtime.createfing
	C:/dev/go/src/runtime/mfinal.go:157 +0x45

goroutine 6 [select]:
runtime.gopark(0xc00006fcf8?, 0x2?, 0x0?, 0x0?, 0xc00006fcd4?)
	C:/dev/go/src/runtime/proc.go:363 +0xd6 fp=0xc00006fb58 sp=0xc00006fb38 pc=0x99ad36
runtime.selectgo(0xc00006fcf8, 0xc00006fcd0, 0x8?, 0x0, 0x976f05?, 0x1)
	C:/dev/go/src/runtime/select.go:328 +0x7dc fp=0xc00006fc98 sp=0xc00006fb58 pc=0x9a8fbc
io.(*pipe).read(0xc000070240, {0xc000086000, 0x1000, 0x96d608?})
	C:/dev/go/src/io/pipe.go:57 +0xb1 fp=0xc00006fd28 sp=0xc00006fc98 pc=0x9f5ef1
io.(*PipeReader).Read(0xc0000576c0?, {0xc000086000?, 0x1000096a273?, 0x1f9593a9c00?})
	C:/dev/go/src/io/pipe.go:136 +0x25 fp=0xc00006fd58 sp=0xc00006fd28 pc=0x9f66a5
bufio.(*Reader).fill(0xc0000702a0)
	C:/dev/go/src/bufio/bufio.go:106 +0xff fp=0xc00006fd90 sp=0xc00006fd58 pc=0xa3267f
bufio.(*Reader).Peek(0xc0000702a0, 0xa)
	C:/dev/go/src/bufio/bufio.go:144 +0x5d fp=0xc00006fdb0 sp=0xc00006fd90 pc=0xa327dd
github.com/SnellerInc/sneller/ion.Peek(0xc00006fe28?)
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/ion/reader.go:39 +0x25 fp=0xc00006fde8 sp=0xc00006fdb0 pc=0xa55c45
github.com/SnellerInc/sneller/ion.ToJSON({0xe3a760?, 0xc00000a018?}, 0xc0000702a0)
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/ion/reader.go:323 +0x1cd fp=0xc00006ff20 sp=0xc00006fde8 pc=0xa5778d
main.main.func1()
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/cmd/sneller/main.go:239 +0x18b fp=0xc00006ffe0 sp=0xc00006ff20 pc=0xcf750b
runtime.goexit()
	C:/dev/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00006ffe8 sp=0xc00006ffe0 pc=0x9c5101
created by main.main
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/cmd/sneller/main.go:237 +0x46c

goroutine 7 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	C:/dev/go/src/runtime/proc.go:363 +0xd6 fp=0xc000069f50 sp=0xc000069f30 pc=0x99ad36
runtime.gcBgMarkWorker()
	C:/dev/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000069fe0 sp=0xc000069f50 pc=0x97bad1
runtime.goexit()
	C:/dev/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000069fe8 sp=0xc000069fe0 pc=0x9c5101
created by runtime.gcBgMarkStartWorkers
	C:/dev/go/src/runtime/mgc.go:1159 +0x25

goroutine 18 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	C:/dev/go/src/runtime/proc.go:363 +0xd6 fp=0xc00008df50 sp=0xc00008df30 pc=0x99ad36
runtime.gcBgMarkWorker()
	C:/dev/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00008dfe0 sp=0xc00008df50 pc=0x97bad1
runtime.goexit()
	C:/dev/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00008dfe8 sp=0xc00008dfe0 pc=0x9c5101
created by runtime.gcBgMarkStartWorkers
	C:/dev/go/src/runtime/mgc.go:1159 +0x25

goroutine 19 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	C:/dev/go/src/runtime/proc.go:363 +0xd6 fp=0xc00008ff50 sp=0xc00008ff30 pc=0x99ad36
runtime.gcBgMarkWorker()
	C:/dev/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00008ffe0 sp=0xc00008ff50 pc=0x97bad1
runtime.goexit()
	C:/dev/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00008ffe8 sp=0xc00008ffe0 pc=0x9c5101
created by runtime.gcBgMarkStartWorkers
	C:/dev/go/src/runtime/mgc.go:1159 +0x25

goroutine 8 [GC worker (idle)]:
runtime.gopark(0x17ddb5909fe94?, 0x1?, 0x38?, 0xdf?, 0x0?)
	C:/dev/go/src/runtime/proc.go:363 +0xd6 fp=0xc00006bf50 sp=0xc00006bf30 pc=0x99ad36
runtime.gcBgMarkWorker()
	C:/dev/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00006bfe0 sp=0xc00006bf50 pc=0x97bad1
runtime.goexit()
	C:/dev/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0x9c5101
created by runtime.gcBgMarkStartWorkers
	C:/dev/go/src/runtime/mgc.go:1159 +0x25

goroutine 34 [GC worker (idle)]:
runtime.gopark(0x17ddb58c321f4?, 0x0?, 0x0?, 0x0?, 0x0?)
	C:/dev/go/src/runtime/proc.go:363 +0xd6 fp=0xc000089f50 sp=0xc000089f30 pc=0x99ad36
runtime.gcBgMarkWorker()
	C:/dev/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000089fe0 sp=0xc000089f50 pc=0x97bad1
runtime.goexit()
	C:/dev/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000089fe8 sp=0xc000089fe0 pc=0x9c5101
created by runtime.gcBgMarkStartWorkers
	C:/dev/go/src/runtime/mgc.go:1159 +0x25

goroutine 20 [GC worker (idle)]:
runtime.gopark(0x17ddb5909fe94?, 0x0?, 0x0?, 0x0?, 0x0?)
	C:/dev/go/src/runtime/proc.go:363 +0xd6 fp=0xc000099f50 sp=0xc000099f30 pc=0x99ad36
runtime.gcBgMarkWorker()
	C:/dev/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000099fe0 sp=0xc000099f50 pc=0x97bad1
runtime.goexit()
	C:/dev/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000099fe8 sp=0xc000099fe0 pc=0x9c5101
created by runtime.gcBgMarkStartWorkers
	C:/dev/go/src/runtime/mgc.go:1159 +0x25

goroutine 35 [GC worker (idle)]:
runtime.gopark(0x17ddb5909fe94?, 0x0?, 0x0?, 0x0?, 0x0?)
	C:/dev/go/src/runtime/proc.go:363 +0xd6 fp=0xc00008bf50 sp=0xc00008bf30 pc=0x99ad36
runtime.gcBgMarkWorker()
	C:/dev/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00008bfe0 sp=0xc00008bf50 pc=0x97bad1
runtime.goexit()
	C:/dev/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00008bfe8 sp=0xc00008bfe0 pc=0x9c5101
created by runtime.gcBgMarkStartWorkers
	C:/dev/go/src/runtime/mgc.go:1159 +0x25

goroutine 21 [GC worker (idle)]:
runtime.gopark(0x17ddb5909fe94?, 0x0?, 0x0?, 0x0?, 0x0?)
	C:/dev/go/src/runtime/proc.go:363 +0xd6 fp=0xc00009bf50 sp=0xc00009bf30 pc=0x99ad36
runtime.gcBgMarkWorker()
	C:/dev/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00009bfe0 sp=0xc00009bf50 pc=0x97bad1
runtime.goexit()
	C:/dev/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00009bfe8 sp=0xc00009bfe0 pc=0x9c5101
created by runtime.gcBgMarkStartWorkers
	C:/dev/go/src/runtime/mgc.go:1159 +0x25

goroutine 10 [runnable]:
github.com/klauspost/compress/zstd.(*bitReader).close(0xc00017a030?)
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/bitreader.go:133 +0xd3 fp=0xc001ff5510 sp=0xc001ff5508 pc=0xcd5453
github.com/klauspost/compress/zstd.(*sequenceDecs).decodeSyncSimple(0xc00013c4d8, {0x0?, 0x0, 0xce0ab8?})
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/seqdec_amd64.go:138 +0x409 fp=0xc001ff5778 sp=0xc001ff5510 pc=0xce3869
github.com/klauspost/compress/zstd.(*sequenceDecs).decodeSync(0xc00013c4d8, {0x0, 0x0, 0x0})
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/seqdec.go:207 +0x4e fp=0xc001ff58e0 sp=0xc001ff5778 pc=0xce1bae
github.com/klauspost/compress/zstd.(*blockDec).decodeCompressed(0xc0001325a0, 0xc00013c4d0)
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:498 +0x19a fp=0xc001ff5938 sp=0xc001ff58e0 pc=0xcd711a
github.com/klauspost/compress/zstd.(*blockDec).decodeBuf(0xc0001325a0, 0xc00013c4d0)
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:262 +0x2db fp=0xc001ff59f0 sp=0xc001ff5938 pc=0xcd5d1b
github.com/klauspost/compress/zstd.(*frameDec).runDecoder(0xc00013c480, {0x1f9fefd0000?, 0x0, 0xa18acc?}, 0xc0001325a0)
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/framedec.go:369 +0x196 fp=0xc001ff5a60 sp=0xc001ff59f0 pc=0xcde036
github.com/klauspost/compress/zstd.(*Decoder).DecodeAll(0xc00016c000, {0xc000280000, 0x29d58, 0x729ebd}, {0x1f9fefd0000, 0x0, 0x100000})
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/decoder.go:378 +0x654 fp=0xc001ff5c28 sp=0xc001ff5a60 pc=0xcda174
github.com/SnellerInc/sneller/compr.(*zstdDecompressor).Decompress(0xe3a440?, {0xc000280000?, 0xc000280000?, 0x29d58?}, {0x1f9fefd0000, 0x100000, 0x0?})
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/compr/compression.go:101 +0x38 fp=0xc001ff5c98 sp=0xc001ff5c28 pc=0xce7f38
github.com/SnellerInc/sneller/ion/blockfmt.(*Decoder).Copy(0xc00009c000, {0xe3a4a0, 0xc00009e000}, {0xe3a440, 0xc0000a0000})
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/ion/blockfmt/compression.go:540 +0x54c fp=0xc001ff5e38 sp=0xc001ff5c98 pc=0xcf0a2c
main.(*readerTable).write(0xc000072050, {0x1fa7f0f89d8?, 0xc0000081b0})
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/cmd/sneller/table.go:85 +0x42c fp=0xc001ff5f10 sp=0xc001ff5e38 pc=0xcf7fac
main.(*readerTable).write-fm({0x1fa7f0f89d8?, 0xc0000081b0?})
	<autogenerated>:1 +0x39 fp=0xc001ff5f38 sp=0xc001ff5f10 pc=0xcf8679
github.com/SnellerInc/sneller/vm.SplitInput.func1(0x1)
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/vm/table.go:125 +0xba fp=0xc001ff5fc8 sp=0xc001ff5f38 pc=0xb3d39a
github.com/SnellerInc/sneller/vm.SplitInput.func2()
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/vm/table.go:134 +0x2a fp=0xc001ff5fe0 sp=0xc001ff5fc8 pc=0xb3d2aa
runtime.goexit()
	C:/dev/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc001ff5fe8 sp=0xc001ff5fe0 pc=0x9c5101
created by github.com/SnellerInc/sneller/vm.SplitInput
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/vm/table.go:123 +0x185

goroutine 11 [runnable]:
github.com/klauspost/compress/zstd.(*bitReader).close(0xc0009f4000?)
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/bitreader.go:133 +0xd3 fp=0xc0009e7510 sp=0xc0009e7508 pc=0xcd5453
github.com/klauspost/compress/zstd.(*sequenceDecs).decodeSyncSimple(0xc00013c958, {0x0?, 0x0, 0xce0ab8?})
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/seqdec_amd64.go:138 +0x409 fp=0xc0009e7778 sp=0xc0009e7510 pc=0xce3869
github.com/klauspost/compress/zstd.(*sequenceDecs).decodeSync(0xc00013c958, {0x0, 0x0, 0x0})
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/seqdec.go:207 +0x4e fp=0xc0009e78e0 sp=0xc0009e7778 pc=0xce1bae
github.com/klauspost/compress/zstd.(*blockDec).decodeCompressed(0xc0001327e0, 0xc00013c950)
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:498 +0x19a fp=0xc0009e7938 sp=0xc0009e78e0 pc=0xcd711a
github.com/klauspost/compress/zstd.(*blockDec).decodeBuf(0xc0001327e0, 0xc00013c950)
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:262 +0x2db fp=0xc0009e79f0 sp=0xc0009e7938 pc=0xcd5d1b
github.com/klauspost/compress/zstd.(*frameDec).runDecoder(0xc00013c900, {0x1f9ff1d0000?, 0x0, 0xa18acc?}, 0xc0001327e0)
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/framedec.go:369 +0x196 fp=0xc0009e7a60 sp=0xc0009e79f0 pc=0xcde036
github.com/klauspost/compress/zstd.(*Decoder).DecodeAll(0xc00016c000, {0xc001fca000, 0x20237, 0x20237}, {0x1f9ff1d0000, 0x0, 0x100000})
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/decoder.go:378 +0x654 fp=0xc0009e7c28 sp=0xc0009e7a60 pc=0xcda174
github.com/SnellerInc/sneller/compr.(*zstdDecompressor).Decompress(0xe3a440?, {0xc001fca000?, 0xc001fca000?, 0x20237?}, {0x1f9ff1d0000, 0x100000, 0x0?})
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/compr/compression.go:101 +0x38 fp=0xc0009e7c98 sp=0xc0009e7c28 pc=0xce7f38
github.com/SnellerInc/sneller/ion/blockfmt.(*Decoder).Copy(0xc00009c050, {0xe3a4a0, 0xc00009e010}, {0xe3a440, 0xc0000a0030})
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/ion/blockfmt/compression.go:540 +0x54c fp=0xc0009e7e38 sp=0xc0009e7c98 pc=0xcf0a2c
main.(*readerTable).write(0xc000072050, {0x1fa7f0f89d8?, 0xc0000081e0})
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/cmd/sneller/table.go:85 +0x42c fp=0xc0009e7f10 sp=0xc0009e7e38 pc=0xcf7fac
main.(*readerTable).write-fm({0x1fa7f0f89d8?, 0xc0000081e0?})
	<autogenerated>:1 +0x39 fp=0xc0009e7f38 sp=0xc0009e7f10 pc=0xcf8679
github.com/SnellerInc/sneller/vm.SplitInput.func1(0x2)
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/vm/table.go:125 +0xba fp=0xc0009e7fc8 sp=0xc0009e7f38 pc=0xb3d39a
github.com/SnellerInc/sneller/vm.SplitInput.func2()
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/vm/table.go:134 +0x2a fp=0xc0009e7fe0 sp=0xc0009e7fc8 pc=0xb3d2aa
runtime.goexit()
	C:/dev/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0009e7fe8 sp=0xc0009e7fe0 pc=0x9c5101
created by github.com/SnellerInc/sneller/vm.SplitInput
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/vm/table.go:123 +0x185

goroutine 12 [runnable]:
github.com/klauspost/compress/zstd.(*bitReader).close(0xc001e00030?)
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/bitreader.go:133 +0xd3 fp=0xc001ff1510 sp=0xc001ff1508 pc=0xcd5453
github.com/klauspost/compress/zstd.(*sequenceDecs).decodeSyncSimple(0xc00013c718, {0x0?, 0x0, 0xce0ab8?})
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/compress@v1.15.4/zstd/seqdec_amd64.go:138 +0x409 fp=0xc001ff1778 sp=0xc001ff1510 pc=0xce3869
github.com/klauspost/compress/zstd.(*sequenceDecs).decodeSync(0xc00013c718, {0x0, 0x0, 0x0})
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/seqdec.go:207 +0x4e fp=0xc001ff18e0 sp=0xc001ff1778 pc=0xce1bae
github.com/klauspost/compress/zstd.(*blockDec).decodeCompressed(0xc0001326c0, 0xc00013c710)
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:498 +0x19a fp=0xc001ff1938 sp=0xc001ff18e0 pc=0xcd711a
github.com/klauspost/compress/zstd.(*blockDec).decodeBuf(0xc0001326c0, 0xc00013c710)
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:262 +0x2db fp=0xc001ff19f0 sp=0xc001ff1938 pc=0xcd5d1b
github.com/klauspost/compress/zstd.(*frameDec).runDecoder(0xc00013c6c0, {0x1f9ff5d0000?, 0x0, 0xa18acc?}, 0xc0001326c0)
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/framedec.go:369 +0x196 fp=0xc001ff1a60 sp=0xc001ff19f0 pc=0xcde036
github.com/klauspost/compress/zstd.(*Decoder).DecodeAll(0xc00016c000, {0xc002096000, 0x2366b, 0x2366b}, {0x1f9ff5d0000, 0x0, 0x100000})
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/decoder.go:378 +0x654 fp=0xc001ff1c28 sp=0xc001ff1a60 pc=0xcda174
github.com/SnellerInc/sneller/compr.(*zstdDecompressor).Decompress(0xe3a440?, {0xc002096000?, 0xc002096000?, 0x2366b?}, {0x1f9ff5d0000, 0x100000, 0x12c210?})
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/compr/compression.go:101 +0x38 fp=0xc001ff1c98 sp=0xc001ff1c28 pc=0xce7f38
github.com/SnellerInc/sneller/ion/blockfmt.(*Decoder).Copy(0xc000104000, {0xe3a4a0, 0xc000106000}, {0xe3a440, 0xc001e00000})
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/ion/blockfmt/compression.go:540 +0x54c fp=0xc001ff1e38 sp=0xc001ff1c98 pc=0xcf0a2c
main.(*readerTable).write(0xc000072050, {0x1fa7f0f89d8?, 0xc000008270})
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/cmd/sneller/table.go:85 +0x42c fp=0xc001ff1f10 sp=0xc001ff1e38 pc=0xcf7fac
main.(*readerTable).write-fm({0x1fa7f0f89d8?, 0xc000008270?})
	<autogenerated>:1 +0x39 fp=0xc001ff1f38 sp=0xc001ff1f10 pc=0xcf8679
github.com/SnellerInc/sneller/vm.SplitInput.func1(0x3)
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/vm/table.go:125 +0xba fp=0xc001ff1fc8 sp=0xc001ff1f38 pc=0xb3d39a
github.com/SnellerInc/sneller/vm.SplitInput.func2()
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/vm/table.go:134 +0x2a fp=0xc001ff1fe0 sp=0xc001ff1fc8 pc=0xb3d2aa
runtime.goexit()
	C:/dev/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc001ff1fe8 sp=0xc001ff1fe0 pc=0x9c5101
created by github.com/SnellerInc/sneller/vm.SplitInput
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/vm/table.go:123 +0x185

goroutine 13 [runnable]:
github.com/klauspost/compress/huff0.(*bitReaderShifted).close(0xc00220f650?)
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/huff0/bitreader.go:233 +0xcc fp=0xc00220f4d8 sp=0xc00220f4d0 pc=0xccc48c
github.com/klauspost/compress/huff0.(*Decoder).Decompress4X(0xc00220f890, {0xc00216a000, 0x1ab89?, 0x286b}, {0xc002034123, 0x20ec, 0x1ab35})
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/huff0/decompress_amd64.go:139 +0x657 fp=0xc00220f720 sp=0xc00220f4d8 pc=0xcd3417
github.com/klauspost/compress/zstd.(*blockDec).decodeLiterals(0xc000132a20, {0xc0020340cb?, 0x0?, 0x0?}, 0xc00013cdd0)
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:462 +0x787 fp=0xc00220f8e0 sp=0xc00220f720 pc=0xcd67c7
github.com/klauspost/compress/zstd.(*blockDec).decodeCompressed(0xc000132a20, 0xc00013cdd0)
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:485 +0x38 fp=0xc00220f938 sp=0xc00220f8e0 pc=0xcd6fb8
github.com/klauspost/compress/zstd.(*blockDec).decodeBuf(0xc000132a20, 0xc00013cdd0)
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:262 +0x2db fp=0xc00220f9f0 sp=0xc00220f938 pc=0xcd5d1b
github.com/klauspost/compress/zstd.(*frameDec).runDecoder(0xc00013cd80, {0x1f9ff3d0000?, 0x0, 0xa18acc?}, 0xc000132a20)
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/framedec.go:369 +0x196 fp=0xc00220fa60 sp=0xc00220f9f0 pc=0xcde036
github.com/klauspost/compress/zstd.(*Decoder).DecodeAll(0xc00016c000, {0xc00202a000, 0x24c58, 0x24c58}, {0x1f9ff3d0000, 0x0, 0x100000})
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/decoder.go:378 +0x654 fp=0xc00220fc28 sp=0xc00220fa60 pc=0xcda174
github.com/SnellerInc/sneller/compr.(*zstdDecompressor).Decompress(0xe3a440?, {0xc00202a000?, 0xc00202a000?, 0x24c58?}, {0x1f9ff3d0000, 0x100000, 0x0?})
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/compr/compression.go:101 +0x38 fp=0xc00220fc98 sp=0xc00220fc28 pc=0xce7f38
github.com/SnellerInc/sneller/ion/blockfmt.(*Decoder).Copy(0xc00009c0f0, {0xe3a4a0, 0xc00009e030}, {0xe3a440, 0xc0000a0090})
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/ion/blockfmt/compression.go:540 +0x54c fp=0xc00220fe38 sp=0xc00220fc98 pc=0xcf0a2c
main.(*readerTable).write(0xc000072050, {0x1fa7f0f89d8?, 0xc0000082a0})
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/cmd/sneller/table.go:85 +0x42c fp=0xc00220ff10 sp=0xc00220fe38 pc=0xcf7fac
main.(*readerTable).write-fm({0x1fa7f0f89d8?, 0xc0000082a0?})
	<autogenerated>:1 +0x39 fp=0xc00220ff38 sp=0xc00220ff10 pc=0xcf8679
github.com/SnellerInc/sneller/vm.SplitInput.func1(0x4)
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/vm/table.go:125 +0xba fp=0xc00220ffc8 sp=0xc00220ff38 pc=0xb3d39a
github.com/SnellerInc/sneller/vm.SplitInput.func2()
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/vm/table.go:134 +0x2a fp=0xc00220ffe0 sp=0xc00220ffc8 pc=0xb3d2aa
runtime.goexit()
	C:/dev/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00220ffe8 sp=0xc00220ffe0 pc=0x9c5101
created by github.com/SnellerInc/sneller/vm.SplitInput
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/vm/table.go:123 +0x185

goroutine 14 [runnable]:
github.com/klauspost/compress/zstd.(*bitReader).close(0xc0000a00f0?)
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/bitreader.go:133 +0xd3 fp=0xc0009e3510 sp=0xc0009e3508 pc=0xcd5453
github.com/klauspost/compress/zstd.(*sequenceDecs).decodeSyncSimple(0xc00013cb98, {0x0?, 0x0, 0xce0ab8?})
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/seqdec_amd64.go:138 +0x409 fp=0xc0009e3778 sp=0xc0009e3510 pc=0xce3869
github.com/klauspost/compress/zstd.(*sequenceDecs).decodeSync(0xc00013cb98, {0x0, 0x0, 0x0})
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/seqdec.go:207 +0x4e fp=0xc0009e38e0 sp=0xc0009e3778 pc=0xce1bae
github.com/klauspost/compress/zstd.(*blockDec).decodeCompressed(0xc000132900, 0xc00013cb90)
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:498 +0x19a fp=0xc0009e3938 sp=0xc0009e38e0 pc=0xcd711a
github.com/klauspost/compress/zstd.(*blockDec).decodeBuf(0xc000132900, 0xc00013cb90)
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:262 +0x2db fp=0xc0009e39f0 sp=0xc0009e3938 pc=0xcd5d1b
github.com/klauspost/compress/zstd.(*frameDec).runDecoder(0xc00013cb40, {0x1f9ff2d0000?, 0x0, 0xa18acc?}, 0xc000132900)
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/framedec.go:369 +0x196 fp=0xc0009e3a60 sp=0xc0009e39f0 pc=0xcde036
github.com/klauspost/compress/zstd.(*Decoder).DecodeAll(0xc00016c000, {0xc002000000, 0x28783, 0x28783}, {0x1f9ff2d0000, 0x0, 0x100000})
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/decoder.go:378 +0x654 fp=0xc0009e3c28 sp=0xc0009e3a60 pc=0xcda174
github.com/SnellerInc/sneller/compr.(*zstdDecompressor).Decompress(0xe3a440?, {0xc002000000?, 0xc002000000?, 0x28783?}, {0x1f9ff2d0000, 0x100000, 0xc000062830?})
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/compr/compression.go:101 +0x38 fp=0xc0009e3c98 sp=0xc0009e3c28 pc=0xce7f38
github.com/SnellerInc/sneller/ion/blockfmt.(*Decoder).Copy(0xc00009c0a0, {0xe3a4a0, 0xc00009e020}, {0xe3a440, 0xc0000a0060})
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/ion/blockfmt/compression.go:540 +0x54c fp=0xc0009e3e38 sp=0xc0009e3c98 pc=0xcf0a2c
main.(*readerTable).write(0xc000072050, {0x1fa7f0f89d8?, 0xc0000082d0})
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/cmd/sneller/table.go:85 +0x42c fp=0xc0009e3f10 sp=0xc0009e3e38 pc=0xcf7fac
main.(*readerTable).write-fm({0x1fa7f0f89d8?, 0xc0000082d0?})
	<autogenerated>:1 +0x39 fp=0xc0009e3f38 sp=0xc0009e3f10 pc=0xcf8679
github.com/SnellerInc/sneller/vm.SplitInput.func1(0x5)
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/vm/table.go:125 +0xba fp=0xc0009e3fc8 sp=0xc0009e3f38 pc=0xb3d39a
github.com/SnellerInc/sneller/vm.SplitInput.func2()
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/vm/table.go:134 +0x2a fp=0xc0009e3fe0 sp=0xc0009e3fc8 pc=0xb3d2aa
runtime.goexit()
	C:/dev/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0009e3fe8 sp=0xc0009e3fe0 pc=0x9c5101
created by github.com/SnellerInc/sneller/vm.SplitInput
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/vm/table.go:123 +0x185

goroutine 15 [runnable]:
github.com/klauspost/compress/zstd.(*bitReader).close(0xc0021c0000?)
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/bitreader.go:133 +0xd3 fp=0xc00220b510 sp=0xc00220b508 pc=0xcd5453
github.com/klauspost/compress/zstd.(*sequenceDecs).decodeSyncSimple(0xc00013d018, {0x0?, 0x0, 0xce0ab8?})
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/seqdec_amd64.go:138 +0x409 fp=0xc00220b778 sp=0xc00220b510 pc=0xce3869
github.com/klauspost/compress/zstd.(*sequenceDecs).decodeSync(0xc00013d018, {0x0, 0x0, 0x0})
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/seqdec.go:207 +0x4e fp=0xc00220b8e0 sp=0xc00220b778 pc=0xce1bae
github.com/klauspost/compress/zstd.(*blockDec).decodeCompressed(0xc000132b40, 0xc00013d010)
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:498 +0x19a fp=0xc00220b938 sp=0xc00220b8e0 pc=0xcd711a
github.com/klauspost/compress/zstd.(*blockDec).decodeBuf(0xc000132b40, 0xc00013d010)
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:262 +0x2db fp=0xc00220b9f0 sp=0xc00220b938 pc=0xcd5d1b
github.com/klauspost/compress/zstd.(*frameDec).runDecoder(0xc00013cfc0, {0x1f9ff4d0000?, 0x0, 0xa18acc?}, 0xc000132b40)
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/framedec.go:369 +0x196 fp=0xc00220ba60 sp=0xc00220b9f0 pc=0xcde036
github.com/klauspost/compress/zstd.(*Decoder).DecodeAll(0xc00016c000, {0xc002050000, 0x219e5, 0x219e5}, {0x1f9ff4d0000, 0x0, 0x100000})
	C:/Users/mattn/go/pkg/mod/github.com/klauspost/[email protected]/zstd/decoder.go:378 +0x654 fp=0xc00220bc28 sp=0xc00220ba60 pc=0xcda174
github.com/SnellerInc/sneller/compr.(*zstdDecompressor).Decompress(0xe3a440?, {0xc002050000?, 0xc002050000?, 0x219e5?}, {0x1f9ff4d0000, 0x100000, 0x0?})
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/compr/compression.go:101 +0x38 fp=0xc00220bc98 sp=0xc00220bc28 pc=0xce7f38
github.com/SnellerInc/sneller/ion/blockfmt.(*Decoder).Copy(0xc00009c140, {0xe3a4a0, 0xc00009e040}, {0xe3a440, 0xc0000a00c0})
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/ion/blockfmt/compression.go:540 +0x54c fp=0xc00220be38 sp=0xc00220bc98 pc=0xcf0a2c
main.(*readerTable).write(0xc000072050, {0x1fa7f0f89d8?, 0xc000008300})
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/cmd/sneller/table.go:85 +0x42c fp=0xc00220bf10 sp=0xc00220be38 pc=0xcf7fac
main.(*readerTable).write-fm({0x1fa7f0f89d8?, 0xc000008300?})
	<autogenerated>:1 +0x39 fp=0xc00220bf38 sp=0xc00220bf10 pc=0xcf8679
github.com/SnellerInc/sneller/vm.SplitInput.func1(0x6)
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/vm/table.go:125 +0xba fp=0xc00220bfc8 sp=0xc00220bf38 pc=0xb3d39a
github.com/SnellerInc/sneller/vm.SplitInput.func2()
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/vm/table.go:134 +0x2a fp=0xc00220bfe0 sp=0xc00220bfc8 pc=0xb3d2aa
runtime.goexit()
	C:/dev/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00220bfe8 sp=0xc00220bfe0 pc=0x9c5101
created by github.com/SnellerInc/sneller/vm.SplitInput
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/vm/table.go:123 +0x185

goroutine 16 [runnable]:
runtime.gopark(0xc0009b1a10?, 0x9aa355?, 0xe0?, 0x73?, 0x9fb8bc?)
	C:/dev/go/src/runtime/proc.go:363 +0xd6 fp=0xc0009b19c0 sp=0xc0009b19a0 pc=0x99ad36
runtime.goparkunlock(...)
	C:/dev/go/src/runtime/proc.go:369
runtime.semacquire1(0xc000005344, 0x1?, 0x3, 0x1)
	C:/dev/go/src/runtime/sema.go:144 +0x20f fp=0xc0009b1a28 sp=0xc0009b19c0 pc=0x9aa0af
sync.runtime_SemacquireMutex(0x0?, 0x80?, 0x9dbd69?)
	C:/dev/go/src/runtime/sema.go:71 +0x25 fp=0xc0009b1a58 sp=0xc0009b1a28 pc=0x9c0b25
sync.(*Mutex).lockSlow(0xc000005340)
	C:/dev/go/src/sync/mutex.go:164 +0x165 fp=0xc0009b1aa8 sp=0xc0009b1a58 pc=0x9dbc05
sync.(*Mutex).Lock(...)
	C:/dev/go/src/sync/mutex.go:83
internal/poll.(*FD).Pread(0x96d608?, {0xc002072000, 0x22ec5?, 0x22ec5}, 0x54e4c)
	C:/dev/go/src/internal/poll/fd_windows.go:542 +0x165 fp=0xc0009b1b90 sp=0xc0009b1aa8 pc=0xa187a5
os.(*File).pread(...)
	C:/dev/go/src/os/file_posix.go:40
os.(*File).ReadAt(0xc00000a060, {0xc002072000?, 0xc0009b1c50?, 0xcefab1?}, 0x10907a0?)
	C:/dev/go/src/os/file.go:136 +0x193 fp=0xc0009b1c18 sp=0xc0009b1b90 pc=0xa1b193
io.(*SectionReader).Read(0xc00017a000, {0xc002072000?, 0x22ec5?, 0x977a5c?})
	C:/dev/go/src/io/io.go:518 +0x54 fp=0xc0009b1c50 sp=0xc0009b1c18 pc=0x9f50f4
io.ReadAtLeast({0xe3a440, 0xc00017a000}, {0xc002072000, 0x22ec5, 0x22ec5}, 0x22ec5)
	C:/dev/go/src/io/io.go:331 +0x9a fp=0xc0009b1c98 sp=0xc0009b1c50 pc=0x9f4a7a
io.ReadFull(...)
	C:/dev/go/src/io/io.go:350
github.com/SnellerInc/sneller/ion/blockfmt.(*Decoder).Copy(0xc000072370, {0xe3a4a0, 0xc000062020}, {0xe3a440, 0xc00017a000})
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/ion/blockfmt/compression.go:533 +0x4ed fp=0xc0009b1e38 sp=0xc0009b1c98 pc=0xcf09cd
main.(*readerTable).write(0xc000072050, {0x1fa7f0f89d8?, 0xc000008330})
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/cmd/sneller/table.go:85 +0x42c fp=0xc0009b1f10 sp=0xc0009b1e38 pc=0xcf7fac
main.(*readerTable).write-fm({0x1fa7f0f89d8?, 0xc000008330?})
	<autogenerated>:1 +0x39 fp=0xc0009b1f38 sp=0xc0009b1f10 pc=0xcf8679
github.com/SnellerInc/sneller/vm.SplitInput.func1(0x7)
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/vm/table.go:125 +0xba fp=0xc0009b1fc8 sp=0xc0009b1f38 pc=0xb3d39a
github.com/SnellerInc/sneller/vm.SplitInput.func2()
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/vm/table.go:134 +0x2a fp=0xc0009b1fe0 sp=0xc0009b1fc8 pc=0xb3d2aa
runtime.goexit()
	C:/dev/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0009b1fe8 sp=0xc0009b1fe0 pc=0x9c5101
created by github.com/SnellerInc/sneller/vm.SplitInput
	C:/Users/mattn/go/src/github.com/SnellerInc/sneller/vm/table.go:123 +0x185

Simple SQL parser

gosqlparser gosqlparser is a simple SQL parser. Installation As simple as: go get github.com/krasun/gosqlparser Usage ... Supported Statements CREATE

38 Dec 21, 2022

Simple SQL escape and format for golang

sqlstring Simple SQL escape and format Escaping sql values //Format sql := sqlstring.Format("select * from users where name=? and age=? limit ?,?", "t

9 Sep 4, 2022

MSSQLDecryptor - Small and simple tool for decrypting encrypted views, functions and stored procdeures in Microsoft SQL Server

MSSQLDecryptor small and simple decryptor for encrypted objects in Microsoft SQL

3 Mar 7, 2022

# github.com/SnellerInc/sneller/usock
usock/conn_other.go:28:7: Implemented redeclared in this block
        usock/conn.go:32:7: other declaration of Implemented
usock/conn_other.go:34:6: Fd redeclared in this block

opened by shumintao 1

go install sneller -> syntax error

Hello

The Quick test drive https://github.com/SnellerInc/sneller#quick-test-drive is not working:

root@perf3 ~ # go install github.com/SnellerInc/sneller/cmd/sneller@latest
go: downloading github.com/SnellerInc/sneller v0.0.0-20220525035318-e886ff071693
go: downloading golang.org/x/sys v0.0.0-20211019181941-9d821ace8654
go: downloading golang.org/x/exp v0.0.0-20220428152302-39d4317da171
go: downloading github.com/dchest/siphash v1.2.3
go: downloading github.com/klauspost/compress v1.15.4
go: downloading golang.org/x/crypto v0.0.0-20220321153916-2c7772ba3064
go: downloading sigs.k8s.io/yaml v1.3.0
go: downloading gopkg.in/yaml.v2 v2.4.0
# github.com/SnellerInc/sneller/heap
go/pkg/mod/github.com/!sneller!inc/[email protected]/heap/heap.go:22:14: syntax error: unexpected [, expecting (
go/pkg/mod/github.com/!sneller!inc/[email protected]/heap/heap.go:22:58: method has multiple receivers
go/pkg/mod/github.com/!sneller!inc/[email protected]/heap/heap.go:24:2: syntax error: non-declaration statement outside function body
go/pkg/mod/github.com/!sneller!inc/[email protected]/heap/heap.go:31:14: syntax error: unexpected [, expecting (
go/pkg/mod/github.com/!sneller!inc/[email protected]/heap/heap.go:31:48: method has multiple receivers
go/pkg/mod/github.com/!sneller!inc/[email protected]/heap/heap.go:32:16: syntax error: unexpected newline, expecting type
go/pkg/mod/github.com/!sneller!inc/[email protected]/heap/heap.go:33:2: syntax error: non-declaration statement outside function body
go/pkg/mod/github.com/!sneller!inc/[email protected]/heap/heap.go:43:15: syntax error: unexpected [, expecting (
go/pkg/mod/github.com/!sneller!inc/[email protected]/heap/heap.go:43:57: method has multiple receivers
go/pkg/mod/github.com/!sneller!inc/[email protected]/heap/heap.go:45:2: syntax error: non-declaration statement outside function body
go/pkg/mod/github.com/!sneller!inc/[email protected]/heap/heap.go:45:2: too many errors
note: module requires Go 1.18
# golang.org/x/exp/constraints
go/pkg/mod/golang.org/x/[email protected]/constraints/constraints.go:13:2: syntax error: unexpected ~, expecting method or interface name
go/pkg/mod/golang.org/x/[email protected]/constraints/constraints.go:20:2: syntax error: unexpected ~, expecting method or interface name
go/pkg/mod/golang.org/x/[email protected]/constraints/constraints.go:27:9: syntax error: unexpected |, expecting semicolon or newline or }
go/pkg/mod/golang.org/x/[email protected]/constraints/constraints.go:34:2: syntax error: unexpected ~, expecting method or interface name
go/pkg/mod/golang.org/x/[email protected]/constraints/constraints.go:41:2: syntax error: unexpected ~, expecting method or interface name
go/pkg/mod/golang.org/x/[email protected]/constraints/constraints.go:49:10: syntax error: unexpected |, expecting semicolon or newline or }
note: module requires Go 1.18

root@perf3 ~ # cat /etc/*release*
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=21.10
DISTRIB_CODENAME=impish
DISTRIB_DESCRIPTION="Ubuntu 21.10"
PRETTY_NAME="Ubuntu 21.10"

and

root@perf3 ~ # go version
go version go1.17 linux/amd64

(the default one which can be installed by apt install golang-go.

opened by sanikolaev 1

Vectorized SQL for JSON at scale: fast, simple, schemaless

Related tags

Overview

Vectorized SQL for JSON at scale: fast, simple, schemaless

Build from source

AVX-512 support

Quick test drive

simple count

search/filter (notice SQL string operations such as LIKE/ILIKE on a nested field repo.name)

standard SQL aggregations/grouping

query custom payloads (see payload.pull_request.created_at only for type = 'PullRequestEvent' rows)

specialized operators like TIME_BUCKET

combine multiple queries

Performance

Spin up stack locally

Spin up sneller stack in the cloud

Documentation

Explore further

Development

Contribute

You might also like...

Simple SQL parser

Simple SQL escape and format for golang

MSSQLDecryptor - Small and simple tool for decrypting encrypted views, functions and stored procdeures in Microsoft SQL Server

Querycrate - A simple library that loads and keeps SQL queries from files

A simple auditor of SQL databases.

Go package for sharding databases ( Supports every ORM or raw SQL )

Prep finds all SQL statements in a Go package and instruments db connection with prepared statements

pggen - generate type safe Go methods from Postgres SQL queries

🐳 A most popular sql audit platform for mysql

Comments

panic on Windows

Can't compile and run on Macpro M1 ?

go install sneller -> syntax error

Owner

write APIs using direct SQL queries with no hassle, let's rethink about SQL

Parses a file and associate SQL queries to a map. Useful for separating SQL from code logic

Go-sql-reader - Go utility to read the externalised sql with predefined tags

CLI tool that can execute SQL queries on CSV, LTSV, JSON and TBLN. Can output to various formats.

Lightweight SQL database written in Go for prototyping and playing with text (CSV, JSON) data

Run SQL queries against JSON, CSV, Excel, Parquet, and more.

Dumpling is a fast, easy-to-use tool written by Go for dumping data from the database(MySQL, TiDB...) to local/cloud(S3, GCP...) in multifarious formats(SQL, CSV...).

Additions to Go's database/sql for super fast performance and convenience.

SQL API is designed to be able to run queries on databases without any configuration by simple HTTP call.

Simple SQL table fuzzing

search/filter (notice SQL string operations such as LIKE/ILIKE on a nested field `repo.name`)

query custom payloads (see `payload.pull_request.created_at` only for `type = 'PullRequestEvent'` rows)

specialized operators like `TIME_BUCKET`