ArcticDB - an embeddable columnar database written in Go

Overview


Go Reference Go Report Card Build

This project is still in its infancy, consider it not production-ready, probably has various consistency and correctness problems and all API will change!

ArcticDB is an embeddable columnar database written in Go. It features semi-structured schemas (could also be described as typed wide-columns), and uses Apache Parquet for storage, and Apache Arrow at query time. Building on top of Apache Arrow, ArcticDB provides a query builder and various optimizers (it reminds of DataFrame-like APIs).

ArcticDB is optimized for use cases where the majority of interactions are writes, and when data is queried, a lot of data is queried at once (our use case at Polar Signals can be broadly described as Observability and specifically for Parca). It could also be described as a wide-column columnar database.

Read the annoucement blog post to learn about what made us create it: https://www.polarsignals.com/blog/posts/2022/05/04/introducing-arcticdb/

Design choices

ArcticDB was specifically built for Observability workloads. This resulted in several characteristics that make it unique in its combination.

Table Of Contents:

Columnar layout

Observability data is most useful when highly dimensional and those dimensions can be searched and aggregated by efficiently. Contrary to many relational databases like (MySQL, PostgreSQL, CockroachDB, TiDB, etc.) that store data all data belonging to a single row together, in a columnar layout all data of the same column in a table is available in one contiguous chunk of data, making it very efficient to scan and more importantly, only the data truly necessary for a query is loaded in the first place. ArcticDB uses Apache Parquet for storage, and Apache Arrow at query time. Apache Parquet is used for storage to make use of its efficient encodings to save on memory and disk space. Apache Arrow is used at query time as a foundation to vectorize the query execution.

Dynamic Columns

While columnar databases already exist, most require a static schema, however, Observability workloads differ in that their schemas are not static, meaning not all columns are pre-defined. On the other hand, wide column databases also already exist, but typically are not strictly typed, and most wide-column databases are row-based databases, not columnar databases.

Take a Prometheus time-series for example. Prometheus time-series are uniquely identified by the combination of their label-sets:

http_requests_total{path="/api/v1/users", code="200"} 12

This model does not map well into a static schema, as label-names cannot be known upfront. The most suitable data-type some columnar databases have to offer is a map, however, maps have the same problems as row-based databases, where all values of a map in a row are stored together, unable to exploit the advantages of a columnar layout. An ArcticDB schema can define a column to be dynamic, causing a column to be created on the fly when a new label-name is seen.

An ArcticDB schema for Prometheus could look like this:

package arcticprometheus

import (
	"github.com/polarsignals/arcticdb/dynparquet"
	"github.com/segmentio/parquet-go"
)

func Schema() *dynparquet.Schema {
	return dynparquet.NewSchema(
		"prometheus",
		[]dynparquet.ColumnDefinition{{
			Name:          "labels",
			StorageLayout: parquet.Encoded(parquet.Optional(parquet.String()), &parquet.RLEDictionary),
			Dynamic:       true,
		}, {
			Name:          "timestamp",
			StorageLayout: parquet.Int(64),
			Dynamic:       false,
		}, {
			Name:          "value",
			StorageLayout: parquet.Leaf(parquet.DoubleType),
			Dynamic:       false,
		}},
		[]dynparquet.SortingColumn{
			dynparquet.NullsFirst(dynparquet.Ascending("labels")),
			dynparquet.Ascending("timestamp"),
		},
	)
}

Note: We are aware that Prometheus uses double-delta encoding for timestamps and XOR encoding for values. This schema is purely an example to highlight the dynamic columns feature.

With this schema, all rows are expected to have a timestamp and a value but can vary in their columns prefixed with labels.. In this schema all dynamically created columns are still Dictionary and run-length encoded and must be of type string.

Immutable & Sorted

There are only writes and reads. All data is immutable and sorted. Having all data sorted allows ArcticDB to avoid maintaining an index per column, and still serve queries with low latency.

To maintain global sorting ArcticDB requires all inserts to be sorted if they contain multiple rows. Combined with immutability, global sorting of all data can be maintained at a reasonable cost. To optimize throughput, it is preferable to perform inserts in as large batches as possible. ArcticDB maintains inserted data in batches of a configurable amount of rows (by default 8192), called a Granule. To directly jump to data needed for a query, ArcticDB maintains a sparse index of Granules. The sparse index is small enough to fully reside in memory, it is currently implemented as a b-tree of Granules.

Sparse index of Granules

At insert time, ArcticDB splits the inserted rows into the appropriate Granule according to their lower and upper bound, to maintain global sorting. Once a Granule exceeds the configured amount, the Granule is split into N new Granules depending.

Split of Granule

Under the hood, Granules are a list of sorted Parts, and only if a query requires it are all parts merged into a sorted stream using a direct k-way merge using a min-heap. An example of an operation that requires the whole Granule to be read as a single sorted stream are the aforementioned Granule splits.

A Granule is organized in Parts

Snapshot isolation

ArcticDB has snapshot isolation, however, it comes with a few caveats that should be well understood. It does not have read-after-write consistency as the intended use is for users reading data that are not the same as the entity writing data to it. To see new data the user re-runs a query. Choosing to trade-off read-after-write consistency allows for mechanisms to increase throughput significantly. ArcticDB releases write transactions in batches. It essentially only ensures write atomicity and that writes are not torn when reading. Since data is immutable, those characteristics together result in snapshot isolation.

More concretely, arcticDB maintains a watermark indicating that all transactions equal and lower to the watermark are safe to be read. Only write transactions obtain a new transaction ID, while reads use the transaction ID of the watermark to identify data that is safe to be read. The watermark is only increased when strictly monotonic, consecutive transactions have finished. This means that a low write transaction can block higher write transactions to become available to be read. To ensure progress is made, write transactions have a timeout.

This mechanism inspired by a mix of Google Spanner, Google Percolator and Highly Available Transactions.

Transactions are released in batches indicated by the watermark

Roadmap

  • Persistence: ArcticDB is currently fully in-memory.

Acknowledgments

ArcticDB stands on the shoulders of giants. Shout out to Segment for creating the incredible parquet-go library as well as InfluxData for starting and various contributors after them working on Go support for Apache Arrow.

Issues
  • Persist data

    Persist data

    Currently, once the configured size of data is reached the active-append table-block is swapped out for a new one and the old one is thrown away. We of course want to persist data in some way. Since we already keep the data in parquet format in memory, it would be great to write that out and memory map it.

    opened by brancz 17
  • Persist data

    Persist data

    Hi, all! I open this as a draft PR because some points need to be refined and improved after discussion, since this is a feature involving several changes. The current implementation uses a block file for each table, where blocks are appended in a log-like fashion. This should be better than storing a single table block on a separate file, because we avoid the cost of creating/opening file each time. There is another point I had to address: when syncing a block to disk, there could be some parts in the block which have not been already committed (by increasing the watermark). It would have been natural to wait for all the txns of the block to be aligned with the watermark to start writing the block to disk, but I decided to follow another approach. I used a pendingWritersWg WaitGroup inside the TableBlock object to track the number of goroutines trying to perform an Insert() operation on the block. Before writing the block to disk we wait on the following wait groups:

    block.pendingWritersWg.Wait()
    block.wg.Wait()
    

    The first wait ensures that all goroutines ended the execution of the Insert() method, so that we can consequently wait for all compactions to finish on the second WaitGroup, because we are sure no one will trigger other compaction operations for that block. At this point, the block will no more get modified and, even if there are pending write transactions, all of them should be committed successfully (it is just a matter of waiting for the watermark to be incremented). So, instead of waiting, I implemented the possibility to iterate on all parts, even those which are not aligned with the watermarks, in order to speed up disk writing.

    Additional points I think should be addressed:

    • I think we should add the possibility to enable/disable persistence when creating a TableConfig, and maybe, we could decide to rotate the BlockFile when a configurable max size is reached for a file.

    • Because now, we store data on disk, the user may also want to configure a root directory where to place all block files.

    • We should add a Close() method to the Table object, in order to do all the necessary clean-up operations when closing a table (wait for all pending blocks to be correctly written to disk and close the current active block file).

    • Moreover, when testing, we should also allow the ActiveMemorySize to be modified. Actually the limit is to high (512MB), so no disk operation will be triggered.

    opened by ostafen 16
  • Sync doesn't guarantee to see the last write

    Sync doesn't guarantee to see the last write

    While trying to add back the write benchmark I realized Sync doesn't guarantee to see the last write even if it has already returned.

    see https://github.com/polarsignals/frostdb/pull/111

    I'm guessing this is a bug, but may be that's not a given guarantee.

    opened by cyriltovena 7
  • Fixes write  benchmarks

    Fixes write benchmarks

    This fixes the write benchmarks with new change in the code and some wrong expectations.

    However running it seems to show that we are missing data when running insert in parallel but not when we run sequentially.

    opened by cyriltovena 7
  • table: handle non existent columns

    table: handle non existent columns

    Co-authored-by: @metalmatze

    While testing system-wide (https://github.com/javierhonduco/parca-agent/tree/system-wide-v2), trying to select a cgroup_id in Parca's UI resulted in the following Panic:

    panic: arrow/array: number of columns/fields mismatch
    
    goroutine 16824 [running]:
    github.com/apache/arrow/go/v8/arrow/array.NewRecord(0xc007a8e960, {0xc00e9cee40, 0x0, 0xc00e9ceee0?}, 0x0)
    	/home/javierhonduco/go/pkg/mod/github.com/apache/arrow/go/[email protected]/arrow/array/record.go:149 +0x173
    github.com/polarsignals/frostdb/pqarrow.contiguousParquetRowGroupToArrowRecord({0x4209b08, 0xc0089ced20}, {0x41fc6f8, 0x5e4c638}, {0x7f6bb853a458, 0xc023f5c228}, 0xc007a8e960, {0x0, 0x0?}, {0xc012cd1400, ...})
    	/home/javierhonduco/go/pkg/mod/github.com/polarsignals/[email protected]/pqarrow/arrow.go:204 +0x60a
    github.com/polarsignals/frostdb/pqarrow.ParquetRowGroupToArrowRecord({0x4209b08?, 0xc0089ced20?}, {0x41fc6f8?, 0x5e4c638?}, {0x7f6bb853a458?, 0xc023f5c228?}, 0x0?, {0x0?, 0x0?}, {0xc012cd1400, ...})
    	/home/javierhonduco/go/pkg/mod/github.com/polarsignals/[email protected]/pqarrow/arrow.go:97 +0xa6
    github.com/polarsignals/frostdb.(*Table).Iterator(0xc0000ac280, {0x4209b08, 0xc0089ced20}, 0xc006d3a730?, {0x41fc6f8, 0x5e4c638}, 0xc007a8e960, {0xc012cd13c0, 0x1, 0x1}, ...)
    	/home/javierhonduco/go/pkg/mod/github.com/polarsignals/[email protected]/table.go:360 +0x61b
    github.com/polarsignals/frostdb/query/physicalplan.(*TableScan).Execute.func1(0xc00e9cf101?)
    	/home/javierhonduco/go/pkg/mod/github.com/polarsignals/[email protected]/query/physicalplan/physicalplan.go:86 +0x225
    github.com/polarsignals/frostdb.(*Table).View(0xc0134e88c0?, 0x374e4d5?)
    	/home/javierhonduco/go/pkg/mod/github.com/polarsignals/[email protected]/table.go:284 +0x29
    github.com/polarsignals/frostdb/query/physicalplan.(*TableScan).Execute(0xc00f40a5a0, {0x4209b08?, 0xc0089ced20}, {0x41fc6f8?, 0x5e4c638})
    	/home/javierhonduco/go/pkg/mod/github.com/polarsignals/[email protected]/query/physicalplan/physicalplan.go:73 +0x131
    github.com/polarsignals/frostdb/query/physicalplan.(*OutputPlan).Execute(...)
    	/home/javierhonduco/go/pkg/mod/github.com/polarsignals/[email protected]/query/physicalplan/physicalplan.go:58
    github.com/polarsignals/frostdb/query.LocalQueryBuilder.Execute({{0x41fc6f8?, 0x5e4c638?}, {0xc0134e8840?}}, {0x4209b08, 0xc0089ced20}, 0xc012cd1310)
    	/home/javierhonduco/go/pkg/mod/github.com/polarsignals/[email protected]/query/engine.go:111 +0x115
    github.com/parca-dev/parca/pkg/query.(*ColumnQueryAPI).Values(0xc0000ac4b0, {0x4209b08, 0xc0089ced20}, 0xc012cd1160?)
    	/home/javierhonduco/code/parca/pkg/query/columnquery.go:127 +0x1a8
    github.com/parca-dev/parca/gen/proto/go/parca/query/v1alpha1._QueryService_Values_Handler.func1({0x4209b08, 0xc0089ced20}, {0x357c360?, 0xc00115ce40})
    	/home/javierhonduco/code/parca/gen/proto/go/parca/query/v1alpha1/query_vtproto.pb.go:271 +0x78
    github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors.UnaryServerInterceptor.func1({0x4209b08, 0xc0089ced20}, {0x357c360, 0xc00115ce40}, 0x0?, 0xc01e14c300)
    	/home/javierhonduco/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/[email protected]/interceptors/server.go:22 +0x21e
    github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x4209b08?, 0xc0089ced20?}, {0x357c360?, 0xc00115ce40?})
    	/home/javierhonduco/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
    github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1({0x4209b08, 0xc0089ced20}, {0x357c360, 0xc00115ce40}, 0x0?, 0xc00f40a4a0)
    	/home/javierhonduco/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/server_metrics.go:107 +0x87
    github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x4209b08?, 0xc0089ced20?}, {0x357c360?, 0xc00115ce40?})
    	/home/javierhonduco/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
    go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1({0x4209b08, 0xc0089ce810}, {0x357c360, 0xc00115ce40}, 0xc00f40a440, 0xc00f40a4c0)
    	/home/javierhonduco/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/[email protected]/interceptor.go:325 +0x664
    github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x4209b08?, 0xc0089ce810?}, {0x357c360?, 0xc00115ce40?})
    	/home/javierhonduco/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
    github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1({0x4209b08, 0xc0089ce810}, {0x357c360, 0xc00115ce40}, 0xc0089a2af0?, 0x30ac120?)
    	/home/javierhonduco/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:34 +0xbf
    github.com/parca-dev/parca/gen/proto/go/parca/query/v1alpha1._QueryService_Values_Handler({0x358c700?, 0xc0000ac4b0}, {0x4209b08, 0xc0089ce810}, 0xc00115ccc0, 0xc0000d6f90)
    	/home/javierhonduco/code/parca/gen/proto/go/parca/query/v1alpha1/query_vtproto.pb.go:273 +0x138
    google.golang.org/grpc.(*Server).processUnaryRPC(0xc001466fc0, {0x42181c0, 0xc01385e1b0}, 0xc0200c5320, 0xc0002c39b0, 0x5dec3f8, 0x0)
    	/home/javierhonduco/go/pkg/mod/google.golang.org/[email protected]/server.go:1283 +0xcfd
    google.golang.org/grpc.(*Server).handleStream(0xc001466fc0, {0x42181c0, 0xc01385e1b0}, 0xc0200c5320, 0x0)
    	/home/javierhonduco/go/pkg/mod/google.golang.org/[email protected]/server.go:1620 +0xa1b
    google.golang.org/grpc.(*Server).serveStreams.func1.2()
    	/home/javierhonduco/go/pkg/mod/google.golang.org/[email protected]/server.go:922 +0x98
    created by google.golang.org/grpc.(*Server).serveStreams.func1
    	/home/javierhonduco/go/pkg/mod/google.golang.org/[email protected]/server.go:920 +0x28a
    

    We believe that this is due to the filter not being able to find any column with this name.

    Test plan

    [[email protected] frostdb]$ go test .
    ok  	github.com/polarsignals/frostdb	(cached)
    

    With Parca Agent rebased off system-wide:

    image
    opened by javierhonduco 6
  • Test_Table_Concurrency and Test_Table_GranuleSplit fail with a low probability

    Test_Table_Concurrency and Test_Table_GranuleSplit fail with a low probability

    OS

    $ sw_vers
    ProductName:	macOS
    ProductVersion:	11.6.4
    BuildVersion:	20G417
    

    Go version

    $ go version
    go version go1.18 darwin/amd64
    

    revision

    latest(ccf34f7bbb98fa1b6af19a0198b81f7b4cd1441e)

    Output

    $ go test -count 300 -run Test_Table_Concurrency -timeout 20h
    --- FAIL: Test_Table_Concurrency (27.61s)
        --- FAIL: Test_Table_Concurrency/8192 (6.71s)
            table_test.go:516:
                	Error Trace:	table_test.go:516
                	Error:      	Not equal:
                	            	expected: 8000
                	            	actual  : 7990
                	Test:       	Test_Table_Concurrency/8192
    --- FAIL: Test_Table_Concurrency (26.70s)
        --- FAIL: Test_Table_Concurrency/8192 (6.65s)
            table_test.go:516:
                	Error Trace:	table_test.go:516
                	Error:      	Not equal:
                	            	expected: 8000
                	            	actual  : 7990
                	Test:       	Test_Table_Concurrency/8192
    FAIL
    exit status 1
    FAIL	github.com/polarsignals/arcticdb	8061.410s
    
    opened by ohkinozomu 6
  • part: added least func

    part: added least func

    Move least of a part into a function. Lifted out all the calls in table where we fetched the least to use this helper function.

    Also changed add part calls in addparttogranule to use the internal form that takes the least row so it doesn't need to be called again.

    This addresses #86

    opened by thorfour 5
  • Fix for concurrency race

    Fix for concurrency race

    If we start a new transaction, reads that started after the last committed write, but before the compaction wont be able to find those commited reads. Instead copy the watermark as the tx id to allow those reads to be able to read all the committed data.

    This is should be the fix for #71

    opened by thorfour 5
  • Add test cases in TestParquetNodeToType

    Add test cases in TestParquetNodeToType

    This may be redundant, but it gives us an understanding of FrostDB's behavior and prevents changes like https://github.com/polarsignals/frostdb/pull/100.

    Reference: https://github.com/segmentio/parquet-go/blob/edd371b528ffa7d9b637160d4fbd3b88f82698fa/type.go

    Interestingly, JSON causes stack overflow, but I haven't looked into that yet.

    === RUN   TestParquetNodeToType
    runtime: goroutine stack exceeds 1000000000-byte limit
    runtime: sp=0xc0201f8388 stack=[0xc0201f8000, 0xc0401f8000]
    fatal error: stack overflow
    
    runtime stack:
    runtime.throw({0x121ef33?, 0x1385f40?})
            /usr/local/Cellar/go/1.18.3/libexec/src/runtime/panic.go:992 +0x71
    runtime.newstack()
            /usr/local/Cellar/go/1.18.3/libexec/src/runtime/stack.go:1101 +0x5cc
    runtime.morestack()
            /usr/local/Cellar/go/1.18.3/libexec/src/runtime/asm_amd64.s:547 +0x8b
    
    goroutine 35 [running]:
    github.com/segmentio/parquet-go.(*jsonType).String(0x13e79b0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x2d fp=0xc0201f8398 sp=0xc0201f8390 pc=0x11a4ead
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f83b0 sp=0xc0201f8398 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f83c8 sp=0xc0201f83b0 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f83e0 sp=0xc0201f83c8 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f83f8 sp=0xc0201f83e0 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8410 sp=0xc0201f83f8 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8428 sp=0xc0201f8410 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8440 sp=0xc0201f8428 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8458 sp=0xc0201f8440 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8470 sp=0xc0201f8458 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8488 sp=0xc0201f8470 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f84a0 sp=0xc0201f8488 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f84b8 sp=0xc0201f84a0 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f84d0 sp=0xc0201f84b8 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f84e8 sp=0xc0201f84d0 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8500 sp=0xc0201f84e8 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8518 sp=0xc0201f8500 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]-0733add4c2cb/type.go:944 +0x19 fp=0xc0201f8530 sp=0xc0201f8518 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8548 sp=0xc0201f8530 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8560 sp=0xc0201f8548 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8578 sp=0xc0201f8560 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8590 sp=0xc0201f8578 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f85a8 sp=0xc0201f8590 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f85c0 sp=0xc0201f85a8 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f85d8 sp=0xc0201f85c0 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f85f0 sp=0xc0201f85d8 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8608 sp=0xc0201f85f0 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8620 sp=0xc0201f8608 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8638 sp=0xc0201f8620 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8650 sp=0xc0201f8638 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8668 sp=0xc0201f8650 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8680 sp=0xc0201f8668 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8698 sp=0xc0201f8680 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f86b0 sp=0xc0201f8698 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f86c8 sp=0xc0201f86b0 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f86e0 sp=0xc0201f86c8 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f86f8 sp=0xc0201f86e0 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8710 sp=0xc0201f86f8 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8728 sp=0xc0201f8710 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8740 sp=0xc0201f8728 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8758 sp=0xc0201f8740 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8770 sp=0xc0201f8758 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8788 sp=0xc0201f8770 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f87a0 sp=0xc0201f8788 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f87b8 sp=0xc0201f87a0 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f87d0 sp=0xc0201f87b8 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f87e8 sp=0xc0201f87d0 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8800 sp=0xc0201f87e8 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8818 sp=0xc0201f8800 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8830 sp=0xc0201f8818 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8848 sp=0xc0201f8830 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8860 sp=0xc0201f8848 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8878 sp=0xc0201f8860 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8890 sp=0xc0201f8878 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f88a8 sp=0xc0201f8890 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f88c0 sp=0xc0201f88a8 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f88d8 sp=0xc0201f88c0 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f88f0 sp=0xc0201f88d8 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8908 sp=0xc0201f88f0 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8920 sp=0xc0201f8908 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8938 sp=0xc0201f8920 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8950 sp=0xc0201f8938 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8968 sp=0xc0201f8950 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8980 sp=0xc0201f8968 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8998 sp=0xc0201f8980 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f89b0 sp=0xc0201f8998 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f89c8 sp=0xc0201f89b0 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f89e0 sp=0xc0201f89c8 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f89f8 sp=0xc0201f89e0 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8a10 sp=0xc0201f89f8 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8a28 sp=0xc0201f8a10 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8a40 sp=0xc0201f8a28 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8a58 sp=0xc0201f8a40 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8a70 sp=0xc0201f8a58 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8a88 sp=0xc0201f8a70 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8aa0 sp=0xc0201f8a88 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8ab8 sp=0xc0201f8aa0 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8ad0 sp=0xc0201f8ab8 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8ae8 sp=0xc0201f8ad0 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8b00 sp=0xc0201f8ae8 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8b18 sp=0xc0201f8b00 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8b30 sp=0xc0201f8b18 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8b48 sp=0xc0201f8b30 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8b60 sp=0xc0201f8b48 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8b78 sp=0xc0201f8b60 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8b90 sp=0xc0201f8b78 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8ba8 sp=0xc0201f8b90 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8bc0 sp=0xc0201f8ba8 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8bd8 sp=0xc0201f8bc0 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8bf0 sp=0xc0201f8bd8 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8c08 sp=0xc0201f8bf0 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8c20 sp=0xc0201f8c08 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8c38 sp=0xc0201f8c20 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8c50 sp=0xc0201f8c38 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8c68 sp=0xc0201f8c50 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8c80 sp=0xc0201f8c68 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8c98 sp=0xc0201f8c80 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8cb0 sp=0xc0201f8c98 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8cc8 sp=0xc0201f8cb0 pc=0x11a4e99
    github.com/segmentio/parquet-go.(*jsonType).String(0x0?)
            /Users/nozomu/go/pkg/mod/github.com/segmentio/[email protected]/type.go:944 +0x19 fp=0xc0201f8ce0 sp=0xc0201f8cc8 pc=0x11a4e99
    ...additional frames elided...
    created by testing.(*T).Run
            /usr/local/Cellar/go/1.18.3/libexec/src/testing/testing.go:1486 +0x35f
    
    goroutine 1 [chan receive]:
    testing.(*T).Run(0xc000105860, {0x1221433?, 0x1274f2ba9f308?}, 0x12315d8)
            /usr/local/Cellar/go/1.18.3/libexec/src/testing/testing.go:1487 +0x37a
    testing.runTests.func1(0xc0001155c0?)
            /usr/local/Cellar/go/1.18.3/libexec/src/testing/testing.go:1839 +0x6e
    testing.tRunner(0xc000105860, 0xc000177cd8)
            /usr/local/Cellar/go/1.18.3/libexec/src/testing/testing.go:1439 +0x102
    testing.runTests(0xc0001355e0?, {0x13acc20, 0x1, 0x1}, {0x1455a68?, 0x40?, 0x13b7da0?})
            /usr/local/Cellar/go/1.18.3/libexec/src/testing/testing.go:1837 +0x457
    testing.(*M).Run(0xc0001355e0)
            /usr/local/Cellar/go/1.18.3/libexec/src/testing/testing.go:1719 +0x5d9
    main.main()
            _testmain.go:47 +0x1aa
    FAIL    github.com/polarsignals/frostdb/pqarrow/convert 2.168s
    FAIL
    
    opened by ohkinozomu 4
  • TableReader returns ArrowSchema for request

    TableReader returns ArrowSchema for request

    It's helpful to return the computed schema for a request. The TableReader now has an ArrowSchema method that under the hood uses the same ParquetRowGroupToArrowRecord function that's used to generate the Arrow Records that now additionally returns the schema belonging to those records.

    opened by metalmatze 4
  • Possible problem inside function `splitGranule`

    Possible problem inside function `splitGranule`

    Hi, all, I noticed that if I increase the number of concurrent writers inside function Test_Table_Concurrency(), I get the following error:

    "failed to delete granule during split"
    

    Thinking about it, this could be due to the following. After we reach the following statement

    // set the newGranules pointer, so new writes will propogate into these new granules	
    granule.newGranules = granules
    

    inside splitGranule(), we propagate new inserts also to child granules. Then, under an high concurrent pressure, compaction could be triggered also for some of the several child granules, even before they were inserted in the index. So, when the control recheases this line (in the context of the child granule)

    deleted := index.Delete(granule)
    if deleted == nil {
    	level.Error(t.logger).Log("msg", "failed to delete granule during split")
    }
    

    we will not find it inside the index. If we simply ignore the error log, we could end with a situation, where some granules are not deleted properly from the index. I think that a possible solution to this could the disabling compaction for newly created granules until we are sure we inserted them inside the index. So, something like:

    
    // we disable compaction for new granules before allowing new insert to be propagated to them
    for _, childGranule := range granules {
            childGranule.metadata.pruned.Store(1)
    }
    
    // we restore the possibility to trigger compaction after we exited the function
    defer func() {
          for _, childGranule := range granules {
    	      childGranule.metadata.pruned.Store(0)
          }
    }()
    
    // set the newGranules pointer, so new writes will propogate into these new granules
    granule.newGranules = granules
    
    ...
    

    If this makes sense for you, I will submit a PR with the fix

    opened by ostafen 4
  • Cold start

    Cold start

    Allows us to open up a database against a bucket storage and start querying data with the database reconstructing table definitions from the parquet file schemas.

    Implements schema functionality to generate a schemapb.Definition from a parquet file.

    opened by thorfour 0
  • Support read-only frostDB instances

    Support read-only frostDB instances

    To allow us to test out the performance of our query paths that utilize block storage without impacting the current read/write path; we should support read only instances of frostDB. Where these can be pointed at LTS and support queries directly.

    opened by thorfour 0
  • Update TableReader interface to return (*Schema,error)?

    Update TableReader interface to return (*Schema,error)?

    It might make sense to update the interface to return the error too? https://github.com/polarsignals/frostdb/blob/b2cc5b02f387b5a9a8da72cb20e01deaf13c3d3e/query/logicalplan/logicalplan.go#L133

    Or should this error really just be handled outside the interface? Basically opening that as a question to everyone else as I'm not 100% sure right now.

    opened by metalmatze 2
  • Wrap database logger with database name

    Wrap database logger with database name

    It would be great to pass down the logger into the database and automatically add the database name as key-value pair: https://github.com/polarsignals/frostdb/blob/b2cc5b02f387b5a9a8da72cb20e01deaf13c3d3e/db.go#L238

    Something like the following would be great:

    log.WithPrefix(s.logger,"db",name)
    
    enhancement good first issue 
    opened by metalmatze 1
  • Fix dynamic schema scan

    Fix dynamic schema scan

    What if we had schema scans actually build a schema instead of returning a single column of name?

    This allows us to do things like schema scans with dynamic projections since our query behavior is based on column names. Otherwise to support something like this we'd need to have each expr know if it's apart of a schema scan to behave differently.

    opened by thorfour 3
Owner
Polar Signals
Polar Signals
BuntDB is an embeddable, in-memory key/value database for Go with custom indexing and geospatial support

BuntDB is a low-level, in-memory, key/value store in pure Go. It persists to disk, is ACID compliant, and uses locking for multiple readers and a sing

Josh Baker 3.8k Aug 7, 2022
A simple, fast, embeddable, persistent key/value store written in pure Go. It supports fully serializable transactions and many data structures such as list, set, sorted set.

NutsDB English | 简体中文 NutsDB is a simple, fast, embeddable and persistent key/value store written in pure Go. It supports fully serializable transacti

徐佳军 2.3k Aug 8, 2022
Owl is a db manager platform,committed to standardizing the data, index in the database and operations to the database, to avoid risks and failures.

Owl is a db manager platform,committed to standardizing the data, index in the database and operations to the database, to avoid risks and failures. capabilities which owl provides include Process approval、sql Audit、sql execute and execute as crontab、data backup and recover .

null 35 Jun 17, 2022
This is a simple graph database in SQLite, inspired by "SQLite as a document database".

About This is a simple graph database in SQLite, inspired by "SQLite as a document database". Structure The schema consists of just two structures: No

Denis Papathanasiou 1.1k Aug 1, 2022
Hard Disk Database based on a former database

Hard Disk Database based on a former database

null 0 Nov 1, 2021
Simple key value database that use json files to store the database

KValDB Simple key value database that use json files to store the database, the key and the respective value. This simple database have two gRPC metho

Francisco Santos 0 Nov 13, 2021
Beerus-DB: a database operation framework, currently only supports Mysql, Use [go-sql-driver/mysql] to do database connection and basic operations

Beerus-DB · Beerus-DB is a database operation framework, currently only supports Mysql, Use [go-sql-driver/mysql] to do database connection and basic

Beerus 6 Mar 5, 2022
Nipo is a powerful, fast, multi-thread, clustered and in-memory key-value database, with ability to configure token and acl on commands and key-regexes written by GO

Welcome to NIPO Nipo is a powerful, fast, multi-thread, clustered and in-memory key-value database, with ability to configure token and acl on command

Morteza Bashsiz 16 Jun 13, 2022
🤔 A minimize Time Series Database, written from scratch as a learning project.

mandodb ?? A minimize Time Series Database, written from scratch as a learning project. 时序数据库(TSDB: Time Series Database)大多数时候都是为了满足监控场景的需求,这里先介绍两个概念:

dongdong 484 Aug 2, 2022
GalaxyDB is a hobbyist key-value database written in Go.

GalaxyDB GalaxyDB is a hobbyist key-value database written in Go Author: Andrew N ([email protected]) Features Data is stored via keys Operations Grafana

nijmeh.xyz 3 Mar 30, 2022
A lightweight document-oriented NoSQL database written in pure Golang.

Lightweight document-oriented NoSQL Database ???? English | ???? 简体中文 | ???? Spanish CloverDB is a lightweight NoSQL database designed for being simpl

Stefano Scafiti 234 Jul 25, 2022
An embedded key/value database for Go.

bbolt bbolt is a fork of Ben Johnson's Bolt key/value store. The purpose of this fork is to provide the Go community with an active maintenance and de

etcd-io 5.7k Jul 31, 2022
CockroachDB - the open source, cloud-native distributed SQL database.

CockroachDB is a cloud-native SQL database for building global, scalable cloud services that survive disasters. What is CockroachDB? Docs Quickstart C

CockroachDB 25.3k Aug 7, 2022
ACID key-value database.

Coffer Simply ACID* key-value database. At the medium or even low latency it tries to provide greater throughput without losing the ACID properties of

Eduard 30 Jul 3, 2022
A decentralized, trusted, high performance, SQL database with blockchain features

中文简介 CovenantSQL(CQL) is a Byzantine Fault Tolerant relational database built on SQLite: ServerLess: Free, High Availabile, Auto Sync Database Service

CovenantSQL 1.4k Aug 2, 2022
Native GraphQL Database with graph backend

The Only Native GraphQL Database With A Graph Backend. Dgraph is a horizontally scalable and distributed GraphQL database with a graph backend. It pro

Dgraph 18.3k Aug 4, 2022
EliasDB a graph-based database.

EliasDB EliasDB is a graph-based database which aims to provide a lightweight solution for projects which want to store their data as a graph. Feature

Matthias Ladkau 930 Jul 24, 2022
LevelDB key/value database in Go.

This is an implementation of the LevelDB key/value database in the Go programming language. Installation go get github.com/syndtr/goleveldb/leveldb R

Suryandaru Triandana 5.3k Jul 31, 2022
immudb - world’s fastest immutable database

immudb Note: The master branch is the joint point for all ongoing development efforts. Thus it may be in an unstable state and should not be used in p

CodeNotary 7.7k Jul 28, 2022