Low-level ClickHouse database driver for Golang

Overview

codecov Actions Status FOSSA Status

chconn - ClickHouse low level Driver

chconn is a pure Go driver for ClickHouse that use Native protocol chconn aims to be low-level, fast, and performant.

Example Usage

package main

import (
	"context"
	"fmt"
	"os"
	"time"

	"github.com/vahid-sohrabloo/chconn/chpool"
	"github.com/vahid-sohrabloo/chconn/column"
)

func main() {
	conn, err := chpool.Connect(context.Background(), os.Getenv("DATABESE_URL"))
	if err != nil {
		panic(err)
	}

	defer conn.Close()

	_, err = conn.Exec(context.Background(), `DROP TABLE IF EXISTS example_table`)
	if err != nil {
		panic(err)
	}

	_, err = conn.Exec(context.Background(), `CREATE TABLE  example_table (
		uint64 UInt64,
		uint64_nullable Nullable(UInt64)
	) Engine=Memory`)
	if err != nil {
		panic(err)
	}

	col1 := column.NewUint64(false)
	col2 := column.NewUint64(true)
	rows := 10000000 // One hundred million rows- insert in 10 times
	numInsert := 10
	startInsert := time.Now()
	for i := 0; i < numInsert; i++ {
		col1.Reset()
		col2.Reset()
		for y := 0; y < rows; y++ {
			col1.Append(uint64(i))
			if i%2 == 0 {
				col2.AppendIsNil(false)
				col2.Append(uint64(i))
			} else {
				col2.AppendIsNil(true)
				col2.AppendEmpty()
			}
		}

		ctxInsert, cancelInsert := context.WithTimeout(context.Background(), time.Second*30)
		// insert data
		insertStmt, err := conn.Insert(ctxInsert, "INSERT INTO example_table (uint64,uint64_nullable) VALUES")
		if err != nil {
			cancelInsert()
			panic(err)
		}
		err = insertStmt.Commit(ctxInsert, col1, col2)
		if err != nil {
			cancelInsert()
			panic(err)
		}
		cancelInsert()
	}
	fmt.Println("inserted 100M rows in ", time.Since(startInsert))

	// select data
	col1Read := column.NewUint64(false)
	col2Read := column.NewUint64(true)

	ctxSelect, cancelSelect := context.WithTimeout(context.Background(), time.Second*30)
	defer cancelSelect()

	startSelect := time.Now()
	// insert data
	selectStmt, err := conn.Select(ctxSelect, "SELECT uint64,uint64_nullable FROM  example_table")
	if err != nil {
		panic(err)
	}

	// make sure close the statement after you are done with it to back it to the pool
	defer selectStmt.Close()

	// next block of data
	// for more information about block, see: https://clickhouse.com/docs/en/development/architecture/#block
	var col1Data []uint64
	var col2DataNil []uint8
	var col2Data []uint64
	for selectStmt.Next() {
		err = selectStmt.NextColumn(col1Read)
		if err != nil {
			panic(err)
		}
		col1Data = col1Data[:0]
		col1Read.ReadAll(&col1Data)

		err = selectStmt.NextColumn(col2Read)
		if err != nil {
			panic(err)
		}
		col2DataNil = col2DataNil[:0]
		col2Read.ReadAllNil(&col2DataNil)

		col2Data = col2Data[:0]
		col2Read.ReadAll(&col2Data)

	}

	// check errors
	if selectStmt.Err() != nil {
		panic(selectStmt.Err())
	}
	fmt.Println("selected 100M rows in ", time.Since(startSelect))

}
inserted 100M rows in  1.206666188s
selected 100M rows in  880.505004ms

For more information please read wiki

Features

  • Connection pool with after-connect hook for arbitrary connection setup similar to pgx (thanks @jackc)
  • Support All ClickHouse data types
  • Read and write data in column-oriendted (like ClickHouse)
  • Do not use interface{} , reflect
  • Batch select and insert
  • Full TLS connection control
  • Read raw binary data
  • Supports profile and progress
  • database url connection very like pgx (thanks @jackc)
  • Code generator for Insert
  • Support LZ4 compresstion protocol

Supported types

  • UInt8, UInt16, UInt32, UInt64, UInt128, UInt256
  • Int8, Int16, Int32, Int64, Int128, Int256
  • Date, Date32, DateTime, DateTime64
  • Decimal32, Decimal64, Decimal128, Decimal256
  • IPv4, IPv6
  • String, FixedString(N)
  • UUID
  • Array(T)
  • Enums
  • LowCardinality(T)
  • Map(K, V)
  • Tuple(T1, T2, ..., Tn)
  • Nullable(T)

TODO

  • Support ExternalTable
  • Support Clickhouse Log
  • Add code generator for select

License

FOSSA Status

Issues
Releases(v1.2.2)
Low-level Go interface to SQLite 3

zombiezen.com/go/sqlite This package provides a low-level Go interface to SQLite 3. It is a fork of crawshaw.io/sqlite that uses modernc.org/sqlite, a

Ross Light 195 Jan 21, 2022
Devcloud-go provides a sql-driver for mysql which named devspore driver and a redis client which named devspore client,

Devcloud-go Devcloud-go provides a sql-driver for mysql which named devspore driver and a redis client which named devspore client, you can use them w

HUAWEI CLOUD 10 Jan 4, 2022
Golang database driver for SQLite

go-sqlite Golang database driver for SQLite. Does not use cgo. This driver is based on pure-Go SQLite implementation (https://gitlab.com/cznic/sqlite)

glebarez 3 Jan 13, 2022
BigQuery database/sql golang driver

BigQuery SQL Driver This library is compatible with Go 1.17+ Please refer to CHA

Viant, Inc 1 Jan 5, 2022
ClickHouse http proxy and load balancer

chproxy English | 简体中文 Chproxy, is an http proxy and load balancer for ClickHouse database. It provides the following features: May proxy requests to

Vertamedia 797 Jan 19, 2022
Collects many small inserts to ClickHouse and send in big inserts

ClickHouse-Bulk Simple Yandex ClickHouse insert collector. It collect requests and send to ClickHouse servers. Installation Download binary for you pl

Nikolay Pavlovich 336 Jan 17, 2022
Bifrost ---- 面向生产环境的 MySQL 同步到Redis,MongoDB,ClickHouse,MySQL等服务的异构中间件

Bifrost ---- 面向生产环境的 MySQL 同步到Redis,ClickHouse等服务的异构中间件 English 漫威里的彩虹桥可以将 雷神 送到 阿斯加德 和 地球 而这个 Bifrost 可以将 你 MySQL 里的数据 全量 , 实时的同步到 : Redis MongoDB Cl

brokerCAP 974 Jan 16, 2022
support clickhouse

Remote storage adapter This is a write adapter that receives samples via Prometheus's remote write protocol and stores them in Graphite, InfluxDB, cli

weetime 13 Dec 28, 2021
Jaeger ClickHouse storage plugin implementation

Jaeger ClickHouse Jaeger ClickHouse gRPC storage plugin. This is WIP and it is based on https://github.com/bobrik/jaeger/tree/ivan/clickhouse/plugin/s

Pavol Loffay 0 Aug 11, 2021
Clickhouse support for GORM

clickhouse Clickhouse support for GORM Quick Start package main import ( "fmt" "github.com/sweetpotato0/clickhouse" "gorm.io/gorm" ) // User

null 1 Oct 24, 2021
Distributed tracing using OpenTelemetry and ClickHouse

Distributed tracing backend using OpenTelemetry and ClickHouse Uptrace is a dist

Uptrace 473 Jan 15, 2022
Mogo: a lightweight browser-based logs analytics and logs search platform for some datasource(ClickHouse, MySQL, etc.)

mogo Mogo is a lightweight browser-based logs analytics and logs search platform

Shimo HQ 30 Jan 18, 2022
A fully-featured AWS Athena database driver (+ athenareader https://github.com/uber/athenadriver/tree/master/athenareader)

?? athenadriver - A fully-featured AWS Athena database driver for Go ?? athenareader - A moneywise command line utililty to query athena in command li

Uber Open Source 94 Jan 1, 2022
Apache H2 Database Go Driver

Apache H2 Database Go Driver This driver is VERY experimental state NOT use for production yet Introduction Apache H2 Database is a very-low footprint

JM Robles 32 Nov 8, 2021
Attach hooks to any database/sql driver

sqlhooks Attach hooks to any database/sql driver. The purpose of sqlhooks is to provide a way to instrument your sql statements, making really easy to

Gustavo Chaín 508 Jan 23, 2022
Re-usable component for subscribing to row-level changes on a postgres database

pgreplicate Re-usable component for subscribing to row-level changes on a postgres database Development In order to run tests the postgres database ne

Crewlinker 0 Dec 21, 2021
[mirror] the database client and tools for the Go vulnerability database

The Go Vulnerability Database golang.org/x/vulndb This repository is a prototype of the Go Vulnerability Database. Read the Draft Design. Neither the

Go 23 Jan 8, 2022
The MongoDB driver for Go

The MongoDB driver for Go This fork has had a few improvements by ourselves as well as several PR's merged from the original mgo repo that are current

GlobalSign 1.9k Jan 17, 2022
go mysql driver, support distributed transaction

Go-MySQL-Driver A MySQL-Driver for Go's database/sql package Features Requirements Installation Usage DSN (Data Source Name) Password Protocol Address

Open Transaction 30 Dec 4, 2021