Go-clickhouse - ClickHouse client for Go

Overview

ClickHouse client for Go 1.18+

build workflow PkgGoDev Documentation

This client uses native protocol to communicate with ClickHouse server. It requires Go 1.18+ in order to use generics. This is not a database/sql driver, but the API is similar.

Main features are:

  • Native protocol support.
  • database/sql-like API.
  • Bun-like query builder.
  • Selecting into scalars, structs, maps, slices of maps/structs/scalars.
  • Efficient inserts.
  • Array(*) including nested arrays.
  • Enums and LowCardinality(String).
  • Migrations.

Not supported:

  • Nullable types.

Resources:

Example

A basic example:

package main

import (
	"context"
	"fmt"
	"time"

	"github.com/uptrace/go-clickhouse/ch"
	"github.com/uptrace/go-clickhouse/extra/chdebug"
)

type Model struct {
	ch.CHModel `ch:"partition:toYYYYMM(time)"`

	ID   uint64
	Text string    `ch:",lc"`
	Time time.Time `ch:",pk,default:now()"`
}

func main() {
	ctx := context.Background()

	db := ch.Connect(ch.WithDatabase("test"))
	db.AddQueryHook(chdebug.NewQueryHook(chdebug.WithVerbose(true)))

	if err := db.Ping(ctx); err != nil {
		panic(err)
	}

	var num int
	if err := db.QueryRowContext(ctx, "SELECT 123").Scan(&num); err != nil {
		panic(err)
	}
	fmt.Println(num)

	if err := db.ResetModel(ctx, (*Model)(nil)); err != nil {
		panic(err)
	}

	src := &Model{ID: 1, Text: "hello"}
	if _, err := db.NewInsert().Model(src).Column("id", "text").Exec(ctx); err != nil {
		panic(err)
	}

	dest := new(Model)
	if err := db.NewSelect().Model(dest).Where("id = ?", src.ID).Limit(1).Scan(ctx); err != nil {
		panic(err)
	}
	fmt.Println(dest)
}
Comments
  • Concerns about failed migrations being marked as applied

    Concerns about failed migrations being marked as applied

    First, thank you for this package and specifically the migration logic (from Bun). That said, I have some concerns about failed migrations being marked as applied. I understand that this allows the rollback logic to be run, but I'm not sure if the desirability of that outweighs the risks.

    The major risk I'm worried about is another "up" being performed before the "rollback" and the failed migration being skipped. CI/CD pipelines and developers not aware of this behavior may try to apply the migrations again, like in the case of a timeout or transient error, and be surprised by the behavior. I would argue it violates the principle of least surprise and that the defacto standard is for a following "up" to try applying the failed migration again, not skipping it.

    opened by ryanrolds 5
  • what is the scaleWithCpu meaning?

    what is the scaleWithCpu meaning?

    I find the scaleWithCpu define the relationship between batchsize and cpu cores. If i have a 2 core machine, then the batchsize is 2000, every 2000 span start a goroutine and save the data. If i have a 8 core machine, then the batchsize is 8000, every 8000 span start a goroutine and save the data.

    I do not understand why it is helpful?

    opened by zdyj3170101136 4
  • Add option that makes migrator only mark migrations applied when migration successful

    Add option that makes migrator only mark migrations applied when migration successful

    Adds an option that instructs the Migrator to mark migrations applied on success, not on both success and failure.

    Something to note, the group is being returned in more of the error paths than before, I think this is consistent with the intent that despite a migration failing some migrations were still applied and that information should be provided to the caller.

    Resolves: https://github.com/uptrace/go-clickhouse/issues/25

    opened by ryanrolds 2
  • import package error

    import package error

    go: finding module for package github.com/uptrace/go-clickhouse/ch go: found github.com/uptrace/go-clickhouse/ch in github.com/uptrace/go-clickhouse/ch v0.0.0-20220308124651-82a5d8d72ef7 go: downloading github.com/uptrace/go-clickhouse/chdebug v0.0.0-00010101000000-000000000000 github.com/uptrace/go-clickhouse/ch tested by github.com/uptrace/go-clickhouse/ch.test imports github.com/uptrace/go-clickhouse/chdebug: github.com/uptrace/go-clickhouse/[email protected]: invalid version: unknown revision 000000000000

    opened by An-uking 2
  • fix: move FINAL modifier to the right place

    fix: move FINAL modifier to the right place

    The documentation (http://devdoc.net/database/ClickhouseDocs_19.4.1.3-docs/query_language/select/) says FINAL has to be placed right after the table name, before SAMPLE: SELECT [DISTINCT] expr_list [FROM [db.]table | (subquery) | table_function] [FINAL] [SAMPLE sample_coeff]

    At the moment it's appended to the end of the query which results in invalid queries being generated when there are WHERE, ORDER BY or any other clauses used.

    opened by tony2001 1
  • Where clause does not work with byte array with zero bytes

    Where clause does not work with byte array with zero bytes

    type Data struct {
    	Str []byte
    }
    
    src, _ := hex.DecodeString("5C00CC")
    d := &Data{Str: src}
    
    dsn := "clickhouse://localhost:9000/test?sslmode=disable"
    db := ch.Connect(ch.WithDSN(dsn), ch.WithAutoCreateDatabase(true))
    
    if err := db.Ping(context.Background()); err != nil {
    	panic(err)
    }
    
    _, err := db.NewDropTable().IfExists().Model(&Data{}).Exec(context.Background())
    if err != nil {
    	panic(err)
    }
    
    _, err = db.NewCreateTable().IfNotExists().Model(&Data{}).Exec(context.Background())
    if err != nil {
    	panic(err)
    }
    
    _, err = db.NewInsert().Model(d).Exec(context.Background())
    if err != nil {
    	panic(err)
    }
    
    // works
    got1 := new(Data)
    err = db.NewSelect().Model(got1).
    	Where(fmt.Sprintf("str = unhex('%s')", hex.EncodeToString(d.Str))).
    	Scan(context.Background())
    if err != nil {
    	panic(errors.Wrap(err, "[1] cannot select"))
    }
    fmt.Println("got1:", got1)
    
    // fails
    got2 := new(Data)
    err = db.NewSelect().Model(got2).
    	Where("str = ?", d.Str).
    	Scan(context.Background())
    if err != nil {
    	panic(errors.Wrap(err, "[2] cannot select"))
    }
    fmt.Println("got2:", got2)
    
    opened by iam047801 1
  • Where clause doesn't work with bytes

    Where clause doesn't work with bytes

    Where clause works well with strings, but fails with sql.ErrNoRows error on byte array.

    Here I attach code to reproduce the issue. It panics with [2] cannot select: sql: no rows in result set error.

    type Data struct {
    	Str []byte
    }
    
    d := &Data{Str: []byte("asdf")}
    
    dsn := "clickhouse://localhost:9000/test?sslmode=disable"
    db := ch.Connect(ch.WithDSN(dsn), ch.WithAutoCreateDatabase(true))
    
    if err := db.Ping(context.Background()); err != nil {
    	panic(err)
    }
    
    _, err := db.NewDropTable().IfExists().Model(&Data{}).Exec(context.Background())
    if err != nil {
    	panic(err)
    }
    
    _, err = db.NewCreateTable().IfNotExists().Model(&Data{}).Exec(context.Background())
    if err != nil {
    	panic(err)
    }
    
    _, err = db.NewInsert().Model(d).Exec(context.Background())
    if err != nil {
    	panic(err)
    }
    
    got1 := new(Data)
    
    err = db.NewSelect().Model(got1).
    	Where(fmt.Sprintf("str = '%s'", d.Str)).
    	Scan(context.Background())
    if err != nil {
    	panic(errors.Wrap(err, "[1] cannot select"))
    }
    fmt.Println("got1:", got1)
    
    got2 := new(Data)
    
    err = db.NewSelect().Model(got2).
    	Where("str = ?", d.Str).
    	Scan(context.Background())
    if err != nil {
    	panic(errors.Wrap(err, "[2] cannot select"))
    }
    fmt.Println("got2:", got2)
    
    opened by iam047801 1
  • chore(deps): bump go.opentelemetry.io/otel/trace from 1.8.0 to 1.9.0

    chore(deps): bump go.opentelemetry.io/otel/trace from 1.8.0 to 1.9.0

    Bumps go.opentelemetry.io/otel/trace from 1.8.0 to 1.9.0.

    Changelog

    Sourced from go.opentelemetry.io/otel/trace's changelog.

    [1.9.0/0.0.3] - 2022-08-01

    Added

    • Add support for Schema Files format 1.1.x (metric "split" transform) with the new go.opentelemetry.io/otel/schema/v1.1 package. (#2999)
    • Add the go.opentelemetry.io/otel/semconv/v1.11.0 package. The package contains semantic conventions from the v1.11.0 version of the OpenTelemetry specification. (#3009)
    • Add the go.opentelemetry.io/otel/semconv/v1.12.0 package. The package contains semantic conventions from the v1.12.0 version of the OpenTelemetry specification. (#3010)
    • Add the http.method attribute to HTTP server metric from all go.opentelemetry.io/otel/semconv/* packages. (#3018)

    Fixed

    • Invalid warning for context setup being deferred in go.opentelemetry.io/otel/bridge/opentracing package. (#3029)
    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies go 
    opened by dependabot[bot] 1
  • chore(deps): bump go.opentelemetry.io/otel/trace from 1.7.0 to 1.8.0

    chore(deps): bump go.opentelemetry.io/otel/trace from 1.7.0 to 1.8.0

    Bumps go.opentelemetry.io/otel/trace from 1.7.0 to 1.8.0.

    Changelog

    Sourced from go.opentelemetry.io/otel/trace's changelog.

    [1.8.0/0.31.0] - 2022-07-08

    Added

    • Add support for opentracing.TextMap format in the Inject and Extract methods of the "go.opentelemetry.io/otel/bridge/opentracing".BridgeTracer type. (#2911)

    Changed

    • The crosslink make target has been updated to use the go.opentelemetry.io/build-tools/crosslink package. (#2886)
    • In the go.opentelemetry.io/otel/sdk/instrumentation package rename Library to Scope and alias Library as Scope (#2976)
    • Move metric no-op implementation form nonrecording to metric package. (#2866)

    Removed

    • Support for go1.16. Support is now only for go1.17 and go1.18 (#2917)

    Deprecated

    • The Library struct in the go.opentelemetry.io/otel/sdk/instrumentation package is deprecated. Use the equivalent Scope struct instead. (#2977)
    • The ReadOnlySpan.InstrumentationLibrary method from the go.opentelemetry.io/otel/sdk/trace package is deprecated. Use the equivalent ReadOnlySpan.InstrumentationScope method instead. (#2977)
    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies go 
    opened by dependabot[bot] 1
  • Fix for upping applying zero migrations

    Fix for upping applying zero migrations

    • Fixed no migrations being applied by having it range the unapplied migration, not the empty applied group
    • Upping and downing creates an empty group and adds migrations as they are applied (was already doing this for ups, applied to downs as well)
    • Renamed the group to applied and unapplied to make the purpose more clear

    Related to https://github.com/uptrace/go-clickhouse/issues/30

    opened by ryanrolds 1
  • Head of master not applying unapplied migrations

    Head of master not applying unapplied migrations

    It looks like the MigrationGroup at https://github.com/uptrace/go-clickhouse/blob/master/chmigrate/migrator.go#L150-L153, is always empty as group.Migrations isn't being set to the unapplied migrations. I assume the range should be on migrations.

    opened by ryanrolds 1
  • Go `[]byte` types mapped to `,type:LowCardinality(String)` yields `<[]uint8 value>` in query results

    Go `[]byte` types mapped to `,type:LowCardinality(String)` yields `<[]uint8 value>` in query results

    given:

    type Model struct {
    ...
      foo []byte `ch:",type:LowCardinality(String)"`
    }
    

    .. and then using the fluent table creation + query mechanism(s) of this library, the resulting data (when foo is queried) shows as <[]uint8 value> instead of the actual string-ish data that was intended for insertion.

    opened by jdef 0
  • panic: unsupported ClickHouse type=

    panic: unsupported ClickHouse type="Map(String, String)"

    My Model

    type Video struct {
    	ch.CHModel `ch:"partition:toYYYYMM(time)"`
    	ID         uint64
    	Title      string            `ch:""`
    	Time       time.Time         `ch:",pk"`
    	Meta       []uint64          `ch:"type:Array(UInt64)"`
    	Chapter    map[string]string `ch:"type:Map(String,String)"`
    }
    
    	video := database.Video{
    		ID:    1,
    		Title: "Video 1",
    		Time:  time.Now(),
    		Meta:  []uint64{10, 10, 10, 10},
    		Chapter: map[string]string{
    			"hello1": "hello",
    			"hello":  "hello",
    			"title":  "title",
    		},
    	}
    	db.NewInsert().Model(&video).Exec(ctx)
    

    panic: unsupported ClickHouse type="Map(String, String)"

    goroutine 1 [running]: github.com/uptrace/go-clickhouse/ch/chschema.goType({0xc0000a4030, 0x13}) /Users/sohelmia/go/pkg/mod/github.com/uptrace/[email protected]/ch/chschema/types.go:307 +0x6ce github.com/uptrace/go-clickhouse/ch/chschema.NewColumnFromCHType({0xc0000a4030, 0x13}, 0xc00009a1e0?) /Users/sohelmia/go/pkg/mod/github.com/uptrace/[email protected]/ch/chschema/column.go:54 +0x28 github.com/uptrace/go-clickhouse/ch/chschema.(*Block).Column(0xc00009c280, {0xc00009a1e0, 0x7}, {0xc0000a4030, 0x13}) /Users/sohelmia/go/pkg/mod/github.com/uptrace/[email protected]/ch/chschema/block.go:46 +0xd4 github.com/uptrace/go-clickhouse/ch.(*DB).readBlock.func1() /Users/sohelmia/go/pkg/mod/github.com/uptrace/[email protected]/ch/proto.go:515 +0x16e github.com/uptrace/go-clickhouse/ch/chproto.(*Reader).WithCompression(0xc00009c000, 0x1, 0x100e4a7?) /Users/sohelmia/go/pkg/mod/github.com/uptrace/[email protected]/ch/chproto/reader.go:44 +0x53 github.com/uptrace/go-clickhouse/ch.(*DB).readBlock(0xc0002c8050, 0xc00009c000, 0xc00009c280, 0x1) /Users/sohelmia/go/pkg/mod/github.com/uptrace/[email protected]/ch/proto.go:481 +0x8f github.com/uptrace/go-clickhouse/ch.(*DB).readSampleBlock(0x13c3820?, 0xc00009c000) /Users/sohelmia/go/pkg/mod/github.com/uptrace/[email protected]/ch/proto.go:382 +0x12c github.com/uptrace/go-clickhouse/ch.(*DB)._insert.func1.2(0xc000094000?) /Users/sohelmia/go/pkg/mod/github.com/uptrace/[email protected]/ch/db.go:405 +0x25 github.com/uptrace/go-clickhouse/ch/chpool.(*Conn).WithReader(0xc0000a0000, {0x131b410?, 0xc00001e0c0?}, 0x13b2c64?, 0xc00035fb50) /Users/sohelmia/go/pkg/mod/github.com/uptrace/[email protected]/ch/chpool/conn.go:70 +0x67 github.com/uptrace/go-clickhouse/ch.(*DB)._insert.func1(0xc0000a0000) /Users/sohelmia/go/pkg/mod/github.com/uptrace/[email protected]/ch/db.go:404 +0x12b github.com/uptrace/go-clickhouse/ch.(*DB)._withConn(0xc0002c8050, {0x131b410?, 0xc00001e0c0}, 0xc00035fcf8) /Users/sohelmia/go/pkg/mod/github.com/uptrace/[email protected]/ch/db.go:219 +0x1ec github.com/uptrace/go-clickhouse/ch.(*DB).withConn(0xc0002c8050, {0x131b410?, 0xc00001e0c0?}, 0x12?) /Users/sohelmia/go/pkg/mod/github.com/uptrace/[email protected]/ch/db.go:176 +0x28 github.com/uptrace/go-clickhouse/ch.(*DB)._insert(0xc0000922a0?, {0x131b410?, 0xc00001e0c0?}, {0x1000?, 0x1000?}, {0xc0000b5000?, 0x14?}, 0x1000?) /Users/sohelmia/go/pkg/mod/github.com/uptrace/[email protected]/ch/db.go:396 +0x8f github.com/uptrace/go-clickhouse/ch.(*DB).insert(0xc0002c8050, {0x131b410, 0xc00001e0c0}, {0x131afa0, 0xc0000922a0}, {0xc0000b5000, 0x46}, {0xc000092090, 0x5, 0x6}) /Users/sohelmia/go/pkg/mod/github.com/uptrace/[email protected]/ch/db.go:383 +0x145

    opened by mdsohelmia 0
  • unsupported ClickHouse type=

    unsupported ClickHouse type="Bool"

    So, trying this lib but it appears that you missed implementing the conversion from BOOLEAN field to a valid golang bool?

    Reproduction is to create a table such as:

    CREATE TABLE IF NOT EXISTS test (
        id VARCHAR,
        flag BOOLEAN
    )
    ...
    
    opened by Mendes11 0
  • Request to support UInt256 with big.Int

    Request to support UInt256 with big.Int

    It would be great if we could use structures like this:

    type Data struct {
    	Int *big.Int `ch:"type:UInt256"`
    }
    

    Currently UInt256 type is not supported and *big.Int maps to String type by default.

    enhancement 
    opened by iam047801 0
  • ch: unsupported compression method: 0x00

    ch: unsupported compression method: 0x00

    With compression switched on client parser throws the following exception:

    ch: unsupported compression method: 0x51
    ...
    ch: unsupported compression method: 0x00
    

    the hex number at the end of the error looks random

    Disabling compression leads to reading timeout issues.

    clickhouse/clickhouse-server:22.6
    

    This error doesn't happen consistently. Some queries do get through and are successfully processed.

    question 
    opened by yandooo 2
Releases(v0.2.9)
Owner
Uptrace
All-in-one tool to optimize performance and monitor errors & logs
Uptrace
ClickHouse http proxy and load balancer

chproxy English | 简体中文 Chproxy, is an http proxy and load balancer for ClickHouse database. It provides the following features: May proxy requests to

Vertamedia 1k Jan 3, 2023
Collects many small inserts to ClickHouse and send in big inserts

ClickHouse-Bulk Simple Yandex ClickHouse insert collector. It collect requests and send to ClickHouse servers. Installation Download binary for you pl

Nikolay Pavlovich 397 Dec 28, 2022
Bifrost ---- 面向生产环境的 MySQL 同步到Redis,MongoDB,ClickHouse,MySQL等服务的异构中间件

Bifrost ---- 面向生产环境的 MySQL 同步到Redis,ClickHouse等服务的异构中间件 English 漫威里的彩虹桥可以将 雷神 送到 阿斯加德 和 地球 而这个 Bifrost 可以将 你 MySQL 里的数据 全量 , 实时的同步到 : Redis MongoDB Cl

brokerCAP 1.4k Dec 30, 2022
support clickhouse

Remote storage adapter This is a write adapter that receives samples via Prometheus's remote write protocol and stores them in Graphite, InfluxDB, cli

weetime 30 Dec 7, 2022
Jaeger ClickHouse storage plugin implementation

Jaeger ClickHouse Jaeger ClickHouse gRPC storage plugin. This is WIP and it is based on https://github.com/bobrik/jaeger/tree/ivan/clickhouse/plugin/s

Pavol Loffay 1 Feb 15, 2022
Clickhouse support for GORM

clickhouse Clickhouse support for GORM Quick Start package main import ( "fmt" "github.com/sweetpotato0/clickhouse" "gorm.io/gorm" ) // User

null 1 Oct 18, 2022
Distributed tracing using OpenTelemetry and ClickHouse

Distributed tracing backend using OpenTelemetry and ClickHouse Uptrace is a dist

Uptrace 1.3k Jan 2, 2023
Mogo: a lightweight browser-based logs analytics and logs search platform for some datasource(ClickHouse, MySQL, etc.)

mogo Mogo is a lightweight browser-based logs analytics and logs search platform

Shimo HQ 986 Dec 30, 2022
Devcloud-go provides a sql-driver for mysql which named devspore driver and a redis client which named devspore client,

Devcloud-go Devcloud-go provides a sql-driver for mysql which named devspore driver and a redis client which named devspore client, you can use them w

HUAWEI CLOUD 11 Jun 9, 2022
Cross-platform client for PostgreSQL databases

pgweb Web-based PostgreSQL database browser written in Go. Overview Pgweb is a web-based database browser for PostgreSQL, written in Go and works on O

Dan Sosedoff 7.6k Dec 30, 2022
Go client for Redis

Redigo Redigo is a Go client for the Redis database. Features A Print-like API with support for all Redis commands. Pipelining, including pipelined tr

null 9.4k Dec 29, 2022
Type-safe Redis client for Golang

Redis client for Golang Join Discord to ask questions. Documentation Reference Examples RealWorld example app Ecosystem Redis Mock. Distributed Locks.

null 16.1k Dec 29, 2022
Go client for AMQP 0.9.1

Go RabbitMQ Client Library This is an AMQP 0.9.1 client with RabbitMQ extensions in Go. Project Maturity This project has been used in production syst

Sean Treadway 4.6k Jan 6, 2023
Interactive client for PostgreSQL and MySQL

dblab Interactive client for PostgreSQL and MySQL. Overview dblab is a fast and lightweight interactive terminal based UI application for PostgreSQL a

Daniel Omar Vergara Pérez 617 Jan 8, 2023
Cross-platform client for PostgreSQL databases

pgweb Web-based PostgreSQL database browser written in Go. Overview Pgweb is a web-based database browser for PostgreSQL, written in Go and works on O

Dan Sosedoff 7.6k Dec 30, 2022
[mirror] the database client and tools for the Go vulnerability database

The Go Vulnerability Database golang.org/x/vulndb This repository is a prototype of the Go Vulnerability Database. Read the Draft Design. Neither the

Go 224 Dec 29, 2022
REST based Redis client built on top of Upstash REST API

An HTTP/REST based Redis client built on top of Upstash REST API.

Andreas Thomas 5 Jul 31, 2022
Migration tool for ksqlDB, which uses the ksqldb-go client.

ksqldb-migrate Migration tool for ksqlDB, which uses the ksqldb-go client.

Thomas Meitz 2 Nov 15, 2022