Go-clickhouse - ClickHouse client for Go

Overview

ClickHouse client for Go 1.18+

build workflow PkgGoDev Documentation

This client uses native protocol to communicate with ClickHouse server. It requires Go 1.18+ in order to use generics. This is not a database/sql driver, but the API is similar.

Main features are:

  • Native protocol support.
  • database/sql-like API.
  • Bun-like query builder.
  • Selecting into scalars, structs, maps, slices of maps/structs/scalars.
  • Efficient inserts.
  • Array(*) including nested arrays.
  • Enums and LowCardinality(String).
  • Migrations.

Not supported:

  • Nullable types.

Resources:

Example

A basic example:

package main

import (
	"context"
	"fmt"
	"time"

	"github.com/uptrace/go-clickhouse/ch"
	"github.com/uptrace/go-clickhouse/extra/chdebug"
)

type Model struct {
	ch.CHModel `ch:"partition:toYYYYMM(time)"`

	ID   uint64
	Text string    `ch:",lc"`
	Time time.Time `ch:",pk,default:now()"`
}

func main() {
	ctx := context.Background()

	db := ch.Connect(ch.WithDatabase("test"))
	db.AddQueryHook(chdebug.NewQueryHook(chdebug.WithVerbose(true)))

	if err := db.Ping(ctx); err != nil {
		panic(err)
	}

	var num int
	if err := db.QueryRowContext(ctx, "SELECT 123").Scan(&num); err != nil {
		panic(err)
	}
	fmt.Println(num)

	if err := db.ResetModel(ctx, (*Model)(nil)); err != nil {
		panic(err)
	}

	src := &Model{ID: 1, Text: "hello"}
	if _, err := db.NewInsert().Model(src).Column("id", "text").Exec(ctx); err != nil {
		panic(err)
	}

	dest := new(Model)
	if err := db.NewSelect().Model(dest).Where("id = ?", src.ID).Limit(1).Scan(ctx); err != nil {
		panic(err)
	}
	fmt.Println(dest)
}
Issues
  • Concerns about failed migrations being marked as applied

    Concerns about failed migrations being marked as applied

    First, thank you for this package and specifically the migration logic (from Bun). That said, I have some concerns about failed migrations being marked as applied. I understand that this allows the rollback logic to be run, but I'm not sure if the desirability of that outweighs the risks.

    The major risk I'm worried about is another "up" being performed before the "rollback" and the failed migration being skipped. CI/CD pipelines and developers not aware of this behavior may try to apply the migrations again, like in the case of a timeout or transient error, and be surprised by the behavior. I would argue it violates the principle of least surprise and that the defacto standard is for a following "up" to try applying the failed migration again, not skipping it.

    opened by ryanrolds 5
  • what is the scaleWithCpu meaning?

    what is the scaleWithCpu meaning?

    I find the scaleWithCpu define the relationship between batchsize and cpu cores. If i have a 2 core machine, then the batchsize is 2000, every 2000 span start a goroutine and save the data. If i have a 8 core machine, then the batchsize is 8000, every 8000 span start a goroutine and save the data.

    I do not understand why it is helpful?

    opened by zdyj3170101136 4
  • Add option that makes migrator only mark migrations applied when migration successful

    Add option that makes migrator only mark migrations applied when migration successful

    Adds an option that instructs the Migrator to mark migrations applied on success, not on both success and failure.

    Something to note, the group is being returned in more of the error paths than before, I think this is consistent with the intent that despite a migration failing some migrations were still applied and that information should be provided to the caller.

    Resolves: https://github.com/uptrace/go-clickhouse/issues/25

    opened by ryanrolds 2
  • import package error

    import package error

    go: finding module for package github.com/uptrace/go-clickhouse/ch go: found github.com/uptrace/go-clickhouse/ch in github.com/uptrace/go-clickhouse/ch v0.0.0-20220308124651-82a5d8d72ef7 go: downloading github.com/uptrace/go-clickhouse/chdebug v0.0.0-00010101000000-000000000000 github.com/uptrace/go-clickhouse/ch tested by github.com/uptrace/go-clickhouse/ch.test imports github.com/uptrace/go-clickhouse/chdebug: github.com/uptrace/go-clickhouse/[email protected]: invalid version: unknown revision 000000000000

    opened by An-uking 2
  • chore(deps): bump actions/setup-go from 2 to 3

    chore(deps): bump actions/setup-go from 2 to 3

    Bumps actions/setup-go from 2 to 3.

    Release notes

    Sourced from actions/setup-go's releases.

    v3.0.0

    What's Changed

    Breaking Changes

    With the update to Node 16, all scripts will now be run with Node 16 rather than Node 12.

    This new major release removes the stable input, so there is no need to specify additional input to use pre-release versions. This release also corrects the pre-release versions syntax to satisfy the SemVer notation (1.18.0-beta1 -> 1.18.0-beta.1, 1.18.0-rc1 -> 1.18.0-rc.1).

    steps:
      - uses: actions/[email protected]
      - uses: actions/[email protected]
        with:
          go-version: '1.18.0-rc.1' 
      - run: go version
    

    Add check-latest input

    In scope of this release we add the check-latest input. If check-latest is set to true, the action first checks if the cached version is the latest one. If the locally cached version is not the most up-to-date, a Go version will then be downloaded from go-versions repository. By default check-latest is set to false. Example of usage:

    steps:
      - uses: actions/[email protected]
      - uses: actions/[email protected]
        with:
          go-version: '1.16'
          check-latest: true
      - run: go version
    

    Moreover, we updated @actions/core from 1.2.6 to 1.6.0

    v2.1.5

    In scope of this release we updated matchers.json to improve the problem matcher pattern. For more information please refer to this pull request

    v2.1.4

    What's Changed

    New Contributors

    Full Changelog: https://github.com/actions/setup-go/compare/v2.1.3...v2.1.4

    v2.1.3

    • Updated communication with runner to use environment files rather then workflow commands

    v2.1.2

    This release includes vendored licenses for this action's npm dependencies.

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies github_actions 
    opened by dependabot[bot] 1
  • chore(deps): bump github.com/stretchr/testify from 1.7.2 to 1.7.3

    chore(deps): bump github.com/stretchr/testify from 1.7.2 to 1.7.3

    Bumps github.com/stretchr/testify from 1.7.2 to 1.7.3.

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies go 
    opened by dependabot[bot] 0
  • chore(deps): bump github.com/pierrec/lz4/v4 from 4.1.14 to 4.1.15

    chore(deps): bump github.com/pierrec/lz4/v4 from 4.1.14 to 4.1.15

    ⚠️ Dependabot is rebasing this PR ⚠️

    Rebasing might not happen immediately, so don't worry if this takes some time.

    Note: if you make any changes to this PR yourself, they will take precedence over the rebase.


    Bumps github.com/pierrec/lz4/v4 from 4.1.14 to 4.1.15.

    Commits
    • c3eaa1e Merge pull request #188 from leehinman/flush_fix
    • 8582079 remove state change after Flush
    • 885e900 Merge pull request #184 from lizthegrey/patch-1
    • ec13d8c do not auto-Close() Writer after ReadFrom()/Copy()
    • c6ed913 Merge pull request #179 from bodgit/doc
    • 9dece95 Fix Reader.Reset() docs
    • b8cae7c Merge pull request #172 from greatroar/decoder-fixes
    • db11e26 internal/lz4block: Handle zero offsets in amd64 shortcut
    • 7c0f44d internal/lz4block: Detect match with no space for offset
    • 24303cf internal/lz4block: arm64 decoder improvements
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies go 
    opened by dependabot[bot] 0
  • chore(deps): bump wagoid/commitlint-github-action from 4 to 5

    chore(deps): bump wagoid/commitlint-github-action from 4 to 5

    Bumps wagoid/commitlint-github-action from 4 to 5.

    Changelog

    Sourced from wagoid/commitlint-github-action's changelog.

    4.1.15 (2022-06-16)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies github_actions 
    opened by dependabot[bot] 0
  • chore(deps): bump github.com/stretchr/testify from 1.7.1 to 1.7.2

    chore(deps): bump github.com/stretchr/testify from 1.7.1 to 1.7.2

    Bumps github.com/stretchr/testify from 1.7.1 to 1.7.2.

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies go 
    opened by dependabot[bot] 0
  • chore(deps): bump golangci/golangci-lint-action from 3.1.0 to 3.2.0

    chore(deps): bump golangci/golangci-lint-action from 3.1.0 to 3.2.0

    Bumps golangci/golangci-lint-action from 3.1.0 to 3.2.0.

    Release notes

    Sourced from golangci/golangci-lint-action's releases.

    v3.2.0

    What's Changed

    misc

    dependencies

    New Contributors

    ... (truncated)

    Commits
    • 537aa19 Expire cache periodically to avoid unbounded size (#466)
    • f70e52d build(deps): bump @​actions/core from 1.6.0 to 1.8.0 (#468)
    • a304692 build(deps-dev): bump @​typescript-eslint/eslint-plugin (#469)
    • eeca7c5 build(deps-dev): bump eslint from 8.14.0 to 8.15.0 (#467)
    • dfbcd2a build(deps): bump github/codeql-action from 1 to 2 (#459)
    • 4421331 build(deps-dev): bump @​typescript-eslint/parser from 5.20.0 to 5.22.0 (#464)
    • 5e6c1bb build(deps-dev): bump typescript from 4.6.3 to 4.6.4 (#461)
    • 44eba43 build(deps-dev): bump @​typescript-eslint/eslint-plugin (#460)
    • 358a5e3 build(deps-dev): bump @​typescript-eslint/eslint-plugin (#457)
    • b9c65a5 build(deps-dev): bump @​typescript-eslint/parser from 5.19.0 to 5.20.0 (#455)
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies github_actions 
    opened by dependabot[bot] 0
  • chore(deps): bump github.com/stretchr/testify from 1.7.0 to 1.7.1

    chore(deps): bump github.com/stretchr/testify from 1.7.0 to 1.7.1

    Bumps github.com/stretchr/testify from 1.7.0 to 1.7.1.

    Commits
    • 083ff1c Fixed didPanic to now detect panic(nil).
    • 1e36bfe Use cross Go version compatible build tag syntax
    • e798dc2 Add docs on 1.17 build tags
    • 83198c2 assert: guard CanConvert call in backward compatible wrapper
    • 087b655 assert: allow comparing time.Time
    • 7bcf74e fix msgAndArgs forwarding
    • c29de71 add tests for correct msgAndArgs forwarding
    • f87e2b2 Update builds
    • ab6dc32 fix linting errors in /assert package
    • edff5a0 fix funtion name
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies go 
    opened by dependabot[bot] 0
  • Support Object/JSON type?

    Support Object/JSON type?

    New data type Object(<schema_format>), which supports storing of semi-structured data (for now JSON only). Data is written to such types as string. Then all paths are extracted according to format of semi-structured data and written as separate columns in most optimal types, that can store all their values. Those columns can be queried by names that match paths in source data. E.g data.key1.key2 or with cast operator data.key1.key2::Int64.

    https://clickhouse.com/docs/en/whats-new/changelog/#experimental-feature-1

    enhancement 
    opened by franchb 0
Releases(v0.2.8)
Owner
Uptrace
All-in-one tool to optimize performance and monitor errors & logs
Uptrace
ClickHouse http proxy and load balancer

chproxy English | 简体中文 Chproxy, is an http proxy and load balancer for ClickHouse database. It provides the following features: May proxy requests to

Vertamedia 922 Jun 21, 2022
Collects many small inserts to ClickHouse and send in big inserts

ClickHouse-Bulk Simple Yandex ClickHouse insert collector. It collect requests and send to ClickHouse servers. Installation Download binary for you pl

Nikolay Pavlovich 364 Jun 20, 2022
Bifrost ---- 面向生产环境的 MySQL 同步到Redis,MongoDB,ClickHouse,MySQL等服务的异构中间件

Bifrost ---- 面向生产环境的 MySQL 同步到Redis,ClickHouse等服务的异构中间件 English 漫威里的彩虹桥可以将 雷神 送到 阿斯加德 和 地球 而这个 Bifrost 可以将 你 MySQL 里的数据 全量 , 实时的同步到 : Redis MongoDB Cl

brokerCAP 1.2k Jun 30, 2022
support clickhouse

Remote storage adapter This is a write adapter that receives samples via Prometheus's remote write protocol and stores them in Graphite, InfluxDB, cli

weetime 28 Jun 21, 2022
Jaeger ClickHouse storage plugin implementation

Jaeger ClickHouse Jaeger ClickHouse gRPC storage plugin. This is WIP and it is based on https://github.com/bobrik/jaeger/tree/ivan/clickhouse/plugin/s

Pavol Loffay 1 Feb 15, 2022
Clickhouse support for GORM

clickhouse Clickhouse support for GORM Quick Start package main import ( "fmt" "github.com/sweetpotato0/clickhouse" "gorm.io/gorm" ) // User

null 1 Oct 24, 2021
Distributed tracing using OpenTelemetry and ClickHouse

Distributed tracing backend using OpenTelemetry and ClickHouse Uptrace is a dist

Uptrace 729 Jun 21, 2022
Mogo: a lightweight browser-based logs analytics and logs search platform for some datasource(ClickHouse, MySQL, etc.)

mogo Mogo is a lightweight browser-based logs analytics and logs search platform

Shimo HQ 670 Jun 22, 2022
Devcloud-go provides a sql-driver for mysql which named devspore driver and a redis client which named devspore client,

Devcloud-go Devcloud-go provides a sql-driver for mysql which named devspore driver and a redis client which named devspore client, you can use them w

HUAWEI CLOUD 11 Jun 9, 2022
Cross-platform client for PostgreSQL databases

pgweb Web-based PostgreSQL database browser written in Go. Overview Pgweb is a web-based database browser for PostgreSQL, written in Go and works on O

Dan Sosedoff 7.3k Jun 23, 2022
Go client for Redis

Redigo Redigo is a Go client for the Redis database. Features A Print-like API with support for all Redis commands. Pipelining, including pipelined tr

null 9.2k Jun 24, 2022
Type-safe Redis client for Golang

Redis client for Golang Join Discord to ask questions. Documentation Reference Examples RealWorld example app Ecosystem Redis Mock. Distributed Locks.

null 14.7k Jun 26, 2022
Go client for AMQP 0.9.1

Go RabbitMQ Client Library This is an AMQP 0.9.1 client with RabbitMQ extensions in Go. Project Maturity This project has been used in production syst

Sean Treadway 4.4k Jun 20, 2022
Interactive client for PostgreSQL and MySQL

dblab Interactive client for PostgreSQL and MySQL. Overview dblab is a fast and lightweight interactive terminal based UI application for PostgreSQL a

Daniel Omar Vergara Pérez 196 Jun 21, 2022
Cross-platform client for PostgreSQL databases

pgweb Web-based PostgreSQL database browser written in Go. Overview Pgweb is a web-based database browser for PostgreSQL, written in Go and works on O

Dan Sosedoff 7.3k Jun 21, 2022
[mirror] the database client and tools for the Go vulnerability database

The Go Vulnerability Database golang.org/x/vulndb This repository is a prototype of the Go Vulnerability Database. Read the Draft Design. Neither the

Go 43 May 28, 2022
REST based Redis client built on top of Upstash REST API

An HTTP/REST based Redis client built on top of Upstash REST API.

Andreas Thomas 4 Jun 2, 2022
Migration tool for ksqlDB, which uses the ksqldb-go client.

ksqldb-migrate Migration tool for ksqlDB, which uses the ksqldb-go client.

Thomas Meitz 1 Jun 14, 2022