Mirror of Apache Calcite - Avatica Go SQL Driver

Overview

Apache Avatica/Phoenix SQL Driver

GoDoc Build Status

Apache Calcite's Avatica Go is a Go database/sql driver for the Avatica server.

Avatica is a sub-project of Apache Calcite.

Quick Start

Install using Go modules:

$ go get github.com/apache/calcite-avatica-go

The Phoenix/Avatica driver implements Go's database/sql/driver interface, so, import the database/sql package and the driver:

import "database/sql"
import _ "github.com/apache/calcite-avatica-go/v5"

db, err := sql.Open("avatica", "http://localhost:8765")

Then simply use the database connection to query some data, for example:

rows := db.Query("SELECT COUNT(*) FROM test")

For more details, see the home page.

Release notes for all published versions are available on the history page.

Issues

We do not use Github to file issues. Please create an issue on Calcite's JIRA and select avatica-go as the component.

Comments
  • Replace ExecuteRequest with ExecuteBatchRequest

    Replace ExecuteRequest with ExecuteBatchRequest

    Where writing a large amount of data, ExecuteRequest need to send multiple http requests. We hope that ExecuteBatchRequest can be used instead of ExecuteRequest to reduce the number of data transmissions as much as possible.

    opened by geange 9
  • Change UUID package to github.com/hashicorp/go-uuid

    Change UUID package to github.com/hashicorp/go-uuid

    The satori UUID package causes problems with go get and vgo when building multiple database drivers in the same Go application. This is because the satori package introduced breaking API changes and has not tagged it a new version yet.

    This change converts the connection ID generation to use the hashicorp UUID package which was already a dependency of this project, as it is a dependency of the gokrb package.

    This has the added benefit of reducing the number of total package dependencies by 1.

    Additionally, this is needed by github.com/xo/usql so that go get and vgo build both work "out of the box".

    The following is a small Go program that demonstrates that the UUIDs generated will be the same format:

    package main
    
    import (
    	"log"
    
    	guuid "github.com/google/uuid"
    	huuid "github.com/hashicorp/go-uuid"
    	suuid "github.com/satori/go.uuid"
    )
    
    func main() {
    	g, err := guuid.NewRandom()
    	if err != nil {
    		log.Fatal(err)
    	}
    
    	h, err := huuid.GenerateUUID()
    	if err != nil {
    		log.Fatal(err)
    	}
    
    	s, err := suuid.NewV4()
    	if err != nil {
    		log.Fatal(err)
    	}
    
    	log.Printf(
    		"google: %s, hashicorp: %s, satori: %s",
    		g, h, s,
    	)
    }
    
    opened by kenshaw 9
  • [CALCITE-5077] ResetSession implements driver.SessionResetter.

    [CALCITE-5077] ResetSession implements driver.SessionResetter.

    avatica sql query fails after running for a period of time on our online loki, this PR attempts to solve this "NoSuchConnectionException" problem.

    level=error ts=2022-03-31T12:51:09.356233496Z caller=table_manager.go:233 msg="error syncing tables" err="An error was encountered while processing your request: NoSuchConnectionException{connectionId='Connection not found: invalid id, closed, or expired: ccd95a16-5496-7001-f99b-1ab0e5f19e05'}"
    
    opened by liguozhong 8
  • [CALCITE-3248] add Connector implementation

    [CALCITE-3248] add Connector implementation

    What

    • adds an implementation for driver.Connector to the project.
    • allows passing in arbitary infos and headers to do all kinds of auth stuff

    Why

    • we want to use that driver to connect to an endpoint behind an auth proxy and need to add some headers.
    • Now its possible to attach other infos than the default ones to the connection to make it possible for a wrapping layer around to act accordingly
    opened by trusch 8
  • Add more nil checks in error handling.

    Add more nil checks in error handling.

    This adds the same nil checks while parsing errors as in connection.go I added a few days ago. I'm greping around to find more missing nil checks, but these are the only three related to GetMetadata() that I could find.

    opened by trusch 5
  • [CALCITE-5077] ResetSession implements driver.SessionResetter.

    [CALCITE-5077] ResetSession implements driver.SessionResetter.

    avatica sql query fails after running for a period of time on our online loki, this PR attempts to solve this "NoSuchConnectionException" problem.

    level=error ts=2022-03-31T12:51:09.356233496Z caller=table_manager.go:233 msg="error syncing tables" err="An error was encountered while processing your request: NoSuchConnectionException{connectionId='Connection not found: invalid id, closed, or expired: ccd95a16-5496-7001-f99b-1ab0e5f19e05'}"

    opened by liguozhong 4
  • Configurable connection parameters of httpclient

    Configurable connection parameters of httpclient

    when i used avatica-go, i can't change the httpclient's idleConnTimeout, it is set to 90 seconds alway, in my project, i need a longer idleConnTimeout value, so i commit this pr to enable configuration some httpclient params. The params can defined in http url params like 'maxRowsTotal' as same as before. When http params is not defines this httpclient params, they also have default value same to before. One thing i confused is, i determine the min value of these params, but i'm not sure if I want to give a limit to the maximum value, and what should be the maximum limit for each value

    opened by huangqian1985 3
  • [CALCITE-4174] avatica-go should handle complex/long URLs (josiahg)

    [CALCITE-4174] avatica-go should handle complex/long URLs (josiahg)

    Enables connecting to longer URLs - for example if a proxy is being used in front of PQS.

    Previous behavior: take any string in the URL path as the schema

    New behavior: take the string following the last / in the URL path as the schema

    opened by josiahg 3
  • [CALCITE-3264] add catch-all type for unknown types in generic adapter

    [CALCITE-3264] add catch-all type for unknown types in generic adapter

    Currently the generic adapter in the calcite-go driver panics when it sees an unknown type.

    I as a user of the driver would like to see it either just working as expected or at least see a meaningfull error.

    This PR adds a default scan type in the case that the column type is not recognized.

    opened by trusch 3
  • forward all query parameters from the dsn in the OpenConnectionRequest info object.

    forward all query parameters from the dsn in the OpenConnectionRequest info object.

    What

    • save all the query parameters from the DSN in the config object
    • map these parameters to the info object of the OpenConnectionRequest message

    Why

    This way we can supply arbitary additional parameters for usecase specific backend behavior. Currently we are using this to specify the scope of the operation (which DB in particular)

    opened by trusch 3
  • [CALCITE-5072] Index out of range when calling rows.Next()

    [CALCITE-5072] Index out of range when calling rows.Next()

    fix index ount of bound panic. https://github.com/grafana/loki/pull/5692/ [new feature] Index: loki support apache calcite avatica index storage. #5692

    I implemented the index storage plugin of apache avatic in the grafana loki community, and found that avatica-go would panic in the online environment. This PR attempts to fix this panic. Our online loki has been fixed.

    panic: runtime error: index out of range [0] with length 0 
    	/usr/local/go/src/runtime/panic.go:1038 +0x215
    github.com/apache/calcite-avatica-go/v5.(*rows).Columns(0x4cc4b7)
    	/Users/fuling/go/pkg/mod/github.com/liguozhong/calcite-avatica-go/[email protected]/rows.go:55 +0x17b
    database/sql.(*Rows).nextLocked(0xc038348000)
    	/usr/local/go/src/database/sql/sql.go:2964 +0xb0
    database/sql.(*Rows).Next.func1()
    	/usr/local/go/src/database/sql/sql.go:2945 +0x2f
    database/sql.withLock({0x32a6008, 0xc038348030}, 0xc004cb76f0)
    	/usr/local/go/src/database/sql/sql.go:3396 +0x8c
    database/sql.(*Rows).Next(0xc038348000)
    	/usr/local/go/src/database/sql/sql.go:2944 +0x6f
    github.com/grafana/loki/pkg/storage/chunk/avatica.(*StorageClient).query(0xc001024ea0, {0x32d98a8, 0xc0188ef320}, {{0xc00589f300, 0x1e}, {0xc01bc60990, 0x2b}, {0x0, 0x0, 0x0}, ...}, ...)
    

    code in loki, for rows.Next() { panic.

    case query.ValueEqual != nil:
    		querySQL = fmt.Sprintf("SELECT range, value FROM %s WHERE hash = ? AND value = ? ALLOW FILTERING",
    			query.TableName)
    		queryFunc = func() error {
    			rows, err = s.readSession.QueryContext(ctx, querySQL, query.HashValue, query.ValueEqual)
    			return err
    		}
    	}
    
    	retries := backoff.New(ctx, s.cfg.BackoffConfig)
    	err = ctx.Err()
    	for retries.Ongoing() {
    		err = s.queryInstrumentation(ctx, querySQL, queryFunc)
    		if err == nil {
    			break
    		}
    		retries.Wait()
    	}
    	if err != nil {
    		level.Error(util_log.Logger).Log("msg", "avatica QUERY fail", "sql", querySQL, "err", err)
    		return errors.WithStack(err)
    	}
    	defer rows.Close()
    	for rows.Next() {
    		b := &readBatch{}
    		err = rows.Scan(&b.rangeValue, &b.value)
    		if err != nil {
    			return errors.WithStack(err)
    		}
    		if !callback(query, b) {
    			return nil
    		}
    	}
    
    func (s *StorageClient) queryInstrumentation(ctx context.Context, query string, queryFunc func() error) error {
    	var start time.Time
    	var end time.Time
    	var err error
    	sp := ot.SpanFromContext(ctx)
    	sp.SetTag("sql", query)
    
    	defer func() {
    		statusCode := "200"
    		if err != nil {
    			level.Warn(util_log.Logger).Log("msg", "avatica query fail", "sql", query, "err", err)
    			statusCode = "500"
    			ext.Error.Set(sp, true)
    			sp.LogFields(otlog.String("event", "error"), otlog.String("message", err.Error()))
    		}
    		parts := strings.SplitN(query, " ", 2)
    		requestDuration.WithLabelValues(parts[0], statusCode).Observe(end.Sub(start).Seconds())
    	}()
    	start = time.Now()
    	err = queryFunc()
    	end = time.Now()
    	if err != nil {
    		return errors.WithStack(err)
    	}
    	return nil
    }
    
    opened by liguozhong 2
Owner
The Apache Software Foundation
The Apache Software Foundation
Qmgo - The Go driver for MongoDB. It‘s based on official mongo-go-driver but easier to use like Mgo.

Qmgo English | 简体中文 Qmgo is a Go driver for MongoDB . It is based on MongoDB official driver, but easier to use like mgo (such as the chain call). Qmg

Qiniu Cloud 970 Sep 21, 2022
Go driver for PostgreSQL over SSH. This driver can connect to postgres on a server via SSH using the local ssh-agent, password, or private-key.

pqssh Go driver for PostgreSQL over SSH. This driver can connect to postgres on a server via SSH using the local ssh-agent, password, or private-key.

mattn 49 Sep 19, 2022
Firebird RDBMS sql driver for Go (golang)

firebirdsql (Go firebird sql driver) Firebird RDBMS http://firebirdsql.org SQL driver for Go Requirements Firebird 2.5 or higher Golang 1.13 or higher

Hajime Nakagami 175 Sep 8, 2022
Microsoft ActiveX Object DataBase driver for go that using exp/sql

go-adodb Microsoft ADODB driver conforming to the built-in database/sql interface Installation This package can be installed with the go get command:

mattn 129 Sep 14, 2022
Microsoft SQL server driver written in go language

A pure Go MSSQL driver for Go's database/sql package Install Requires Go 1.8 or above. Install with go get github.com/denisenkom/go-mssqldb . Connecti

null 1.6k Sep 26, 2022
Oracle driver for Go using database/sql

go-oci8 Description Golang Oracle database driver conforming to the Go database/sql interface Installation Install Oracle full client or Instant Clien

mattn 593 Sep 20, 2022
sqlite3 driver for go using database/sql

go-sqlite3 Latest stable version is v1.14 or later not v2. NOTE: The increase to v2 was an accident. There were no major changes or features. Descript

mattn 6k Sep 20, 2022
Go Sql Server database driver.

gofreetds Go FreeTDS wrapper. Native Sql Server database driver. Features: can be used as database/sql driver handles calling stored procedures handle

minus5 106 Jan 23, 2022
Pure Go Postgres driver for database/sql

pq - A pure Go postgres driver for Go's database/sql package Install go get github.com/lib/pq Features SSL Handles bad connections for database/sql S

null 7.6k Sep 26, 2022
Attach hooks to any database/sql driver

sqlhooks Attach hooks to any database/sql driver. The purpose of sqlhooks is to provide a way to instrument your sql statements, making really easy to

Gustavo Chaín 578 Sep 15, 2022
GO DRiver for ORacle DB

Go DRiver for ORacle godror is a package which is a database/sql/driver.Driver for connecting to Oracle DB, using Anthony Tuininga's excellent OCI wra

null 390 Sep 15, 2022
PostgreSQL driver and toolkit for Go

pgx - PostgreSQL Driver and Toolkit pgx is a pure Go driver and toolkit for PostgreSQL. pgx aims to be low-level, fast, and performant, while also ena

Jack Christensen 6k Sep 22, 2022
Lightweight Golang driver for ArangoDB

Arangolite Arangolite is a lightweight ArangoDB driver for Go. It focuses on pure AQL querying. See AranGO for a more ORM-like experience. IMPORTANT:

Fabien Herfray 72 Jun 17, 2022
Go language driver for RethinkDB

RethinkDB-go - RethinkDB Driver for Go Go driver for RethinkDB Current version: v6.2.1 (RethinkDB v2.4) Please note that this version of the driver on

RethinkDB 1.6k Sep 2, 2022
goriak - Go language driver for Riak KV

goriak Current version: v3.2.1. Riak KV version: 2.0 or higher, the latest version of Riak KV is always recommended. What is goriak? goriak is a wrapp

Gustav Westling 27 Jan 23, 2022
Mongo Go Models (mgm) is a fast and simple MongoDB ODM for Go (based on official Mongo Go Driver)

Mongo Go Models Important Note: We changed package name from github.com/Kamva/mgm/v3(uppercase Kamva) to github.com/kamva/mgm/v3(lowercase kamva) in v

kamva 564 Sep 23, 2022
The MongoDB driver for Go

The MongoDB driver for Go This fork has had a few improvements by ourselves as well as several PR's merged from the original mgo repo that are current

GlobalSign 1.9k Sep 18, 2022
The Go driver for MongoDB

MongoDB Go Driver The MongoDB supported driver for Go. Requirements Installation Usage Bugs / Feature Reporting Testing / Development Continuous Integ

mongodb 6.9k Sep 26, 2022
RethinkDB-go - RethinkDB Driver for Go

Go language driver for RethinkDB

RethinkDB 1.6k Sep 26, 2022