A db proxy for distributed transaction, read write splitting and sharding! Support any language! It can be deployed as a sidecar in a pod.

Overview

DBPack

LICENSE codecov

dbpack.svg

DBPack means a database cluster tool pack. It can be deployed as a sidecar in a pod, it shields complex basic logic, so that business development does not need to rely on a specific SDK, simplifying the development process and improving development efficiency.

Feature

  • support MYSQL protocol
  • simple and easy distributed transaction solution inspired by kubernetes
  • support read write splitting,supports custom SQL routing via Hint
  • deployed as a sidecar,support any language
  • sharding: support query on sharded table, support order by,support limit
  • more features on the road

Metrics

using prometheus metrics, default metric server run at :18888

Documentation

Requirements

  • Go 1.17 or higher
  • MYSQL (5.7+)

Contribution Guide

Architecture

architecture

  • Listener: responsible for parsing the SQL protocol.
  • Executor: responsible for forwarding SQL requests to the backend physical DB.
  • Filter: responsible for metrics statistics, SQL interception, encryption and decryption of sensitive information, etc.
  • ConnectionFilter: process the intercepted SQL on the connection.

WeChat group

To enter the WeChat discussion group, please scan the QR code below and reply with the code "join the group".

image-20220427091848831

License

DBPack is licensed under the Apache License, Version 2.0.

Issues
  • Test: unit test for `FormatTimeMillis`

    Test: unit test for `FormatTimeMillis`

    Added unit-tests for FormatTimeMillis func

    ref: https://github.com/cectc/dbpack/issues/70

    Ⅰ. Describe what this PR did

    Adding unit-tests for FormatTimeMillis func

    Ⅱ. Does this pull request fix one issue?

    I am not sure, but i think yes

    Ⅲ. Why don't you add test cases (unit test/integration test)?

    tests added

    Ⅳ. Describe how to verify it

    Ⅴ. Special notes for reviews

    this PR is ok for you, i will continue for others, let me know if you have any comments for PR

    opened by tanryberdi 19
  • feat: add prometheus metric

    feat: add prometheus metric

    ref: https://github.com/cectc/dbpack/issues/25

    Ⅰ. Describe what this PR did

    add prometheus metric for dbpack. opentelemetry metric currently unstable image

    Ⅱ. Does this pull request fix one issue?

    #25

    Ⅲ. Why don't you add test cases (unit test/integration test)?

    no need tests

    Ⅳ. Describe how to verify it

    in my computer, i verify it

    Ⅴ. Special notes for reviews

    connection metric filter config image

    opened by fatelei 7
  • test: unit test for `pkg/filter/dt/executor.go`

    test: unit test for `pkg/filter/dt/executor.go`

    Is your feature request related to a problem? Please describe. A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

    Describe the solution you'd like A clear and concise description of what you want to happen.

    Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

    Additional context Add any other context or screenshots about the feature request here.

    good first issue help wanted 
    opened by dk-lockdown 6
  • feat: add  python examples

    feat: add python examples

    Signed-off-by: zhu733756 [email protected]

    ref: https://github.com/CECTC/dbpack/issues/31

    Ⅰ. Describe what this PR did

    add python examples

    Ⅱ. Does this pull request fix one issue?

    https://github.com/CECTC/dbpack/issues/31

    Ⅲ. Why don't you add test cases (unit test/integration test)?

    No need

    Ⅳ. Describe how to verify it

    See README

    Ⅴ. Special notes for reviews

    None.

    opened by zhu733756 6
  • support tcc branch commit & rollback

    support tcc branch commit & rollback

    ref: https://github.com/cectc/dbpack/issues/12

    Ⅰ. Describe what this PR did

    support tcc mode distributed transaction

    Ⅱ. Does this pull request fix one issue?

    #12

    opened by bohehe 6
  • add unit test cases for `convertAssignRows`

    add unit test cases for `convertAssignRows`

    ref: https://github.com/cectc/dbpack/issues/

    Ⅰ. Describe what this PR did

    Ⅱ. Does this pull request fix one issue?

    Ⅲ. Why don't you add test cases (unit test/integration test)?

    Ⅳ. Describe how to verify it

    Ⅴ. Special notes for reviews

    opened by dk-lockdown 5
  • feat: add readiness and liveness probe

    feat: add readiness and liveness probe

    Signed-off-by: hzliangbin [email protected]

    ref: https://github.com/cectc/dbpack/issues/52

    Ⅰ. Describe what this PR did

    add readiness and liveness probe

    Ⅱ. Does this pull request fix one issue?

    #52

    Ⅲ. Why don't you add test cases (unit test/integration test)?

    Ⅳ. Describe how to verify it

    Ⅴ. Special notes for reviews

    opened by hzliangbin 5
  • WIP feat: intergrate opentelemetry tracing

    WIP feat: intergrate opentelemetry tracing

    ref: https://github.com/cectc/dbpack/issues/

    Ⅰ. Describe what this PR did

    #50

    • add opentelemetry support in dbpack

    Ⅱ. Does this pull request fix one issue?

    Ⅲ. Why don't you add test cases (unit test/integration test)?

    Ⅳ. Describe how to verify it

    Ⅴ. Special notes for reviews

    opened by fatelei 4
  • [WIP] feature: support delete on sharded table

    [WIP] feature: support delete on sharded table

    ref: https://github.com/cectc/dbpack/issues/44

    Ⅰ. Describe what this PR did

    Todo: Need to handle distributed transactions across databases

    Ⅱ. Does this pull request fix one issue?

    Ⅲ. Why don't you add test cases (unit test/integration test)?

    there are unit test cases and integration test cases

    Ⅳ. Describe how to verify it

    check ci result

    Ⅴ. Special notes for reviews

    opened by dk-lockdown 4
  • Test: unit test for pkg/misc/other.go

    Test: unit test for pkg/misc/other.go

    ref: https://github.com/cectc/dbpack/issues/68

    Ⅰ. Describe what this PR did

    Added unit-test for pkg/misc/other.go

    Ⅱ. Does this pull request fix one issue?

    yes

    Ⅲ. Why don't you add test cases (unit test/integration test)?

    Ⅳ. Describe how to verify it

    Ⅴ. Special notes for reviews

    opened by tanryberdi 4
  • refactor: refact distributed transaction filter support comQuery

    refactor: refact distributed transaction filter support comQuery

    ref: https://github.com/cectc/dbpack/issues/104

    Ⅰ. Describe what this PR did

    Ⅱ. Does this pull request fix one issue?

    Ⅲ. Why don't you add test cases (unit test/integration test)?

    no need

    Ⅳ. Describe how to verify it

    Ⅴ. Special notes for reviews

    opened by dk-lockdown 4
  • feature: merge result support limit without orderby

    feature: merge result support limit without orderby

    Is your feature request related to a problem? Please describe. A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

    https://github.com/CECTC/dbpack/blob/41f110519603feb32ff834c507fbf6d92ca9de94/pkg/plan/result.go#L177

    such as : select id, name, age from employees where id > 1000 limit 100,10

    Describe the solution you'd like A clear and concise description of what you want to happen.

    Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

    Additional context Add any other context or screenshots about the feature request here.

    enhancement good first issue help wanted 
    opened by dk-lockdown 0
  • [WIP] support distributed pk

    [WIP] support distributed pk

    Signed-off-by: Bin Liang [email protected]

    ref: https://github.com/cectc/dbpack/issues/42

    Ⅰ. Describe what this PR did

    Ⅱ. Does this pull request fix one issue?

    Ⅲ. Why don't you add test cases (unit test/integration test)?

    Ⅳ. Describe how to verify it

    Ⅴ. Special notes for reviews

    opened by hzliangbin 1
  • feat: support com_long_data

    feat: support com_long_data

    ref: https://github.com/cectc/dbpack/issues/

    Ⅰ. Describe what this PR did

    Ⅱ. Does this pull request fix one issue?

    Ⅲ. Why don't you add test cases (unit test/integration test)?

    Ⅳ. Describe how to verify it

    Ⅴ. Special notes for reviews

    opened by dk-lockdown 2
  • test: unit test for `pkg/function/function.go`

    test: unit test for `pkg/function/function.go`

    Is your feature request related to a problem? Please describe. A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

    Describe the solution you'd like A clear and concise description of what you want to happen.

    Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

    Additional context Add any other context or screenshots about the feature request here.

    good first issue help wanted 
    opened by dk-lockdown 0
  • feature: support query from global table

    feature: support query from global table

    Is your feature request related to a problem? Please describe. In sharding mode, global tables are the same on every db

    Describe the solution you'd like Support query from global table

    Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

    Additional context Add any other context or screenshots about the feature request here.

    good first issue help wanted 
    opened by dk-lockdown 0
  • feature: support multiple columns `order by` when query sharded table

    feature: support multiple columns `order by` when query sharded table

    Is your feature request related to a problem? Please describe. A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] Currently only one column order by is supported

    Describe the solution you'd like A clear and concise description of what you want to happen.

    Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

    Additional context Add any other context or screenshots about the feature request here.

    enhancement good first issue help wanted 
    opened by dk-lockdown 0
Releases(v0.1.2)
  • v0.1.2(Jun 13, 2022)

  • v0.1.1(Jun 9, 2022)

    Added dotnet samples

    Bug Fixes

    Features

    Changes

    • refactor: update global session status, branch session status in txn (120) (9829387)
    Source code(tar.gz)
    Source code(zip)
  • v0.1.0(May 24, 2022)

    dbpack support any languages, there are gojavaphppython samples.

    Bug Fixes

    • add terminationDrainDuration config (#4) (5c607d4)
    • should use db connection rather than tx connection exec sql request (#8) (52d78ca)
    • when receive ComQuit request, should return connection (#51) (e8f0708)
    • close statement after read result from conn (#71) (4c9a292)
    • ping should put resource back to the pool (#74) (c1c7771)
    • process global session after timeout should refresh global session status (#86) (6bf4090)
    • release tx when undologs doesn't exists (#93) (99df036)

    Features

    • distributed transaction support etcd watch (#11) (e991050)
    • support tcc branch commit & rollback (#12) (feab7ae)
    • support GlobalLock hint (#14) (5c7c967)
    • support leader election, only leader can commit and rollback (#19) (d7ab60b)
    • add prometheus metric (#25) (627adc2)
    • add readiness and liveness probe (#52) (f43ab5f)

    Changes

    • update branch session process logic (#17) (06d6245)

    New Contributors

    • @rocymp made their first contribution in https://github.com/CECTC/dbpack/pull/3
    • @gorexlv made their first contribution in https://github.com/CECTC/dbpack/pull/5
    • @zackzhangkai made their first contribution in https://github.com/CECTC/dbpack/pull/10
    • @yx9o made their first contribution in https://github.com/CECTC/dbpack/pull/32
    • @bohehe made their first contribution in https://github.com/CECTC/dbpack/pull/41
    • @fatelei made their first contribution in https://github.com/CECTC/dbpack/pull/48
    • @zhu733756 made their first contribution in https://github.com/CECTC/dbpack/pull/58
    • @wybrobin made their first contribution in https://github.com/CECTC/dbpack/pull/72
    • @tanryberdi made their first contribution in https://github.com/CECTC/dbpack/pull/75
    • @JuwanXu made their first contribution in https://github.com/CECTC/dbpack/pull/81
    • @hzliangbin made their first contribution in https://github.com/CECTC/dbpack/pull/83
    Source code(tar.gz)
    Source code(zip)
Example golang microservices deployed on kubernetes.

Tech Stack Golang RabbitMQ Docker K8S MongoDB Services There are two services which communicate via http(sync) and rabbitmq(async). Services opened to

Serkan 3 Dec 24, 2021
Blog microservices deployed in an Istio-enabled kubernetes cluster

blog-microservices blog microservices deployed in an Istio-enabled kubernetes cl

jxlwqq 96 Jun 29, 2022
Global Financial Transaction Network Services

Global Financial Transaction Network Services This code was developed at IBM during 2017-2020, and contributed to open source in September 2021. Overv

null 14 Feb 26, 2022
A microservice gateway developed based on golang.With a variety of plug-ins which can be expanded by itself, plug and play. what's more,it can quickly help enterprises manage API services and improve the stability and security of API services.

Goku API gateway is a microservice gateway developed based on golang. It can achieve the purposes of high-performance HTTP API forwarding, multi tenant management, API access control, etc. it has a powerful custom plug-in system, which can be expanded by itself, and can quickly help enterprises manage API services and improve the stability and security of API services.

Eolink 194 Jun 22, 2022
A Micro-UTP, plug-able sanity checker for any on-prem JFrog platform instance

hello-frog About this plugin This plugin is a template and a functioning example for a basic JFrog CLI plugin. This README shows the expected structur

rdar 0 Dec 7, 2021
A gateway to expose s3 standard API for any storage type.

s3-gateway this project is used to be compatible with any other storage type. there are tons of object storage services in the internat. However, many

xylon 0 Nov 28, 2021
Omh-users-management - A go microservice that can enables us to create, modify, fetch, and delete users

Users Management System A go microservice that can enables us to create, modify, fetch, and delete users Usage To Run the application $ go run main.

Pulkit Tanwar 0 Jan 20, 2022
Just a quick demo of how you can use automatically generated protobuffer and gRPC code from buf.build

buf.build demo The purpose of this repository is to demonstrate how to use the services offered by buf.build for hosting protobuffer definitions and a

Bjørn Borud 0 Jan 4, 2022
Go-explosion - Distributed indexing and searching in Go/Golang

go-explosion Distributed indexing and searching in Go/Golang. This library does

Mooij Technologies 1 Jan 1, 2022
This project extends the go-chi router to support OpenAPI 3, bringing to you a simple interface to build a router conforming your API contract.

Go OpenAPI This project extends the go-chi router to support OpenAPI 3, bringing to you a simple interface to build a router conforming your API contr

Tiago Angelo 2 Mar 27, 2022
gRPC to JSON proxy generator following the gRPC HTTP spec

The gRPC-Gateway is a plugin of the Google protocol buffers compiler protoc. It reads protobuf service definitions and generates a reverse-proxy server which translates a RESTful HTTP API into gRPC. This server is generated according to the google.api.http annotations in your service definitions.

gRPC Ecosystem 13.6k Jun 28, 2022
Go Micro is a framework for distributed systems development

Go Micro Go Micro is a framework for distributed systems development. Overview Go Micro provides the core requirements for distributed systems develop

Asim Aslam 18.6k Jun 25, 2022
NewSQL distributed storage database based on micro service framework

QLite 是基于微服务的 NewSQL 型数据库系统,与传统的一体化数据库不同,该系统将本该内置的多种数据结构(STL)拆分成多个服务模块,每个模块都是独立的一个节点,每个节点都与其主网关进行连接,从而形成分布式存储结构。

null 36 Jun 19, 2022
CNCF Jaeger, a Distributed Tracing Platform

Jaeger - a Distributed Tracing System Jaeger, inspired by Dapper and OpenZipkin, is a distributed tracing platform created by Uber Technologies and do

Jaeger - Distributed Tracing Platform 16k Jun 30, 2022
Go Micro: a standard library for distributed systems development

Go Micro Go Micro is a standard library for distributed systems development. Ove

Pixelmatic 1 May 2, 2022
Labs for MIT 6.824 Distributed Systems (Spring 2020)

6.824-2020-labs This repo contains labs for MIT 6.824 Distributed Systems (Spring 2020) The master branch contains the source code with no implementat

Junwang Zhao 0 Jan 18, 2022
Demonstrate a bounded context distributed over multiple repositories. In `go`

Contextive Demo - Go - Service This repository illustrates the use of Contextive in an environment where multiple repositories are part of the same bo

Dev Cycles 0 Feb 12, 2022
This tool generates Go language bindings of services in protobuf definition files for go-kit

protoc-gen-go-kit This tool generates Go language bindings of services in protobuf definition files for go-kit. Installation $ go install github.com/x

X64FUN 1 Nov 9, 2021
Microservice Boilerplate for Golang with gRPC and RESTful API. Multiple database and client supported

Go Microservice Starter A boilerplate for flexible Go microservice. Table of contents Features Installation Todo List Folder Structures Features: Mult

Ahmad Saugi 11 May 27, 2022