Gorm-sharding - High performance table sharding plugin for Gorm

Related tags

ORM gorm-sharding
Overview

Gorm Sharding

This project has moved to Gorm offical organization:

https://github.com/go-gorm/sharding

Go Go Reference

Gorm Sharding plugin using SQL parser and replace for splits large tables into smaller ones, redirects Query into sharding tables. Give you a high performance database access.

Gorm Sharding 是我们基于 Longbridge 的业务场景以及我们的实践经验构建出来的一个高性能的数据库分表中间件。

它基于 Conn 层做 SQL 拦截、AST 解析、分表路由、自增主键填充,带来的额外开销极小。对开发者友好、透明,使用上与普通 SQL、Gorm 查询无差别,只需要额外注意一下分表键条件。

Features

  • Non-intrusive design. Load the plugin, specify the config, and all done.
  • Lighting-fast. No network based middlewares, as fast as Go.
  • Multiple database support. PostgreSQL tested, MySQL and SQLite is coming.
  • Allows you custom the Primary Key generator (Sequence, UUID, Snowflake ...).

Sharding process

This graph show up how Gorm Sharding works.

Example

Install

go get -u github.com/longbridgeapp/gorm-sharding

Usage

Open a db session.

dsn := "postgres://localhost:5432/sharding-db?sslmode=disable"
db, err := gorm.Open(postgres.New(postgres.Config{DSN: dsn}))

Config the sharding middleware, register the tables which you want to shard. See Godoc for config details.

db.Use(&sharding.Register(map[string]sharding.Resolver{
    "orders": {
        ShardingColumn: "user_id",
        ShardingAlgorithm: func(value interface{}) (suffix string, err error) {
            if uid, ok := value.(int64); ok {
                return fmt.Sprintf("_%02d", uid%4), nil
            }
            return "", errors.New("invalid user_id")
        },
        PrimaryKeyGenerate: func(tableIdx int64) int64 {
            return node.Generate()
        },
    },
    "notifications": {
       // ... sharding rule for notifications table
    },
}))

Use the db session as usual. Just note that the query should have the Sharding Key when operate sharding tables.

// Gorm create example, this will insert to orders_01
db.Create(&Order{UserID: 2})
// sql: INSERT INTO orders_2 ...

// Show have use Raw SQL to insert, this will insert into orders_04
db.Exec("INSERT INTO orders(user_id) VALUES(?)", int64(3))

// This will throw ErrMissingShardingKey error, because there not have sharding key presented.
db.Create(&Order{Amount: 10, ProductID: 100})
fmt.Println(err)

// Find, this will redirect query to orders_03
var orders []Order
db.Model(&Order{}).Where("user_id", int64(2)).Find(&orders)
fmt.Printf("%#v\n", orders)

// This will throw ErrMissingShardingKey error, because WHERE conditions not included sharding key
err = db.Model(&Order{}).Where("product_id", "1").Find(&orders).Error
fmt.Println(err)

// Update and Delete are similar to create and query
db.Exec("UPDATE orders SET product_id = ? WHERE user_id = ?", 2, int64(3))
err = db.Exec("DELETE FROM orders WHERE product_id = 3").Error
fmt.Println(err) // ErrMissingShardingKey

The full example is here.

Primary Key

When you sharding tables, you need consider how the primary key generate.

Recommend options:

License

This project under MIT license.

You might also like...
CURD using go fiber - gorm - mysql

GO Fiber - CRUD - GORM - Mysql Folder Structure - database | database config |- migration | migration config - middleware | mid

Fiber Clean Architecture With GORM

Fiber Clean Architecture With GORM I offer this repository as a proposal for a c

Ormtool - 将数据库表转换为golang的结构体,自定义生成tag,如json,gorm,xorm和简单的db信息

数据库表转换为golang 结构体 1. 获取方式 2. 配置说明 # 保存路径 SavePath: "./models/test.go", # 是

Simple-crm-system - Simple CRM system CRUD backend using Go, Fiber, SQLite, Gorm

Simple CRM system CRUD backend using GO, Fiber, Gorm, SQLite Developent go mod t

Api-project - Api project with Golang, Gorm, Gorilla-Mux, Postgresql

TECHNOLOGIES GOLANG 1.14 GORM GORILLA-MUX POSTGRESQL API's PATHS For Product Ser

This library is a complementary library for Gorm (v2) which resolves the first available pool passed to it.

This library is a complementary library for Gorm (v2) which resolves the first available pool passed to it.

EZCoin is a control panel for Bitfinex funding, backend is build by Golang, Gin and GORM, frontend is build by angular

EZCoin server is backend for Bitfinex funding, it build by Golang, Gin and GORM.

RestAPI Starter Template Using Go (Gin + Gorm)

go-gin-gorm-restapi-template Go (Gin + Gorm) を使用した RestAPI 開発のスターターテンプレート RestAPI Starter Template Using Golang (Gin + Gorm) 主要な依存ライブラリ Gin (Web フレームワ

A simple CRUD API made with Go, Postgres, FIber, Gorm and Docker.

golang-test-api A simple CRUD API made with Go, Postgres, FIber, Gorm and Docker. Cloning the repository To clone the repository run the following com

Releases(v0.1.0)
Owner
Longbridge
Long Bridge Securities
Longbridge
A plugin to allow telemetry by NewRelic Go Agent for GORM

GORM NewRelic Telemetry Plugin A plugin to allow telemetry by NewRelic Go Agent for GORM Overview Plugin implementation to add datastore segments on a

Rafael Holanda de Lima 12 Aug 19, 2022
OpenTelemetry plugin for GORM v2

gorm-opentelemetry OpenTelemetry plugin for GORM v2 Traces all queries along with the query SQL. Usage Example: // Copyright The OpenTelemetry Authors

ZopSmart 0 Jan 11, 2022
Golang struct-to-table database mapper

Structable: Struct-Table Mapping for Go Warning: This is the Structable 4 development branch. For a stable release, use version 3.1.0. Structable deve

null 267 Nov 14, 2022
Crud - A mysql crud code generate tool from table DDL sql file

crud is a mysql crud code generate tool 中文文档 Getting Started Overview Crud is a

null 20 Oct 13, 2022
100% type-safe ORM for Go (Golang) with code generation and MySQL, PostgreSQL, Sqlite3, SQL Server support. GORM under the hood.

go-queryset 100% type-safe ORM for Go (Golang) with code generation and MySQL, PostgreSQL, Sqlite3, SQL Server support. GORM under the hood. Contents

Denis Isaev 688 Nov 16, 2022
opentracing integration with GORM

gorm-opentracing opentracing support for gorm2. Features Record SQL in span logs. Record Result in span logs. Record Table in span tags. Record Error

null 79 Nov 18, 2022
A code generator base on GORM

GORM/GEN The code generator base on GORM, aims to be developer friendly. Overview CRUD or DIY query method code generation Auto migration from databas

null 1.4k Nov 19, 2022
Converts a database into gorm structs and RESTful api

gen The gen tool produces a CRUD (Create, read, update and delete) REST api project template from a given database. The gen tool will connect to the d

smallnest 1.3k Nov 27, 2022
Scope function for GORM queries provides easy filtering with query parameters

Gin GORM filter Scope function for GORM queries provides easy filtering with query parameters Usage go get github.com/ActiveChooN/gin-gorm-filter Mod

Dmitry Kalinin 8 Oct 12, 2022
Gorm firebird driver

gorm-firebird GORM firebird driver import: "github.com/flylink888/gorm-firebird" Example: var products []Product dsn := "SYSDBA:[email protected]/sy

null 8 Sep 27, 2022