Stateless Postgres Query Router

Related tags

Routers spqr
Overview

Github CI/CD GitHub go.mod Go version Go Report Github Repository Size Lines of code

Stateless Postgres Query Router

Development

How to build

go get golang.org/x/tools/cmd/goyacc
go get -u github.com/golang/protobuf/protoc-gen-go
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc

make yaccgen
make build
Issues
  • Balancer App

    Balancer App

    • implement the Shard service
    • run balancer: register shards, init the metadata DB, dockerize balancer
    • list key ranges to use in the balancer
    • fix balancer db connections and queries to manage actions
    • fix key range lock, data transfer, key range move, delete old key range
    • implement key ranges uniting
    • add sharding key to the config, rewrite installation queries by removing hardcoded table values
    • make possible to run parallel tasks and fix data duplication on move a table
    opened by agneum 4
  • TLS and rrouter small refactoring

    TLS and rrouter small refactoring

    I just tried to make the shard configs work, but it didn't work.

    • Removed unnecessary passing of sslmode to different functions. If *tls.Config is not nil, then ssl is enabled.
    • Fixed those code with initializing frontend and backend rules
    opened by Denchick 0
  • Fix problem with multiline init sql

    Fix problem with multiline init sql

    If init sql file is multiline, then it is no actually executed well

    LOG: 2022/05/26 16:49:20.878624 init.sql found
    INFO: 2022/05/26 16:49:20.882685 executing init sql cmd CREATE TABLE articles (
    ERROR: 2022/05/26 16:49:20.882716 syntax error
    INFO: 2022/05/26 16:49:20.882720 syntax error
    INFO: 2022/05/26 16:49:20.882722 executing init sql cmd id BIGSERIAL NOT NULL PRIMARY KEY,
    ERROR: 2022/05/26 16:49:20.882726 syntax error
    INFO: 2022/05/26 16:49:20.882728 syntax error
    INFO: 2022/05/26 16:49:20.882730 executing init sql cmd url TEXT NOT NULL UNIQUE,
    ERROR: 2022/05/26 16:49:20.882735 syntax error
    INFO: 2022/05/26 16:49:20.882737 syntax error
    INFO: 2022/05/26 16:49:20.882739 executing init sql cmd title TEXT NOT NULL,
    ERROR: 2022/05/26 16:49:20.882743 syntax error
    INFO: 2022/05/26 16:49:20.882745 syntax error
    INFO: 2022/05/26 16:49:20.882746 executing init sql cmd description TEXT NOT NULL
    ERROR: 2022/05/26 16:49:20.882752 syntax error
    INFO: 2022/05/26 16:49:20.882754 syntax error
    INFO: 2022/05/26 16:49:20.882755 executing init sql cmd );
    ERROR: 2022/05/26 16:49:20.882760 syntax error
    INFO: 2022/05/26 16:49:20.882762 syntax error
    INFO: 2022/05/26 16:49:20.882765 Successfully init 6 queries from init.sql
    
    opened by Denchick 0
  • Balancer console

    Balancer console

    • init console interface for Balancer App
    • support commands:
      • show key ranges
      • split key range
    • use spqrlog package to log messages

    Examples:

    Connection

    $ psql "host=::1 port=8433 dbname=db1 sslmode=disable user=user1"
    Welcome
    psql (14.2 (Ubuntu 14.2-1.pgdg20.04+1), server console)
    Type "help" for help.
    

    Show key ranges

    db1=?> show key_ranges;
     Key range ID | Shard ID | Lower bound | Upper bound 
    --------------+----------+-------------+-------------
     krid1        | 1        | 1           | 1000
     krid2        | 2        | 1000        | 2000
    (2 rows)
    

    Split key ranges

    db1=?> split key range krid3 from krid2 by 1500;
            split key range        
    -------------------------------
     split key range krid2 by 1500
    (1 row)
    
    db1=?> show key_ranges;
     Key range ID | Shard ID | Lower bound | Upper bound 
    --------------+----------+-------------+-------------
     krid1        | 1        | 1           | 1000
     krid2        | 2        | 1000        | 1500
     krid3        | 2        | 1500        | 2000
    (3 rows)
    
    opened by agneum 0
  • fix: add a mock router to avoid panic in the coordinator client process

    fix: add a mock router to avoid panic in the coordinator client process

    Fixed panic:

    panic: runtime error: invalid memory address or nil pointer dereference
    spqr-coordinator-1  | [signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x8cd99e]
    spqr-coordinator-1  | 
    spqr-coordinator-1  | goroutine 67 [running]:
    spqr-coordinator-1  | github.com/pg-sharding/spqr/router/pkg/route.(*Route).Params(0x0)
    spqr-coordinator-1  | 	/home/go/src/github.com/pg-sharding/spqr/router/pkg/route/route.go:59 +0x5e
    spqr-coordinator-1  | github.com/pg-sharding/spqr/router/pkg/client.(*PsqlClient).Auth(0xc0005b8db8, 0xda0f26?)
    spqr-coordinator-1  | 	/home/go/src/github.com/pg-sharding/spqr/router/pkg/client/client.go:511 +0x30a
    spqr-coordinator-1  | github.com/pg-sharding/spqr/coordinator/provider.(*qdbCoordinator).ProcClient(0xc00007e140?, {0xeb1430, 0xc000038040}, {0xeb3d70, 0xc0003a0000})
    spqr-coordinator-1  | 	/home/go/src/github.com/pg-sharding/spqr/coordinator/provider/coordinator.go:351 +0x306
    spqr-coordinator-1  | github.com/pg-sharding/spqr/coordinator/app.(*App).ServePsql(0xc000592020, 0x0?)
    spqr-coordinator-1  | 	/home/go/src/github.com/pg-sharding/spqr/coordinator/app/app.go:63 +0x1ea
    spqr-coordinator-1  | github.com/pg-sharding/spqr/coordinator/app.(*App).Run.func2(0x0?)
    spqr-coordinator-1  | 	/home/go/src/github.com/pg-sharding/spqr/coordinator/app/app.go:40 +0x35
    spqr-coordinator-1  | created by github.com/pg-sharding/spqr/coordinator/app.(*App).Run
    spqr-coordinator-1  | 	/home/go/src/github.com/pg-sharding/spqr/coordinator/app/app.go:39 +0x18d
    
    opened by agneum 0
  • fix(proxy): process transactions to run benchmarks

    fix(proxy): process transactions to run benchmarks

    Process transactions to run benchmarks:

    • support queries in a transaction
    • support the truncate DDL query
    • add debug info
    • adjust configuration files
    • fix e2e tests
    opened by agneum 0
  • Automatic configuration of proxy routers using the Coordinator app

    Automatic configuration of proxy routers using the Coordinator app

    • register/unregister proxy routers
    • update proto files and open ports to run coordinator
    • configure a new proxy router using coordinator
    • store ShardingRules and apply the existing rules to the router after registration
    • implement the ShardingRules service in proxy routers
    opened by agneum 1
  • fix(router): run world and coordinator apps

    fix(router): run world and coordinator apps

    • proxy queries that do not match any shard to the world storage
    • send DDL queries to all shards
    • update proto-files and open ports to run coordinator
    • fix typos
    opened by agneum 0
Bxog is a simple and fast HTTP router for Go (HTTP request multiplexer).

Bxog is a simple and fast HTTP router for Go (HTTP request multiplexer). Usage An example of using the multiplexer: package main import ( "io" "net

Eduard 103 Apr 10, 2022
A high performance fasthttp request router that scales well

FastHttpRouter FastHttpRouter is forked from httprouter which is a lightweight high performance HTTP request router (also called multiplexer or just m

招牌疯子 866 Jun 16, 2022
FastRouter is a fast, flexible HTTP router written in Go.

FastRouter FastRouter is a fast, flexible HTTP router written in Go. FastRouter contains some customizable options, such as TrailingSlashesPolicy, Pan

Razon Yang 21 Apr 19, 2022
Go Router + Middleware. Your Contexts.

gocraft/web gocraft/web is a Go mux and middleware package. We deal with casting and reflection so YOUR code can be statically typed. And we're fast.

null 1.5k Jun 24, 2022
Go Server/API micro framework, HTTP request router, multiplexer, mux

?? gorouter Go Server/API micro framework, HTTP request router, multiplexer, mux. ?? ABOUT Contributors: Rafał Lorenz Want to contribute ? Feel free t

Rafał Lorenz 128 Jun 13, 2022
A high performance HTTP request router that scales well

HttpRouter HttpRouter is a lightweight high performance HTTP request router (also called multiplexer or just mux for short) for Go. In contrast to the

Julien Schmidt 14.2k Jun 26, 2022
High-speed, flexible tree-based HTTP router for Go.

httptreemux High-speed, flexible, tree-based HTTP router for Go. This is inspired by Julien Schmidt's httprouter, in that it uses a patricia tree, but

Daniel Imfeld 545 Jun 23, 2022
:rotating_light: Is a lightweight, fast and extensible zero allocation HTTP router for Go used to create customizable frameworks.

LARS LARS is a fast radix-tree based, zero allocation, HTTP router for Go. view examples. If looking for a more pure Go solution, be sure to check out

Go Playgound 387 Jun 11, 2022
A powerful HTTP router and URL matcher for building Go web servers with 🦍

gorilla/mux https://www.gorillatoolkit.org/pkg/mux Package gorilla/mux implements a request router and dispatcher for matching incoming requests to th

Gorilla Web Toolkit 16.8k Jun 22, 2022
An extremely fast Go (golang) HTTP router that supports regular expression route matching. Comes with full support for building RESTful APIs.

ozzo-routing You may consider using go-rest-api to jumpstart your new RESTful applications with ozzo-routing. Description ozzo-routing is a Go package

Ozzo Framework 438 Jun 15, 2022
Pure is a fast radix-tree based HTTP router

package pure Pure is a fast radix-tree based HTTP router that sticks to the native implementations of Go's "net/http" package; in essence, keeping the

Go Playgound 126 Jun 25, 2022
Echo Inspired Stand Alone URL Router

Vestigo - A Standalone Golang URL Router Abstract Many fast Golang URL routers are often embedded inside frameworks. Vestigo is a stand alone url rout

null 267 Jun 13, 2022
Go HTTP router

violetear Go HTTP router http://violetear.org Design Goals Keep it simple and small, avoiding extra complexity at all cost. KISS Support for static an

Nicolas Embriz 105 Mar 21, 2022
xujiajun/gorouter is a simple and fast HTTP router for Go. It is easy to build RESTful APIs and your web framework.

gorouter xujiajun/gorouter is a simple and fast HTTP router for Go. It is easy to build RESTful APIs and your web framework. Motivation I wanted a sim

徐佳军 525 May 29, 2022
lightweight, idiomatic and composable router for building Go HTTP services

chi is a lightweight, idiomatic and composable router for building Go HTTP services. It's especially good at helping you write large REST API services

go-chi 11.7k Jun 26, 2022
Router socks. One port socks for all the others.

Router socks The next step after compromising a machine is to enumerate the network behind. Many tools exist to expose a socks port on the attacker's

null 60 Apr 5, 2022
:tongue: CleverGo is a lightweight, feature rich and high performance HTTP router for Go.

CleverGo CleverGo is a lightweight, feature rich and trie based high performance HTTP request router. go get -u clevergo.tech/clevergo English 简体中文 Fe

CleverGo Web Framework 251 May 30, 2022
Fast and flexible HTTP router

treemux - fast and flexible HTTP router Basic example Debug logging CORS example Error handling Rate limiting using Redis Gzip compression OpenTelemet

Vladimir Mihailenco 441 Jun 25, 2022
Fast, simple, and lightweight HTTP router for Golang

Sariaf Fast, simple and lightweight HTTP router for golang Install go get -u github.com/majidsajadi/sariaf Features Lightweight compatible with net/ht

defectivepixel 33 Mar 3, 2022