Stateless Postgres Query Router

Related tags

Routers spqr
Overview

Github CI/CD GitHub go.mod Go version Go Report Github Repository Size Lines of code

Stateless Postgres Query Router

Development

How to build

go get golang.org/x/tools/cmd/goyacc
go get -u github.com/golang/protobuf/protoc-gen-go
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc

make yaccgen
make build
Comments
  • Balancer App

    Balancer App

    • implement the Shard service
    • run balancer: register shards, init the metadata DB, dockerize balancer
    • list key ranges to use in the balancer
    • fix balancer db connections and queries to manage actions
    • fix key range lock, data transfer, key range move, delete old key range
    • implement key ranges uniting
    • add sharding key to the config, rewrite installation queries by removing hardcoded table values
    • make possible to run parallel tasks and fix data duplication on move a table
    opened by agneum 4
  • Support sync messages in multi-shard setup

    Support sync messages in multi-shard setup

    Process sync message correctly in cases when there is no active backend connections to any shard, instead of replying with error 'too complex to parse'. This fixed the case when sync message is issued right after parse message, or similar, in which router fill fail to 'route' sync to shard, because this is un-routabale messafe.

    opened by reshke 0
  • Do not close connection with disabled ssl

    Do not close connection with disabled ssl

    Continue client-sever intercation after ssl requirest rejection Send ssl negotionation message and wait for next client message. Also fix incorrect auth rule handling in AuthBackend()

    opened by reshke 0
  • Handle `sslmode` correctly

    Handle `sslmode` correctly

    It was impossible for the router to connect to shards using a root certificate. It works:

    [email protected]:/spqr# psql "host=localhost sslmode=disable user=sbtest dbname=sbtest port=6432"
psql (14.5 (Ubuntu 14.5-201-yandex.52547.2fabebd33c))
Type "help" for help.

sbtest=> select * from articles;
NOTICE: send query to shard(s) : shard01,shard02
 id 
----
(0 rows)

    SPQR logs:

    [email protected]:/spqr# ./spqr-router -c router.yaml run
2022/11/07 06:36:45 Running config: {
  "log_level": "",
  "host": "localhost",
  "router_port": "6432",
  "admin_console_port": "7432",
  "grpc_api_port": "7000",
  "world_shard_fallback": false,
  "show_notice_messages": true,
  "auto_conf": "",
  "init_sql": "init.sql",
  "under_coordinator": false,
  "router_mode": "PROXY",
  "jaeger_url": "",
  "frontend_rules": [
    {
      "db": "sbtest",
      "usr": "sbtest",
      "auth_rule": {
        "auth_method": "ok",
        "password": ""
      },
      "pool_mode": "TRANSACTION",
      "pool_discard": false,
      "pool_rollback": false,
      "pool_prepared_statement": false,
      "pool_default": false
    }
  ],
  "frontend_tls": null,
  "backend_rules": [
    {
      "db": "sbtest",
      "usr": "sbtest",
      "auth_rule": {
        "auth_method": "md5",
        "password": "password"
      },
      "pool_default": false
    }
  ],
  "shards": {
    "shard01": {
      "target_session_attrs": "",
      "hosts": [
        "sas-xhxplfw3lnnqbf5c.db.yandex.net:6432"
      ],
      "type": "DATA",
      "tls": {
        "sslmode": "require",
        "key_file": "",
        "cert_file": "",
        "root_cert_file": "/spqr/root.crt"
      }
    },
    "shard02": {
      "target_session_attrs": "",
      "hosts": [
        "sas-v0df2vny5r5x5u77.db.yandex.net:6432"
      ],
      "type": "DATA",
      "tls": {
        "sslmode": "require",
        "key_file": "",
        "cert_file": "",
        "root_cert_file": "/spqr/root.crt"
      }
    }
  }
}
LOG: 2022/11/07 06:36:45.348305 adding node shard01
LOG: 2022/11/07 06:36:45.348322 adding node shard02
LOG: 2022/11/07 06:36:45.348354 init.sql found
INFO: 2022/11/07 06:36:45.348412 executing init sql
INFO: 2022/11/07 06:36:45.348418 query: SHOW shards;
INFO: 2022/11/07 06:36:45.348453 query: ADD SHARDING RULE rule1 COLUMNS id;
INFO: 2022/11/07 06:36:45.348483 query: ADD KEY RANGE krid1 FROM 1 TO 1073741823  ROUTE TO shard01;
INFO: 2022/11/07 06:36:45.348511 query: ADD KEY RANGE krid2 FROM 1073741824 TO 2147483647  ROUTE TO shard02;
INFO: 2022/11/07 06:36:45.348529 query: SHOW sharding_rules;
INFO: 2022/11/07 06:36:45.348545 query: SHOW key_ranges;
INFO: 2022/11/07 06:36:45.348563 Successfully init 6 queries from init.sql
INFO: 2022/11/07 06:36:45.349055 SPQR Administative Console is ready on localhost:7432
INFO: 2022/11/07 06:36:45.349087 SPQR GRPC API is ready on localhost:7000
INFO: 2022/11/07 06:36:45.349106 SPQR Router is ready on localhost:6432 by postgresql proto
LOG: 2022/11/07 06:36:46.736241 init client connection with ssl: false
LOG: 2022/11/07 06:36:46.736598 Processing auth for sbtest sbtest
LOG: 2022/11/07 06:36:46.736642 acquire conn to shard02
LOG: 2022/11/07 06:36:46.736650 acquire new connection to sas-v0df2vny5r5x5u77.db.yandex.net:6432
LOG: 2022/11/07 06:36:46.746281 client auth OK
LOG: 2022/11/07 06:36:46.746296 pre route ok
INFO: 2022/11/07 06:36:46.746306 process frontend for route sbtest sbtest
LOG: 2022/11/07 06:36:53.972523 acquire conn to shard01
LOG: 2022/11/07 06:36:53.972541 acquire new connection to sas-xhxplfw3lnnqbf5c.db.yandex.net:6432
LOG: 2022/11/07 06:36:53.977892 acquire conn to shard02
LOG: 2022/11/07 06:36:53.977908 got cached connection from pool
LOG: 2022/11/07 06:36:53.982652 multishard server: enter rfq await mode
LOG: 2022/11/07 06:36:53.982691 end of tx unrouting from [{shard01 true} {shard02 true}]
LOG: 2022/11/07 06:36:53.982698 unrouting from datashard shard01
LOG: 2022/11/07 06:36:53.982703 unrouting from datashard shard02
LOG: 2022/11/07 06:37:00.541583 acquire conn to shard01
LOG: 2022/11/07 06:37:00.541602 acquire new connection to sas-xhxplfw3lnnqbf5c.db.yandex.net:6432
LOG: 2022/11/07 06:37:00.546895 end of tx unrouting from [{shard01 false}]
LOG: 2022/11/07 06:37:00.546905 unrouting from datashard shard01
    opened by Denchick 0
  • Online reload for router configuration

    Online reload for router configuration

    Supported router configuragtion online-reloading by SIGHUP singal, refactor and cleanup code to avoid using config.RouterConfig() code pattern as it wil be more appropiate to store currently running configuration inside RuleRouter struct.

    Also dropped watchdog code as its functionality is not needed and/or should be re-implemented with sync of Target Session Attrs storage feature

    opened by reshke 0
  • FYI: experiment with postgres regression tests

    FYI: experiment with postgres regression tests

    My idea was to up one-shard SPQR installation and run Postgresql regression tests against the router. I ran into several problems:

    • There are indeed a lot of tests.
    • Before every test, I should set up sharding rules cause there are a lot of test tables.
    • SPQR does not support yet a lot of queries that should be run on every shard.
    ➜  installcheck git:(postgres-regression-tests) ✗ docker-compose up installcheck
installcheck_shard_1 is up-to-date
installcheck_router_1 is up-to-date
Starting installcheck_installcheck_1 ... done
Attaching to installcheck_installcheck_1
installcheck_1  | make -C ./src/backend generated-headers
installcheck_1  | make[1]: Entering directory '/build/postgres/src/backend'
installcheck_1  | make -C catalog distprep generated-header-symlinks
installcheck_1  | make[2]: Entering directory '/build/postgres/src/backend/catalog'
installcheck_1  | make[2]: Nothing to be done for 'distprep'.
installcheck_1  | make[2]: Nothing to be done for 'generated-header-symlinks'.
installcheck_1  | make[2]: Leaving directory '/build/postgres/src/backend/catalog'
installcheck_1  | make -C utils distprep generated-header-symlinks
installcheck_1  | make[2]: Entering directory '/build/postgres/src/backend/utils'
installcheck_1  | make[2]: Nothing to be done for 'distprep'.
installcheck_1  | make[2]: Nothing to be done for 'generated-header-symlinks'.
installcheck_1  | make[2]: Leaving directory '/build/postgres/src/backend/utils'
installcheck_1  | make[1]: Leaving directory '/build/postgres/src/backend'
installcheck_1  | make -C src/test/regress installcheck
installcheck_1  | make[1]: Entering directory '/build/postgres/src/test/regress'
installcheck_1  | make -C ../../../src/port all
installcheck_1  | make[2]: Entering directory '/build/postgres/src/port'
installcheck_1  | make[2]: Nothing to be done for 'all'.
installcheck_1  | make[2]: Leaving directory '/build/postgres/src/port'
installcheck_1  | make -C ../../../src/common all
installcheck_1  | make[2]: Entering directory '/build/postgres/src/common'
installcheck_1  | make[2]: Nothing to be done for 'all'.
installcheck_1  | make[2]: Leaving directory '/build/postgres/src/common'
installcheck_1  | make -C ../../../contrib/spi
installcheck_1  | make[2]: Entering directory '/build/postgres/contrib/spi'
installcheck_1  | make[2]: Nothing to be done for 'all'.
installcheck_1  | make[2]: Leaving directory '/build/postgres/contrib/spi'
installcheck_1  | rm -rf ./testtablespace
installcheck_1  | mkdir ./testtablespace
installcheck_1  | ../../../src/test/regress/pg_regress --inputdir=. --bindir='/usr/local/pgsql/bin'    --dlpath=. --max-concurrent-tests=20  --schedule=./serial_schedule 
installcheck_1  | (using postmaster on installcheck_router_1, port 6432)
installcheck_1  | ============== dropping database "regression"         ==============
installcheck_1  | DROP DATABASE
installcheck_1  | ============== creating database "regression"         ==============
installcheck_1  | CREATE DATABASE
installcheck_1  | ERROR:  too complex query to parse
installcheck_1  | command failed: "/usr/local/pgsql/bin/psql" -X -c "ALTER DATABASE \"regression\" SET lc_messages TO 'C';ALTER DATABASE \"regression\" SET lc_monetary TO 'C';ALTER DATABASE \"regression\" SET lc_numeric TO 'C';ALTER DATABASE \"regression\" SET lc_time TO 'C';ALTER DATABASE \"regression\" SET bytea_output TO 'hex';ALTER DATABASE \"regression\" SET timezone_abbreviations TO 'Default';" "regression"
installcheck_1  | make[1]: *** [GNUmakefile:138: installcheck] Error 2
installcheck_1  | make[1]: Leaving directory '/build/postgres/src/test/regress'
installcheck_1  | make: *** [GNUmakefile:69: installcheck] Error 2
installcheck_installcheck_1 exited with code 2
    opened by Denchick 0
  • Add target-session-attrs option to shard config

    Add target-session-attrs option to shard config

    Implement target session attrs option and policies for backend connections. Now ew can specify in shard`s config to match replica/primary hosts while acquiring new connections

    opened by reshke 0
  • There is no console user validation

    There is no console user validation

    You can connect to the console as anybody and not understand that you are doing something wrong.

    ➜  ~ psql "host=localhost sslmode=disable user=username dbname=dbname port=7432"

		SQPR router admin console
	Here you can configure your routing rules
------------------------------------------------
	You can find documentation here
https://github.com/pg-sharding/spqr/tree/master/docs

psql (14.5 (Homebrew), server console)
Type "help" for help.

dbname=> SHOW SHARDS;
ERROR:  unknown coordinator cmd
    opened by Denchick 0
  • More informative ERROR level logs

    More informative ERROR level logs 

    INFO: 2022/11/21 08:41:57.545152 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.545522 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.545630 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.545674 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.545701 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.545732 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.545766 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.545806 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.545864 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.545887 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.545910 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.545952 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.545999 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.546024 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.546060 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.546082 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.546110 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.546139 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.546184 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.546240 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.546278 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.546301 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.546324 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.546397 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.546461 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.546482 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.546501 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.546795 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.546911 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.546989 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.547014 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.547283 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.547600 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.547623 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.548003 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.548397 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.549474 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.549915 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.550407 match route {denchick denchick}
INFO: 2022/11/21 08:41:57.550886 match route {denchick denchick}
ERROR: 2022/11/21 08:42:07.598418 <nil>
ERROR: 2022/11/21 08:42:07.600009 <nil>
ERROR: 2022/11/21 08:42:07.600029 <nil>
ERROR: 2022/11/21 08:42:07.600098 <nil>
ERROR: 2022/11/21 08:42:07.600135 <nil>
ERROR: 2022/11/21 08:42:07.600217 <nil>
ERROR: 2022/11/21 08:42:07.600240 <nil>
ERROR: 2022/11/21 08:42:07.600104 <nil>
ERROR: 2022/11/21 08:42:07.600281 <nil>
ERROR: 2022/11/21 08:42:07.600034 <nil>
ERROR: 2022/11/21 08:42:07.600296 <nil>
ERROR: 2022/11/21 08:42:07.600298 <nil>
ERROR: 2022/11/21 08:42:07.600631 <nil>
ERROR: 2022/11/21 08:42:07.600640 <nil>
ERROR: 2022/11/21 08:42:07.600744 <nil>
ERROR: 2022/11/21 08:42:07.600754 <nil>
ERROR: 2022/11/21 08:42:07.600779 <nil>
ERROR: 2022/11/21 08:42:07.600784 <nil>
ERROR: 2022/11/21 08:42:07.600990 <nil>
ERROR: 2022/11/21 08:42:07.601042 <nil>
ERROR: 2022/11/21 08:42:07.600999 <nil>
ERROR: 2022/11/21 08:42:07.601298 <nil>
ERROR: 2022/11/21 08:42:07.601399 <nil>
ERROR: 2022/11/21 08:42:07.601595 <nil>
ERROR: 2022/11/21 08:42:07.601704 <nil>
ERROR: 2022/11/21 08:42:07.601732 <nil>
ERROR: 2022/11/21 08:42:07.601788 <nil>
ERROR: 2022/11/21 08:42:07.601866 <nil>
ERROR: 2022/11/21 08:42:07.602564 <nil>
ERROR: 2022/11/21 08:42:07.602736 <nil>
ERROR: 2022/11/21 08:42:07.603088 <nil>
ERROR: 2022/11/21 08:42:07.603506 <nil>
ERROR: 2022/11/21 08:42:07.604151 <nil>
ERROR: 2022/11/21 08:42:07.604207 <nil>
ERROR: 2022/11/21 08:42:07.604720 <nil>
ERROR: 2022/11/21 08:42:07.604978 <nil>
ERROR: 2022/11/21 08:42:07.605137 <nil>
ERROR: 2022/11/21 08:42:07.605317 <nil>
ERROR: 2022/11/21 08:42:07.605332 <nil>
ERROR: 2022/11/21 08:42:07.605414 <nil>
    opened by Denchick 0
  • panic: implement me

    panic: implement me 

    panic: implement me

goroutine 78 [running]:
github.com/pg-sharding/spqr/router/pkg/server.(*MultiShardServer).HasPrepareStatement(0xc0005ea000?, 0xc000ac8c80?)
        /spqr/router/pkg/server/multishard.go:49 +0x27
github.com/pg-sharding/spqr/router/pkg/rulerouter.(*RelayStateImpl).PrepareStatement(0xc00112a240, 0x2?, {{0xc0012b6270?, 0x1?}, {0xc0008aa1d0?, 0xc0011280d8?}})
        /spqr/router/pkg/rulerouter/relay.go:91 +0x68
github.com/pg-sharding/spqr/router/pkg.Frontend.func1({0xdf0210, 0xc000720090}, 0xc000567ea8, {0xdee628?, 0xc00112a240?}, {0xdebcb8, 0xc0011280d8})
        /spqr/router/pkg/frontend.go:276 +0x428
github.com/pg-sharding/spqr/router/pkg.Frontend({0xdeed80?, 0xc0001cb800}, {0xdf0210, 0xc000720090}, {0xdebcb8?, 0xc0011280d8}, 0xc0003f8000)
        /spqr/router/pkg/frontend.go:291 +0x56c
github.com/pg-sharding/spqr/router/pkg.(*InstanceImpl).serv(0xc000181360, {0xdec3f8, 0xc000b0c100})
        /spqr/router/pkg/instance.go:130 +0x114
github.com/pg-sharding/spqr/router/pkg.(*InstanceImpl).Run.func3()
        /spqr/router/pkg/instance.go:163 +0x29
created by github.com/pg-sharding/spqr/router/pkg.(*InstanceImpl).Run
        /spqr/router/pkg/instance.go:162 +0x32a
    opened by Denchick 0
  • Unknown bug with prepared statements

    Unknown bug with prepared statements

    Some logs

    DEBUG3: 2022/11/18 07:33:43.316073 recv msg from server *pgproto3.RowDescription
DEBUG2: 2022/11/18 07:33:43.316081 got msg type from server: *pgproto3.RowDescription
DEBUG3: 2022/11/18 07:33:43.316085 sending *pgproto3.RowDescription to client
DEBUG3: 2022/11/18 07:33:43.316106 recv msg from server *pgproto3.DataRow
DEBUG2: 2022/11/18 07:33:43.316111 got msg type from server: *pgproto3.DataRow
DEBUG3: 2022/11/18 07:33:43.316115 sending *pgproto3.DataRow to client
DEBUG3: 2022/11/18 07:33:43.316129 recv msg from server *pgproto3.CommandComplete
DEBUG2: 2022/11/18 07:33:43.316134 got msg type from server: *pgproto3.CommandComplete
DEBUG3: 2022/11/18 07:33:43.316138 sending *pgproto3.CommandComplete to client
DEBUG3: 2022/11/18 07:33:43.316150 recv msg from server *pgproto3.ReadyForQuery
DEBUG1: 2022/11/18 07:33:43.316155 complete relay iter with TX active
DEBUG3: 2022/11/18 07:33:43.316160 sending *pgproto3.ReadyForQuery to client
DEBUG1: 2022/11/18 07:33:43.316172 active shards are 0x9ebee0
DEBUG3: 2022/11/18 07:33:43.316187 Received *pgproto3.Query from client
DEBUG1: 2022/11/18 07:33:43.316193 received *pgproto3.Query msg, 0xc0004bca38
DEBUG1: 2022/11/18 07:33:43.316200 received query SELECT c FROM sbtest1 WHERE id=5138
DEBUG2: 2022/11/18 07:33:43.316251 parsed query stmt is *pg_query.ParseResult
DEBUG2: 2022/11/18 07:33:43.316314 [stmt:{select_stmt:{target_list:{res_target:{val:{column_ref:{fields:{string:{str:"c"}}  location:7}}  location:7}}  from_clause:{range_var:{relname:"sbtest1"  inh:true  relpersistence:"p"  location:14}}  where_clause:{a_expr:{kind:AEXPR_OP  name:{string:{str:"="}}  lexpr:{column_ref:{fields:{string:{str:"id"}}  location:28}}  rexpr:{a_const:{val:{integer:{ival:5138}}  location:31}}  location:30}}  limit_option:LIMIT_OPTION_DEFAULT  op:SETOP_NONE}}]
DEBUG1: 2022/11/18 07:33:43.316324 adding pgproto3.Query
DEBUG1: 2022/11/18 07:33:43.316329 preparing relay step for denchick denchick
DEBUG1: 2022/11/18 07:33:43.316333 flush message buff
DEBUG1: 2022/11/18 07:33:43.316339 flushing {String:SELECT c FROM sbtest1 WHERE id=5138} waitForResp: true replyCl: true
DEBUG2: 2022/11/18 07:33:43.316345 process query &{SELECT c FROM sbtest1 WHERE id=5138}
DEBUG3: 2022/11/18 07:33:43.316618 recv msg from server *pgproto3.RowDescription
DEBUG2: 2022/11/18 07:33:43.316632 got msg type from server: *pgproto3.RowDescription
DEBUG3: 2022/11/18 07:33:43.316639 sending *pgproto3.RowDescription to client
DEBUG3: 2022/11/18 07:33:43.316666 recv msg from server *pgproto3.DataRow
DEBUG2: 2022/11/18 07:33:43.316672 got msg type from server: *pgproto3.DataRow
DEBUG3: 2022/11/18 07:33:43.316678 sending *pgproto3.DataRow to client
DEBUG3: 2022/11/18 07:33:43.316694 recv msg from server *pgproto3.CommandComplete
DEBUG2: 2022/11/18 07:33:43.316699 got msg type from server: *pgproto3.CommandComplete
DEBUG3: 2022/11/18 07:33:43.316704 sending *pgproto3.CommandComplete to client
DEBUG3: 2022/11/18 07:33:43.316708 send msg to server &{String:SELECT c FROM sbtest1 WHERE id=5138}
DEBUG3: 2022/11/18 07:33:43.316721 recv msg from server *pgproto3.ReadyForQuery
DEBUG1: 2022/11/18 07:33:43.316769 complete relay iter with TX active
DEBUG3: 2022/11/18 07:33:43.316780 sending *pgproto3.ReadyForQuery to client
DEBUG1: 2022/11/18 07:33:43.316819 active shards are 0x9ebee0
DEBUG3: 2022/11/18 07:33:43.316890 Received *pgproto3.Query from client
DEBUG1: 2022/11/18 07:33:43.316900 received *pgproto3.Query msg, 0xc0004bc878
DEBUG1: 2022/11/18 07:33:43.316909 received query COMMIT
DEBUG2: 2022/11/18 07:33:43.316943 parsed query stmt is *pg_query.ParseResult
DEBUG3: 2022/11/18 07:33:43.317033 recv msg from server *pgproto3.RowDescription
DEBUG2: 2022/11/18 07:33:43.317046 got msg type from server: *pgproto3.RowDescription
DEBUG3: 2022/11/18 07:33:43.317055 sending *pgproto3.RowDescription to client
DEBUG3: 2022/11/18 07:33:43.317091 recv msg from server *pgproto3.DataRow
DEBUG2: 2022/11/18 07:33:43.317098 got msg type from server: *pgproto3.DataRow
DEBUG3: 2022/11/18 07:33:43.317103 sending *pgproto3.DataRow to client
DEBUG3: 2022/11/18 07:33:43.317131 recv msg from server *pgproto3.CommandComplete
DEBUG2: 2022/11/18 07:33:43.317138 got msg type from server: *pgproto3.CommandComplete
DEBUG3: 2022/11/18 07:33:43.317146 sending *pgproto3.CommandComplete to client
DEBUG2: 2022/11/18 07:33:43.317140 [stmt:{transaction_stmt:{kind:TRANS_STMT_COMMIT}}]
DEBUG1: 2022/11/18 07:33:43.317210 adding pgproto3.Query
DEBUG1: 2022/11/18 07:33:43.317220 preparing relay step for denchick denchick
DEBUG1: 2022/11/18 07:33:43.317226 flush message buff
DEBUG1: 2022/11/18 07:33:43.317238 flushing {String:COMMIT} waitForResp: true replyCl: true
DEBUG2: 2022/11/18 07:33:43.317245 process query &{COMMIT}
DEBUG3: 2022/11/18 07:33:43.317194 recv msg from server *pgproto3.ReadyForQuery
DEBUG1: 2022/11/18 07:33:43.317261 complete relay iter with TX active
DEBUG3: 2022/11/18 07:33:43.317266 sending *pgproto3.ReadyForQuery to client
DEBUG3: 2022/11/18 07:33:43.317254 send msg to server &{String:COMMIT}
DEBUG1: 2022/11/18 07:33:43.317291 active shards are 0x9ebee0
DEBUG3: 2022/11/18 07:33:43.317521 Received *pgproto3.Query from client
DEBUG1: 2022/11/18 07:33:43.317528 received *pgproto3.Query msg, 0xc0004bca38
DEBUG1: 2022/11/18 07:33:43.317534 received query SELECT c FROM sbtest1 WHERE id=1958
DEBUG2: 2022/11/18 07:33:43.317652 parsed query stmt is *pg_query.ParseResult
DEBUG2: 2022/11/18 07:33:43.317737 [stmt:{select_stmt:{target_list:{res_target:{val:{column_ref:{fields:{string:{str:"c"}}  location:7}}  location:7}}  from_clause:{range_var:{relname:"sbtest1"  inh:true  relpersistence:"p"  location:14}}  where_clause:{a_expr:{kind:AEXPR_OP  name:{string:{str:"="}}  lexpr:{column_ref:{fields:{string:{str:"id"}}  location:28}}  rexpr:{a_const:{val:{integer:{ival:1958}}  location:31}}  location:30}}  limit_option:LIMIT_OPTION_DEFAULT  op:SETOP_NONE}}]
DEBUG1: 2022/11/18 07:33:43.317748 adding pgproto3.Query
DEBUG1: 2022/11/18 07:33:43.317753 preparing relay step for denchick denchick
DEBUG1: 2022/11/18 07:33:43.317758 flush message buff
DEBUG1: 2022/11/18 07:33:43.317765 flushing {String:SELECT c FROM sbtest1 WHERE id=1958} waitForResp: true replyCl: true
DEBUG3: 2022/11/18 07:33:43.317767 recv msg from server *pgproto3.CommandComplete
DEBUG2: 2022/11/18 07:33:43.317772 process query &{SELECT c FROM sbtest1 WHERE id=1958}
DEBUG3: 2022/11/18 07:33:43.317798 send msg to server &{String:SELECT c FROM sbtest1 WHERE id=1958}
DEBUG2: 2022/11/18 07:33:43.317784 got msg type from server: *pgproto3.CommandComplete
DEBUG3: 2022/11/18 07:33:43.317848 sending *pgproto3.CommandComplete to client
DEBUG3: 2022/11/18 07:33:43.317868 recv msg from server *pgproto3.ReadyForQuery
DEBUG1: 2022/11/18 07:33:43.317875 complete relay iter with TX idle
DEBUG3: 2022/11/18 07:33:43.317880 sending *pgproto3.ReadyForQuery to client
LOG: 2022/11/18 07:33:43.317898 end of tx unrouting from [{shard01 false}]
LOG: 2022/11/18 07:33:43.317903 unrouting from datashard shard01
DEBUG1: 2022/11/18 07:33:43.317912 put connection to sas-vewvjvext6fqguug.db.yandex.net:6432 back to pool
DEBUG1: 2022/11/18 07:33:43.317920 active shards are 0x9ebee0
DEBUG3: 2022/11/18 07:33:43.317959 Received *pgproto3.Query from client
DEBUG1: 2022/11/18 07:33:43.317968 received *pgproto3.Query msg, 0xc0004bc878
DEBUG1: 2022/11/18 07:33:43.317977 received query BEGIN
DEBUG2: 2022/11/18 07:33:43.318010 parsed query stmt is *pg_query.ParseResult
DEBUG2: 2022/11/18 07:33:43.318040 [stmt:{transaction_stmt:{kind:TRANS_STMT_BEGIN}}]
DEBUG1: 2022/11/18 07:33:43.318049 adding silent pgproto3.Query
DEBUG4: 2022/11/18 07:33:43.318056 start new params set
DEBUG3: 2022/11/18 07:33:43.318067 sending *pgproto3.CommandComplete to client
DEBUG3: 2022/11/18 07:33:43.318090 sending *pgproto3.ReadyForQuery to client
DEBUG3: 2022/11/18 07:33:43.318125 recv msg from server *pgproto3.RowDescription
DEBUG2: 2022/11/18 07:33:43.318134 got msg type from server: *pgproto3.RowDescription
DEBUG3: 2022/11/18 07:33:43.318140 sending *pgproto3.RowDescription to client
DEBUG3: 2022/11/18 07:33:43.318163 recv msg from server *pgproto3.DataRow
DEBUG2: 2022/11/18 07:33:43.318168 got msg type from server: *pgproto3.DataRow
DEBUG3: 2022/11/18 07:33:43.318174 sending *pgproto3.DataRow to client
DEBUG3: 2022/11/18 07:33:43.318194 recv msg from server *pgproto3.CommandComplete
DEBUG2: 2022/11/18 07:33:43.318200 got msg type from server: *pgproto3.CommandComplete
DEBUG3: 2022/11/18 07:33:43.318205 sending *pgproto3.CommandComplete to client
DEBUG3: 2022/11/18 07:33:43.318223 recv msg from server *pgproto3.ReadyForQuery
DEBUG1: 2022/11/18 07:33:43.318229 complete relay iter with TX active
DEBUG3: 2022/11/18 07:33:43.318233 sending *pgproto3.ReadyForQuery to client
DEBUG1: 2022/11/18 07:33:43.318247 active shards are 0x9ebee0
DEBUG3: 2022/11/18 07:33:43.318251 Received *pgproto3.Query from client
DEBUG1: 2022/11/18 07:33:43.318264 received *pgproto3.Query msg, 0xc0004bc878
DEBUG1: 2022/11/18 07:33:43.318271 received query SELECT c FROM sbtest1 WHERE id=5968
DEBUG3: 2022/11/18 07:33:43.318304 Received *pgproto3.Query from client
DEBUG1: 2022/11/18 07:33:43.318311 received *pgproto3.Query msg, 0xc0004bca38
DEBUG1: 2022/11/18 07:33:43.318316 received query COMMIT
DEBUG2: 2022/11/18 07:33:43.318340 parsed query stmt is *pg_query.ParseResult
DEBUG2: 2022/11/18 07:33:43.318367 [stmt:{transaction_stmt:{kind:TRANS_STMT_COMMIT}}]
DEBUG1: 2022/11/18 07:33:43.318374 adding pgproto3.Query
DEBUG1: 2022/11/18 07:33:43.318379 preparing relay step for denchick denchick
DEBUG1: 2022/11/18 07:33:43.318383 flush message buff
DEBUG1: 2022/11/18 07:33:43.318390 flushing {String:COMMIT} waitForResp: true replyCl: true
DEBUG2: 2022/11/18 07:33:43.318396 process query &{COMMIT}
DEBUG3: 2022/11/18 07:33:43.318402 send msg to server &{String:COMMIT}
DEBUG2: 2022/11/18 07:33:43.318340 parsed query stmt is *pg_query.ParseResult
DEBUG2: 2022/11/18 07:33:43.318500 [stmt:{select_stmt:{target_list:{res_target:{val:{column_ref:{fields:{string:{str:"c"}}  location:7}}  location:7}}  from_clause:{range_var:{relname:"sbtest1"  inh:true  relpersistence:"p"  location:14}}  where_clause:{a_expr:{kind:AEXPR_OP  name:{string:{str:"="}}  lexpr:{column_ref:{fields:{string:{str:"id"}}  location:28}}  rexpr:{a_const:{val:{integer:{ival:5968}}  location:31}}  location:30}}  limit_option:LIMIT_OPTION_DEFAULT  op:SETOP_NONE}}]
DEBUG1: 2022/11/18 07:33:43.318510 adding pgproto3.Query
DEBUG1: 2022/11/18 07:33:43.318515 preparing relay step for denchick denchick
DEBUG5: 2022/11/18 07:33:43.318552 mathcing qstmt *pg_query.Node_TransactionStmt
DEBUG3: 2022/11/18 07:33:43.318569 sending *pgproto3.ErrorResponse to client
DEBUG3: 2022/11/18 07:33:43.318591 sending *pgproto3.ReadyForQuery to client
DEBUG5: 2022/11/18 07:33:43.318606 client iter done with too complex query to parse
DEBUG3: 2022/11/18 07:33:43.318592 recv msg from server *pgproto3.CommandComplete
DEBUG2: 2022/11/18 07:33:43.318620 got msg type from server: *pgproto3.CommandComplete
DEBUG3: 2022/11/18 07:33:43.318624 sending *pgproto3.CommandComplete to client
DEBUG3: 2022/11/18 07:33:43.318640 recv msg from server *pgproto3.ReadyForQuery
DEBUG1: 2022/11/18 07:33:43.318645 complete relay iter with TX idle
DEBUG3: 2022/11/18 07:33:43.318650 sending *pgproto3.ReadyForQuery to client
LOG: 2022/11/18 07:33:43.318664 end of tx unrouting from [{shard01 false}]
LOG: 2022/11/18 07:33:43.318669 unrouting from datashard shard01
DEBUG1: 2022/11/18 07:33:43.318675 put connection to sas-vewvjvext6fqguug.db.yandex.net:6432 back to pool
DEBUG1: 2022/11/18 07:33:43.318681 active shards are 0x9ebee0
DEBUG3: 2022/11/18 07:33:43.325298 Received <nil> from client
DEBUG3: 2022/11/18 07:33:43.325371 Received <nil> from client
DEBUG3: 2022/11/18 07:33:43.325388 sending *pgproto3.ErrorResponse to client
DEBUG1: 2022/11/18 07:33:43.325417 received <nil> msg, %!p(<nil>)
DEBUG3: 2022/11/18 07:33:43.325425 Received <nil> from client
    opened by Denchick 3
  • Some small log fixes

    Some small log fixes 

    LOG: 2022/11/18 07:27:59.380872 unrouting from datashard shard01
DEBUG1: 2022/11/18 07:27:59.380908 put connection to sas-vewvjvext6fqguug.db.yandex.net:6432 back to pool
DEBUG1: 2022/11/18 07:27:59.380925 active shards are 0x9ebee0
    opened by Denchick 0
Releases(0.0.2)

  • 0.0.2(Oct 26, 2022)

    What's changed

    Coordinator

    • Router-side key range moves by @reshke (#29)
    • Coordinator fixes by @reshke (#30, #47)
    • Implement drop key ranges by @reshke (#31)
    • Balancer console by @reshke (#33)
    • Simple mover by @reshke (#39)
    • Add sharding rules support in coordinator by @reshke (#48)
    • Add drop sharding rules statement support by @reshke (#51)
    • Add missing coordinator and router functionality by @reshke (#52)
    • Fix bug with add key range using coordinator by @reshke (#53)

    Router

    • TLS config and router small refactoring by @denchick (#38)
    • Multishard feature by @reshke (#45)
    • Simplify configs, add docs by @denchick (#46)
    • Fix json and toml config formats support by @denchick (#54)
    • Add simple COPY support by @denchick and @reshke (#56, #58)
    • Add target-session-attrs option to shard config by @reshke (#65)
    • Online reload for router configuration by @reshke (#77)

    Docs, tests

    • Add regression PG-style tests by @EinKrebs by @denchick @reshke (#43, #62, #67, #72, #73, #74)
    • Add docs for coordinator, router and world by @denchick (814622e, 6a77d9b, 2267506, #49)
    • Add new test set with joins-relation queryies tests by @reshke (#63)
    Source code(tar.gz)
    Source code(zip)
Owner
null
GitHub
Bxog is a simple and fast HTTP router for Go (HTTP request multiplexer).

Bxog is a simple and fast HTTP router for Go (HTTP request multiplexer). Usage An example of using the multiplexer: package main import ( "io" "net

Eduard 106 Dec 26, 2022
A high performance fasthttp request router that scales well

FastHttpRouter FastHttpRouter is forked from httprouter which is a lightweight high performance HTTP request router (also called multiplexer or just m

招牌疯子 871 Dec 1, 2022
FastRouter is a fast, flexible HTTP router written in Go.

FastRouter FastRouter is a fast, flexible HTTP router written in Go. FastRouter contains some customizable options, such as TrailingSlashesPolicy, Pan

Razon Yang 22 Sep 27, 2022
Go Router + Middleware. Your Contexts.

gocraft/web gocraft/web is a Go mux and middleware package. We deal with casting and reflection so YOUR code can be statically typed. And we're fast.

null 1.5k Dec 28, 2022
Go Server/API micro framework, HTTP request router, multiplexer, mux

?? gorouter Go Server/API micro framework, HTTP request router, multiplexer, mux. ?? ABOUT Contributors: Rafał Lorenz Want to contribute ? Feel free t

Rafał Lorenz 143 Dec 16, 2022
A high performance HTTP request router that scales well

HttpRouter HttpRouter is a lightweight high performance HTTP request router (also called multiplexer or just mux for short) for Go. In contrast to the

Julien Schmidt 14.8k Dec 28, 2022
High-speed, flexible tree-based HTTP router for Go.

httptreemux High-speed, flexible, tree-based HTTP router for Go. This is inspired by Julien Schmidt's httprouter, in that it uses a patricia tree, but

Daniel Imfeld 575 Dec 28, 2022
:rotating_light: Is a lightweight, fast and extensible zero allocation HTTP router for Go used to create customizable frameworks.

LARS LARS is a fast radix-tree based, zero allocation, HTTP router for Go. view examples. If looking for a more pure Go solution, be sure to check out

Go Playgound 388 Dec 27, 2022
A powerful HTTP router and URL matcher for building Go web servers with 🦍

gorilla/mux https://www.gorillatoolkit.org/pkg/mux Package gorilla/mux implements a request router and dispatcher for matching incoming requests to th

Gorilla Web Toolkit 18k Jan 9, 2023
An extremely fast Go (golang) HTTP router that supports regular expression route matching. Comes with full support for building RESTful APIs.

ozzo-routing You may consider using go-rest-api to jumpstart your new RESTful applications with ozzo-routing. Description ozzo-routing is a Go package

Ozzo Framework 447 Dec 31, 2022
Pure is a fast radix-tree based HTTP router

package pure Pure is a fast radix-tree based HTTP router that sticks to the native implementations of Go's "net/http" package; in essence, keeping the

Go Playgound 132 Dec 1, 2022
Echo Inspired Stand Alone URL Router

Vestigo - A Standalone Golang URL Router Abstract Many fast Golang URL routers are often embedded inside frameworks. Vestigo is a stand alone url rout

null 268 Dec 1, 2022
Go HTTP router

violetear Go HTTP router http://violetear.org Design Goals Keep it simple and small, avoiding extra complexity at all cost. KISS Support for static an

Nicolas Embriz 106 Dec 10, 2022
xujiajun/gorouter is a simple and fast HTTP router for Go. It is easy to build RESTful APIs and your web framework.

gorouter xujiajun/gorouter is a simple and fast HTTP router for Go. It is easy to build RESTful APIs and your web framework. Motivation I wanted a sim

徐佳军 533 Dec 8, 2022
lightweight, idiomatic and composable router for building Go HTTP services

chi is a lightweight, idiomatic and composable router for building Go HTTP services. It's especially good at helping you write large REST API services

go-chi 13k Jan 8, 2023
Router socks. One port socks for all the others.

Router socks The next step after compromising a machine is to enumerate the network behind. Many tools exist to expose a socks port on the attacker's

null 62 Dec 13, 2022
:tongue: CleverGo is a lightweight, feature rich and high performance HTTP router for Go.

CleverGo CleverGo is a lightweight, feature rich and trie based high performance HTTP request router. go get -u clevergo.tech/clevergo English 简体中文 Fe

CleverGo Web Framework 248 Nov 17, 2022
Fast and flexible HTTP router

treemux - fast and flexible HTTP router Basic example Debug logging CORS example Error handling Rate limiting using Redis Gzip compression OpenTelemet

Vladimir Mihailenco 534 Dec 27, 2022
Fast, simple, and lightweight HTTP router for Golang

Sariaf Fast, simple and lightweight HTTP router for golang Install go get -u github.com/majidsajadi/sariaf Features Lightweight compatible with net/ht

defectivepixel 33 Aug 19, 2022