MatrixOne is a planet scale, cloud-edge native big data engine crafted for heterogeneous workloads.

Overview

LICENSE Language CodeFactor

What is MatrixOne?

MatrixOne is a planet scale, cloud-edge native big data engine crafted for heterogeneous workloads. It provides an end-to-end data processing platform that is highly autonomous and easy to use, to empower users to store, manipulate, and analyze data across devices, edges, and clouds, with minimal operational overheads.

Features

Planet Scalability

MatrixOne cluster can easily expand capacity during SQL processing, computation, and storage, by adding nodes to the cluster on the fly.

Cloud-Edge Native

Not limited to public clouds, hybrid clouds, on-premise servers, or smart devices, MatrixOne accommodates itself to myriads of infrastructure while still providing top services with low latency and high throughput.

Hybrid Streaming, Transactional and Analytical Processing Engine

By converging multiple engines, MatrixOne can support hybrid streaming, transactional, and analytical workloads; with its pluggable architecture, MatrixOne allows for easy integration with third-party engines.

High Availability

MatrixOne uses a RAFT-based consensus algorithm to provide fault tolerance in one zone. And a more advanced state-machine replication protocol is planned for the future to achieve geo-distributed active-active.

Ease of Use

An important goal of MatrixOne is to make it easy for users to operate and manage data, making daily work almost effortless.

  • No Dependency: Download, install, or start MatrixOne straightforwardly without depending on external toolings.
  • Simplify Administration: Re-balancing, failover, system tuning, and other administrative tasks are fully automatic.
  • MySQL-compatible Syntax: MatrixOne allows you to query data using traditional SQL queries.

End-to-End Automated Data Science

By streaming SQL and user-defined functions, MatrixOne provides end-to-end data processing pipelines to deliver productive data science applications.

Architecture

Architecture

Query Parser Layer

  • Parser: Parses SQL, Streaming Query, or Python language into an abstract syntax tree for further processing.
  • Planner: Finds the best execution plan through rule-based, cost-based optimization algorithms, and transfers abstract syntax tree to plan tree.
  • IR Generator: Converts Python code into an intermediate representation.

Computation Layer

  • JIT Compilation: Turns SQL plan tree or IR code into a native program using LLVM at runtime.
  • Vectorized Execution: MatrixOne leverages SIMD instructions to construct vectorized execution pipelines.
  • Cache: Multi-version cache of data, indexes, and metadata for queries.

Cluster Management Layer

MatrixCube is a fundamental library for building distributed systems, which offers guarantees about reliability, consistency, and scalability. It is designed to facilitate distributed, stateful application building to allow developers only need to focus on the business logic on a single node. MatrixCube is currently built upon multi-raft to provide replicated state machine and will migrate to Paxos families to increase friendliness to scenarios spanning multiple data centers.

  • Prophet: Used by MatrixCube to manage and schedule the MatrixOne cluster.
  • Transaction Manager: MatrixOne supports distributed transaction of snapshot isolation level.
  • Replicated State Machine: MatrixOne uses RAFT-based consensus algorithms and hyper logic clocks to implement strong consistency of the clusters. Introduction of more advanced state-machine replication protocols is yet to come.

Replicated Storage Layer

  • Row Storage: Stores serving workload, metadata, and catalog.
  • Column Storage: Stores analytical workload and materialized views.

Storage Provision Layer

MatrixOne stores data in shared storage of S3 / HDFS, or the local disk, on-premise server, hybrid and any cloud, or even smart devices.

Quick Start

Get started with MatrixOne quickly by the following steps.

Installation

You can install MatrixOne either by building from source or using docker.

Building from source

  1. Install Go (version 1.17 is required).

  2. Get the MatrixOne code:

$ git clone https://github.com/matrixorigin/matrixone.git
$ cd matrixone
  1. Run make:

    You can run make debug, make clean, or anything else our Makefile offers.

$ make config
$ make build
  1. Boot MatrixOne server:
$ ./mo-server system_vars_config.toml

Using docker

  1. Install Docker, then verify that Docker daemon is running in the background:
$ docker --version
  1. Create and run the container for the latest release of MatrixOne. It will pull the image from Docker Hub if not exists.
$ docker run -d -p 6001:6001 --name matrixone matrixorigin/matrixone:latest

Connecting MatrixOne server

  1. Install MySQL client.

    MatrixOne supports the MySQL wire protocol, so you can use MySQL client drivers to connect from various languages. Currently, MatrixOne is only compatible with Oracle MySQL client. This means that some features might not work with MariaDB client.

  2. Connect MatrixOne server:

$ mysql -h IP -P PORT -uUsername -p

The connection string is the same format as MySQL accepts. You need to provide a user name and a password.

Use the built-in test account for example:

  • user: dump
  • password: 111
$ mysql -h 127.0.0.1 -P 6001 -udump -p
Enter password:

Now, MatrixOne only supports the TCP listener.

Contributing

See Contributing Guide for details on contribution workflows.

Roadmap

Check out Roadmap for MatrixOne development plan.

Community

You can join MatrixOne community on Slack to discuss and ask questions.

License

MatrixOne is licensed under the Apache License, Version 2.0.

Issues
  • [Bug]:  mo-server stucked when handling sql, aoe, plan2

    [Bug]: mo-server stucked when handling sql, aoe, plan2

    Is there an existing issue for the same bug?

    • [X] I have checked the existing issues.

    Environment

    - Version or commit-id (e.g. v0.1.0 or 8b23a93): d257eee
    - Hardware parameters:
    - OS type:
    - Others:
    

    Actual Behavior

    The connection occasionally hangs if there is an uncaught runtime error during executing SQL.

    {"level":"INFO","time":"2022/06/07 16:49:26.652427 +0000","caller":"frontend/mysql_cmd_executor.go:2131","msg":"cmd 3"}
    {"level":"INFO","time":"2022/06/07 16:49:26.654911 +0000","caller":"frontend/mysql_cmd_executor.go:2154","msg":"query:drop table if exists t2;\n"}
    {"level":"DEBUG","time":"2022/06/07 16:49:26.561731 +0000","name":"cube.prophet.rpc.goetty","caller":"[email protected]/application.go:231","msg":"session read","node":"node0","listen-address":"127.0.0.1:30000","session-id":1,"addr":"127.0.0.1:49068","msg":"id:16525 type:TypeShardHeartbeatReq shardHeartbeat:<storeID:1 shard:\"\\010\\\\\\022\\014aoe-\\000\\000\\000\\000\\000\\000\\000,\\032\\014aoe-\\000\\000\\000\\000\\000\\000\\000-\\\"\\002\\010\\001(\\0012\\006\\010]\\020\\001 \\0018\\001B\\0061-1-44R\\020\\n\\nLabelTable\\022\\002t1\" term:2 leader:<id:93 storeID:1 initialMember:true > stats:<writtenBytes:946 writtenKeys:3 readKeys:17 approximateSize:946 interval:<start:1654620549 end:1654620565 > > groupKey:\"\\000\\000\\000\\000\\000\\000\\000\\001\\000t1\" > storeHeartbeat:<stats:<> > putStore:<> getStore:<> allocID:<> askBatchSplit:<> createDestroying:<> ReportDestroyed:<> getDestroying:<> createWatcher:<> createShards:<> removeShards:<> checkShardState:<> putPlacementRule:<rule:<> > getAppliedRules:<> createJob:<job:<> > removeJob:<job:<> > executeJob:<job:<> > addScheduleGroupRule:<rule:<> > getScheduleGroupRule:<> "}
    {"level":"DEBUG","time":"2022/06/07 16:49:26.657940 +0000","name":"cube.prophet","caller":"prophet/prophet_handler.go:50","msg":"rpc request received","node":"node0","id":16525,"from":"127.0.0.1:49068","type":"TypeShardHeartbeatReq"}
    {"level":"DEBUG","time":"2022/06/07 16:49:26.658852 +0000","name":"cube.store.router","caller":"raftstore/router.go:412","msg":"shard route updated","store":"node0","shard":"shard-id: 2406, shard-group: 1, shard-epoch: 0/1, shard-range: [616f652d00000000000004b1, 616f652d00000000000004b2), replicas: [2407/1/Voter], shard-state: Creating, shard-unique: 1-1-1201, shard-labels: [LabelTable:t2]","leader":2407}
    {"level":"INFO","time":"2022/06/07 16:49:26.660405 +0000","name":"cube.store.router","caller":"raftstore/router.go:448","msg":"shard leader updated","store":"node0","shard-id":2406,"leader-replica":"2407/1/Voter","address":"localhost:20000"}
    {"level":"INFO","time":"2022/06/07 16:49:26.592376 +0000","name":"cube.prophet.raft-cluster","caller":"cluster/cluster.go:285","msg":"coordinator has been stopped","node":"node0"}
    {"level":"DEBUG","time":"2022/06/07 16:49:26.661607 +0000","name":"cube.store","caller":"raftstore/replica_event_proposal.go:76","msg":"push to proposal batch","store":"node0","store-id":1,"shard-id":2,"replica-id":3,"id":"4210e647bf0c4980b3bc128f37fa0753"}
    {"level":"INFO","time":"2022/06/07 16:49:26.660735 +0000","name":"cube.prophet.rpc.goetty","caller":"[email protected]/application.go:226","msg":"session read failed","node":"node0","listen-address":"127.0.0.1:30000","session-id":2,"addr":"127.0.0.1:49070","error":"read tcp4 127.0.0.1:30000->127.0.0.1:49070: use of closed network connection"}
    {"level":"INFO","time":"2022/06/07 16:49:26.662674 +0000","name":"cube.prophet.watch-notify","caller":"prophet/event_notifier.go:122","msg":"watcher removed","node":"node0","address":"127.0.0.1:49070"}
    {"level":"INFO","time":"2022/06/07 16:49:26.662688 +0000","name":"cube.prophet.watch-notify","caller":"prophet/event_notifier.go:168","msg":"watcher notifier stopped","node":"node0"}
    {"level":"ERROR","time":"2022/06/07 16:49:26.748947 +0000","name":"cube.prophet.client.watcher","caller":"prophet/event_watcher.go:124","msg":"fail to read events","node":"node0","error":"EOF"}
    {"level":"ERROR","time":"2022/06/07 16:49:26.752093 +0000","name":"cube.prophet.client.watcher","caller":"prophet/event_watcher.go:96","msg":"fail to reset connection, retry later","node":"node0"}
    {"level":"DEBUG","time":"2022/06/07 16:49:26.662793 +0000","name":"cube.store","caller":"raftstore/replica_event_proposal.go:173","msg":"call read index","store":"node0","store-id":1,"shard-id":2,"replica-id":3,"id":"3daf2044150940728417ad7eb5be1192"}
    {"level":"DEBUG","time":"2022/06/07 16:49:26.753497 +0000","name":"cube.store","caller":"raftstore/read_index_queue.go:74","msg":"read index ready","store":"node0","store-id":1,"shard-id":2,"replica-id":3,"index":7815,"batch-id":"3daf2044150940728417ad7eb5be1192"}
    {"level":"DEBUG","time":"2022/06/07 16:49:26.755170 +0000","name":"cube.store","caller":"raftstore/replica.go:483","msg":"begin to exec read requests","store":"node0","store-id":1,"shard-id":2,"replica-id":3,"request-id":"4210e647bf0c4980b3bc128f37fa0753","request":"id: 4210e647bf0c4980b3bc128f37fa0753, key: 6d657461000000000000000150726553706c6974, pid: 0, type: Read, custom-type: 3, cmd: 44 bytes, to-shard: 0"}
    {"level":"DEBUG","time":"2022/06/07 16:49:26.759745 +0000","name":"cube.store.proxy","caller":"raftstore/proxy.go:309","msg":"requests done","store":"node0","store-id":1,"resp":"id: 4210e647bf0c4980b3bc128f37fa0753, pid: 0, type: Read, error: , value: 0 bytes"}
    {"level":"DEBUG","time":"2022/06/07 16:49:26.761208 +0000","name":"cube.cube-client","caller":"client/client.go:300","msg":"response received","request-id":"4210e647bf0c4980b3bc128f37fa0753"}
    {"level":"INFO","time":"2022/06/07 16:49:26.761775 +0000","name":"cube.prophet","caller":"prophet/prophet_job.go:89","msg":"stop jobs","node":"node0"}
    {"level":"INFO","time":"2022/06/07 16:49:26.761796 +0000","name":"cube.prophet","caller":"prophet/prophet_job.go:95","msg":"stop jobs","node":"node0","count":1}
    {"level":"INFO","time":"2022/06/07 16:49:26.762358 +0000","name":"cube.store.shard-pool","caller":"raftstore/store_shards_pool.go:292","msg":"dynamic shards pool job stopped","store":"node0"}
    {"level":"DEBUG","time":"2022/06/07 16:49:26.755960 +0000","name":"cube.prophet.rpc.goetty","caller":"[email protected]/application.go:231","msg":"session read","node":"node0","listen-address":"127.0.0.1:30000","session-id":1,"addr":"127.0.0.1:49068","msg":"id:16526 type:TypeShardHeartbeatReq shardHeartbeat:<storeID:1 shard:\"\\010\\326\\004\\022\\014aoe-\\000\\000\\000\\000\\000\\000\\001)\\032\\014aoe-\\000\\000\\000\\000\\000\\000\\001*\\\"\\002\\010\\001(\\0012\\007\\010\\327\\004\\020\\001 \\0018\\001B\\0071-1-297R\\020\\n\\nLabelTable\\022\\002t1\" term:2 leader:<id:599 storeID:1 initialMember:true > stats:<writtenBytes:832 writtenKeys:3 readKeys:6 approximateSize:832 interval:<start:1654620549 end:1654620565 > > groupKey:\"\\000\\000\\000\\000\\000\\000\\000\\001\\000t1\" > storeHeartbeat:<stats:<> > putStore:<> getStore:<> allocID:<> askBatchSplit:<> createDestroying:<> ReportDestroyed:<> getDestroying:<> createWatcher:<> createShards:<> removeShards:<> checkShardState:<> putPlacementRule:<rule:<> > getAppliedRules:<> createJob:<job:<> > removeJob:<job:<> > executeJob:<job:<> > addScheduleGroupRule:<rule:<> > getScheduleGroupRule:<> "}
    {"level":"DEBUG","time":"2022/06/07 16:49:26.763393 +0000","name":"cube.prophet","caller":"prophet/prophet_handler.go:50","msg":"rpc request received","node":"node0","id":16526,"from":"127.0.0.1:49068","type":"TypeShardHeartbeatReq"}
    
    

    The complete log can be downloaded by https://shared-dev-data-1308875761.cos.ap-chengdu.myqcloud.com/mo-server.log.zip

    Expected Behavior

    No response

    Steps to Reproduce

    No response

    Additional information

    No response

    kind/bug severity/s0 
    opened by sukki37 15
  • [MatrixCamp] Add built-in function rpad

    [MatrixCamp] Add built-in function rpad

    What type of PR is this?

    • [ ] API-change
    • [ ] BUG
    • [ ] Improvement
    • [ ] Documentation
    • [x] Feature
    • [ ] Test and CI
    • [ ] Code Refactoring

    Which issue(s) this PR fixes:

    issue #1986

    What this PR does / why we need it:

    Add built-in function rpad(str,len,padstr)

    Special notes for your reviewer:

    Not Available

    Additional documentation (e.g. design docs, usage docs, etc.):

    Not Available

    good first issue reviewed MatrixCamp 
    opened by Fungx 13
  • [Bug]: EXPLAIN failed when aggregation function without GROUP BY

    [Bug]: EXPLAIN failed when aggregation function without GROUP BY

    Is there an existing issue for the same bug?

    • [X] I have checked the existing issues.

    Environment

    - Version or commit-id (e.g. v0.1.0 or 8b23a93):
    - Hardware parameters:
    - OS type:
    - Others:
    

    Actual Behavior

    select sum(ti) from t1 order by ti; +---------+ | sum(ti) | +---------+ | 39 | +---------+ 1 row in set (0.03 sec)

    explain select sum(ti) from t1 order by ti; ERROR 1105 (HY000): unknown error:[42803]'ti' must appear in the GROUP BY clause or be used in an aggregate function

    Expected Behavior

    No response

    Steps to Reproduce

    No response

    Additional information

    No response

    kind/bug needs-triage 
    opened by domingozhang 11
  • [MatrixCamp] Add Aggregate function bit_and

    [MatrixCamp] Add Aggregate function bit_and

    What type of PR is this?

    • [ ] API-change
    • [ ] BUG
    • [ ] Improvement
    • [ ] Documentation
    • [x] Feature
    • [ ] Test and CI
    • [ ] Code Refactoring

    Which issue(s) this PR fixes:

    issue #1993

    What this PR does / why we need it:

    Add aggregate function bit_and

    Special notes for your reviewer:

    Not Available

    Additional documentation (e.g. design docs, usage docs, etc.):

    Not Available

    good first issue reviewed MatrixCamp 
    opened by noneback 11
  • [Bug]:  Server crashed after loading data into server

    [Bug]: Server crashed after loading data into server

    Is there an existing issue for the same bug?

    • [X] I have checked the existing issues.

    Environment

    - Version or commit-id (e.g. v0.1.0 or 8b23a93):  d099c08c0b9abd7659dc6ac8b785214e121e3639
    - Hardware parameters: 
    - OS type: archlinux
    - Others:
    

    Actual Behavior

    {"level":"INFO","time":"2022/02/13 15:11:13.117327 +0800","caller":"aoe/storage.go:473","msg":"GetInitialStates LogIndex is 1, ShardID is 45, Metadata is shard:<id:45 start:\"aoe-\\000\\000\\000\\000\\000\\000\\000\\025\" end:\"aoe-\\000\\000\\000\\000\\000\\000\\000\\026\" epoch:<confVer:1 > state:Creating replicas:<id:46 containerID:1 initialMember:true > group:1 unique:\"1-1-21\" > \n"}
    panic: runtime error: index out of range [-1]
    
    goroutine 1 [running]:
    github.com/matrixorigin/matrixone/pkg/vm/driver/aoe.(*Storage).GetInitialStates(0xc002dae1b0)
            /home/sundy/work/matrixone/pkg/vm/driver/aoe/storage.go:450 +0xba5
    github.com/matrixorigin/matrixcube/raftstore.(*store).startShards.func1({0x2809778, 0xc002dae1b0})
            /home/sundy/gopath/pkg/mod/github.com/matrixorigin/[email protected]/raftstore/store.go:430 +0x8e
    github.com/matrixorigin/matrixone/pkg/vm/driver.NewCubeDriverWithFactory.func2(0xc002ccb900)
            /home/sundy/work/matrixone/pkg/vm/driver/cube_driver.go:192 +0x53
    github.com/matrixorigin/matrixcube/raftstore.(*store).startShards(0xc0001ce380)
            /home/sundy/gopath/pkg/mod/github.com/matrixorigin/[email protected]/raftstore/store.go:429 +0x19e
    github.com/matrixorigin/matrixcube/raftstore.(*store).Start(0xc0001ce380)
            /home/sundy/gopath/pkg/mod/github.com/matrixorigin/[email protected]/raftstore/store.go:185 +0x6f2
    github.com/matrixorigin/matrixcube/server.(*Application).Start(0xc003058a80)
            /home/sundy/gopath/pkg/mod/github.com/matrixorigin/[email protected]/server/server.go:84 +0x86
    github.com/matrixorigin/matrixone/pkg/vm/driver.(*driver).Start(0xc002d9d180)
            /home/sundy/work/matrixone/pkg/vm/driver/cube_driver.go:251 +0x3a
    main.main()
            /home/sundy/work/matrixone/cmd/db-server/main.go:253 +0xe56
    ❯
    

    Expected Behavior

    No response

    Steps to Reproduce

    Follow the ssb benchmark docs in https://docs.matrixorigin.io/0.2.0/MatrixOne/Get-Started/Tutorial/SSB-test-with-matrixone/#3-create-tables-in-matrixone
    
    Load the data using:
    
    
    load data infile '/ssb-dbgen-path/lineorder_flat.tbl ' into table lineorder_flat FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' LINES TERMINATED BY '\n';
    
    

    Then the server crashed and I restarted the matrix-server, client can't connect to the server again. BTW how to cleanup the data && metadata in matrixone?

    
    
    ### Additional information
    
    _No response_
    kind/bug triage/accepted 
    opened by sundy-li 11
  • [MatrixCamp] Add new feature : now

    [MatrixCamp] Add new feature : now

    What type of PR is this?

    • [ ] API-change
    • [ ] BUG
    • [ ] Improvement
    • [ ] Documentation
    • [x] Feature
    • [ ] Test and CI
    • [ ] Code Refactoring

    Which issue(s) this PR fixes:

    issue #https://github.com/matrixorigin/matrixone/issues/1830

    What this PR does / why we need it:

    Implemented now() function. Notice : types.DateTime hasn't yet implemented fsp(fractional seconds precision), so, don't consider fsp when format DateTime to string.

    Special notes for your reviewer:

    Not Available

    Additional documentation (e.g. design docs, usage docs, etc.):

    Not Available

    MatrixCamp 
    opened by ikenchina 10
  • [Feature Request]: Datetime Built-in function utc_time()

    [Feature Request]: Datetime Built-in function utc_time()

    Is there an existing issue for the same feature request?

    • [X] I have checked the existing issues.

    Is your feature request related to a problem?

    Describe the feature you'd like

    Function syntax

    UTC_TIME()

    Returns the current UTC time as a value in 'hh:mm:ss' format.

    mysql> SELECT UTC_TIME();
            -> '18:07:53'
    

    Describe implementation you've considered

    Hint: we have already some useful functions implemented at for date and datetime type.

    Documentation, Adoption, Use Case, Migration Strategy

    Please refer to the tutorial about how to develop a builtin function.

    Additional information

    built-in funtcion 
    opened by dengn 10
  • [Bug]: operation imprecision when round with float numbers

    [Bug]: operation imprecision when round with float numbers

    Is there an existing issue for the same bug?

    • [X] I have checked the existing issues.

    Environment

    - Version or commit-id (e.g. v0.1.0 or 8b23a93):
    - Hardware parameters:
    - OS type:
    - Others:
    

    Actual Behavior

    mysql> select round(999999999.0, -9); +------------------------+ | round(999999999.0, -9) | +------------------------+ | 999999999.9999999 | +------------------------+ 1 row in set (0.00 sec)

    mysql> select round(999999999999999999.0, -18); +----------------------------------+ | round(999999999999999999.0, -18) | +----------------------------------+ | 999999999999999900 | +----------------------------------+ 1 row in set (0.00 sec)

    Expected Behavior

    No response

    Steps to Reproduce

    No response

    Additional information

    No response

    kind/bug needs-triage bvt 
    opened by broccoliSpicy 9
  • [Bug]: `SHOW xxx` failed when running plan2 on TAE

    [Bug]: `SHOW xxx` failed when running plan2 on TAE

    Is there an existing issue for the same bug?

    • [X] I have checked the existing issues.

    Environment

    - Version or commit-id (e.g. v0.1.0 or 8b23a93): 8c8674e
    - Hardware parameters:
    - OS type:
    - Others:
       special config:
        usePlan2 = true
        storageEngine = "aoe"
    

    Actual Behavior

    image

    Expected Behavior

    No response

    Steps to Reproduce

    No response

    Additional information

    No response

    kind/bug severity/s0 
    opened by sukki37 9
  • [MatrixCamp] Add built-in function REVERSE

    [MatrixCamp] Add built-in function REVERSE

    What type of PR is this?

    • [ ] API-change
    • [ ] BUG
    • [ ] Improvement
    • [ ] Documentation
    • [x] Feature
    • [ ] Test and CI
    • [ ] Code Refactoring

    Which issue(s) this PR fixes:

    issue #1989

    What this PR does / why we need it:

    Add string reverse(str) built-in function

    Special notes for your reviewer:

    Not Available

    Additional documentation (e.g. design docs, usage docs, etc.):

    Not Available

    good first issue reviewed MatrixCamp 
    opened by noneback 9
  • [LongTermTests]: runtime error: index out of range [3] with length 3

    [LongTermTests]: runtime error: index out of range [3] with length 3

    Is there an existing issue for the same bug?

    • [X] I have checked the existing issues.

    Environment

    - MatrixOne Commit ID: d7ff30a98d8dc9aaf441b02d15f357f57b91f8f2
    - MatrixCube Version: v0.1.0
    - OS type: CentOS 7
    - Go Version: 1.17
    

    Actual Behavior

    Found connection painc with below error:

    2021/11/23 12:38:28.500378 +0800 ERROR [email protected]/application.go:242 connection painc {"err": "runtime error: index out of range [3] with length 3", "stack": "goroutine 197270 [running]:\ngithub.com/fagongzi/goetty.(*server).doStart.func1.1()\n\t/home/devops/go/pkg/  mod/github.com/fagongzi/[email protected]/application.go:241 +0x7b\npanic({0x2252380, 0xc01496f9f8})\n\t/usr/local/go/src/runtime/panic.go:1038 +0x215\ngithub.com/matrixorigin/matrixone/pkg/frontend.(*MysqlProtocolImpl).analyseHandshakeResponse320(0xc00bdd6820,              {0xc01ccba5b8, 0x0, 0xc05f39f11dc48bc4})\n\t/home/devops/matrixone/pkg/frontend/mysql_protocol.go:763 +0x4a5\ngithub.com/matrixorigin/matrixone/pkg/frontend.(*MysqlProtocolImpl).handleHandshake(0xc00bdd6820, {0xc01ccba5b8, 0x0, 0x3})\n\t/home/devops/matrixone/pkg/        frontend/mysql_protocol.go:519 +0x3a8\ngithub.com/matrixorigin/matrixone/pkg/frontend.(*RoutineManager).Handler(0xc0081056e0, {0x26225a0, 0xc024586420}, {0x203b2a0, 0xc01bd691a0}, 0xc)\n\t/home/devops/matrixone/pkg/frontend/routine_manager.go:117 +0x41e\ngithub.com/      fagongzi/goetty.(*server).doConnection(0xc00810e840, {0x26225a0, 0xc024586420})\n\t/home/devops/go/pkg/mod/github.com/fagongzi/[email protected]/application.go:287 +0x503\ngithub.com/fagongzi/goetty.(*server).doStart.func1()\n\t/home/devops/go/pkg/mod/github.com/fagongzi/   [email protected]/application.go:258 +0xc8\ncreated by github.com/fagongzi/goetty.(*server).doStart\n\t/home/devops/go/pkg/mod/github.com/fagongzi/[email protected]/application.go:236 +0x192\n"}
    

    Expected Behavior

    No index out of range exception

    Steps to Reproduce

    1. Start up 5 threads to random execute sql statements without service failure for long term.
    2. Found below panic error from cube:
    
    +++++++++
    :3
    +++++++++
    2021/11/23 12:38:28.498604 +0800 INFO v1/collection.go:156 Append logindex: S-867:708:<0+0/1>
    2021/11/23 12:38:28.499368 +0800 INFO [email protected]/application.go:267 session connected {"session-id": 2396, "addr": "45.9.20.111:63054"}
    2021/11/23 12:38:28.499407 +0800 INFO [email protected]/application.go:272 session read {"session-id": 2396, "addr": "45.9.20.111:63054", "msg": {"Length":3,"SequenceID":47,"Payload":"KuAA"}}
    2021/11/23 12:38:28.499420 +0800 INFO frontend/routine_manager.go:116 HANDLE HANDSHAKE
    2021/11/23 12:38:28.500281 +0800 INFO frontend/routine_manager.go:68 will close iosession
    2021/11/23 12:38:28.500378 +0800 ERROR [email protected]/application.go:242 connection painc {"err": "runtime error: index out of range [3] with length 3", "stack": "goroutine 197270 [running]:\ngithub.com/fagongzi/goetty.(*server).doStart.func1.1()\n\t/home/devops/go/pkg/  mod/github.com/fagongzi/[email protected]/application.go:241 +0x7b\npanic({0x2252380, 0xc01496f9f8})\n\t/usr/local/go/src/runtime/panic.go:1038 +0x215\ngithub.com/matrixorigin/matrixone/pkg/frontend.(*MysqlProtocolImpl).analyseHandshakeResponse320(0xc00bdd6820,              {0xc01ccba5b8, 0x0, 0xc05f39f11dc48bc4})\n\t/home/devops/matrixone/pkg/frontend/mysql_protocol.go:763 +0x4a5\ngithub.com/matrixorigin/matrixone/pkg/frontend.(*MysqlProtocolImpl).handleHandshake(0xc00bdd6820, {0xc01ccba5b8, 0x0, 0x3})\n\t/home/devops/matrixone/pkg/        frontend/mysql_protocol.go:519 +0x3a8\ngithub.com/matrixorigin/matrixone/pkg/frontend.(*RoutineManager).Handler(0xc0081056e0, {0x26225a0, 0xc024586420}, {0x203b2a0, 0xc01bd691a0}, 0xc)\n\t/home/devops/matrixone/pkg/frontend/routine_manager.go:117 +0x41e\ngithub.com/      fagongzi/goetty.(*server).doConnection(0xc00810e840, {0x26225a0, 0xc024586420})\n\t/home/devops/go/pkg/mod/github.com/fagongzi/[email protected]/application.go:287 +0x503\ngithub.com/fagongzi/goetty.(*server).doStart.func1()\n\t/home/devops/go/pkg/mod/github.com/fagongzi/   [email protected]/application.go:258 +0xc8\ncreated by github.com/fagongzi/goetty.(*server).doStart\n\t/home/devops/go/pkg/mod/github.com/fagongzi/[email protected]/application.go:236 +0x192\n"}
    2021/11/23 12:38:28.501170 +0800 INFO frontend/mysql_cmd_executor.go:897 time of Exec.Run : 4.885774ms
    2021/11/23 12:38:28.501218 +0800 INFO frontend/mysql_cmd_executor.go:924 time of SendResponse 23.231µs
    2021/11/23 12:38:28.501240 +0800 INFO frontend/routine.go:111 connection id 3391 , the time of handling the request 6.29072ms
    2021/11/23 12:38:28.501468 +0800 INFO [email protected]/application.go:272 session read {"session-id": 2391, "addr": "127.0.0.1:54990", "msg": {"Length":78,"SequenceID":0,"Payload":                                                                                              "A0lOU0VSVCBJTlRPIHQxNDMgVkFMVUVTICg3MDIsMTk4MjAxLDM3LCdhbWVuaXRpZXMnLCdjbG9ja2VycycsJ3BpY3R1cmVzJywnQScp"}}
    2021/11/23 12:38:28.501513 +0800 INFO frontend/mysql_cmd_executor.go:935 cmd 3
    2021/11/23 12:38:28.501525 +0800 INFO frontend/mysql_cmd_executor.go:963 query:INSERT INTO t143 VALUES (702,198201,37,'amenities'...
    2021/11/23 12:38:28.502824 +0800 INFO frontend/mysql_cmd_executor.go:806 time of Exec.Build : 1.183159ms
    

    Start at: 2021/11/23 12:05:35.258148863 +0800 Found panic at: 2021/11/23 12:38:28.500378 +0800

    
    
    ### Additional information
    
    _No response_
    kind/bug priority/p1 needs-triage severity/s2 
    opened by lignay 9
  • logservice: fixed various HAKeeper client connectivity issues

    logservice: fixed various HAKeeper client connectivity issues

    What type of PR is this?

    • [ ] API-change
    • [x] BUG
    • [ ] Improvement
    • [ ] Documentation
    • [ ] Feature
    • [ ] Test and CI
    • [ ] Code Refactoring

    Which issue(s) this PR fixes:

    issue #

    What this PR does / why we need it:

    • use constant connection timeout value which is much smaller than the LOG node heartbeat timeout. this ensures we get chance to try most specified HAKeeper nodes.
    • during bootstrap, it is possible for the HAKeeper replica to be removed by a KILL command, terminate the local bootstrap procedure when that happens.
    • always keep the ticker worker even after the HAKeeper replica is stopped.
    kind/bug size/s 
    opened by lni 0
  • Fix union bug

    Fix union bug

    What type of PR is this?

    • [ ] API-change
    • [x] BUG
    • [ ] Improvement
    • [ ] Documentation
    • [ ] Feature
    • [ ] Test and CI
    • [ ] Code Refactoring

    Which issue(s) this PR fixes:

    issue #

    What this PR does / why we need it:

    fix bug: right node's columns will be Pruning in union clause

    kind/bug size/m 
    opened by ouyuanning 0
  • [Feature Request]: Decimal support operator if mod

    [Feature Request]: Decimal support operator if mod

    Is there an existing issue for the same feature request?

    • [X] I have checked the existing issues.

    Is your feature request related to a problem?

    No response

    Describe the feature you'd like

    select cast(10000002383263201056 as decimal) mod 50 as result;

    Describe implementation you've considered

    No response

    Documentation, Adoption, Use Case, Migration Strategy

    No response

    Additional information

    No response

    kind/feature bvt 
    opened by iamlinjunhong 0
  • fix onReplayAppendCmd

    fix onReplayAppendCmd

    What type of PR is this?

    • [ ] API-change
    • [x] BUG
    • [ ] Improvement
    • [ ] Documentation
    • [ ] Feature
    • [ ] Test and CI
    • [ ] Code Refactoring

    Which issue(s) this PR fixes:

    issue #4428

    What this PR does / why we need it:

    Fix onReplayAppendCmd. Check whether there exists active block before load data.

    kind/bug size/s 
    opened by jiangxinmeng1 0
  • [Bug]: Syscheck sends `shutdown` incorrectly

    [Bug]: Syscheck sends `shutdown` incorrectly

    Is there an existing issue for the same bug?

    • [X] I have checked the existing issues.

    Environment

    - Version or commit-id (e.g. v0.1.0 or 8b23a93):
    - Hardware parameters:
    - OS type:
    - Others:
    

    Actual Behavior

    Syscheck sends shutdown command incorrectly if there were two replicas in the cluster expired.

    Syscheck should not regard repaired replicas as expired.

    Expected Behavior

    No response

    Steps to Reproduce

    No response

    Additional information

    No response

    kind/bug needs-triage 
    opened by w-zr 0
  • [Bug]: group not found

    [Bug]: group not found

    Is there an existing issue for the same bug?

    • [X] I have checked the existing issues.

    Environment

    - Version or commit-id (e.g. v0.1.0 or 8b23a93):
    - Hardware parameters:
    - OS type:
    - Others:
    

    Actual Behavior

    /mo-server system_vars_config.toml Initialize the TAE engine ... panic: group not found

    goroutine 1 [running]: github.com/matrixorigin/matrixone/pkg/vm/engine/tae/db.(*DB).onReplayAppendCmd(0xc000178300, 0xc0052d2080, {0x18c5f48, 0xc0005400a0}) /tmp/test/matrixone/pkg/vm/engine/tae/db/replay.go:268 +0x516 github.com/matrixorigin/matrixone/pkg/vm/engine/tae/db.(*Replayer).OnReplayCmd(0xc0005400a0, {0x18f4d20?, 0xc0052d2080?}, 0x0) /tmp/test/matrixone/pkg/vm/engine/tae/db/replay.go:250 +0x5bf github.com/matrixorigin/matrixone/pkg/vm/engine/tae/db.(*Replayer).OnReplayCmd(0xc0005400a0, {0x18f4c00?, 0xc0065a38e0?}, 0xc006cb1b40) /tmp/test/matrixone/pkg/vm/engine/tae/db/replay.go:240 +0x647 github.com/matrixorigin/matrixone/pkg/vm/engine/tae/db.(*Replayer).OnReplayEntry(0xc0005400a0, 0xb?, 0xea, {0xc00018a800, 0x20e, 0x400}, 0x108?, {0xea?, 0xc0000bbfa8?}) /tmp/test/matrixone/pkg/vm/engine/tae/db/replay.go:209 +0x125 github.com/matrixorigin/matrixone/pkg/vm/engine/tae/wal.(*walDriver).replayhandle.func1(0x13c120?, 0xc0005400f0?, {0xc00018a800, 0x20e, 0x400}, 0x0?, {0x1531a00?, 0xc000540140?}) /tmp/test/matrixone/pkg/vm/engine/tae/wal/driver.go:89 +0xb1 github.com/matrixorigin/matrixone/pkg/vm/engine/tae/logstore/store.(*StoreImpl).replayEntry(0x557460082499f9b7?, 0xc00053c220?, 0xc0002bf638) /tmp/test/matrixone/pkg/vm/engine/tae/logstore/store/replay.go:55 +0x105 github.com/matrixorigin/matrixone/pkg/vm/engine/tae/logstore/store.(*StoreImpl).Replay.func1(0x14a0c00?) /tmp/test/matrixone/pkg/vm/engine/tae/logstore/store/replay.go:23 +0x27 github.com/matrixorigin/matrixone/pkg/vm/engine/tae/logstore/driver/batchstoredriver.(*replayer).replayHandler(0xc000535410, 0xc000200000) /tmp/test/matrixone/pkg/vm/engine/tae/logstore/driver/batchstoredriver/replay.go:90 +0xd5 github.com/matrixorigin/matrixone/pkg/vm/engine/tae/logstore/driver/batchstoredriver.(*vFile).Replay(0xc000118020?, 0xf?) /tmp/test/matrixone/pkg/vm/engine/tae/logstore/driver/batchstoredriver/vfile.go:247 +0x3c github.com/matrixorigin/matrixone/pkg/vm/engine/tae/logstore/driver/batchstoredriver.(*rotateFile).Replay(0xc000322160, 0x18?) /tmp/test/matrixone/pkg/vm/engine/tae/logstore/driver/batchstoredriver/file.go:184 +0x145 github.com/matrixorigin/matrixone/pkg/vm/engine/tae/logstore/driver/batchstoredriver.(*baseStore).Replay(0xc0002bc180, 0xc0002bf650) /tmp/test/matrixone/pkg/vm/engine/tae/logstore/driver/batchstoredriver/store.go:274 +0xc7 github.com/matrixorigin/matrixone/pkg/vm/engine/tae/logstore/store.(*StoreImpl).Replay(0xc000322210, 0xc0002bf638) /tmp/test/matrixone/pkg/vm/engine/tae/logstore/store/replay.go:22 +0xaa github.com/matrixorigin/matrixone/pkg/vm/engine/tae/wal.(*walDriver).Replay(0xc00012ee80, 0x48?) /tmp/test/matrixone/pkg/vm/engine/tae/wal/driver.go:93 +0x37 github.com/matrixorigin/matrixone/pkg/vm/engine/tae/db.(*Replayer).Replay(0xc0005400a0) /tmp/test/matrixone/pkg/vm/engine/tae/db/replay.go:104 +0x6d github.com/matrixorigin/matrixone/pkg/vm/engine/tae/db.(*DB).Replay(0xc000178300, 0xc000339580) /tmp/test/matrixone/pkg/vm/engine/tae/db/db.go:100 +0x14b github.com/matrixorigin/matrixone/pkg/vm/engine/tae/db.Open({0xc000157020, 0xb}, 0x0) /tmp/test/matrixone/pkg/vm/engine/tae/db/open.go:79 +0x75c main.initTae() /tmp/test/matrixone/cmd/db-server/main.go:110 +0x9c main.main() /tmp/test/matrixone/cmd/db-server/main.go:211 +0x916

    Expected Behavior

    No response

    Steps to Reproduce

    No response

    Additional information

    No response

    kind/bug needs-triage 
    opened by nnsgmsone 0
Releases(v0.5.0)
  • v0.5.0(Jul 17, 2022)

    We are excited to announce MatrixOne 0.5.0 release. In this release, MatrixOne has a stand-alone columnar storage engine which can support HTAP workloads. Cheers!

    Docker

    docker pull matrixorigin/matrixone:0.5.0

    Features

    • Support ACID transaction with a Snapshot Isolation level.
    • Support UPDATE, DELETE and INSERT INTO ... SELECT ... statement.
    • Support BOOL and Timestamp data type.
    • Support LEFT / RIGHT / OUTER / NATURAL JOIN.
    • Support Having expression.
    • Support subquery.
    • Support Common Table Expression.
    • Support CASE ... WHEN expression.
    • Support Interval expression.
    • Support Explain plan tree.
    • Support new aggregate function: any_value
    • Lots of new functions are supported.

    Known issues

    • Hotfix on 0.5.0 may cause data format incompatable.
    • 'Too many open files' error might be issued when user insert/update/delete data or create/drop table frequently. User need to increase the 'max open files' to resolve the issue.
    • Running 1GB TPCH benchmark test with memory less than 64GB may result in an out of memory error.
    • Loading large size of CSV file than 100GB might lead to system hanging. #3858
    • Mixed TP and AP work loads running for a long time, might cause system panic. #3947 #3961

    Contributors

    This release includes 811 commits by 73 authors.

    New Contributors

    • @lawrshen
    • @lyfer233
    • @wuliuqii
    • @ericsyh
    • @dongdongyang33
    • @aylei
    • @richelleguice
    • @aressu1985
    • @mklzl

    We appreciate your contribution! Welcome to MatrixOne community.

    Full Changelog

    https://github.com/matrixorigin/matrixone/compare/v0.4.0...v0.5.0

    Source code(tar.gz)
    Source code(zip)
    mo-server-v0.5.0-darwin-x86_64.zip(17.37 MB)
    mo-server-v0.5.0-linux-amd64.zip(15.02 MB)
  • v0.4.0(May 5, 2022)

    We are excited to announce MatrixOne 0.4.0 release. Cheers!

    Docker

    docker pull matrixorigin/matrixone:0.4.0

    Features

    • Support inner join with equality operator.
    • Support subquery within from clause.
    • Support decimal data type.
    • Support following builtin functions:
      • Mathematical: Abs, Log, Ln, Ceil, Exp, Power, Pi, Sin, Sinh, Cos, ACos, Tan, ATan, Cot
      • Datetime: Month, Weekday, Date, DayOfYear
      • String: Space, Reverse, Substring, Ltrim, Rtrim, StartsWith, EndsWith, Lpad, Rpad, Empty, LengthUTF8
    • Support following aggregate functions:
      • Bit_and, Bit_or, Bit_xor, Stddev_pop, Var

    Known issues

    • For continuous table creation actions in a script, the parameter of pre-allocated-group-num should be set to a bigger value such in cluster scenarios. Otherwise, "no available raft group" may be notified during tables creation sequence.

    Contributors

    This release includes 259 commits by 50 authors.

    New Contributors

    • @BePPPower
    • @JackTan25
    • @Charlie17Li
    • @domingozhang
    • @Fungx
    • @JasonPeng1310
    • @jiajunhuang
    • @NTH19
    • @noneback
    • @RinChanNOWWW
    • @chaixuqing
    • @Y7n05h
    • @yuxubinchen
    • @adlternative
    • @ajian2002
    • @bxiiiiii
    • @coderzc
    • @e11jah
    • @fengttt
    • @florashi181
    • @hiyoyolumi
    • @jinfuchiang
    • @ouyuanning
    • @qingxinhome
    • @supermario1990
    • @whileskies
    • @xiw5
    • @yclchuxue
    • @ZtXavier

    We appreciate your contribution! Welcome to MatrixOne community.

    Full Changelog

    https://github.com/matrixorigin/matrixone/compare/v0.3.0...v0.4.0

    Source code(tar.gz)
    Source code(zip)
    mo-server-v0.4.0-darwin-x86_64.zip(34.19 MB)
    mo-server-v0.4.0-linux-amd64.zip(34.75 MB)
  • v0.3.0(Mar 10, 2022)

    We are excited to announce MatrixOne 0.3.0 release. Cheers!

    Docker

    docker pull matrixorigin/matrixone:0.3.0

    Features

    • Exports data into CSV file.
    • Introduces parallel execution and improve performance for projection query.
    • Introduces IN operator and allows you to specify multiple values in a WHERE clause.
    • Support NULLABLE column in group by expression.
    • Introduces new functions: round and floor.
    • Introduces a Chaos Testing framework for distributed system.

    Known issues

    • For continuous table creation actions in a script, the parameter of pre-allocated-group-num should be set to a bigger value such in cluster scenarios. Otherwise, "no available raft group" may be notified during tables creation sequence.

    Contributors

    This release includes 157 commits by 21 authors.

    • broccoliSpicy(@broccoliSpicy)
    • Chen Mingsong(@m-schen)
    • hanfang (@aptend)
    • O2 (@ikenchina)
    • Jin Hai (@JinHai-CN)
    • Jiang xinmeng (@jiangxinmeng1)
    • Lin Junhong (@iamlinjunhong)
    • Long Ran (@aunjgr)
    • Nan Deng (@dengn)
    • Otter(@WenhaoKong2001)
    • Peng Zhen (@daviszhen)
    • Qin Shuqi (@sukki37)
    • Sundy Li(@sundy-li)
    • Shen JiangWei(@LeftHandCold)
    • Jian Wang(@jianwan0214)
    • Wan Hanbo(@wanhanbo )
    • Xu Peng (@XuPeng-SH)
    • Yan Wenze(@nnsgmsone)
    • Yuesheng Li(@reusee)
    • Zilong Zhou (@zzl200012)
    • Zhang Yingfeng (@yingfeng)

    New Contributors

    • @wanhanbo made their first contribution in https://github.com/matrixorigin/matrixone/pull/1600
    • @ikenchina made their first contribution in https://github.com/matrixorigin/matrixone/pull/1685
    • @sundy-li made their first contribution in https://github.com/matrixorigin/matrixone/pull/1704
    • @WenhaoKong2001 made their first contribution in https://github.com/matrixorigin/matrixone/pull/1838

    We appreciate your contribution! Welcome to MatrixOne community.

    Full Changelog

    https://github.com/matrixorigin/matrixone/compare/v0.2.0...v0.3.0

    Source code(tar.gz)
    Source code(zip)
    mo-server-v0.3.0-linux-amd64.zip(31.50 MB)
  • v0.2.0(Jan 6, 2022)

    We are excited to announce MatrixOne 0.2.0 release. Cheers!

    Docker

    docker pull matrixorigin/matrixone:0.2.0

    Features

    • Support automatic rebalancing for the built-in AOE columnar storage engine.
    • Introduce new SQL parser.
    • Introduce factorized SQL execution engine which is the fastest MPP in Golang.
    • Support CREATE / DROP INDEX.
    • Support PRIMARY KEY in CREATE TABLE.
    • Support more data types for SQL binary operators.
    • Support equi join with group by or aggregation statements.
    • Support new types of DATE and DATETIME.

    Known issues

    • For continuous table creation actions in a script, the parameter of pre-allocated-group-num should be set to a bigger value such in cluster scenarios. Otherwise, "no available raft group" may be notified during tables creation sequence.

    Contributors

    This release includes 243 commits by 21 authors.

    • BingLin Chang (@decster)
    • Chen Mingsong(@m-schen)
    • Nan Deng (@dengn)
    • Jin Hai (@JinHai-CN)
    • Jiang xinmeng (@jiangxinmeng1)
    • Li Yang (@lignay)
    • Lin Junhong (@iamlinjunhong)
    • lni (@lni)
    • Long Ran (@aunjgr)
    • Peng Zhen (@daviszhen)
    • Qin Shuqi (@sukki37)
    • Shen JiangWei(@LeftHandCold)
    • Jian Wang(@jianwan0214)
    • broccoliSpicy(@broccoliSpicy)
    • Ryan Wang(@wanglei4687)
    • Xiong Jingjuan (@anitajjx)
    • Xu Peng (@XuPeng-SH)
    • Yan Wenze(@nnsgmsone)
    • Yuesheng Li(@reusee)
    • Zilong Zhou (@zzl200012)
    • Zhang Yingfeng (@yingfeng)
    Source code(tar.gz)
    Source code(zip)
  • v0.1.0(Oct 24, 2021)

    We are excited to announce the first release of MatrixOne 0.1.0 on 2021/10/24. Cheers!

    Docker

    docker pull matrixorigin/matrixone:0.1.0

    Features

    SQL

    Following SQL statements are supported in this release.

    DDL

    • CREATE / DROP DATABASE
    • CREATE / DROP TABLE

    DML

    • INSERT
    • LOAD DATA
    • SELECT
      • WHERE
      • GROUP BY
      • ORDER BY
      • LIMIT, OFFSET
    • SHOW
      • DATABASES
      • TABLES
    • USE

    Data Types

    • TINYINT / SMALLINT / INT / BIGINT, SIGNED / UNSIGNED
    • FLOAT / DOUBLE
    • CHAR / VARCHAR

    Operators

    • >, >=, <, <=, <>, !=, =
    • BETWEEN … AND …, NOT BETWEEN … AND …
    • AND, &&, OR, ||
    • +, -, *, /, %, MOD, DIV, NEG
    • CAST

    Aggregation Functions

    • COUNT
    • SUM
    • AVG
    • MAX
    • MIN

    Database Engine

    • MySQL dialect is supported.
    • Build-in Append only columnar engine(AOE) has been supported as the first fusion database engine.
    • Real-time analytical queries are supported.
    • Vectorized query execution engine, based on MPP solution, is introduced.
    • Part of the expression execution has been vectorized through SIMD instructions.
    • Distributed AOE with strong consistency guaranteed by RAFT has been supported.
    • The performance of replicated state machine has been greatly improved through the unique design of reusing RAFT log as well as Write-Ahead log without any redundancy.

    Contributors

    This release includes 453 commits by 16 authors. Special thanks to the first author: Yan Wenze (@nnsgmsone)

    • Chen Mingsong(@m-schen)
    • Jin Hai (@JinHai-CN)
    • Jiang xinmeng (@jiangxinmeng1)
    • Li Yang (@lignay)
    • Lin Junhong (@iamlinjunhong)
    • lni (@lni)
    • Long Ran (@aunjgr)
    • Peng Zhen (@daviszhen)
    • Qin Shuqi (@sukki37)
    • Shen JiangWei(@LeftHandCold)
    • Wei Ziran (@w-zr)
    • Xiong Jingjuan (@anitajjx)
    • Xu Peng (@XuPeng-SH)
    • Yan Wenze(@nnsgmsone)
    • Zilong Zhou (@zzl200012)
    • Zhang Yingfeng (@yingfeng)
    Source code(tar.gz)
    Source code(zip)
Owner
Matrix Origin
Matrix Origin
Collects many small inserts to ClickHouse and send in big inserts

ClickHouse-Bulk Simple Yandex ClickHouse insert collector. It collect requests and send to ClickHouse servers. Installation Download binary for you pl

Nikolay Pavlovich 375 Aug 8, 2022
Vectorized SQL for JSON at scale: fast, simple, schemaless

Vectorized SQL for JSON at scale: fast, simple, schemaless Sneller is a high-performance vectorized SQL engine for JSON that runs directly on object s

null 632 Aug 2, 2022
Minimal memory usage, cloud native logstash alternative

Mr-Plow Tiny and minimal tool to export data from relational db (postgres or mysql) to elasticsearch. The tool does not implement all the logstash fea

RingLoop 28 Apr 27, 2022
Dumpling is a fast, easy-to-use tool written by Go for dumping data from the database(MySQL, TiDB...) to local/cloud(S3, GCP...) in multifarious formats(SQL, CSV...).

?? Dumpling Dumpling is a tool and a Go library for creating SQL dump from a MySQL-compatible database. It is intended to replace mysqldump and mydump

PingCAP 261 Jul 14, 2022
Constant Database native golang implementation

CDB golang implementation cdb is a fast, reliable, simple package for creating and reading constant databases see docs for more details Advantages Ite

Alexander Petrov 32 Jul 15, 2022
A MySQL-compatible relational database with a storage agnostic query engine. Implemented in pure Go.

go-mysql-server go-mysql-server is a SQL engine which parses standard SQL (based on MySQL syntax) and executes queries on data sources of your choice.

DoltHub 862 Aug 5, 2022
MySQL Storage engine conversion,Support mutual conversion between MyISAM and InnoDB engines.

econvert MySQL Storage engine conversion 简介 此工具用于MySQL存储引擎转换,支持CTAS和ALTER两种模式,目前只支持MyISAM和InnoDB存储引擎相互转换,其它引擎尚不支持。 注意:当对表进行引擎转换时,建议业务停止访问或者极少量访问时进行。 原

null 5 Oct 25, 2021
It's a Go console utility for migration from MSSQL to MySQL engine.

A tool for migration the databases to MySQL It's a Go console utility for migration from MSSQL to MySQL engine. The databases should have prepopulated

Eugen Vasilyeu 0 Jan 4, 2022
Zinc Search engine. A lightweight alternative to elasticsearch that requires minimal resources, written in Go.

Zinc Zinc is a search engine that does full text indexing. It is a lightweight alternative to elasticsearch and runs in less than 100 MB of RAM. It us

Prabhat Sharma 9.9k Aug 8, 2022
Hybrid Engine for emulate trading flow

Deridex Backend This is a backend implementation for Deridex market. The system

Deridex 0 Dec 15, 2021
Query and Provision Cloud Infrastructure using an extensible SQL based grammar

Deploy, Manage and Query Cloud Infrastructure using SQL [Documentation] [Developer Guide] Cloud infrastructure coding using SQL InfraQL allows you to

InfraQL 22 Apr 5, 2022
CloudQuery extracts, transforms, and loads your cloud assets into normalized PostgreSQL tables.

The open-source cloud asset inventory backed by SQL. CloudQuery extracts, transforms, and loads your cloud assets into normalized PostgreSQL tables. C

CloudQuery 2.4k Aug 13, 2022
Crossplane provider for InfluxDB Cloud

provider-template provider-template is a minimal Crossplane Provider that is meant to be used as a template for implementing new Providers. It comes w

Crossplane Contrib 3 Jan 10, 2022
Google Cloud Spanner driver for Go's database/sql package.

go-sql-spanner Google Cloud Spanner driver for Go's database/sql package. This support is currently in the Preview release status. import _ "github.co

Google APIs 52 Jul 31, 2022
A fast data generator that's multi-table aware and supports multi-row DML.

If you need to generate a lot of random data for your database tables but don't want to spend hours configuring a custom tool for the job, then datage

Coding Concepts 49 Aug 8, 2022
Sync MySQL data into elasticsearch

go-mysql-elasticsearch is a service syncing your MySQL data into Elasticsearch automatically. It uses mysqldump to fetch the origin data at first, the

siddontang 3.8k Aug 5, 2022
db-recovery is a tool for recovering MySQL data.

db-recovery is a tool for recovering MySQL data. It is used in scenarios where the database has no backup or binlog. It can parse data files and redo/undo logs to recover data.

null 23 Jul 5, 2022
Convert data exports from various services to a single SQLite database

Bionic Bionic is a tool to convert data exports from web apps to a single SQLite database. Bionic currently supports data exports from Google, Apple H

Bionic 138 Jun 29, 2022
OctoSQL is a query tool that allows you to join, analyse and transform data from multiple databases and file formats using SQL.

OctoSQL OctoSQL is a query tool that allows you to join, analyse and transform data from multiple databases, streaming sources and file formats using

Jacob Martin 3.9k Aug 9, 2022