CnosDB 用于存储物联网数据的开源时间序列数据库。

Overview

CnosDB

一款高性能、高压缩率、高可玩性的开源时序数据库。

功能特性

  • 快速的批量写入
  • 数据高压缩比
  • 丰富的计算函数

加入社区

欢迎所有热爱时序数据库的开发者/用户参与到CnosDB User Group中。扫描下方二维码,加CC为好友,即可入群。

入群前请查看入群须知

快速开始

如果需要完整的入门指南,请查看入门指南

构建

  1. 克隆项目

    git clone https://github.com/cnosdatabase/cnosdb.git
    
  2. 编译

    go install ./...
    

运行

  1. 启动

    $GOPATH/bin/cnosdb
  2. 使用

    $GOPATH/bin/cnosdb-cli

使用指南

创建数据库

curl -i -XPOST http://localhost:8086/query --data-urlencode "q=CREATE DATABASE mydb"

写入数据

curl -i -XPOST 'http://localhost:8086/write?db=db' --data-binary 'cpu,host=server01,region=Beijing idle=0.72 1434055562000000000'

查询数据

curl -G 'http://localhost:8086/query?pretty=true' --data-urlencode "db=db" --data-urlencode "q=SELECT \"idle\" FROM \"cpu\" WHERE \"region\"='Beijing'"

如何贡献

请参照贡献指南为CnosDB做贡献。

许可证

MIT License

Comments
  • 数据库当中最少需要保留一个保留策略

    数据库当中最少需要保留一个保留策略

    Closes #

    Describe your proposed changes here.

    opened by adofsauron 9
  • [Feature]Configurable Data Compaction Algorithm

    [Feature]Configurable Data Compaction Algorithm

    Is your feature request related to a problem? Please describe. Compaction Algorithm in CnosDB is like a box of chocolates. You never know what you're gonna get... kidding Nevertheless, Compaction Algorithm is still a black box to us and we would like to configure it based on our own understanding of dataset.

    Describe the solution you'd like Similar to commercial version of Cassandra here, we could have WITH compaction keyword:

    1. Disable compaction via CREATE SQL: CREATE TABLE DEMO(...) WITH compaction= { 'enabled' : false };
    2. Enable compaction via CREATE SQL: CREATE TABLE DEMO(...) WITH compaction= { 'class' : 'LZ4Compressor' };
    3. Disable compaction via ALTER TABLE SQL: ALTER TABLE DEMO WITH compaction= { 'enabled' : false };
    4. Enable compaction via ALTER TABLE SQL: ALTER TABLE DEMO WITH compaction ={ 'class' : 'LZ4Compressor' }; ALTER TABLE DEMO WITH compaction ={ 'class' : 'DeflateCompressor', 'chunk_length_in_kb' : 64 };

    Or you could try MariaDB way, using COMMENT keyword here:

    1. Set Global Compaction Type in configuration file: columnstore_compression_type=LZ4
    2. Set Table Compaction Type in SQL's COMMENT: CREATE TABLE DEMO(...) COMMENT 'compression=3';
    3. Set Column Compaction Type in SQL's COMMENT:
    CREATE TABLE DEMO(
        invoice_id BIGINT UNSIGNED NOT NULL,
        invoice_total DECIMAL(13, 2) COMMENT 'compression=3',
    );
    

    Describe alternatives you've considered We could disable compaction once for all in the configuration instead of DDL but DDL provides more granular way.

    Additional context More granularity should be provided as well:

    1. Global
    2. Database
    3. Table

    This feature should be integrated in the cloud version as well.

    Other alternatives:

    1. TecentCloud's COMPRESSED keyword: https://www.tencentcloud.com/document/product/236/43480
    2. ClickHouse's CODEC keyword: https://clickhouse.com/docs/en/sql-reference/statements/create/table#column-compression-codecs
    component/tsdb Community kv store 
    opened by muninshen 4
  • [Feature] Schema Query Statements

    [Feature] Schema Query Statements

    Is your feature request related to a problem? Please describe. support those schema query statements

    • [x] DESCRIBE DATABASE db_name;
    • [x] DESCRIBE TABLE [db_name.] table_name;
    • [x] SHOW DATABASES;
    • [x] SHOW TABLES [ON <db_name>];
    • [x] SHOW SERIES [ON <db_name>] FROM table_name [WHERE <tag_key> [ '<tag_value>' | ]] [LIMIT_clause]
    • [x] SHOW TAG VALUES [ON <db_name>] FROM table_name WITH KEY [ [ "<tag_key>" | <regular_expression>] | [IN (""] [WHERE <tag_key> ['<tag_value>' | ]] [LIMIT_clause]
    good first issue component/query 
    opened by roseboy-liu 3
  • refactor  tskv : Error passed up in async function

    refactor tskv : Error passed up in async function

    Errors inside the [run_wal_job,run_flush_job,run_compact_job,run_summary_job] functions are passed up

    避免以上四个函数中的错误在内部消耗而外部不可知而向上传递,并将方法改为异步函数形式,另外项目Error设计不是很优雅

    opened by Babanmei 3
  • [User]Compile issue need help

    [User]Compile issue need help

    Hello, in docs/get-started.md document in the last step git clone https://github.com/cnosdb/cnosdb-isipho.git && cd cnosdb-isipho following address I can not pull, and then I pulled https://github. com/cnosdb/cnosdb.git, cargo build and then run the error as follows image Can you tell me what the cause is?

    Chinese original: 您好,在docs/get-started.md文档中最后一步git clone https://github.com/cnosdb/cnosdb-isipho.git && cd cnosdb-isipho如下地址我无法拉取,而后我拉取了https://github.com/cnosdb/cnosdb.git,cargo build后运行报错如下 方便问一下大概是什么原因嘛

    component/build Community 
    opened by xin321 3
  • [Bug]parallel kvcore test will panic

    [Bug]parallel kvcore test will panic

    Describe the bug parallel kvcore test will panic,we add serial for each test,the test will pass

    To Reproduce cargo test --package tskv --lib kvcore::test

    Expected behavior should parallel

    Additional context none

    type/bug test kv store 
    opened by oyzg 3
  • tombstone success

    tombstone success

    Closes #

    Describe your proposed changes here.

    opened by 120L020410 3
  • RewriteRegexConditions can panic instead of erroring

    RewriteRegexConditions can panic instead of erroring

    Describe the bug.

    RewriteRegexConditions can panic when given an invalid regex, e.g.

    select * from cpu where ( host =~ /foo/ ^ other AND env =~ /bar/ ) and time >= now()-15m
    

    Gives a panic:

    panic:interface conversion: cnosql.Expr is *cnosql.BinaryExpr, not *cnosql.RegexLiteral
    

    This should return an error without panic'ing

    type/bug component/query v1 
    opened by IvanGao01 3
  • [Compile] compile error

    [Compile] compile error

    Describe the bug the latest branch compile error in my machine.

    Arch Linux 6.1.1 x86_64
    rustc version 1.64.0
    cargo version: 1.64.0
    

    cargo ouput:

    Caused by:
      process didn't exit successfully: `/home/fanr/Github/cnosdb/target/debug/build/protos-98787f2df9789b3c/build-script-build` (exit status: 101)
      --- stderr
      thread 'main' panicked at 'Failed to generate file 'models.rs' by flatc(path: 'flatc'): Os { code: 2, kind: NotFound, message: "No such file or directory" }.', common/protos/build.rs:123:21
      stack backtrace:
         0: rust_begin_unwind
                   at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/panicking.rs:584:5
         1: core::panicking::panic_fmt
                   at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/panicking.rs:142:14
         2: build_script_build::main::{{closure}}
                   at ./build.rs:123:21
         3: core::result::Result<T,E>::unwrap_or_else
                   at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/result.rs:1493:23
         4: build_script_build::main
                   at ./build.rs:111:26
         5: core::ops::function::FnOnce::call_once
                   at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/ops/function.rs:248:5
      note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
    warning: build failed, waiting for other jobs to finish...
    

    To Reproduce Steps to reproduce the behavior:

    Expected behavior A clear and concise description of what you expected to happen.

    Additional context Add any other context about the problem here.

    opened by fansehep 2
  • [QUESTION] compile error

    [QUESTION] compile error

    compile the cnosdb following the get-started page: error reported: type Scalar is not a member of trait flatbuffers::EndianScalar --> common/protos/src/generated/flatbuffers_generated/models.rs:106:9 | 106 | type Scalar = i32; | ^^^^^^^^^^^^^^^^^^ not a member of trait flatbuffers::EndianScalar

    opened by guoxiangCN 2
  • [Refactor] Refactor parsing line_protocol.

    [Refactor] Refactor parsing line_protocol.

    Parsing lineprotocol may have performance issues. There are the following optimization measures. 1.Use the nom library for parsing 2.Add some error messages. 3.Use smallvec that allocates memory on the stack

    opened by ZuoTiJia 2
  • Refactor: Refactor implicit_type_conversion's error message

    Refactor: Refactor implicit_type_conversion's error message

    Now using our optimization rules implicit_type_conversion throws a datafusion internal error. We need to improve this error message

    sql

    explain select time from test_double_conv where fa >= '1997-01-31';
    

    response

    422 Unprocessable Entity
    {"error_code":"010000","error_message":"Datafusion: Internal error: Optimizer rule 'implicit_type_conversion' failed due to unexpected error: Arrow error: Cast error: Cannot cast string '1997-01-31' to value of Float64 type. This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker"}
    
    opened by ZuoTiJia 0
  • [Feature] Implement TagScan support for Coodinator Service

    [Feature] Implement TagScan support for Coodinator Service

    Is your feature request related to a problem? Please describe. A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

    Describe the solution you'd like Implement TagScan support for Coodinator Service

    Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

    Additional context Add any other context or screenshots about the feature request here.

    opened by roseboy-liu 0
  • [Feature] Add operation command

    [Feature] Add operation command

    Is your feature request related to a problem? Please describe. A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

    Describe the solution you'd like SQL command

    • [ ] Move Vnode [vnode_id] to Node [node_id]
    • [ ] Copy Vnode [vnode_id] to Node [node_id]
    • [ ] Drop Vnode [vnode_id]
    • [ ] Compact Vnode [vnode_id1, vnode_id2...]
    • [ ] CheckSum Group [replication_set_id]

    Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

    Additional context Add any other context or screenshots about the feature request here.

    opened by roseboy-liu 0
  • [Improve] Add lru cache module for our column file to improve query

    [Improve] Add lru cache module for our column file to improve query

    Is your feature request related to a problem? Please describe. When reading a file, spend a lot of time opening the file and reading the index from disk Describe the solution you'd like Add lru cache module for our column file to improve query

    Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

    Additional context Add any other context or screenshots about the feature request here.

    opened by roseboy-liu 0
  • [Feature] Support Python API

    [Feature] Support Python API

    Since 2019, "AI for Systems and Systems for AI" has become a trend.

    As we all know, TensorFlow and PyTorch are the most popular AI frameworks nowadays.

    However, I have noticed that CnosDB does not yet support a Python API, which could potentially lead to a loss of users such as data scientists and data engineers.

    In contrast, InfluxDB, a competitor of CnosDB, released the first version of a Python Client for interacting with InfluxDB in 2013. (Reference: https://github.com/influxdata/influxdb-python).

    I believe it is time for CnosDB to support a Python API.

    ecosystem 
    opened by DjangoPeng 7
Releases(v2.0.1)
  • v2.0.1(Nov 8, 2022)

    What's Changed

    Bug Fixes

    https://github.com/cnosdb/cnosdb/pull/715 Fix object name. https://github.com/cnosdb/cnosdb/pull/718 Fixes some bugs among wal. https://github.com/cnosdb/cnosdb/pull/724 Fix external table not found. https://github.com/cnosdb/cnosdb/pull/727 TableWriterExec only one partition can be written. https://github.com/cnosdb/cnosdb/pull/728 Use wrong schema when flush data to tsm lead to query mess. https://github.com/cnosdb/cnosdb/pull/731 Projection validity check of TskvScan & optimize column pruning. https://github.com/cnosdb/cnosdb/pull/726 Fix the bug of querying duplicate data. https://github.com/cnosdb/cnosdb/pull/740 Check empty field. https://github.com/cnosdb/cnosdb/pull/738 Failed parsing request contains string field with commas or equal sign. https://github.com/cnosdb/cnosdb/pull/739 Check field and tag name. https://github.com/cnosdb/cnosdb/pull/742 Compact read locked VersionSet and TsFamily too long. https://github.com/cnosdb/cnosdb/pull/744 Add tsfid for global context. https://github.com/cnosdb/cnosdb/pull/741 Only the tag column is included in the tablescan projection, no data is returned. https://github.com/cnosdb/cnosdb/pull/745 Reject cross join.

    Features https://github.com/cnosdb/cnosdb/pull/733 Limit queries connections and queries content size. https://github.com/cnosdb/cnosdb/pull/747 Add usage collect.

    Other https://github.com/cnosdb/cnosdb/pull/720 Make key of HashMap clearer. https://github.com/cnosdb/cnosdb/pull/721 Upgrade datafusion version to 13.0.0. https://github.com/cnosdb/cnosdb/pull/729 All crates in workspace share the same versions of dependencies. https://github.com/cnosdb/cnosdb/pull/723 Add log_err macro. https://github.com/cnosdb/cnosdb/pull/725 Add compress for wal. https://github.com/cnosdb/cnosdb/pull/730 Input check point. https://github.com/cnosdb/cnosdb/pull/743 Improve performance. https://github.com/cnosdb/cnosdb/pull/746 Rename Point::table to tab, Points::database to db.

    Full Changelog: https://github.com/cnosdb/cnosdb/compare/v2.0.0...v2.0.1

    Source code(tar.gz)
    Source code(zip)
  • v2.0.0(Oct 24, 2022)

    2022年10月24日,CnosDB 发布 v2.0.0 版本。 这次的版本发布是一个新的开始,我们基于Rust重新构建了CnosDB时序数据库,提升了CnosDB的性能,增强了CnosDB的易用性,也使CnosDB更加适应云生态的要求。

    特色功能

    • 专为时序数据设计的存储引擎,优化写操作,支持删除和更新操作;
    • 压缩算法由用户灵活指定,压缩比可调;
    • 基于Apache Arrow及Datafusion 实现了查询引擎;
    • 支持标准SQL,支持Schemaless 写入;
    • 多索引优化了查询效率;
    • 生态友好,支持RESTful接口,支持Telegraf、Grafana等通用第三方生态组件;

    On October 24, 2022, CnosDB released version v2.0.0. This version release is a new beginning. We have rebuilt the CnosDB time series database based on Rust, which has improved the performance of CnosDB, enhanced the usability of CnosDB, and made CnosDB more suitable for the requirements of the cloud ecosystem.

    Features

    • A storage engine designed for time series data, optimized for write operations, and supports delete and update operations;
    • The compression algorithm is flexibly specified by the user, and the compression ratio is adjustable;
    • Implemented the query engine based on Apache Arrow and Datafusion;
    • Supports standard SQL and Schemaless writing;
    • Multiple indexes optimize query efficiency;
    • Eco-friendly, supports RESTful interface, and supports common third-party ecological components such as Telegraf and Grafana;

    GitHub Repo: https://github.com/cnosdb/cnosdb Quick Start Guide: http://docs.cnosdb.com

    Source code(tar.gz)
    Source code(zip)
Owner
cnosdb
cnosdb