A GPU-powered real-time analytics storage and query engine.

Overview

Build Status Coverage Status license FOSSA Status


AresDB Logo

AresDB

AresDB is a GPU-powered real-time analytics storage and query engine. It features low query latency, high data freshness and highly efficient in-memory and on disk storage management. Please see AresDB's features, architecture design described in the Uber Engineering Blog.

This repo contains the source code of AresDB and debug UI.

Legal Note

AresDB requires the CUDA Toolkit. Please ensure you read, acknowledge, and accept the CUDA End User License Agreement.

Getting started

To get AresDB:

git clone --recursive https://github.com/uber/aresdb.git $GOPATH/src/github.com/uber/aresdb

NVIDIA Driver and CUDA Setup

AresDB needs NVIDIA driver version >= 390.48 and CUDA version 9.1.

Environment Variables

Run the following to make sure the following environment variables are correctly set:

export PATH=/path/to/cuda/bin:${PATH}
export LD_LIBRARY_PATH=/path/to/cuda/lib64:/path/to/aresdb/lib:${LD_LIBRARY_PATH}
export PKG_CONFIG_PATH=${LD_LIBRARY_PATH}/pkgconfig:${PKG_CONFIG_PATH}

Language Requirements

Building and running AresDB requires:

  • golang 1.11+
  • C++ compiler that support c++14
  • cmake 3.12+
  • nvcc version 9.1

Configure

Run following commands to generate makefile:

cmake -DQUERY_MODE=DEVICE .

Alternatively, if you want to run the query in CPU mode, run following commands:

cmake -DQUERY_MODE=HOST .

Local Test

AresDB is written in C++ (query engine) and Golang (mem store, disk store and other query components). Because of this, we break testing into two parts:

Test Golang Code

Ginkgo

We use Ginkgo as the test framework for running the Golang unit test and coverage. Install Ginkgo first and run

make test-golang

Test C++ Code

google-test

We use google-test as the test framework to test C++ code. Install google-test and set the environment variable, GTEST_ROOT, to the installed location.

After you have installed properly, run

make test-cuda

Run AresDB Server

The following command will start an AresDB server locally. You can start to query the server using a curl command or swagger page.

make run_server

Run AresDB Docker

Please read the Docker page.

Documentation

Interested in learning more about AresDB? Read the blog post

License

Apache 2.0 License, please see LICENSE for details.

Comments
  • Modified the Travis configuration file to enable automated FOSSA scans.

    Modified the Travis configuration file to enable automated FOSSA scans.

    I'm from FOSSA and I'm working with the Uber OSPO to automate license scanning. In addition to the changes in this PR, we may also need to add an API key as an environmental variable in the build environment.

    opened by xinding33 5
  • fix docker file error for update to go 1.11

    fix docker file error for update to go 1.11

    fix docker file error in different go version

    errors in https://github.com/uber/aresdb/issues/244

    1. different go path in change go version

    in this PR

    just change the go version from 1.9 to 1.11.4

    however the gopath in DockerFile not been changed

    already fixed

    2. go 1.11+'s rule, we should set GO111MODULE

    ENV GO111MODULE=on

    already fixed

    success output

    after that:

    [email protected]:~/go/src/github.com/uber/aresdb# make run_server
    [ 11%] Built target mem
    [100%] Built target algorithm
    [100%] Built target lib
    [100%] Built target aresd
    Scanning dependencies of target run_server
    Using config file:  config/ares.yaml
    {"level":"info","msg":"Bootstrapping service","config":{"Port":9374,"DebugPort":43202,"RootPath":"ares-root","TotalMemorySize":161061273600,"SchedulerOff":false,"Version":"","Env":"","Distributed":false,"Query":{"DeviceMemoryUtilization":0.95,"DeviceChoosingTimeout":10,"TimezoneTable":{"TableName":"api_cities"},"EnableHashReduction":false},"DiskStore":{"WriteSync":true},"HTTP":{"MaxConnections":300,"ReadTimeOutInSeconds":20,"WriteTimeOutInSeconds":300},"Cluster":{"Enable":false,"ClusterName":"","InstanceName":"60eadc19dd6d"},"Gateway":{"Controller":null},"RedoLogConfig":{"Namespace":"","DiskConfig":{"Disabled":false},"KafkaConfig":{"Enabled":false,"Brokers":null,"TopicSuffix":""}},"InstanceConfig":{"ID":"","Namespace":"","Etcd":{"Zone":"","Env":"","Service":"","CacheDir":"","ETCDClusters":null,"SDConfig":{"InitTimeout":null},"WatchWithRevision":0},"HeartbeatConfig":{"Timeout":0,"Interval":0}}}}
    {"level":"info","msg":"Reading schema from local MetaStore ares-root/metastore"}
    {"level":"info","msg":"Initialized device manager","utilization":0.949999988079071,"timeout":10}
    {"level":"info","msg":"DeviceInfo[0]={DeviceID:0 QueryCount:0 TotalMemory:25576865792 TotalAvailableMemory:24298022912 FreeMemory:24298022912 QueryMemoryUsageMap:map[]}\n"}
    {"level":"info","msg":"Bootstrapping device"}
    {"level":"info","msg":"Finish bootstrapping device"}
    {"level":"info","msg":"Initializing shards from local DiskStore ares-root"}
    {"level":"info","msg":"HostMemoryManager: initial preloading done"}
    {"level":"info","msg":"Start loading snapshots for all table shards"}
    {"level":"info","msg":"Finish loading snapshots for all table shards"}
    {"level":"info","msg":"Starting archiving scheduler"}
    {"level":"info","msg":"Start replaying redo logs for all table shards"}
    {"level":"info","msg":"Finish replaying redo logs for all table shards"}
    {"level":"info","msg":"Starting HTTP server on port 9374 with max connection 300"}
    {"level":"info","msg":"Starting HTTP server on dbg-port 43202"}
    
    
    opened by yutiansut 4
  • Base docker and examples throws error at data ingest

    Base docker and examples throws error at data ingest "panic: Missing primary key column"

    Describe the bug A clear and concise description of what the bug is. Running basic docker and examples throws error: panic: Missing primary key column

    To Reproduce Steps to reproduce the behavior: Follow docker instructions Follow example instructions inside docker container

    Expected behavior A clear and concise description of what you expected to happen. Be able to run examples in docker container, so I can get to a place where I can query data via swagger interface. Would also be useful to have a plain description of how to get data ingested via swagger interface as trying to dig through go code inside a docker container to trouble shoot is cumbersome.

    Full details============= [email protected]:$ sudo nvidia-docker run -p 9374:9374/tcp -p 43202:43202/tcp -it aresdb:latest [sudo] password for s: [email protected]:/go/src/github.com/uber/aresdb# make run_server >> ./log/aresdb.log 2>&1 & [1] 15 [email protected]:/go/src/github.com/uber/aresdb# cd examples/ [email protected]:/go/src/github.com/uber/aresdb/examples# go build examples.go [email protected]:/go/src/github.com/uber/aresdb/examples# ./examples tables --dataset 1k_trips table cities created table trips created [email protected]:/go/src/github.com/uber/aresdb/examples# ./examples data --dataset 1k_trips panic: Missing primary key column goroutine 1 [running]: github.com/uber/aresdb/utils.StackError(0x0, 0x0, 0x8f01c9, 0x1a, 0x0, 0x0, 0x0, 0x0) /root/go/src/github.com/uber/aresdb/utils/error.go:61 +0x46e github.com/uber/aresdb/client.checkPrimaryKeys(0xc42033a140, 0xc420316ce0, 0x2, 0x2, 0x0, 0x0) /root/go/src/github.com/uber/aresdb/client/connector.go:241 +0x14b github.com/uber/aresdb/client.(*connector).prepareUpsertBatch(0xc42026c9a0, 0xc4200149f0, 0x6, 0xc420316ce0, 0x2, 0x2, 0xc420018fb0, 0x2, 0x2, 0xc4203a6000, ...) /root/go/src/github.com/uber/aresdb/client/connector.go:223 +0xae github.com/uber/aresdb/client.(*connector).Insert(0xc42026c9a0, 0xc4200149f0, 0x6, 0xc420316ce0, 0x2, 0x2, 0xc4203a6000, 0xa, 0x10, 0x0, ...) /root/go/src/github.com/uber/aresdb/client/connector.go:169 +0x247 main.ingestDataForTable(0xb3b020, 0xc42026c9a0, 0xc4200149f0, 0x6, 0xc420014d00, 0x1a) /root/go/src/github.com/uber/aresdb/examples/examples.go:187 +0x961 main.ingestDataForDataSet() /root/go/src/github.com/uber/aresdb/examples/examples.go:115 +0x370 main.main.func1(0xc4200af900, 0xc420316740, 0x0, 0x2) /root/go/src/github.com/uber/aresdb/examples/examples.go:214 +0x20 github.com/uber/aresdb/vendor/github.com/spf13/cobra.(*Command).execute(0xc4200af900, 0xc4203166e0, 0x2, 0x2, 0xc4200af900, 0xc4203166e0) /root/go/src/github.com/uber/aresdb/vendor/github.com/spf13/cobra/command.go:766 +0x2c1 github.com/uber/aresdb/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc4200af680, 0x8e1260, 0xc4200af7d0, 0xc42014df68) /root/go/src/github.com/uber/aresdb/vendor/github.com/spf13/cobra/command.go:852 +0x334 github.com/uber/aresdb/vendor/github.com/spf13/cobra.(*Command).Execute(0xc4200af680, 0xc42014df58, 0x3) /root/go/src/github.com/uber/aresdb/vendor/github.com/spf13/cobra/command.go:800 +0x2b main.main() /root/go/src/github.com/uber/aresdb/examples/examples.go:239 +0x524

    goroutine 1 [running]: main.panicIfErr(0xb3b0a0, 0xc4203193e0) /root/go/src/github.com/uber/aresdb/examples/examples.go:67 +0x4a main.ingestDataForTable(0xb3b020, 0xc42026c9a0, 0xc4200149f0, 0x6, 0xc420014d00, 0x1a) /root/go/src/github.com/uber/aresdb/examples/examples.go:188 +0x986 main.ingestDataForDataSet() /root/go/src/github.com/uber/aresdb/examples/examples.go:115 +0x370 main.main.func1(0xc4200af900, 0xc420316740, 0x0, 0x2) /root/go/src/github.com/uber/aresdb/examples/examples.go:214 +0x20 github.com/uber/aresdb/vendor/github.com/spf13/cobra.(*Command).execute(0xc4200af900, 0xc4203166e0, 0x2, 0x2, 0xc4200af900, 0xc4203166e0) /root/go/src/github.com/uber/aresdb/vendor/github.com/spf13/cobra/command.go:766 +0x2c1 github.com/uber/aresdb/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc4200af680, 0x8e1260, 0xc4200af7d0, 0xc42014df68) /root/go/src/github.com/uber/aresdb/vendor/github.com/spf13/cobra/command.go:852 +0x334 github.com/uber/aresdb/vendor/github.com/spf13/cobra.(*Command).Execute(0xc4200af680, 0xc42014df58, 0x3) /root/go/src/github.com/uber/aresdb/vendor/github.com/spf13/cobra/command.go:800 +0x2b main.main() /root/go/src/github.com/uber/aresdb/examples/examples.go:239 +0x524 [email protected]:~/go/src/github.com/uber/aresdb/examples#

    opened by shaunstoltz 4
  • Add license scan report and status

    Add license scan report and status

    Your FOSSA integration was successful! Attached in this PR is a badge and license report to track scan status in your README.

    Below are docs for integrating FOSSA license checks into your CI:

    opened by fossabot 4
  • Broker skeleton

    Broker skeleton

    changes included:

    • minor refactoring to prepare for broker implementation: memstore + aql context
    • skeleton code for broker

    TODOs are added including:

    • implement BrokerSchemaMutator
    • implement non agg query executor
    • implement agg query executor
    • implement different query options in handler
    • logging and metrics where needed
    opened by shz117 3
  • Failed to use swagger in docker

    Failed to use swagger in docker

    Describe the bug Failed to use swagger in docker

    To Reproduce

    • Follow: https://github.com/uber/aresdb/blob/master/docker/README.md
    • Visit http://my-ip:9374/swagger/ (my-ip is the docker host's ip)

    Chrome console output:

    Refused to apply style from 'http://my-ip:9374/node_modules/swagger-ui-dist/swagger-ui.css' because its MIME type ('text/plain') is not a supported stylesheet MIME type, and strict MIME checking is enabled.
    (index):14 GET http://my-ip:9374/node_modules/swagger-ui-dist/swagger-ui-bundle.js net::ERR_ABORTED 404 (Not Found)
    (index):15 GET http://my-ip:9374/node_modules/swagger-ui-dist/swagger-ui-standalone-preset.js net::ERR_ABORTED 404 (Not Found)
    
    opened by zingdle 2
  • broker bug fix: 1. send rewritten query to datanode 2. return empty hll result correctly 3. resultSize is not len(result)

    broker bug fix: 1. send rewritten query to datanode 2. return empty hll result correctly 3. resultSize is not len(result)

    1. send rewritten query to data node which will rewrite enum string value to number
    2. return empty hll result correctly
    3. resultSize is not len(result)
    4. rewritten dim not passed back
    opened by jshencode 2
  • [NLJ] implementation layout

    [NLJ] implementation layout

    • add comments for existing many to one join logic
    • add TODOs for code change to support NLJ
    • add query context fields needed for NLJ
    • compiler changes for matching many to many joins
    opened by shz117 2
  • Suns

    Suns

    1. Update etcdConfig to support uns
    2. Update etcdConfig loading
    3. Refactor etcd heartbeat registration
    4. Only load aresdb instances from jobAssignment if sinkmode is aresDB
    5. Update GetDataValue if type is not string.
    6. Set sarama publisher manual partition for fact table
    opened by lxning 2
  • support ad-hoc polygons in query

    support ad-hoc polygons in query

    • support ad-hoc polygon in a new filter in_polygon(<point_column>, <polygon in WKT format>)
    • allow exporting polygon as virtual dimension "_geoshape"
    opened by shz117 2
  • docker build error

    docker build error

    docker build error in last step RUN make aresd -j

    Step 27/27 : RUN make aresd -j
     ---> Running in 2bdfb1ea1d1d
    Scanning dependencies of target mem
    [  5%] Building C object CMakeFiles/mem.dir/cgoutils/memory/malloc.c.o
    [ 11%] Linking C shared library lib/libmem.so
    [ 11%] Built target mem
    Scanning dependencies of target algorithm
    [ 16%] Building CUDA object CMakeFiles/algorithm.dir/query/algorithm.cu.o
    [ 22%] Building CUDA object CMakeFiles/algorithm.dir/query/dimension_transform.cu.o
    [ 27%] Building CUDA object CMakeFiles/algorithm.dir/query/filter.cu.o
    [ 33%] Building CUDA object CMakeFiles/algorithm.dir/query/functor.cu.o
    [ 38%] Building CUDA object CMakeFiles/algorithm.dir/query/geo_intersects.cu.o
    [ 44%] Building CUDA object CMakeFiles/algorithm.dir/query/hash_lookup.cu.o
    [ 50%] Building CUDA object CMakeFiles/algorithm.dir/query/hash_reduction.cu.o
    [ 55%] Building CUDA object CMakeFiles/algorithm.dir/query/hll.cu.o
    [ 61%] Building CUDA object CMakeFiles/algorithm.dir/query/memory.cu.o
    [ 66%] Building CUDA object CMakeFiles/algorithm.dir/query/measure_transform.cu.o
    [ 72%] Building CUDA object CMakeFiles/algorithm.dir/query/scratch_space_transform.cu.o
    [ 77%] Building CUDA object CMakeFiles/algorithm.dir/query/sort_reduce.cu.o
    [ 83%] Building CUDA object CMakeFiles/algorithm.dir/query/transform.cu.o
    [ 88%] Building CUDA object CMakeFiles/algorithm.dir/query/utils.cu.o
    /root/go/src/github.com/uber/aresdb/query/utils.hpp(114): warning: floating-point value does not fit in required integral type
              detected during:
                instantiation of "Value ares::get_identity_value<Value>(AggregateFunction) [with Value=int32_t]"
    /root/go/src/github.com/uber/aresdb/query/iterator.hpp(608): here
                instantiation of "ares::MeasureOutputIterator<Value>::MeasureOutputIterator(Value *, const uint32_t *, const uint32_t *, AggregateFunction) [with Value=int32_t]"
    /root/go/src/github.com/uber/aresdb/query/iterator.hpp(650): here
                instantiation of "ares::MeasureOutputIterator<Value> ares::make_measure_output_iterator(Value *, uint32_t *, uint32_t *, AggregateFunction) [with Value=int32_t]"
    /root/go/src/github.com/uber/aresdb/query/measure_transform.cu(31): here
                instantiation of "int ares::OutputVectorBinder<NInput, FunctorType>::transformMeasureOutput(MeasureOutputVector) const [with NInput=1, FunctorType=UnaryFunctorType]"
    /root/go/src/github.com/uber/aresdb/query/measure_transform.cu(43): here
    
    /root/go/src/github.com/uber/aresdb/query/utils.hpp(114): warning: floating-point value does not fit in required integral type
              detected during instantiation of "Value ares::get_identity_value<Value>(AggregateFunction) [with Value=uint32_t]"
    /root/go/src/github.com/uber/aresdb/query/hash_reduction.cu(364): here
    
    /usr/local/cuda/bin/../targets/x86_64-linux/include/thrust/iterator/iterator_facade.h(128): warning: calling a __host__ function from a __host__ __device__ function is not allowed
              detected during:
                instantiation of "Facade::reference thrust::iterator_core_access::dereference(const Facade &) [with Facade=ares::GeoBatchIntersectIterator<ares::ColumnIterator<GeoPointT>>]"
    (310): here
                instantiation of "thrust::iterator_facade<Derived, Value, System, Traversal, Reference, Difference>::reference thrust::iterator_facade<Derived, Value, System, Traversal, Reference, Difference>::operator*() const [with Derived=ares::GeoBatchIntersectIterator<ares::ColumnIterator<GeoPointT>>, Value=ares::EmptyStruct, System=thrust::host_system_tag, Traversal=thrust::random_access_traversal_tag, Reference=ares::EmptyStruct, Difference=std::ptrdiff_t]"
    /usr/local/cuda/bin/../targets/x86_64-linux/include/thrust/system/detail/sequential/for_each.h(56): here
                instantiation of "InputIterator thrust::system::detail::sequential::for_each(thrust::system::detail::sequential::execution_policy<DerivedPolicy> &, InputIterator, InputIterator, UnaryFunction) [with DerivedPolicy=thrust::system::cpp::detail::par_t, InputIterator=ares::GeoBatchIntersectIterator<ares::ColumnIterator<GeoPointT>>, UnaryFunction=ares::VoidFunctor]"
    /usr/local/cuda/bin/../targets/x86_64-linux/include/thrust/detail/for_each.inl(44): here
                instantiation of "InputIterator thrust::for_each(const thrust::detail::execution_policy_base<DerivedPolicy> &, InputIterator, InputIterator, UnaryFunction) [with DerivedPolicy=thrust::system::cpp::detail::par_t, InputIterator=ares::GeoBatchIntersectIterator<ares::ColumnIterator<GeoPointT>>, UnaryFunction=ares::VoidFunctor]"
    /root/go/src/github.com/uber/aresdb/query/geo_intersects.cu(282): here
                instantiation of "void ares::calculateBatchIntersection(GeoShapeBatch, InputIterator, uint32_t *, int, uint32_t, uint32_t *, __nv_bool, cudaStream_t) [with InputIterator=ares::ColumnIterator<GeoPointT>]"
    /root/go/src/github.com/uber/aresdb/query/geo_intersects.cu(292): here
                instantiation of "int ares::GeoIntersectionContext::run(uint32_t *, InputIterator) [with InputIterator=ares::ColumnIterator<GeoPointT>]"
    /root/go/src/github.com/uber/aresdb/query/geo_intersects.cu(171): here
                instantiation of "int ares::InputVectorBinder<ares::GeoIntersectionContext, 1>::bind(InputIterators...) [with InputIterators=<>]"
    /root/go/src/github.com/uber/aresdb/query/geo_intersects.cu(107): here
    
    /root/go/src/github.com/uber/aresdb/query/utils.hpp(114): warning: floating-point value does not fit in required integral type
              detected during instantiation of "Value ares::get_identity_value<Value>(AggregateFunction) [with Value=uint64_t]"
    /root/go/src/github.com/uber/aresdb/query/hash_reduction.cu(366): here
    
    /root/go/src/github.com/uber/aresdb/query/utils.hpp(114): warning: floating-point value does not fit in required integral type
              detected during instantiation of "Value ares::get_identity_value<Value>(AggregateFunction) [with Value=int32_t]"
    /root/go/src/github.com/uber/aresdb/query/hash_reduction.cu(370): here
    
    /usr/local/cuda/bin/../targets/x86_64-linux/include/thrust/iterator/iterator_facade.h(128): warning: calling a __host__ function from a __host__ __device__ function is not allowed
              detected during:
                instantiation of "Facade::reference thrust::iterator_core_access::dereference(const Facade &) [with Facade=ares::GeoBatchIntersectIterator<ares::RecordIDJoinIterator<GeoPointT>>]"
    (310): here
                instantiation of "thrust::iterator_facade<Derived, Value, System, Traversal, Reference, Difference>::reference thrust::iterator_facade<Derived, Value, System, Traversal, Reference, Difference>::operator*() const [with Derived=ares::GeoBatchIntersectIterator<ares::RecordIDJoinIterator<GeoPointT>>, Value=ares::EmptyStruct, System=thrust::host_system_tag, Traversal=thrust::random_access_traversal_tag, Reference=ares::EmptyStruct, Difference=std::ptrdiff_t]"
    /usr/local/cuda/bin/../targets/x86_64-linux/include/thrust/system/detail/sequential/for_each.h(56): here
                instantiation of "InputIterator thrust::system::detail::sequential::for_each(thrust::system::detail::sequential::execution_policy<DerivedPolicy> &, InputIterator, InputIterator, UnaryFunction) [with DerivedPolicy=thrust::system::cpp::detail::par_t, InputIterator=ares::GeoBatchIntersectIterator<ares::RecordIDJoinIterator<GeoPointT>>, UnaryFunction=ares::VoidFunctor]"
    /usr/local/cuda/bin/../targets/x86_64-linux/include/thrust/detail/for_each.inl(44): here
                instantiation of "InputIterator thrust::for_each(const thrust::detail::execution_policy_base<DerivedPolicy> &, InputIterator, InputIterator, UnaryFunction) [with DerivedPolicy=thrust::system::cpp::detail::par_t, InputIterator=ares::GeoBatchIntersectIterator<ares::RecordIDJoinIterator<GeoPointT>>, UnaryFunction=ares::VoidFunctor]"
    /root/go/src/github.com/uber/aresdb/query/geo_intersects.cu(282): here
                instantiation of "void ares::calculateBatchIntersection(GeoShapeBatch, InputIterator, uint32_t *, int, uint32_t, uint32_t *, __nv_bool, cudaStream_t) [with InputIterator=ares::RecordIDJoinIterator<GeoPointT>]"
    /root/go/src/github.com/uber/aresdb/query/geo_intersects.cu(292): here
                instantiation of "int ares::GeoIntersectionContext::run(uint32_t *, InputIterator) [with InputIterator=ares::RecordIDJoinIterator<GeoPointT>]"
    /root/go/src/github.com/uber/aresdb/query/geo_intersects.cu(205): here
                instantiation of "int ares::InputVectorBinder<ares::GeoIntersectionContext, 1>::bind(InputIterators...) [with InputIterators=<>]"
    /root/go/src/github.com/uber/aresdb/query/geo_intersects.cu(107): here
    
    /root/go/src/github.com/uber/aresdb/query/utils.hpp(114): warning: floating-point value does not fit in required integral type
              detected during:
                instantiation of "Value ares::get_identity_value<Value>(AggregateFunction) [with Value=uint32_t]"
    /root/go/src/github.com/uber/aresdb/query/iterator.hpp(608): here
                instantiation of "ares::MeasureOutputIterator<Value>::MeasureOutputIterator(Value *, const uint32_t *, const uint32_t *, AggregateFunction) [with Value=uint32_t]"
    /root/go/src/github.com/uber/aresdb/query/iterator.hpp(650): here
                instantiation of "ares::MeasureOutputIterator<Value> ares::make_measure_output_iterator(Value *, uint32_t *, uint32_t *, AggregateFunction) [with Value=uint32_t]"
    /root/go/src/github.com/uber/aresdb/query/measure_transform.cu(32): here
                instantiation of "int ares::OutputVectorBinder<NInput, FunctorType>::transformMeasureOutput(MeasureOutputVector) const [with NInput=1, FunctorType=UnaryFunctorType]"
    /root/go/src/github.com/uber/aresdb/query/measure_transform.cu(43): here
    
    /root/go/src/github.com/uber/aresdb/query/utils.hpp(114): warning: floating-point value does not fit in required integral type
              detected during instantiation of "Value ares::get_identity_value<Value>(AggregateFunction) [with Value=int64_t]"
    /root/go/src/github.com/uber/aresdb/query/hash_reduction.cu(372): here
    
    /root/go/src/github.com/uber/aresdb/query/utils.hpp(114): warning: floating-point value does not fit in required integral type
              detected during:
                instantiation of "Value ares::get_identity_value<Value>(AggregateFunction) [with Value=int64_t]"
    /root/go/src/github.com/uber/aresdb/query/iterator.hpp(608): here
                instantiation of "ares::MeasureOutputIterator<Value>::MeasureOutputIterator(Value *, const uint32_t *, const uint32_t *, AggregateFunction) [with Value=int64_t]"
    /root/go/src/github.com/uber/aresdb/query/iterator.hpp(650): here
                instantiation of "ares::MeasureOutputIterator<Value> ares::make_measure_output_iterator(Value *, uint32_t *, uint32_t *, AggregateFunction) [with Value=int64_t]"
    /root/go/src/github.com/uber/aresdb/query/measure_transform.cu(34): here
                instantiation of "int ares::OutputVectorBinder<NInput, FunctorType>::transformMeasureOutput(MeasureOutputVector) const [with NInput=1, FunctorType=UnaryFunctorType]"
    /root/go/src/github.com/uber/aresdb/query/measure_transform.cu(43): here
    
    /usr/local/cuda/bin/../targets/x86_64-linux/include/thrust/iterator/iterator_facade.h(310): warning: calling a __host__ function("ares::GeoBatchIntersectIterator< ::thrust::permutation_iterator< ::ares::VectorPartyIterator< ::GeoPointT> , unsigned int *> > ::dereference const") from a __host__ __device__ function("thrust::iterator_facade< ::ares::GeoBatchIntersectIterator< ::thrust::permutation_iterator< ::ares::VectorPartyIterator< ::GeoPointT> , unsigned int *> > ,  ::ares::EmptyStruct,  ::thrust::system::cpp::detail::tag,  ::thrust::random_access_traversal_tag,  ::ares::EmptyStruct, long> ::operator * const") is not allowed
    
    /usr/local/cuda/bin/../targets/x86_64-linux/include/thrust/iterator/iterator_facade.h(310): warning: calling a __host__ function("ares::GeoBatchIntersectIterator< ::ares::RecordIDJoinIterator< ::GeoPointT> > ::dereference const") from a __host__ __device__ function("thrust::iterator_facade< ::ares::GeoBatchIntersectIterator< ::ares::RecordIDJoinIterator< ::GeoPointT> > ,  ::ares::EmptyStruct,  ::thrust::system::cpp::detail::tag,  ::thrust::random_access_traversal_tag,  ::ares::EmptyStruct, long> ::operator * const") is not allowed
    
    /root/go/src/github.com/uber/aresdb/query/utils.hpp: In instantiation of 'Value ares::get_identity_value(AggregateFunction) [with Value = unsigned int]':
    /root/go/src/github.com/uber/aresdb/query/hash_reduction.cu:364:163:   required from here
    /root/go/src/github.com/uber/aresdb/query/utils.hpp:114:47: warning: overflow in implicit constant conversion [-Woverflow]
    /root/go/src/github.com/uber/aresdb/query/utils.hpp: In instantiation of 'Value ares::get_identity_value(AggregateFunction) [with Value = long unsigned int]':
    /root/go/src/github.com/uber/aresdb/query/hash_reduction.cu:366:178:   required from here
    /root/go/src/github.com/uber/aresdb/query/utils.hpp:114:47: warning: overflow in implicit constant conversion [-Woverflow]
    /root/go/src/github.com/uber/aresdb/query/utils.hpp: In instantiation of 'Value ares::get_identity_value(AggregateFunction) [with Value = int]':
    /root/go/src/github.com/uber/aresdb/query/hash_reduction.cu:370:148:   required from here
    /root/go/src/github.com/uber/aresdb/query/utils.hpp:114:47: warning: overflow in implicit constant conversion [-Woverflow]
    /root/go/src/github.com/uber/aresdb/query/utils.hpp: In instantiation of 'Value ares::get_identity_value(AggregateFunction) [with Value = long int]':
    /root/go/src/github.com/uber/aresdb/query/hash_reduction.cu:372:151:   required from here
    /root/go/src/github.com/uber/aresdb/query/utils.hpp:114:47: warning: overflow in implicit constant conversion [-Woverflow]
    /root/go/src/github.com/uber/aresdb/query/utils.hpp: In instantiation of 'Value ares::get_identity_value(AggregateFunction) [with Value = int]':
    /root/go/src/github.com/uber/aresdb/query/iterator.hpp:608:41:   required from 'ares::MeasureOutputIterator<Value>::MeasureOutputIterator(Value*, const uint32_t*, const uint32_t*, AggregateFunction) [with Value = int; uint32_t = unsigned int]'
    /root/go/src/github.com/uber/aresdb/query/iterator.hpp:649:74:   required from 'ares::MeasureOutputIterator<Value> ares::make_measure_output_iterator(Value*, uint32_t*, uint32_t*, AggregateFunction) [with Value = int; uint32_t = unsigned int]'
    /root/go/src/github.com/uber/aresdb/query/measure_transform.cu:31:65:   required from 'int ares::OutputVectorBinder<NInput, FunctorType>::transformMeasureOutput(MeasureOutputVector) const [with int NInput = 1; FunctorType = UnaryFunctorType]'
    /root/go/src/github.com/uber/aresdb/query/measure_transform.cu:43:108:   required from here
    /root/go/src/github.com/uber/aresdb/query/utils.hpp:114:47: warning: overflow in implicit constant conversion [-Woverflow]
    /root/go/src/github.com/uber/aresdb/query/utils.hpp: In instantiation of 'Value ares::get_identity_value(AggregateFunction) [with Value = unsigned int]':
    /root/go/src/github.com/uber/aresdb/query/iterator.hpp:608:41:   required from 'ares::MeasureOutputIterator<Value>::MeasureOutputIterator(Value*, const uint32_t*, const uint32_t*, AggregateFunction) [with Value = unsigned int; uint32_t = unsigned int]'
    /root/go/src/github.com/uber/aresdb/query/iterator.hpp:649:74:   required from 'ares::MeasureOutputIterator<Value> ares::make_measure_output_iterator(Value*, uint32_t*, uint32_t*, AggregateFunction) [with Value = unsigned int; uint32_t = unsigned int]'
    /root/go/src/github.com/uber/aresdb/query/measure_transform.cu:32:66:   required from 'int ares::OutputVectorBinder<NInput, FunctorType>::transformMeasureOutput(MeasureOutputVector) const [with int NInput = 1; FunctorType = UnaryFunctorType]'
    /root/go/src/github.com/uber/aresdb/query/measure_transform.cu:43:108:   required from here
    /root/go/src/github.com/uber/aresdb/query/utils.hpp:114:47: warning: overflow in implicit constant conversion [-Woverflow]
    /root/go/src/github.com/uber/aresdb/query/utils.hpp: In instantiation of 'Value ares::get_identity_value(AggregateFunction) [with Value = long int]':
    /root/go/src/github.com/uber/aresdb/query/iterator.hpp:608:41:   required from 'ares::MeasureOutputIterator<Value>::MeasureOutputIterator(Value*, const uint32_t*, const uint32_t*, AggregateFunction) [with Value = long int; uint32_t = unsigned int]'
    /root/go/src/github.com/uber/aresdb/query/iterator.hpp:649:74:   required from 'ares::MeasureOutputIterator<Value> ares::make_measure_output_iterator(Value*, uint32_t*, uint32_t*, AggregateFunction) [with Value = long int; uint32_t = unsigned int]'
    /root/go/src/github.com/uber/aresdb/query/measure_transform.cu:34:65:   required from 'int ares::OutputVectorBinder<NInput, FunctorType>::transformMeasureOutput(MeasureOutputVector) const [with int NInput = 1; FunctorType = UnaryFunctorType]'
    /root/go/src/github.com/uber/aresdb/query/measure_transform.cu:43:108:   required from here
    /root/go/src/github.com/uber/aresdb/query/utils.hpp:114:47: warning: overflow in implicit constant conversion [-Woverflow]
    
    
    [ 94%] Linking CUDA device code CMakeFiles/algorithm.dir/cmake_device_link.o
    [100%] Linking CUDA shared library lib/libalgorithm.so
    [100%] Built target algorithm
    Scanning dependencies of target lib
    [100%] Built target lib
    Scanning dependencies of target aresd
    make[3]: go: Command not found
    CMakeFiles/aresd.dir/build.make:480: recipe for target 'CMakeFiles/aresd' failed
    make[3]: *** [CMakeFiles/aresd] Error 127
    CMakeFiles/Makefile2:546: recipe for target 'CMakeFiles/aresd.dir/all' failed
    make[2]: *** [CMakeFiles/aresd.dir/all] Error 2
    CMakeFiles/Makefile2:553: recipe for target 'CMakeFiles/aresd.dir/rule' failed
    make[1]: *** [CMakeFiles/aresd.dir/rule] Error 2
    Makefile:311: recipe for target 'aresd' failed
    make: *** [aresd] Error 2
    The command '/bin/sh -c make aresd -j' returned a non-zero code: 2
    
    opened by yutiansut 2
  • Status of aresdb

    Status of aresdb

    1. What is the status of this product?

    2. Is this product still being maintained by Uber?

    3. Is this product still in active use at Uber?

    4. Are there any benchmarks anywhere?

    5. Anyone know of an active fork?

    The last active commits was back in 2019.

    opened by tildeleb 0
  • Java Support

    Java Support

    Is your feature request related to a problem? Please describe. No

    Describe the solution you'd like N/A

    Describe alternatives you've considered N/A

    Additional context Does AresDB support Java? If yes, could you please share some java examples/instructions?

    opened by h117561964 2
  • kaungmyat182804.wordpress

    kaungmyat182804.wordpress

    Describe the bug A clear and concise description of what the bug is.

    To Reproduce Steps to reproduce the behavior:

    1. Go to '...'
    2. Click on '....'
    3. Scroll down to '....'
    4. See error

    Expected behavior A clear and concise description of what you expected to happen.

    Screenshots If applicable, add screenshots to help explain your problem.

    Desktop (please complete the following information):

    • OS: [e.g. iOS]
    • Browser [e.g. chrome, safari]
    • Version [e.g. 22]

    Smartphone (please complete the following information):

    • Device: [e.g. iPhone6]
    • OS: [e.g. iOS8.1]
    • Browser [e.g. stock browser, safari]
    • Version [e.g. 22]

    Additional context Add any other context about the problem here.

    opened by NetSoeNetSoe 0
  • kaungmyat182804.wordpress

    kaungmyat182804.wordpress

    Describe the bug A clear and concise description of what the bug is.

    To Reproduce Steps to reproduce the behavior:

    1. Go to '...'
    2. Click on '....'
    3. Scroll down to '....'
    4. See error

    Expected behavior A clear and concise description of what you expected to happen.

    Screenshots If applicable, add screenshots to help explain your problem.

    Desktop (please complete the following information):

    • OS: [e.g. iOS]
    • Browser [e.g. chrome, safari]
    • Version [e.g. 22]

    Smartphone (please complete the following information):

    • Device: [e.g. iPhone6]
    • OS: [e.g. iOS8.1]
    • Browser [e.g. stock browser, safari]
    • Version [e.g. 22]

    Additional context Add any other context about the problem here.

    opened by NetSoeNetSoe 0
  • Error populating query using the examples

    Error populating query using the examples

    Describe the bug I was following the examples given in aresdb/examples to populate the database with the tables, data and queries. Both the table and data worked fine, but the example for the query resulted in panic: query failed with status code 400. I have attached the full error here: error.log.

    To Reproduce Follow the example code given at aresdb/examples:

    1. cd examples
    2. go build examples.go
    3. ./examples tables --dataset 1k_trips
    4. ./examples data --dataset 1k_trips
    5. ./examples query --dataset 1k_trips

    Expected behavior Expected the database to respond with the response from the queries.

    Desktop (please complete the following information):

    • OS: Ubuntu 18.04.4 LTS
    • Browser: Firefox 76.0.1
    • Docker: v19.03.11
    • nvidia-docker: v19.03.11

    Additional context I am using the docker image of the AresDB.

    opened by sianghwee 1
Releases(v0.0.1)
Owner
Uber Open Source
Open Source Software at Uber
Uber Open Source
Scalable datastore for metrics, events, and real-time analytics

InfluxDB InfluxDB is an open source time series platform. This includes APIs for storing and querying data, processing it in the background for ETL or

InfluxData 24.7k Jan 5, 2023
Scalable datastore for metrics, events, and real-time analytics

InfluxDB InfluxDB is an open source time series platform. This includes APIs for storing and querying data, processing it in the background for ETL or

InfluxData 24.7k Jan 4, 2023
A MySQL-compatible relational database with a storage agnostic query engine. Implemented in pure Go.

go-mysql-server is a SQL engine which parses standard SQL (based on MySQL syntax) and executes queries on data sources of your choice. A simple in-memory database and table implementation are provided, and you can query any data source you want by implementing a few interfaces.

DoltHub 947 Dec 27, 2022
Real-time Geospatial and Geofencing

Tile38 is an open source (MIT licensed), in-memory geolocation data store, spatial index, and realtime geofence. It supports a variety of object types

Josh Baker 8.4k Dec 30, 2022
VectorSQL is a free analytics DBMS for IoT & Big Data, compatible with ClickHouse.

NOTICE: This project have moved to fuse-query VectorSQL is a free analytics DBMS for IoT & Big Data, compatible with ClickHouse. Features High Perform

VectorEngine 251 Jan 6, 2023
A high performance NoSQL Database Server powered by Go

LedisDB Ledisdb is a high-performance NoSQL database library and server written in Go. It's similar to Redis but store data in disk. It supports many

LedisDB 3.9k Dec 26, 2022
Multitiered file storage API built on Filecoin and IPFS

Powergate Powergate is a multitiered file storage API built on Filecoin and IPFS, and an index builder for Filecoin data. It's designed to be modular

textile.io 368 Dec 20, 2022
IceFireDB - Distributed disk storage system based on Raft and RESP protocol.

Distributed disk storage database based on Raft and Redis protocol.

GITSRC 947 Dec 27, 2022
Secure storage for personal records built to comply with GDPR

Databunker Databunker is a Personally Identifiable Information (PII) Data Storage Service built to Comply with GDPR and CCPA Privacy Requirements. Pro

null 1k Jan 8, 2023
moss - a simple, fast, ordered, persistable, key-val storage library for golang

moss moss provides a simple, fast, persistable, ordered key-val collection implementation as a 100% golang library. moss stands for "memory-oriented s

null 895 Dec 18, 2022
A distributed MySQL binlog storage system built on Raft

What is kingbus? 中文 Kingbus is a distributed MySQL binlog store based on raft. Kingbus can act as a slave to the real master and as a master to the sl

Fei Chen 856 Dec 31, 2022
Key-Value Storage written in Go.

kvs kvs is an in-memory key-value storage written in Go. It has 2 different usage. It can be used as a package by importing it to your code or as a se

Gökhan Özeloğlu 4 Jun 15, 2022
The Prometheus monitoring system and time series database.

Prometheus Visit prometheus.io for the full documentation, examples and guides. Prometheus, a Cloud Native Computing Foundation project, is a systems

Prometheus 46.1k Dec 31, 2022
VictoriaMetrics: fast, cost-effective monitoring solution and time series database

VictoriaMetrics VictoriaMetrics is a fast, cost-effective and scalable monitoring solution and time series database. It is available in binary release

VictoriaMetrics 7.6k Jan 8, 2023
LinDB is an open-source Time Series Database which provides high performance, high availability and horizontal scalability.

LinDB is an open-source Time Series Database which provides high performance, high availability and horizontal scalability. LinDB stores all monitoring data of ELEME Inc, there is 88TB incremental writes per day and 2.7PB total raw data.

LinDB 2.3k Jan 1, 2023
TalariaDB is a distributed, highly available, and low latency time-series database for Presto

TalariaDB is a distributed, highly available, and low latency time-series database that stores real-time data. It's built on top of Badger DB.

Grab 104 Nov 16, 2022
Export output from pg_stat_activity and pg_stat_statements from Postgres into a time-series database that supports the Influx Line Protocol (ILP).

pgstat2ilp pgstat2ilp is a command-line program for exporting output from pg_stat_activity and pg_stat_statements (if the extension is installed/enabl

Zikani Nyirenda Mwase 4 Dec 15, 2021
Lightweight RESTful database engine based on stack data structures

piladb [pee-lah-dee-bee]. pila means stack or battery in Spanish. piladb is a lightweight RESTful database engine based on stack data structures. Crea

Fernando Álvarez 200 Nov 27, 2022
Fast Database engine in Go.

gaeadb gaeadb is a pure Go Database engine designed by nnsgmsone. The goal of the project is to provide a database engine for table or other complex d

null 14 Oct 14, 2022