Vald. A Highly Scalable Distributed Vector Search Engine

Overview

License: Apache 2.0 release Go Reference Codacy Badge Go Report Card DepShield Badge FOSSA Status DeepSource CLA Artifact Hub Slack Twitter

What is Vald?

Vald is a highly scalable distributed fast approximate nearest neighbor dense vector search engine.

Vald is designed and implemented based on Cloud-Native architecture.

It uses the fastest ANN Algorithm NGT to search neighbors.

Vald has automatic vector indexing and index backup, and horizontal scaling which made for searching from billions of feature vector data.

Vald is easy to use, feature-rich and highly customizable as you needed.

Go to Get Started page to try out Vald :)

(If you are interested in ANN benchmarks, please refer to the official website.)

Main Features

  • Asynchronous Auto Indexing

    • Usually the graph requires locking during indexing, which causes stop-the-world. But Vald uses distributed index graphs so it continues to work during indexing.
  • Customizable Ingress/Egress Filtering

    • Vald implements it's own highly customizable Ingress/Egress filter.
    • Which can be configured to fit the gRPC interface.
      • Ingress Filter: Ability to Vectorize through filter on request.
      • Egress Filter: rerank or filter the searching result with your own algorithm.
  • Cloud-native based vector searching engine

    • Horizontal scalable on memory and CPU for your demand.
  • Auto Backup for Index data

    • Vald has a feature to store the backup of the index data using MySQL or Cassandra which enables disaster recovery.
  • Distributed Indexing

    • Vald distribute vector index to multiple agents, each agent stores different index.
  • Index Replication

    • Vald stores each index in multiple agents which enables index replicas.
    • Automatically rebalance the replica when some Vald agent goes down.
  • Easy to use

    • Vald can be easily installed in a few steps.
  • Highly customizable

    • You can configure the number of vector dimensions, the number of replica and etc.
  • Multi language supported

    • Go, Java, Clojure, Node.js, and Python client library are supported.
    • gRPC APIs can be triggered by any programming languages which support gRPC.
    • REST API is also supported.

Requirements

  • Kubernetes 1.17~
  • AVX2 instructions (required by Vald Agent NGT)

Get Started

Please refer to Get Started.

Installation

Using Helm

helm repo add vald https://vald.vdaas.org/charts
helm install vald-cluster vald/vald

If you use the default values.yaml, the nightly images will be installed.

Docker image tagging policy

  • nightly ... latest build of master branch
  • vX.X.X ... released versions
  • latest ... latest build of release versions
  • stable ... latest long-term supported version

Using Helm-operator

vald-helm-operator

Example

Write example here

Architecture Overview

Please refer here for more details of the architecture overview in the future.

Development

Before your first commit to this repository, it is strongly recommended to run the commands below.

make init

Components

Component Docker image
Agent NGT
Agent Sidecar
Discoverer
Gateways






Backup Managers


Compressor
Metas


Index Manager
Helm Operator

Contribution

Please read the contribution guide

Contributors

All Contributors

Thanks goes to these wonderful people (emoji key):


Yusuke Kato

πŸ’» 🎨 🚧 πŸ“†

Rintaro Okamura

πŸ’» πŸ“– 🚧 πŸ“¦

Kosuke Morimoto

πŸ’» πŸ’‘ πŸ”§ ⚠️

Kiichiro YUKAWA

πŸ“– 🚧 ⚠️ βœ…

datelier

πŸ’» πŸ€”

Kevin Diu

πŸ“– πŸ’‘ ⚠️ βœ…

Hiroto Funakoshi

πŸ“– πŸ”§ ⚠️ βœ…

taisho

🎨 πŸ“– πŸ’‘

Pierre Grimaud

πŸ“–

Omer Katz

πŸ“– βœ…

LICENSE

vald released under Apache 2.0 license, refer LICENSE file.

FOSSA Status

Issues
  • create test template for using gotests

    create test template for using gotests

    Signed-off-by: vankichi [email protected]

    Description:

    I added make command and testing template for generating missing test files using gotests . This PR is related to #325

    Note:

    We will update the below test files using our template. Missing test files are below (check filled test file will be generated):

    files
    • [x] cmd/agent/ngt/main_test.go
    • [ ] cmd/cli/vdctl/main_test.go
    • [x] cmd/discoverer/k8s/main_test.go
    • [x] cmd/gateway/vald/main_test.go
    • [x] cmd/manager/backup/cassandra/main_test.go
    • [x] cmd/manager/backup/mysql/main_test.go
    • [x] cmd/manager/compressor/main_test.go
    • [x] cmd/manager/index/main_test.go
    • [x] cmd/manager/replication/agent/main_test.go
    • [x] cmd/manager/replication/controller/main_test.go
    • [x] cmd/meta/cassandra/main_test.go
    • [x] cmd/meta/redis/main_test.go
    • [x] hack/benchmark/internal/assets/dataset_test.go
    • [x] hack/benchmark/internal/assets/loader_test.go
    • [ ] hack/benchmark/internal/client/ngtd/grpc/client_test.go
    • [ ] hack/benchmark/internal/client/ngtd/grpc/option_test.go
    • [ ] hack/benchmark/internal/client/ngtd/rest/client_test.go
    • [ ] hack/benchmark/internal/client/ngtd/rest/option_test.go
    • [x] hack/benchmark/internal/e2e/e2e_test.go
    • [x] hack/benchmark/internal/e2e/option_test.go
    • [x] hack/benchmark/internal/e2e/strategy/create_index_option_test.go
    • [x] hack/benchmark/internal/e2e/strategy/create_index_test.go
    • [x] hack/benchmark/internal/e2e/strategy/insert_option_test.go
    • [x] hack/benchmark/internal/e2e/strategy/insert_test.go
    • [x] hack/benchmark/internal/e2e/strategy/remove_option_test.go
    • [x] hack/benchmark/internal/e2e/strategy/remove_test.go
    • [x] hack/benchmark/internal/e2e/strategy/search_option_test.go
    • [x] hack/benchmark/internal/e2e/strategy/search_test.go
    • [x] hack/benchmark/internal/e2e/strategy/stream_insert_test.go
    • [x] hack/benchmark/internal/e2e/strategy/stream_remove_test.go
    • [x] hack/benchmark/internal/e2e/strategy/stream_search_option_test.go
    • [x] hack/benchmark/internal/e2e/strategy/stream_search_test.go
    • [ ] hack/benchmark/internal/starter/agent/ngt/ngt_test.go
    • [ ] hack/benchmark/internal/starter/agent/ngt/option_test.go
    • [ ] hack/benchmark/internal/starter/external/ngtd/ngtd_test.go
    • [ ] hack/benchmark/internal/starter/external/ngtd/option_test.go
    • [ ] hack/benchmark/internal/starter/gateway/vald/vald_test.go
    • [ ] hack/license/gen/main_test.go
    • [ ] hack/swagger/main_test.go
    • [ ] hack/tools/config/agent/ngt/main_test.go
    • [ ] hack/tools/config/discoverer/k8s/main_test.go
    • [ ] hack/tools/config/gateway/vald/main_test.go
    • [ ] hack/tools/config/manager/backup/mysql/main_test.go
    • [ ] hack/tools/config/meta/redis/main_test.go
    • [ ] hack/tools/metrics/main_test.go
    • [x] internal/cache/cache_test.go
    • [x] internal/cache/cacher/cacher_test.go
    • [x] internal/cache/gache/gache_test.go
    • [x] internal/cache/gache/option_test.go
    • [x] internal/cache/option_test.go
    • [x] internal/client/agent/ngt/grpc/client_test.go
    • [x] internal/client/agent/ngt/grpc/option_test.go
    • [x] internal/client/agent/ngt/rest/client_test.go
    • [x] internal/client/agent/ngt/rest/option_test.go
    • [x] internal/client/compressor/client_test.go
    • [x] internal/client/compressor/option.go
    • [x] internal/client/compressor/option_test.go
    • [x] internal/client/discoverer/discover_test.go
    • [x] internal/client/discoverer/option_test.go
    • [x] internal/client/gateway/vald/grpc/client_test.go
    • [x] internal/client/gateway/vald/grpc/option_test.go
    • [x] internal/client/gateway/vald/rest/client_test.go
    • [x] internal/client/gateway/vald/rest/option_test.go
    • [x] internal/config/backoff_test.go
    • [x] internal/config/backup_test.go
    • [x] internal/config/cassandra_test.go
    • [x] internal/config/client_test.go
    • [x] internal/config/compress.go
    • [x] internal/config/compress_test.go
    • [x] internal/config/config_test.go
    • [x] internal/config/debug_test.go
    • [x] internal/config/discoverer_test.go
    • [x] internal/config/filter_test.go
    • [x] internal/config/gateway_test.go
    • [x] internal/config/grpc_test.go
    • [x] internal/config/index_test.go
    • [x] internal/config/log_test.go
    • [x] internal/config/meta_test.go
    • [x] internal/config/mysql_test.go
    • [x] internal/config/ngt_test.go
    • [x] internal/config/observability_test.go
    • [x] internal/config/redis_test.go
    • [x] internal/config/server_test.go
    • [x] internal/config/tcp_test.go
    • [x] internal/config/tls_test.go
    • [x] internal/config/transport_test.go
    • [x] internal/core/converter/tensorflow/option_test.go
    • [x] internal/core/converter/tensorflow/tensorflow_test.go
    • [x] internal/db/kvs/redis/option_test.go
    • [x] internal/db/kvs/redis/redis_test.go
    • [x] internal/db/nosql/cassandra/cassandra_test.go
    • [x] internal/db/nosql/cassandra/conviction_test.go
    • [x] internal/db/nosql/cassandra/option_test.go
    • [x] internal/db/rdb/mysql/model_test.go
    • [x] internal/db/rdb/mysql/mysql_test.go
    • [x] internal/db/rdb/mysql/option_test.go
    • [x] internal/errgroup/group_test.go
    • [x] internal/errors/cassandra_test.go
    • [x] internal/errors/compressor.go
    • [x] internal/errors/mysql_test.go
    • [x] internal/errors/redis_test.go
    • [x] internal/info/info_test.go
    • [x] internal/k8s/metrics/node/node_test.go
    • [x] internal/k8s/metrics/node/option_test.go
    • [x] internal/k8s/metrics/pod/option_test.go
    • [x] internal/k8s/metrics/pod/pod_test.go
    • [x] internal/k8s/node/node_test.go
    • [x] internal/k8s/node/option_test.go
    • [x] internal/k8s/option_test.go
    • [x] internal/k8s/pod/option_test.go
    • [x] internal/k8s/pod/pod_test.go
    • [x] internal/k8s/reconciler_test.go
    • [x] internal/log/mock/logger_test.go
    • [x] internal/log/mock/retry_test.go
    • [x] internal/net/grpc/metric/client_test.go
    • [x] internal/net/grpc/metric/server_test.go
    • [x] internal/net/grpc/proto/proto_test.go
    • [x] internal/net/grpc/status/status_test.go
    • [x] internal/net/net_test.go
    • [x] internal/net/tcp/control_darwin_test.go
    • [x] internal/net/tcp/control_other_test.go
    • [x] internal/net/tcp/control_unix_test.go
    • [x] internal/net/tcp/control_windows_test.go
    • [x] internal/net/tcp/dialer_test.go
    • [x] internal/net/tcp/option_dialer_test.go
    • [x] internal/observability/collector/collector_option_test.go
    • [x] internal/observability/collector/collector_test.go
    • [x] internal/observability/exporter/jaeger/jaeger_option_test.go
    • [x] internal/observability/exporter/jaeger/jaeger_test.go
    • [x] internal/observability/exporter/prometheus/prometheus_option_test.go
    • [x] internal/observability/exporter/prometheus/prometheus_test.go
    • [x] internal/observability/metrics/agent/ngt/ngt_test.go
    • [x] internal/observability/metrics/manager/compressor/compressor_test.go
    • [x] internal/observability/metrics/mem/mem_test.go
    • [x] internal/observability/metrics/metrics_test.go
    • [x] internal/observability/metrics/runtime/cgo/cgo_test.go
    • [x] internal/observability/metrics/runtime/goroutine/goroutine_test.go
    • [x] internal/observability/metrics/version/version_test.go
    • [x] internal/observability/observability_option_test.go
    • [x] internal/observability/observability_test.go
    • [x] internal/observability/trace/status_test.go
    • [x] internal/observability/trace/trace_option_test.go
    • [x] internal/observability/trace/trace_test.go
    • [x] internal/params/option_test.go
    • [x] internal/params/params_test.go
    • [x] internal/runner/option_test.go
    • [x] internal/runner/runner_test.go
    • [x] internal/singleflight/singleflight_test.go
    • [x] internal/tls/option_test.go
    • [x] internal/tls/tls_test.go
    • [x] internal/worker/queue_test.go
    • [x] internal/worker/queue_option_test.go
    • [x] internal/worker/worker_test.go
    • [x] internal/worker/worker_option_test.go
    • [x] pkg/agent/ngt/config/config_test.go
    • [x] pkg/agent/ngt/handler/grpc/handler_test.go
    • [x] pkg/agent/ngt/handler/grpc/option_test.go
    • [x] pkg/agent/ngt/handler/rest/handler_test.go
    • [x] pkg/agent/ngt/handler/rest/option_test.go
    • [x] pkg/agent/ngt/router/option_test.go
    • [x] pkg/agent/ngt/router/router_test.go
    • [x] pkg/agent/ngt/service/kvs/kvs_test.go
    • [x] pkg/agent/ngt/service/kvs/ou_test.go
    • [x] pkg/agent/ngt/service/kvs/uo_test.go
    • [x] pkg/agent/ngt/service/ngt_test.go
    • [x] pkg/agent/ngt/service/vcaches_test.go
    • [x] pkg/agent/ngt/usecase/agentd_test.go
    • [x] pkg/discoverer/k8s/config/config_test.go
    • [x] pkg/discoverer/k8s/handler/grpc/handler_test.go
    • [x] pkg/discoverer/k8s/handler/grpc/option_test.go
    • [x] pkg/discoverer/k8s/handler/rest/handler_test.go
    • [x] pkg/discoverer/k8s/handler/rest/option_test.go
    • [x] pkg/discoverer/k8s/router/option_test.go
    • [x] pkg/discoverer/k8s/router/router_test.go
    • [x] pkg/discoverer/k8s/service/discover_test.go
    • [x] pkg/discoverer/k8s/service/nodemap_test.go
    • [x] pkg/discoverer/k8s/service/nodemetricsmap_test.go
    • [x] pkg/discoverer/k8s/service/option_test.go
    • [x] pkg/discoverer/k8s/service/podmetricsmap_test.go
    • [x] pkg/discoverer/k8s/service/podsmap_test.go
    • [x] pkg/discoverer/k8s/usecase/discovered_test.go
    • [x] pkg/gateway/vald/config/config_test.go
    • [x] pkg/gateway/vald/handler/grpc/checklist_test.go
    • [x] pkg/gateway/vald/handler/grpc/handler_test.go
    • [x] pkg/gateway/vald/handler/grpc/option_test.go
    • [x] pkg/gateway/vald/handler/rest/handler_test.go
    • [x] pkg/gateway/vald/handler/rest/option_test.go
    • [x] pkg/gateway/vald/router/option_test.go
    • [x] pkg/gateway/vald/router/router_test.go
    • [x] pkg/gateway/vald/service/backup_option_test.go
    • [x] pkg/gateway/vald/service/backup_test.go
    • [x] pkg/gateway/vald/service/filter_option_test.go
    • [x] pkg/gateway/vald/service/filter_test.go
    • [x] pkg/gateway/vald/service/gateway_option_test.go
    • [x] pkg/gateway/vald/service/gateway_test.go
    • [x] pkg/gateway/vald/service/meta_option_test.go
    • [x] pkg/gateway/vald/service/meta_test.go
    • [x] pkg/gateway/vald/usecase/vald_test.go
    • [x] pkg/manager/backup/cassandra/config/config_test.go
    • [x] pkg/manager/backup/cassandra/handler/grpc/handler_test.go
    • [x] pkg/manager/backup/cassandra/handler/grpc/option_test.go
    • [x] pkg/manager/backup/cassandra/handler/rest/handler_test.go
    • [x] pkg/manager/backup/cassandra/handler/rest/option_test.go
    • [x] pkg/manager/backup/cassandra/router/option_test.go
    • [x] pkg/manager/backup/cassandra/router/router_test.go
    • [x] pkg/manager/backup/cassandra/service/cassandra_test.go
    • [x] pkg/manager/backup/cassandra/usecase/backupd_test.go
    • [x] pkg/manager/backup/mysql/config/config_test.go
    • [x] pkg/manager/backup/mysql/handler/grpc/handler_test.go
    • [x] pkg/manager/backup/mysql/handler/grpc/option_test.go
    • [x] pkg/manager/backup/mysql/handler/rest/handler_test.go
    • [x] pkg/manager/backup/mysql/handler/rest/option_test.go
    • [x] pkg/manager/backup/mysql/model/model_test.go
    • [x] pkg/manager/backup/mysql/router/option_test.go
    • [x] pkg/manager/backup/mysql/router/router_test.go
    • [x] pkg/manager/backup/mysql/service/mysql_test.go
    • [x] pkg/manager/backup/mysql/usecase/backupd_test.go
    • [x] pkg/manager/compressor/config/config.go
    • [x] pkg/manager/compressor/config/config_test.go
    • [x] pkg/manager/compressor/handler/grpc/handler.go
    • [x] pkg/manager/compressor/handler/grpc/handler_test.go
    • [x] pkg/manager/compressor/handler/grpc/option.go
    • [x] pkg/manager/compressor/handler/grpc/option_test.go
    • [x] pkg/manager/compressor/handler/rest/handler_test.go
    • [x] pkg/manager/compressor/handler/rest/option_test.go
    • [x] pkg/manager/compressor/router/option_test.go
    • [x] pkg/manager/compressor/router/router_test.go
    • [x] pkg/manager/compressor/service/backup_option_test.go
    • [x] pkg/manager/compressor/service/backup_test.go
    • [x] pkg/manager/compressor/service/compress.go
    • [x] pkg/manager/compressor/service/compress_option.go
    • [x] pkg/manager/compressor/service/compress_option_test.go
    • [x] pkg/manager/compressor/service/compress_test.go
    • [x] pkg/manager/compressor/service/registerer.go
    • [x] pkg/manager/compressor/service/registerer_option.go
    • [x] pkg/manager/compressor/usecase/compressord.go
    • [x] pkg/manager/compressor/usecase/compressord_test.go
    • [x] pkg/manager/index/config/config_test.go
    • [x] pkg/manager/index/handler/grpc/checklist_test.go
    • [x] pkg/manager/index/handler/grpc/handler_test.go
    • [x] pkg/manager/index/handler/grpc/option_test.go
    • [x] pkg/manager/index/handler/rest/handler_test.go
    • [x] pkg/manager/index/handler/rest/option_test.go
    • [x] pkg/manager/index/router/option_test.go
    • [x] pkg/manager/index/router/router_test.go
    • [x] pkg/manager/index/service/indexer_test.go
    • [x] pkg/manager/index/service/indexinfos_test.go
    • [x] pkg/manager/index/service/option_test.go
    • [x] pkg/manager/index/usecase/indexer_test.go
    • [x] pkg/manager/replication/agent/config/config_test.go
    • [x] pkg/manager/replication/agent/handler/grpc/handler_test.go
    • [x] pkg/manager/replication/agent/handler/rest/handler_test.go
    • [x] pkg/manager/replication/agent/handler/rest/option_test.go
    • [x] pkg/manager/replication/agent/router/option_test.go
    • [x] pkg/manager/replication/agent/router/router_test.go
    • [x] pkg/manager/replication/agent/usecase/backupd_test.go
    • [x] pkg/manager/replication/controller/config/config_test.go
    • [x] pkg/manager/replication/controller/handler/grpc/handler_test.go
    • [x] pkg/manager/replication/controller/handler/grpc/option_test.go
    • [x] pkg/manager/replication/controller/handler/rest/handler_test.go
    • [x] pkg/manager/replication/controller/handler/rest/option_test.go
    • [x] pkg/manager/replication/controller/router/option_test.go
    • [x] pkg/manager/replication/controller/router/router_test.go
    • [x] pkg/manager/replication/controller/service/discover_test.go
    • [x] pkg/manager/replication/controller/service/nodemap_test.go
    • [x] pkg/manager/replication/controller/service/nodemetricsmap_test.go
    • [x] pkg/manager/replication/controller/service/option_test.go
    • [x] pkg/manager/replication/controller/service/podmetricsmap_test.go
    • [x] pkg/manager/replication/controller/service/podsmap_test.go
    • [x] pkg/manager/replication/controller/usecase/discovered_test.go
    • [x] pkg/meta/cassandra/config/config_test.go
    • [x] pkg/meta/cassandra/handler/grpc/handler_test.go
    • [x] pkg/meta/cassandra/handler/grpc/option_test.go
    • [x] pkg/meta/cassandra/handler/rest/handler_test.go
    • [x] pkg/meta/cassandra/handler/rest/option_test.go
    • [x] pkg/meta/cassandra/router/option_test.go
    • [x] pkg/meta/cassandra/router/router_test.go
    • [x] pkg/meta/cassandra/service/cassandra_test.go
    • [x] pkg/meta/cassandra/usecase/meta_test.go
    • [x] pkg/meta/redis/config/config_test.go
    • [x] pkg/meta/redis/handler/grpc/handler_test.go
    • [x] pkg/meta/redis/handler/grpc/option_test.go
    • [x] pkg/meta/redis/handler/rest/handler_test.go
    • [x] pkg/meta/redis/handler/rest/option_test.go
    • [x] pkg/meta/redis/router/option_test.go
    • [x] pkg/meta/redis/router/router_test.go/
    • [x] pkg/meta/redis/service/redis_test.go
    • [x] pkg/meta/redis/usecase/meta_test.go

    Related Issue:

    #325

    How Has This Been Tested?:

    Environment:

    • Golang Version: 1.14
    • Docker Version: 19.03.5
    • Kubernetes Version: 1.17.3
    • NGT Version: 1.9.1

    Types of changes:

    • [ ] Bug fix [type/bug]
    • [ ] New feature [type/feature]
    • [x] Add tests [type/test]
    • [ ] Security related changes [type/security]
    • [ ] Add documents [type/documentation]
    • [ ] Refactoring [type/refactoring]
    • [ ] Update dependencies [type/dependency]
    • [ ] Update benchmarks and performances [type/bench]
    • [ ] Update CI [type/ci]

    Changes to Core Features:

    • [ ] Have you added an explanation of what your changes do and why you'd like us to include them?
    • [ ] Have you written new tests for your core changes, as applicable?
    • [ ] Have you successfully ran tests with your changes locally?

    Checklist:

    • [x] I have read the CONTRIBUTING document.
    • [ ] I have checked open Pull Requests for the similar feature or fixes?
    • [ ] I have added tests and benchmarks to cover my changes.
    • [ ] I have ensured all new and existing tests passed.
    • [ ] I have commented my code, particularly in hard-to-understand areas
    • [ ] I have updated the documentation accordingly.
    team/set team/sre type/test priority/low area/agent area/gateway area/discoverer area/meta area/manager/backup area/manager/index area/manager/replication area/internal status/author-review size/XXXL area/makefile 
    opened by vankichi 72
  • [patch] Test/internal/tcp

    [patch] Test/internal/tcp

    Author review required!!!! Maybe performance test required before merging.

    Description:

    Sorry for this PR become so huge :(

    This PR includes the following changes:

    • enhancement: fetch DNS cache in round robin order instead of random order (which may cause performance problem since it is not balanced)
    • enhancement: do not lookup DNS when the address passed to cachedDialer() is IP address
    • bug fix: DNS cache expire hook not executed
    • bug fix: dialerKeepAlive did not set properly
    • internal/tcp package test
    • code refactoring

    The test coverage is over 90% in internal/net/tcp package.

    When I working on this PR, I found a bug about sharing cache. and I created a PR for this bug fix. https://github.com/vdaas/vald/pull/560

    Performance result:

    Gateway replicas: 10 Agent replicas: 30 Dataset: random-786-100000 Concurrency: 32 Batch size: 100

    Insert mode (100000 random vectors)

    Nightly branch (20200713) Average VPS for each run:

    5395.421578
    6424.425950
    6425.912179
    6461.443722
    5988.220358
    6493.199904
    6486.264278
    6335.389072
    6129.457926
    6659.673990
    

    Variance

    Total Average: 6279.9408957


    PR-501 Average VPS for each run:

    6217.897276
    6673.575530
    6976.387795
    6890.498418
    6487.006260
    6585.224161
    6734.012306
    6966.123523
    5937.095263
    6751.386072
    

    Variance

    Total Average: 6621.9206604 (+ 5.4455 % )

    Search Mode

    Nightly branch (20200714) Average VPS for each run:

    55342.011671
    55041.653094
    54863.910108
    55507.475455
    54847.253156
    55087.350757
    54773.136435
    55560.020191
    55438.817729
    55180.390630
    

    Variance

    Total Average: 55164.2019226


    PR501 Average VPS for each run:

    53006.025749
    54954.973641
    55533.508719
    55152.016128
    55107.615004
    55149.334190
    55255.700455
    55688.780070
    54430.523454
    54487.434100
    

    Variance

    Total Average: 54876.591151 (- 0.521372124%)

    Search Mode (with 20000 random data & 3 index replica)

    Nightly branch (20200715): Average VPS for each run:

    54754.772502
    54210.456157
    55277.095015
    54375.751102
    54606.793579
    54980.820024
    55043.223001
    54279.938913
    54407.296268
    54939.336378
    

    Variance

    Total Average: 54687.5482939


    PR 501: Average VPS for each run:

    54635.047079
    54744.703738
    55256.844790
    54916.153493
    55260.394869
    54986.154799
    55572.723950
    54315.390605
    55549.817220
    53304.934344
    

    Variance

    Total Average: 54854.2164888 (+ 3%)

    Related Issue:

    How Has This Been Tested?:

    Environment:

    • Go Version: 1.14.3
    • Docker Version: 19.03.8
    • Kubernetes Version: 1.18.2
    • NGT Version: 1.11.5

    Types of changes:

    • [x] Bug fix [type/bug]
    • [ ] New feature [type/feature]
    • [x] Add tests [type/test]
    • [ ] Security related changes [type/security]
    • [ ] Add documents [type/documentation]
    • [x] Refactoring [type/refactoring]
    • [ ] Update dependencies [type/dependency]
    • [ ] Update benchmarks and performances [type/bench]
    • [ ] Update CI [type/ci]

    Changes to Core Features:

    • [x] Have you added an explanation of what your changes do and why you'd like us to include them?
    • [x] Have you written new tests for your core changes, as applicable?
    • [x] Have you successfully ran tests with your changes locally?

    Checklist:

    • [x] I have read the CONTRIBUTING document.
    • [x] I have checked open Pull Requests for the similar feature or fixes?
    • [x] I have added tests and benchmarks to cover my changes.
    • [x] I have ensured all new and existing tests passed.
    • [x] I have commented my code, particularly in hard-to-understand areas
    • [ ] I have updated the documentation accordingly.
    team/set type/test priority/low area/internal status/author-review size/XXL 
    opened by kevindiu 57
  • internal/tls test

    internal/tls test

    Just in case I think we need author review for the tls/tls.go file.

    Description:

    This PR implements the test case of internal/tls package. The test coverage is 93%. The missing test case is as the follow:

    1. Option will never return error so for example tls.go:48 cannot be covered.
    2. no error return from x509.SystemCertPool() call so tls.go:113 cannot be covered.

    It also include the new package implementation called test to get the test data path.

    Related Issue:

    How Has This Been Tested?:

    Environment:

    • Go Version: 1.14.3
    • Docker Version: 19.03.8
    • Kubernetes Version: 1.18.2
    • NGT Version: 1.11.5

    Types of changes:

    • [ ] Bug fix [type/bug]
    • [ ] New feature [type/feature]
    • [x] Add tests [type/test]
    • [ ] Security related changes [type/security]
    • [ ] Add documents [type/documentation]
    • [ ] Refactoring [type/refactoring]
    • [ ] Update dependencies [type/dependency]
    • [ ] Update benchmarks and performances [type/bench]
    • [ ] Update CI [type/ci]

    Changes to Core Features:

    • [x] Have you added an explanation of what your changes do and why you'd like us to include them?
    • [x] Have you written new tests for your core changes, as applicable?
    • [x] Have you successfully ran tests with your changes locally?

    Checklist:

    • [x] I have read the CONTRIBUTING document.
    • [x] I have checked open Pull Requests for the similar feature or fixes?
    • [x] I have added tests and benchmarks to cover my changes.
    • [ ] I have ensured all new and existing tests passed.
    • [ ] I have commented my code, particularly in hard-to-understand areas
    • [ ] I have updated the documentation accordingly.
    team/set type/test priority/low area/internal status/author-review size/XXL 
    opened by kevindiu 55
  • [proposal] Test code of the testing guideline

    [proposal] Test code of the testing guideline

    We suggest the following example test template of the testing guideline. We suggest to use gotests to generate the test code, and we will modify the template of gotests to generate the following test code.

    The example implementation is here. https://github.com/vdaas/vald/tree/proposal/tls_test/internal/tls

    Please note that:

    1. The args struct will only be generated if target function accept arguments.
    2. The fields struct will only be generated if target struct have fields.
    3. For the option.go template, the comments will be generated with the source code, user need to decide which source code to use.
    func Test_person_Hoge4(t *testing.T) {
    	type args struct { 
    		err error
    	}
    	type fields struct {
    		name string
    	}
    	type want struct {
    		want  string
    		want1 string
    		err   error
    	}
    	type test struct {
    		name       string
    		args       args
    		fields     fields
    		want       want
    		checkFunc  func(want, string, string, error) error
    		beforeFunc func(args)
    		afterFunc  func(args)
    	}
    	defaultCheckFunc := func(w want, got string, got1 string, err error) error {
    		if !errors.Is(err, w.err) {
    			return errors.Errorf("got error = %v, want %v", err, w.err)
    		}
    		if !reflect.DeepEqual(got, w.want) {
    			return errors.Errorf("got = %v, want %v", got, w.want)
    		}
    		if !reflect.DeepEqual(got1, w.want1) {
    			return errors.Errorf("got = %v, want %v", got1, w.want1)
    		}
    		return nil
    	}
    	tests := []test{
    		// TODO test cases
    		{
    			name:      "test_case_1",
    			args:      args{},
    			fields:    fields{},
    			want:      want{},
    			checkFunc: defaultCheckFunc,
    		},
                     // TODO test cases
    		func() test {
    			return test {
    				name:      "test_case_2",
    				args:      args{},
    				fields:    fields{},
    				want:      want{},
    				checkFunc: defaultCheckFunc,
    			}
    		}(),
    	}
    
    	for _, tt := range tests {
    		t.Run(tt.name, func(t *testing.T) {
    			if tt.beforeFunc != nil {
    				tt.beforeFunc(tt.args)
    			}
    			if tt.afterFunc != nil {
    				defer tt.afterFunc(tt.args)
    			}
    			if tt.checkFunc == nil {
    				tt.checkFunc = defaultCheckFunc
    			}
    			p := &person{
    				name: tt.fields.name,
    			}
    
    			got, got1, err := p.Hoge4(tt.args.err)
    			if err := tt.checkFunc(tt.want, got, got1, err); err != nil {
    				t.Errorf("error = %v", err)
    			}
    
    		})
    	}
    }
    

    Option.go template:

    func TestWithAddr(t *testing.T) {
    	type T = interface{}
    	type args struct {
    		addr string
    		a    int
    		b    int
    		c    int
    		d    int
    	}
    	type want struct {
    		obj *T
    		// Uncomment this line if the option returns an error, otherwise delete it
    		// err error
    	}
    	type test struct {
    		name string
    		args args
    		want want
    		// Use the first line if the option returns an error. otherwise use the second line
    		// checkFunc  func(want, *T, error) error
    		// checkFunc  func(want, *T) error
    		beforeFunc func(args)
    		afterFunc  func(args)
    	}
    
    	// Uncomment this block if the option returns an error, otherwise delete it
    	/*
    	   defaultCheckFunc := func(w want, obj *T, err error) error {
    	       if !errors.Is(err, w.err) {
    	           return errors.Errorf("got error = %v, want %v", err, w.err)
    	       }
    	       if !reflect.DeepEqual(obj, w.obj) {
    	           return errors.Errorf("got = %v, want %v", obj, w.obj)
    	       }
    	       return nil
    	   }
    	*/
    
    	// Uncomment this block if the option do not returns an error, otherwise delete it
    	/*
    	   defaultCheckFunc := func(w want, obj *T) error {
    	       if !reflect.DeepEqual(obj, w.obj) {
    	           return fmt.Errorf("got = %v, want %v", obj, w.c)
    	       }
    	       return nil
    	   }
    	*/
    
    	tests := []test{
                     // TODO test cases
    		{
    			name:      "test_case_1",
    			args:      args{},
    			want:      want{},
    			// checkFunc: defaultCheckFunc,
    		},
                     // TODO test cases
    		func() test {
    			return test {
    				name:      "test_case_2",
    				args:      args{},
    				fields:    fields{},
    				want:      want{},
    				checkFunc: defaultCheckFunc,
    			}
    		}(),
    	}
    
    	for _, tt := range tests {
    		t.Run(tt.name, func(t *testing.T) {
    			if tt.beforeFunc != nil {
    				tt.beforeFunc(tt.args)
    			}
    			if tt.afterFunc != nil {
    				defer tt.afterFunc(tt.args)
    			}
    
    			// Uncomment this block if the option returns an error, otherwise delete it
    			/*
    			   if tt.checkFunc == nil {
    			       tt.checkFunc = defaultCheckFunc
    			   }
    
    			   got := WithAddr(tt.args.addr, tt.args.a, tt.args.b, tt.args.c, tt.args.d)
    			   obj := new(T)
    			   if err := tt.checkFunc(tt.want, obj, got(obj)); err != nil {
    			       t.Errorf("error = %v", err)
    			   }
    			*/
    
    			// Uncomment this block if the option do not returns an error, otherwise delete it
    			/*
    			   if tt.checkFunc == nil {
    			       tt.checkFunc = defaultCheckFunc
    			   }
    
    			   got := WithAddr(tt.args.addr, tt.args.a, tt.args.b, tt.args.c, tt.args.d)
    			   obj := new(T)
    			   got(obj)
    			   if err := tt.checkFunc(tt.want, obj); err != nil {
    			       t.Errorf("error = %v", err)
    			   }
    			*/
    		})
    	}
    }
    
    type/test status/proposal 
    opened by kevindiu 53
  • WIP [patch] divide gateway

    WIP [patch] divide gateway

    Signed-off-by: kpango [email protected]

    WIP

    please do not review for now

    Description

    This PR includes a lot of feature.

    Related Issue:

    How Has This Been Tested?:

    Environment:

    • Golang Version: 1.14.3
    • Docker Version: 19.03.8
    • Kubernetes Version: 1.18.2
    • NGT Version: 1.11.5

    Types of changes:

    • [ ] Bug fix [type/bug]
    • [x] New feature [type/feature]
    • [ ] Add tests [type/test]
    • [ ] Security related changes [type/security]
    • [ ] Add documents [type/documentation]
    • [x] Refactoring [type/refactoring]
    • [ ] Update dependencies [type/dependency]
    • [ ] Update benchmarks and performances [type/bench]
    • [ ] Update CI [type/ci]

    Changes to Core Features:

    • [x] Have you added an explanation of what your changes do and why you'd like us to include them?
    • [ ] Have you written new tests for your core changes, as applicable?
    • [ ] Have you successfully ran tests with your changes locally?

    Checklist:

    • [x] I have read the CONTRIBUTING document.
    • [x] I have checked open Pull Requests for the similar feature or fixes?
    • [ ] I have added tests and benchmarks to cover my changes.
    • [ ] I have ensured all new and existing tests passed.
    • [ ] I have commented my code, particularly in hard-to-understand areas
    • [ ] I have updated the documentation accordingly.
    team/sre type/feature priority/medium area/filter/ingress type/ci size/XXXL area/helm area/makefile area/agent/core area/tools/cli/loadtest 
    opened by kpango 51
  • ♻️ Refactoring compressor

    ♻️ Refactoring compressor

    Signed-off-by: Rintaro Okamura [email protected]

    Description:

    Refactoring manager-compressor.

    • extract worker loop
    • add Registerer service that processes requested meta vector and registers it asynchronously
    • in the PreStop phase, the terminated compressor sends meta_vectors in its queue to other compressor.

    Related Issue:

    Nothing.

    How Has This Been Tested?:

    local KinD.

    Environment:

    • Golang Version: 1.14
    • Docker Version: 19.03.5
    • Kubernetes Version: 1.17.3
    • NGT Version: 1.9.1

    Types of changes:

    • [ ] Bug fix [type/bug]
    • [ ] New feature [type/feature]
    • [ ] Add tests [type/test]
    • [ ] Security related changes [type/security]
    • [ ] Add documents [type/documentation]
    • [X] Refactoring [type/refactoring]
    • [ ] Update dependencies [type/dependency]
    • [ ] Update benchmarks and performances [type/bench]
    • [ ] Update CI [type/ci]

    Changes to Core Features:

    • [ ] Have you added an explanation of what your changes do and why you'd like us to include them?
    • [ ] Have you written new tests for your core changes, as applicable?
    • [ ] Have you successfully ran tests with your changes locally?

    Checklist:

    • [X] I have read the CONTRIBUTING document.
    • [X] I have checked open Pull Requests for the similar feature or fixes?
    • [ ] I have added tests and benchmarks to cover my changes.
    • [ ] I have ensure all new and existing tests passed.
    • [ ] I have commented my code, particularly in hard-to-understand areas
    • [ ] I have updated the documentation accordingly.
    team/sre priority/low type/refactoring status/in-progress area/compressor area/internal type/ci size/XXL area/helm 
    opened by rinx 51
  • Add gRPC interceptor and exporters for Prometheus and Jaeger

    Add gRPC interceptor and exporters for Prometheus and Jaeger

    Signed-off-by: Rintaro Okamura [email protected]

    Description:

    Introducing Prometheus and Jaeger exporter using OpenCensus-go.

    • gRPC interceptor and trace spans are implemented to each components.
    • deployments of Prometheus, Jaeger, and Grafana (for development use)
    • two default Grafana dashboards are provided.
      • Vald Cluster Overview
      • Vald Agent

    Vald Cluster Overview dashboard vald-cluster-overview

    Vald Agent dashboard vald-agent

    Related Issue:

    Nothing.

    How Has This Been Tested?:

    In my local KinD cluster. By running

    $ make k8s/metrics/prometheus/deploy
    $ make k8s/metrics/jaeger/deploy
    $ helm install --values vald/values.yaml \
    --set defaults.observability.jaeger.enabled=true \
    --set gateway.minReplicas=1 \
    --set gateway.hpa.enabled=false \
    --set gateway.gateway_config.index_replica=3 \
    --set gateway.image.tag=pr-131 \
    --set agent.minReplicas=6 \
    --set agent.hpa.enabled=false \
    --set agent.image.tag=pr-131 \
    --set agent.ngt.dimension=6 \
    --set discoverer.image.tag=pr-131 \
    --set compressor.minReplicas=1 \
    --set compressor.hpa.enabled=false \
    --set compressor.image.tag=pr-131 \
    --set backupManager.minReplicas=1 \
    --set backupManager.hpa.enabled=false \
    --set backupManager.image.tag=pr-131 \
    --set meta.minReplicas=1 \
    --set meta.hpa.enabled=false \
    --set meta.image.tag=pr-131 \
    --generate-name vald
    

    UI ports

    • Grafana: 3000
    • Prometheus: 9090
    • Jaeger: 16686

    Environment:

    • Golang Version: 1.13.8
    • Docker Version: 19.03.5
    • Kubernetes Version: 1.16.3
    • NGT Version: 1.8.4

    Types of changes:

    • [ ] Bug fix [type/bug]
    • [X] New feature [type/feature]
    • [ ] Add tests [type/test]
    • [ ] Security related changes [type/security]
    • [ ] Add documents [type/documentation]
    • [ ] Refactoring [type/refactoring]
    • [ ] Update dependencies [type/dependency]
    • [ ] Update benchmarks and performances [type/bench]
    • [ ] Update CI [type/ci]

    Changes to Core Features:

    • [ ] Have you added an explanation of what your changes do and why you'd like us to include them?
    • [ ] Have you written new tests for your core changes, as applicable?
    • [ ] Have you successfully ran tests with your changes locally?

    Checklist:

    • [X] I have read the CONTRIBUTING document.
    • [X] I have checked open Pull Requests for the similar feature or fixes?
    • [ ] I have added tests and benchmarks to cover my changes.
    • [ ] I have ensure all new and existing tests passed.
    • [ ] I have commented my code, particularly in hard-to-understand areas
    • [ ] I have updated the documentation accordingly.
    team/sre type/feature priority/low status/in-review area/agent area/gateway area/discoverer area/meta area/manager/backup area/manager/index area/manager/replication area/internal size/XXXL 
    opened by rinx 50
  • bugfix discoverer & add discovery cache & bugfix agent index count & add some feature

    bugfix discoverer & add discovery cache & bugfix agent index count & add some feature

    Signed-off-by: kpango [email protected]

    Description:

    in this PR includes 2 bugfixes and 6 features and 3 prototypes

    bugfix
    • discoverer returns the same pointer of grpc payload it makes problem so I fixed about that
    • agent reports index count using non-atomic counter it makes the wrong value, so I fixed that using atomic variable
    • agent crashes when observability is nil
    feature
    • discoverer now caches each response
    • internal Cassandra now supports tls authentication and completely support DC aware routing
    • grpc client now supports rebalanced reconnecting
    • grpc client now supports DNS optimized balanced connection pooling
    • internal/net package now overrides official net package
    • index manager now supports custom creation pool size
    prototype
    • replication manager controller
    • replication manager agent
    • internal compressor client

    Related Issue:

    How Has This Been Tested?:

    Environment:

    • Golang Version: 1.14
    • Docker Version: 19.03.6-rc2
    • Kubernetes Version: 1.17.4
    • NGT Version: 1.9.1
    • [ ] Bug fix [type/bug]
    • [x] New feature [type/feature]
    • [ ] Add tests [type/test]
    • [ ] Security related changes [type/security]
    • [ ] Add documents [type/documentation]
    • [ ] Refactoring [type/refactoring]
    • [ ] Update dependencies [type/dependency]
    • [ ] Update benchmarks and performances [type/bench]
    • [ ] Update CI [type/ci]

    Changes to Core Features:

    • [ ] Have you added an explanation of what your changes do and why you'd like us to include them?
    • [ ] Have you written new tests for your core changes, as applicable?
    • [ ] Have you successfully ran tests with your changes locally?

    Checklist:

    • [x] I have read the CONTRIBUTING document.
    • [x] I have checked open Pull Requests for the similar feature or fixes?
    • [ ] I have added tests and benchmarks to cover my changes.
    • [ ] I have ensured all new and existing tests passed.
    • [ ] I have commented my code, particularly in hard-to-understand areas
    • [ ] I have updated the documentation accordingly.
    team/set team/sre type/feature priority/medium area/discoverer area/manager/replication area/internal type/ci size/XXL area/helm area/makefile 
    opened by kpango 49
  • ✨[patch] Implementation of agent-sidecar storage backup logic

    ✨[patch] Implementation of agent-sidecar storage backup logic

    Description:

    In this PR, a blob storage backup logic of agent-sidecar is implemented. Currently, it only supports S3.

    Bugfix:

    • internal/core/ngt/ngt.go ... remove useless C.frees

    Related Issue:

    Nothing.

    How Has This Been Tested?:

    Nothing.

    Environment:

    • Golang Version: 1.14.3
    • Docker Version: 19.03.8
    • Kubernetes Version: 1.18.2
    • NGT Version: 1.11.5

    Types of changes:

    • [ ] Bug fix [type/bug]
    • [X] New feature [type/feature]
    • [ ] Add tests [type/test]
    • [ ] Security related changes [type/security]
    • [ ] Add documents [type/documentation]
    • [ ] Refactoring [type/refactoring]
    • [ ] Update dependencies [type/dependency]
    • [ ] Update benchmarks and performances [type/bench]
    • [ ] Update CI [type/ci]

    Changes to Core Features:

    • [ ] Have you added an explanation of what your changes do and why you'd like us to include them?
    • [ ] Have you written new tests for your core changes, as applicable?
    • [ ] Have you successfully ran tests with your changes locally?

    Checklist:

    • [X] I have read the CONTRIBUTING document.
    • [X] I have checked open Pull Requests for the similar feature or fixes?
    • [ ] I have added tests and benchmarks to cover my changes.
    • [ ] I have ensured all new and existing tests passed.
    • [ ] I have commented my code, particularly in hard-to-understand areas
    • [ ] I have updated the documentation accordingly.
    team/set team/sre type/feature priority/medium area/internal type/ci size/XXXL area/helm area/makefile 
    opened by rinx 48
  • [WIP] add replication manager and replicator

    [WIP] add replication manager and replicator

    Signed-off-by: kpango [email protected]

    Description:

    create replication manager

    Related Issue:

    How Has This Been Tested?:

    Environment:

    • Golang Version: 1.13.5
    • Docker Version: 19.03.5
    • Kubernetes Version: 1.16.3
    • NGT Version: 1.8.4

    Types of changes:

    • [ ] Bug fix [type/bug]
    • [x] New feature [type/feature]
    • [ ] Add tests [type/test]
    • [ ] Security related changes [type/security]
    • [ ] Add documents [type/documentation]
    • [ ] Refactoring [type/refactoring]
    • [ ] Update dependencies [type/dependency]
    • [ ] Update benchmarks and performances [type/bench]
    • [ ] Update CI [type/ci]

    Changes to Core Features:

    • [ ] Have you added an explanation of what your changes do and why you'd like us to include them?
    • [ ] Have you written new tests for your core changes, as applicable?
    • [ ] Have you successfully ran tests with your changes locally?

    Checklist:

    • [x] I have read the CONTRIBUTING document.
    • [x] I have checked open Pull Requests for the similar feature or fixes?
    • [ ] I have added tests and benchmarks to cover my changes.
    • [ ] I have ensure all new and existing tests passed.
    • [ ] I have commented my code, particularly in hard-to-understand areas
    • [ ] I have updated the documentation accordingly.
    team/set team/sre type/feature priority/medium area/agent area/gateway area/manager/index area/manager/replication area/internal type/ci size/XXL area/makefile 
    opened by kpango 47
  • Vald architecture document

    Vald architecture document

    #106 # Description: This PR create a Vald Architecture document including the summary of each components in Vald and the data flow explanation.

    Remaining task:

    • Update and Delete flow (another PR)
    • [ ] Fix image design

    Related Issue:

    How Has This Been Tested?:

    Environment:

    • Golang Version: 1.14
    • Docker Version: 19.03.5
    • Kubernetes Version: 1.17.3
    • NGT Version: 1.9.1

    Types of changes:

    • [ ] Bug fix [type/bug]
    • [ ] New feature [type/feature]
    • [ ] Add tests [type/test]
    • [ ] Security related changes [type/security]
    • [x] Add documents [type/documentation]
    • [ ] Refactoring [type/refactoring]
    • [ ] Update dependencies [type/dependency]
    • [ ] Update benchmarks and performances [type/bench]
    • [ ] Update CI [type/ci]

    Changes to Core Features:

    • [x] Have you added an explanation of what your changes do and why you'd like us to include them?
    • [x] Have you written new tests for your core changes, as applicable?
    • [x] Have you successfully ran tests with your changes locally?

    Checklist:

    • [x] I have read the CONTRIBUTING document.
    • [x] I have checked open Pull Requests for the similar feature or fixes?
    • [x] I have added tests and benchmarks to cover my changes.
    • [x] I have ensured all new and existing tests passed.
    • [x] I have commented my code, particularly in hard-to-understand areas
    • [x] I have updated the documentation accordingly.
    team/sre priority/low type/documentation size/L type/ci status/author-review area/helm area/makefile 
    opened by kevindiu 46
  • Add capacity plannnig document

    Add capacity plannnig document

    Description:

    SSIA

    Related Issue:

    How Has This Been Tested?:

    Environment:

    • Go Version: 1.18.3
    • Docker Version: 20.10.8
    • Kubernetes Version: 1.22.0
    • NGT Version: 1.14.6

    Types of changes:

    • [ ] Bug fix [type/bug]
    • [ ] New feature [type/feature]
    • [ ] Add tests [type/test]
    • [ ] Security related changes [type/security]
    • [x] Add documents [type/documentation]
    • [ ] Refactoring [type/refactoring]
    • [ ] Update dependencies [type/dependency]
    • [ ] Update benchmarks and performances [type/bench]
    • [ ] Update CI [type/ci]

    Changes to Core Features:

    • [ ] Have you added an explanation of what your changes do and why you'd like us to include them?
    • [ ] Have you written new tests for your core changes, as applicable?
    • [ ] Have you successfully ran tests with your changes locally?

    Checklist:

    • [ ] I have read the CONTRIBUTING document.
    • [ ] I have checked open Pull Requests for the similar feature or fixes?
    • [ ] I have added tests and benchmarks to cover my changes.
    • [ ] I have ensured all new and existing tests passed.
    • [ ] I have commented my code, particularly in hard-to-understand areas
    • [ ] I have updated the documentation accordingly.
    priority/low type/documentation size/L 
    opened by vankichi 1
  • [WIP] Implement agent handler createIndex test case

    [WIP] Implement agent handler createIndex test case

    Signed-off-by: kevindiu [email protected]

    Description:

    Related Issue:

    How Has This Been Tested?:

    Environment:

    • Go Version: 1.18.3
    • Docker Version: 20.10.8
    • Kubernetes Version: 1.22.0
    • NGT Version: 1.14.6

    Types of changes:

    • [ ] Bug fix [type/bug]
    • [ ] New feature [type/feature]
    • [ ] Add tests [type/test]
    • [ ] Security related changes [type/security]
    • [ ] Add documents [type/documentation]
    • [ ] Refactoring [type/refactoring]
    • [ ] Update dependencies [type/dependency]
    • [ ] Update benchmarks and performances [type/bench]
    • [ ] Update CI [type/ci]

    Changes to Core Features:

    • [ ] Have you added an explanation of what your changes do and why you'd like us to include them?
    • [ ] Have you written new tests for your core changes, as applicable?
    • [ ] Have you successfully ran tests with your changes locally?

    Checklist:

    • [ ] I have read the CONTRIBUTING document.
    • [ ] I have checked open Pull Requests for the similar feature or fixes?
    • [ ] I have added tests and benchmarks to cover my changes.
    • [ ] I have ensured all new and existing tests passed.
    • [ ] I have commented my code, particularly in hard-to-understand areas
    • [ ] I have updated the documentation accordingly.
    team/set type/test priority/low size/S area/agent/core 
    opened by kevindiu 2
  • [WIP] Implement agent handler indexInfo test case

    [WIP] Implement agent handler indexInfo test case

    Signed-off-by: kevindiu [email protected]

    Description:

    Related Issue:

    How Has This Been Tested?:

    Environment:

    • Go Version: 1.18.3
    • Docker Version: 20.10.8
    • Kubernetes Version: 1.22.0
    • NGT Version: 1.14.6

    Types of changes:

    • [ ] Bug fix [type/bug]
    • [ ] New feature [type/feature]
    • [ ] Add tests [type/test]
    • [ ] Security related changes [type/security]
    • [ ] Add documents [type/documentation]
    • [ ] Refactoring [type/refactoring]
    • [ ] Update dependencies [type/dependency]
    • [ ] Update benchmarks and performances [type/bench]
    • [ ] Update CI [type/ci]

    Changes to Core Features:

    • [ ] Have you added an explanation of what your changes do and why you'd like us to include them?
    • [ ] Have you written new tests for your core changes, as applicable?
    • [ ] Have you successfully ran tests with your changes locally?

    Checklist:

    • [ ] I have read the CONTRIBUTING document.
    • [ ] I have checked open Pull Requests for the similar feature or fixes?
    • [ ] I have added tests and benchmarks to cover my changes.
    • [ ] I have ensured all new and existing tests passed.
    • [ ] I have commented my code, particularly in hard-to-understand areas
    • [ ] I have updated the documentation accordingly.
    team/set type/test priority/low size/S area/agent/core 
    opened by kevindiu 3
  • Implement agent handler getObject test case

    Implement agent handler getObject test case

    Signed-off-by: kevindiu [email protected]

    Description:

    Same as title

    Related Issue:

    How Has This Been Tested?:

    Environment:

    • Go Version: 1.18.3
    • Docker Version: 20.10.8
    • Kubernetes Version: 1.22.0
    • NGT Version: 1.14.6

    Types of changes:

    • [ ] Bug fix [type/bug]
    • [ ] New feature [type/feature]
    • [x] Add tests [type/test]
    • [ ] Security related changes [type/security]
    • [ ] Add documents [type/documentation]
    • [ ] Refactoring [type/refactoring]
    • [ ] Update dependencies [type/dependency]
    • [ ] Update benchmarks and performances [type/bench]
    • [ ] Update CI [type/ci]

    Changes to Core Features:

    • [x] Have you added an explanation of what your changes do and why you'd like us to include them?
    • [x] Have you written new tests for your core changes, as applicable?
    • [x] Have you successfully ran tests with your changes locally?

    Checklist:

    • [x] I have read the CONTRIBUTING document.
    • [x] I have checked open Pull Requests for the similar feature or fixes?
    • [x] I have added tests and benchmarks to cover my changes.
    • [x] I have ensured all new and existing tests passed.
    • [x] I have commented my code, particularly in hard-to-understand areas
    • [ ] I have updated the documentation accordingly.
    team/set type/test priority/low size/XL area/agent/core 
    opened by kevindiu 3
  • [WIP] add values.go implementation

    [WIP] add values.go implementation

    Signed-off-by: kpango [email protected]

    Description:

    Related Issue:

    How Has This Been Tested?:

    Environment:

    • Go Version: 1.18.2
    • Docker Version: 20.10.8
    • Kubernetes Version: 1.22.0
    • NGT Version: 1.14.5

    Types of changes:

    • [ ] Bug fix [type/bug]
    • [ ] New feature [type/feature]
    • [ ] Add tests [type/test]
    • [ ] Security related changes [type/security]
    • [ ] Add documents [type/documentation]
    • [ ] Refactoring [type/refactoring]
    • [ ] Update dependencies [type/dependency]
    • [ ] Update benchmarks and performances [type/bench]
    • [ ] Update CI [type/ci]

    Changes to Core Features:

    • [ ] Have you added an explanation of what your changes do and why you'd like us to include them?
    • [ ] Have you written new tests for your core changes, as applicable?
    • [ ] Have you successfully ran tests with your changes locally?

    Checklist:

    • [ ] I have read the CONTRIBUTING document.
    • [ ] I have checked open Pull Requests for the similar feature or fixes?
    • [ ] I have added tests and benchmarks to cover my changes.
    • [ ] I have ensured all new and existing tests passed.
    • [ ] I have commented my code, particularly in hard-to-understand areas
    • [ ] I have updated the documentation accordingly.
    team/set team/sre type/feature priority/medium area/manager/index area/internal size/XXL area/helm area/makefile area/agent/core area/agent/sidecar area/gateway/lb area/gateway/filter 
    opened by kpango 3
  • [WIP] Add circuit breaker

    [WIP] Add circuit breaker

    Description:

    WIP

    Related Issue:

    How Has This Been Tested?:

    Environment:

    • Go Version: 1.18.2
    • Docker Version: 20.10.8
    • Kubernetes Version: 1.22.0
    • NGT Version: 1.14.3

    Types of changes:

    • [ ] Bug fix [type/bug]
    • [x] New feature [type/feature]
    • [ ] Add tests [type/test]
    • [ ] Security related changes [type/security]
    • [ ] Add documents [type/documentation]
    • [ ] Refactoring [type/refactoring]
    • [ ] Update dependencies [type/dependency]
    • [ ] Update benchmarks and performances [type/bench]
    • [ ] Update CI [type/ci]

    Changes to Core Features:

    • [ ] Have you added an explanation of what your changes do and why you'd like us to include them?
    • [ ] Have you written new tests for your core changes, as applicable?
    • [ ] Have you successfully ran tests with your changes locally?

    Checklist:

    • [x] I have read the CONTRIBUTING document.
    • [x] I have checked open Pull Requests for the similar feature or fixes?
    • [x] I have added tests and benchmarks to cover my changes.
    • [ ] I have ensured all new and existing tests passed.
    • [ ] I have commented my code, particularly in hard-to-understand areas
    • [ ] I have updated the documentation accordingly.
    type/feature priority/medium size/L area/internal area/helm 
    opened by hlts2 3
Releases(v1.5.4)
Owner
Vector Data as a Service
Everything becomes Vector
Vector Data as a Service
txtai: AI-powered search engine for Go

txtai builds an AI-powered index over sections of text. txtai supports building text indices to perform similarity searches and create extractive question-answering based systems. txtai also has functionality for zero-shot classification.

NeuML 40 Apr 5, 2022
A highly flexible blockchain architecture with great transaction performance.

XuperChain δΈ­ζ–‡θ―΄ζ˜Ž What is XuperChain XuperChain, the first open source project of XuperChain Lab, introduces a underlying solution to build the super al

null 1.6k Jun 29, 2022
Recommendation engine for Go

regommend Recommendation engine for Go Installation Make sure you have a working Go environment (Go 1.2 or higher is required). See the install instru

Christian Muehlhaeuser 299 Jun 18, 2022
Casbin Neo (neo for new engine option)

A Casbin-compatible engine Casbin NEO Casbin NEO(neo for new engine option), A Casbin-compatible engine. In this project, we would go to restructure t

Casbin Mesh 5 May 22, 2022
Distributed hyperparameter optimization framework, inspired by Optuna.

Goptuna Distributed hyperparameter optimization framework, inspired by Optuna [1]. This library is particularly designed for machine learning, but eve

Masashi SHIBATA 214 Jun 28, 2022
PaddleDTX is a solution that focused on distributed machine learning technology based on decentralized storage.

δΈ­ζ–‡ | English PaddleDTX PaddleDTX is a solution that focused on distributed machine learning technology based on decentralized storage. It solves the d

null 58 Jun 23, 2022
Cadence is a distributed, scalable, durable, and highly available orchestration engine to execute asynchronous long-running business logic in a scalable and resilient way.

Cadence Visit cadenceworkflow.io to learn about Cadence. This repo contains the source code of the Cadence server. To implement workflows, activities

Uber Open Source 6.1k Jun 29, 2022
Weaviate is a cloud-native, modular, real-time vector search engine

Weaviate is a cloud-native, real-time vector search engine (aka neural search engine or deep search engine). There are modules for specific use cases such as semantic search, plugins to integrate Weaviate in any application of your choice, and a console to visualize your data.

SeMI Technologies 2.5k Jun 24, 2022
Weaviate is a cloud-native, modular, real-time vector search engine

Weaviate is a cloud-native, real-time vector search engine (aka neural search engine or deep search engine). There are modules for specific use cases such as semantic search, plugins to integrate Weaviate in any application of your choice, and a console to visualize your data.

SeMI Technologies 2.5k Jun 24, 2022
An open source embedding vector similarity search engine powered by Faiss, NMSLIB and Annoy

Click to take a quick look at our demos! Image search Chatbots Chemical structure search Milvus is an open-source vector database built to power AI ap

The Milvus Project 11.1k Jun 26, 2022
BlobStore is a highly reliable,highly available and ultra-large scale distributed storage system

BlobStore Overview Documents Build BlobStore Deploy BlobStore Manage BlobStore License Overview BlobStore is a highly reliable,highly available and ul

CubeFS 14 Apr 28, 2022
Consul is a distributed, highly available, and data center aware solution to connect and configure applications across dynamic, distributed infrastructure.

Consul Website: https://www.consul.io Tutorials: HashiCorp Learn Forum: Discuss Consul is a distributed, highly available, and data center aware solut

HashiCorp 24.9k Jun 22, 2022
A dead simple, highly performant, highly customizable sessions middleware for go http servers.

If you're interested in jwt's, see my jwt library! Sessions A dead simple, highly performant, highly customizable sessions service for go http servers

Adam Hanna 65 May 4, 2022
Scalable Distributed Game Server Engine with Hot Swapping in Golang

GoWorld Scalable Distributed Game Server Engine with Hot Reload in Golang Features Architecture Introduction Get GoWorld Manage GoWorld Servers Demos

Nan Lin 2.1k Jun 28, 2022
A powerful go web framework for highly scalable and resource efficient web application

webfr A powerful go web framework for highly scalable and resource efficient web application Installation: go get -u github.com/krishpranav/webfr Exa

Krisna Pranav 13 Nov 28, 2021
A powerful go web framework for highly scalable and resource efficient web application

A powerful go web framework for highly scalable and resource efficient web application

null 21 May 25, 2022
A highly-scalable, entity-resolution technology that was originally developed to connect internal data together

TiloRes CLI What is TiloRes? TiloRes is a highly-scalable, β€œentity-resolution” technology that was originally developed to connect internal data toget

Tilo Tech GmbH 4 Jun 17, 2022
Go-Web-Dev - Golang helps the developer to develop highly scalable applications

Go-Web-Dev Golang helps the developer to develop highly scalable applications. T

Mohamed Dhik 0 Feb 5, 2022
Grafana Mimir provides horizontally scalable, highly available, multi-tenant, long-term storage for Prometheus.

Grafana Mimir Grafana Mimir is an open source software project that provides a scalable long-term storage for Prometheus. Some of the core strengths o

Grafana Labs 2.2k Jul 1, 2022
Go Open Source, Distributed, Simple and efficient Search Engine

Go Open Source, Distributed, Simple and efficient full text search engine.

ego 6.1k Jun 25, 2022
Flash - A highly optimised typeahead(search recommendation) server

flash Flash is highly optimised typeahead(search recommendation) server. We are

Shubham Mishra 6 Apr 4, 2022
Search and store the best cryptos for the best scalable and modern application development.

Invst Hunt Search and store the best cryptos for the best scalable and modern application development. Layout Creating... Project Challenge The Techni

FΓ‘bio Morais 1 Nov 12, 2021
Sabre is highly customisable, embeddable LISP engine for Go. :computer:

Sabre DEPRECATED: This repository is deprecated in favour much better slurp project and will be archived/removed soon. Sabre is highly customizable, e

Shivaprasad Bhat 28 May 23, 2021
Distributed-Services - Distributed Systems with Golang to consequently build a fully-fletched distributed service

Distributed-Services This project is essentially a result of my attempt to under

Hamza Yusuff 6 Jun 1, 2022
TalariaDB is a distributed, highly available, and low latency time-series database for Presto

TalariaDB is a distributed, highly available, and low latency time-series database that stores real-time data. It's built on top of Badger DB.

Grab 97 Jun 18, 2022
Redwood is a highly-configurable, distributed, realtime database that manages a state tree shared among many peers

Redwood is a highly-configurable, distributed, realtime database that manages a state tree shared among many peers. Imagine something like a Redux store, but distributed across all users of an application, that offers offline editing and is resilient to poor connectivity.

Redwood 634 Jun 28, 2022
Easy to use Raft library to make your app distributed, highly available and fault-tolerant

An easy to use customizable library to make your Go application Distributed, Highly available, Fault Tolerant etc... using Hashicorp's Raft library wh

Richard Bertok 58 May 29, 2022
Publish Your GIS Data(Vector Data) to PostGIS and Geoserver

GISManager Publish Your GIS Data(Vector Data) to PostGIS and Geoserver How to install: go get -v github.com/hishamkaram/gismanager Usage: testdata fol

Hisham waleed karam 45 Jun 16, 2022
Rasterx is an SVG 2.0 path compliant rasterizer that can use either the golang vector or a derivative of the freetype anti-aliaser.

rasterx Rasterx is a golang rasterizer that implements path stroking functions capable of SVG 2.0 compliant 'arc' joins and explicit loop closing. Pat

Steven R Wiley 97 Jun 11, 2022