Versioned model registry suitable for temporary in-training storage and permanent storage

Overview

Cogment Model Registry

Latest Release Apache 2 License Changelog

Cogment is an innovative open source AI platform designed to leverage the advent of AI to benefit humankind through human-AI collaboration developed by AI Redefined. Cogment enables AI researchers and engineers to build, train and operate AI agents in simulated or real environments shared with humans. For the full user documentation visit https://docs.cogment.ai

This module, cogment Model Registry, is a versioned key value store dedicated to the storage of models.

Usage

$ docker run -p 9000:9000 -v $(pwd)/relative/path/to/model/archive:/data cogment/model-registry

Configuration

The following environment variables can be used to configure the server:

  • COGMENT_MODEL_REGISTRY_PORT: The port to listen on. Defaults to 9000.
  • COGMENT_MODEL_REGISTRY_ARCHIVE_DIR: The directory to store model archives. Docker images defaults to /data.
  • COGMENT_MODEL_REGISTRY_SENT_MODEL_VERSION_DATA_CHUNK_SIZE: The size of the model version data chunk sent by the server. Defaults to 510241024 (5MB).
  • COGMENT_MODEL_REGISTRY_GRPC_REFLECTION: Set to start a gRPC reflection server. Defaults to false.

API

The Model Registry exposes a gRPC defined in the Model Registry API

Create or update a model - cogment.ModelRegistrySP/CreateOrUpdateModel( .cogment.CreateOrUpdateModelRequest ) returns ( .cogment.CreateOrUpdateModelReply );

This example requires COGMENT_MODEL_REGISTRY_GRPC_REFLECTION to be enabled and requires grpcurl

$ echo "{\"model_info\":{\"model_id\":\"my_model\",\"user_data\":{\"type\":\"my_model_type\"}}}" | grpcurl -plaintext -d @ localhost:9000 cogment.ModelRegistrySP/CreateOrUpdateModel
{

}

Delete a model - cogment.ModelRegistrySP/DeleteModel( .cogment.DeleteModelRequest ) returns ( .cogment.DeleteModelReply );

This example requires COGMENT_MODEL_REGISTRY_GRPC_REFLECTION to be enabled and requires grpcurl

$ echo "{\"model_id\":\"my_model\"}" | grpcurl -plaintext -d @ localhost:9000 cogment.ModelRegistrySP/DeleteModel
{

}

Create a model version - cogment.ModelRegistrySP/CreateVersion( stream .cogment.CreateVersionRequestChunk ) returns ( .cogment.CreateVersionReply );

$ echo "{\"header\":{\"version_info\":{
    \"model_id\":\"my_model\",\
    \"archived\":true,\
    \"data_size\":$(printf chunk_1chunk_2 | wc -c)\
  }}}\
  {\"body\":{\
    \"data_chunk\":\"$(printf chunk_1 | base64)\"\
  }}\
  {\"body\":{\
    \"data_chunk\":\"$(printf chunk_2 | base64)\"\
  }}" | grpcurl -plaintext -d @ localhost:9000 cogment.ModelRegistrySP/CreateVersion
{
  "versionInfo": {
    "modelId": "my_model",
    "versionNumber": 2,
    "creationTimestamp": "907957639",
    "archived": true,
    "dataHash": "jY0g3VkUK62ILPr2JuaW5g7uQi0EcJVZJu8IYp3yfhI=",
    "dataSize": "14"
  }
}

Retrieve model versions infos - cogment.ModelRegistrySP/RetrieveVersionInfos ( .cogment.RetrieveVersionInfosRequest ) returns ( .cogment.RetrieveVersionInfosReply );

List the versions of a model

$ echo "{\"model_id\":\"my_model\"}" | grpcurl -plaintext -d @ localhost:9000 cogment.ModelRegistrySP/RetrieveVersionInfos
{
  "versionInfos": [
    {
      "modelId": "my_model",
      "versionNumber": 1,
      "creationTimestamp": "1633119005107454620",
      "archived": true,
      "dataHash": "jY0g3VkUK62ILPr2JuaW5g7uQi0EcJVZJu8IYp3yfhI=",
      "dataSize": "14"
    },
    {
      "modelId": "my_model",
      "versionNumber": 2,
      "creationTimestamp": "1633119625907957639",
      "archived": true,
      "dataHash": "jY0g3VkUK62ILPr2JuaW5g7uQi0EcJVZJu8IYp3yfhI=",
      "dataSize": "14"
    }
  ],
  "nextVersionHandle": "2"
}

Retrieve specific versions of a model

$ echo "{\"model_id\":\"my_model\", \"version_numbers\":[1]}" | grpcurl -plaintext -d @ localhost:9000 cogment.ModelRegistrySP/RetrieveVersionInfos
{
  "versionInfos": [
    {
      "modelId": "my_model",
      "versionNumber": 1,
      "creationTimestamp": "1633119005107454620",
      "archived": true,
      "dataHash": "jY0g3VkUK62ILPr2JuaW5g7uQi0EcJVZJu8IYp3yfhI=",
      "dataSize": "14"
    }
  ],
  "nextVersionHandle": "1"
}

Retrieve the latest version of a model

$ echo "{\"model_id\":\"my_model\", \"version_numbers\":[-1]}" | grpcurl -plaintext -d @ localhost:9000 cogment.ModelRegistrySP/RetrieveVersionInfos
{
  "versionInfos": [
    {
      "modelId": "my_model",
      "versionNumber": 2,
      "creationTimestamp": "1633119625907957639",
      "archived": true,
      "dataHash": "jY0g3VkUK62ILPr2JuaW5g7uQi0EcJVZJu8IYp3yfhI=",
      "dataSize": "14"
    }
  ],
  "nextVersionHandle": "1"
}

Retrieve given version data - cogment.ModelRegistrySP/RetrieveVersionData ( .cogment.RetrieveVersionDataRequest ) returns ( stream .cogment.RetrieveVersionDataReplyChunk );

$ echo "{\"model_id\":\"my_model\", \"version_number\":1}" | grpcurl -plaintext -d @ localhost:9000 cogment.ModelRegistrySP/RetrieveVersionData
{
  "dataChunk": "Y2h1bmtfMWNodW5rXzI="
}

To retrieve the latest version, use version_number:-1.

Developers

With a local Go installation

Linting is based on the "meta" linter golangci-lint it needs to be installed locally:

$ make lint

Testing:

$ make test

Testing with JUnit style reports (for the CI):

$ make test-with-report

Build a binary in build/cogment-model-registry:

$ make build

With Docker

Build image

$ ./scripts/build_docker.sh

Release process

People having maintainers rights of the repository can follow these steps to release a version MAJOR.MINOR.PATCH. The versioning scheme follows Semantic Versioning.

  1. Run ./scripts/create_release_branch.sh automatically compute and update the version of the package, create the release branch and update the changelog from the commit history,
  2. On the release branch, check and update the changelog if needed
  3. Update .cogment-api.yaml (in particular make sure it doesn't rely on the "latest" version)
  4. Make sure everything's fine on CI,
  5. Run ./scripts/tag_release.sh MAJOR.MINOR.PATCH to create the specific version section in the changelog, merge the release branch in main, create the release tag and update the develop branch with those.

The rest, publishing the packages to github releases and dockerhub and updating the mirror repositories, is handled directly by the CI.


Owner
Cogment
Cogment
Training materials and labs for a "Getting Started" level course on COBOL

COBOL Programming Course This project is a set of training materials and labs for COBOL on z/OS. The following books are available within this reposit

Open Mainframe Project 2.3k Aug 9, 2022
Spice.ai is an open source, portable runtime for training and using deep learning on time series data.

Spice.ai Spice.ai is an open source, portable runtime for training and using deep learning on time series data. ⚠️ DEVELOPER PREVIEW ONLY Spice.ai is

Spice.ai 741 Jul 30, 2022
A multilayer perceptron network implemented in Go, with training via backpropagation.

Neural Go I'm in the process of making significant changes to this package, particularly, to make it more modular, and to base it around an actual lin

Schuyler Erle 63 Mar 5, 2022
Go Training Class Material :

Go Training Review our different courses and material To learn about Corporate training events, options and special pricing please contact: William Ke

Ardan Labs 10.7k Aug 10, 2022
Incentivized AI Training Casino using ISCP for the Agri-D Hackaton!

Welcome to the Wasp repository! Wasp is a node software developed by the IOTA Foundation to run the IOTA Smart Contract Protocol (ISCP in short) on to

Zignar Technologies 1 May 15, 2022
SGPM: A coroutine scheduling model for wound-wait concurrency control optimization

Environment set up off the mod go env -w GO111MODULE=off change to GPATH to project directory go env -w GOPATH=$HOME/sgpm Usage This project serves th

Jack Wang 0 Dec 12, 2021
Bayesian text classifier with flexible tokenizers and storage backends for Go

Shield is a bayesian text classifier with flexible tokenizer and backend store support Currently implemented: Redis backend English tokenizer Example

Erik Aigner 152 Jun 30, 2022
PaddleDTX is a solution that focused on distributed machine learning technology based on decentralized storage.

中文 | English PaddleDTX PaddleDTX is a solution that focused on distributed machine learning technology based on decentralized storage. It solves the d

null 60 Jul 14, 2022
The open source, end-to-end computer vision platform. Label, build, train, tune, deploy and automate in a unified platform that runs on any cloud and on-premises.

End-to-end computer vision platform Label, build, train, tune, deploy and automate in a unified platform that runs on any cloud and on-premises. onepa

Onepanel, Inc. 611 Aug 9, 2022
Go types, funcs, and utilities for working with cards, decks, and evaluating poker hands (Holdem, Omaha, Stud, more)

cardrank.io/cardrank Package cardrank.io/cardrank provides a library of types, funcs, and utilities for working with playing cards, decks, and evaluat

null 56 Jul 30, 2022
Genetic Algorithm and Particle Swarm Optimization

evoli Genetic Algorithm and Particle Swarm Optimization written in Go Example Problem Given f(x,y) = cos(x^2 * y^2) * 1/(x^2 * y^2 + 1) Find (x,y) suc

Guillaume Simonneau 24 Jul 4, 2022
k-modes and k-prototypes clustering algorithms implementation in Go

go-cluster GO implementation of clustering algorithms: k-modes and k-prototypes. K-modes algorithm is very similar to well-known clustering algorithm

e-Xpert Solutions 31 Mar 14, 2022
Probability distributions and associated methods in Go

godist godist provides some Go implementations of useful continuous and discrete probability distributions, as well as some handy methods for working

Edd Robinson 33 Apr 1, 2022
On-line Machine Learning in Go (and so much more)

goml Golang Machine Learning, On The Wire goml is a machine learning library written entirely in Golang which lets the average developer include machi

Conner DiPaolo 1.4k Aug 1, 2022
A curated list of Awesome Go performance libraries and tools

Awesome Go performance Collection of the Awesome™ Go libraries, tools, project around performance. Contents Algorithm Assembly Benchmarks Compiling Co

Oleg Kovalov 272 Jul 28, 2022
Deploy, manage, and scale machine learning models in production

Deploy, manage, and scale machine learning models in production. Cortex is a cloud native model serving platform for machine learning engineering teams.

Cortex Labs 7.8k Aug 4, 2022
The Go kernel for Jupyter notebooks and nteract.

gophernotes - Use Go in Jupyter notebooks and nteract gophernotes is a Go kernel for Jupyter notebooks and nteract. It lets you use Go interactively i

GopherData 3.3k Aug 7, 2022
Library for multi-armed bandit selection strategies, including efficient deterministic implementations of Thompson sampling and epsilon-greedy.

Mab Multi-Armed Bandits Go Library Description Installation Usage Creating a bandit and selecting arms Numerical integration with numint Documentation

Stitch Fix Technology 30 Jul 2, 2022
A program that generates a folder structure with challenges and projects for mastering a programming language.

Challenge Generator A program that generates a folder structure with challenges and projects for mastering a programming language. Explore the docs »

João Freitas 69 Jul 20, 2022