💧 Visual Data Preparation (VDP) is an open-source tool to seamlessly integrate Vision AI with the modern data stack

Overview

Instill AI - Visual Data Preparation Made for All

Website | Community | Blog

Get Early Access


Visual Data Preparation (VDP) is an open-source tool to streamline the end-to-end visual data processing pipeline:

  1. Ingest unstructured visual data from data sources such as data lakes or IoT devices;
  2. Transform visual data to meaningful structured data representations by Vision AI models;
  3. Load the structured data into warehouses, applications, or other destinations.

The goal of VDP is to seamlessly bring Vision AI into the modern data stack with a standardised framework. Check our blog post Missing piece in modern data stack: visual data preparation on how this tool is proposed to streamline unstructured visual data processing across different stakeholders.

Table of contents

Code in the main branch tracks under-development progress towards the next release and may not work as expected. If you are looking for a stable alpha version, please use latest release.

How VDP works

The core concept of VDP is pipeline. A pipeline is an end-to-end workflow that automates a sequence of tasks to process visual data. Each pipeline consists of three ordered components:

  1. data source: where the pipeline starts. It connects the source of image and video data to be processed.
  2. model: a deployed Vision AI model to process the ingested visual data and generate structured outputs
  3. data destination: where to send the structured outputs

Based on the mode of a pipeline, it will ingest and process the visual data, send the outputs to the destination every time the trigger event occurs.

We use data connector as a general term to represent data source and data destination. Please find the supported data connectors here.

Quick start

Download and run VDP locally

Execute the following commands to start pre-built images with all the dependencies:

$ git clone https://github.com/instill-ai/vdp.git && cd vdp

# Build instill/vdp:dev local development image
$ make build

# Launch all services.
$ make all

âš ī¸ Downloading the Triton server image will take a while, but it should be just a one-time effort.

Run the samples to trigger an object detection pipeline

We provide sample codes on how to build and trigger an object detection pipeline. Run it with the local VDP:

$ cd examples-go

# Download a YOLOv4 ONNX model for object detection task (GPU not required)
$ curl -o yolov4-onnx-cpu.zip https://artifacts.instill.tech/vdp/sample-models/yolov4-onnx-cpu.zip

# [optional] Download a test image or use your own images
$ curl -o dog.jpg https://artifacts.instill.tech/dog.jpg

# Deploy the model
$ go run deploy-model/main.go --model-path yolov4-onnx-cpu.zip --model-name yolov4

# Test the model
$ go run test-model/main.go --model-name yolov4 --test-image dog.jpg

# Create an object detection pipeline
$ go run create-pipeline/main.go --pipeline-name hello-pipeline --model-name yolov4

# Trigger the pipeline by using the same test image
$ go run trigger-pipeline/main.go --pipeline-name hello-pipeline --test-image dog.jpg

Create a pipeline with your own models

Please follow the guideline "Prepare your own model to deploy on VDP ". Based on the above sample codes, you can deploy a prepared model and create your own pipeline.

Clean up

To clean up all running services:

$ make prune

Documentation

The gRPC protocols in protobufs provide the single source of truth for the VDP APIs. To view the generated OpenAPI spec on http://localhost:3000:

$ make doc

Community support

For general help using VDP, you can use one of these channels:

  • GitHub (bug reports, feature requests, project discussions and contributions)
  • Discord (live discussion with the community and the Instill AI Team)

License

See the LICENSE file for licensing information.

Comments
  • any default value for .env?

    any default value for .env?

    any example value for .env ?

    if missing , displayed some warning messages as following

    WARNING: The TRITONCONDAENV_IMAGE_TAG variable is not set. Defaulting to a blank string.
    WARNING: The TRITONSERVER_IMAGE_TAG variable is not set. Defaulting to a blank string.
    WARNING: The REDIS_IMAGE_TAG variable is not set. Defaulting to a blank string.
    

    ERROR: build path vdp/dev/pipeline-backend either does not exist, is not accessible, or is not a valid URL. and will not helping with quick start.

    opened by yangboz 11
  • docs(pipeline): add SYNC and ASYNC diagram and section for connectors

    docs(pipeline): add SYNC and ASYNC diagram and section for connectors

    Because

    • we need a document for users to get to know more about how the pipeline works

    This commit

    • add SYNC and ASYNC diagram and section for connectors
    • close #52
    opened by bochengyang 7
  • Wrong model type

    Wrong model type

    The model type is mistakenly set as tensorrt. https://github.com/instill-ai/vdp/blob/814ea12e94b7ebb4656c7208f5c160c646f8523a/examples-go/deploy-model/main.go#L65

    bug 
    opened by pinglin 5
  • feat: add console e2e test into vdp

    feat: add console e2e test into vdp

    Because

    • The console integration test is ready

    This commit

    • add console e2e test into vdp
    • update ITMODE=true when running integration-test. When the flag ITMODE is enabled, the integration-test use dummy models instead of pulling from GitHub, HuggingFace, or ArtiVC to reduce the impact of the internet connection.

    co-author: @Phelan164

    opened by EiffelFly 4
  • test: update integration test

    test: update integration test

    Because

    • need integration test to make sure feature work correctly

    This commit

    • add integration test for model-backend and update for pipeline-backend
    • closes #67
    • closes #69

    Limitation: error code is not covered in this PR. It will be added during working this ticket

    Note: docker image for model-backend is dev which should update when have a new release

    opened by Phelan164 4
  • chore: add pipeine-backend integration test and refactor example and docker-compose

    chore: add pipeine-backend integration test and refactor example and docker-compose

    Because

    • we need an integration test for our ci
    • there is a wrong depend configuration in docker-compose
    • refactor example to be more self-contained

    This commit

    • add condition when waiting for pg_sql ready
    • add pipeline integration test by k6
    • add argument when creating pipeline and correct README.md
    opened by bochengyang 3
  • [doc]: There has no make build command

    [doc]: There has no make build command

    Issue

    • According to the quick start of the repo: if you want to develop locally, you could do the following
    $ git clone https://github.com/instill-ai/vdp.git && cd vdp
    
    # Build instill/vdp:dev local development image
    $ make build
    
    # Launch all services.
    $ make all
    

    But actually, there has no make build command in the Makefile

    https://github.com/instill-ai/vdp/blob/main/Makefile#L15

    Only exist make dev and make all.

    Solution

    Please update the readme quick start guideline.

    documentation 
    opened by EiffelFly 2
  • [release] v0.1.4-alpha

    [release] v0.1.4-alpha

    Protobufs

    • [x] https://github.com/instill-ai/protobufs/issues/17
    • [x] https://github.com/instill-ai/protobufs/issues/33

    model-backend

    • [x] https://github.com/instill-ai/model-backend/issues/44
    • [x] https://github.com/instill-ai/model-backend/issues/33
    • [x] https://github.com/instill-ai/model-backend/issues/45
    • [x] https://github.com/instill-ai/model-backend/issues/30

    pipeline-backend

    • [x] https://github.com/instill-ai/pipeline-backend/issues/32
    • [x] https://github.com/instill-ai/pipeline-backend/issues/33

    vdp

    • [x] Use the latest docker images in vdp
    next-release 
    opened by xiaofei-du 2
  • Quick start sample code bug

    Quick start sample code bug

    Deploy the model

    If I run the following script in quick start

    # Deploy the model
    go run deploy-model/main.go --model-path yolov4-onyx-cpu.zip --model-name yolov4
    

    I get a model with model name yolov4 with 1 version.

    2022/02/21 00:16:13 model has been created, the response is: id:1 name:"yolov4" full_Name:"local-user/yolov4" cv_task:DETECTION versions:{version:1 model_id:1 description:"YoloV4 for object detection" created_at:{seconds:1645402547 nanos:80792000} updated_at:{seconds:1645402547 nanos:80807000}}

    The response missed the status of version 1 of the model.

    If I run the above script the second time, I get a model with model name yolov4 with 2 versions.

    bash 2022/02/21 00:20:07 model has been created, the response is: id:1 name:"yolov4" full_Name:"local-user/yolov4" cv_task:DETECTION versions:{version:1 model_id:1 description:"YoloV4 for object detection" created_at:{seconds:1645402547 nanos:80792000} updated_at:{seconds:1645402625 nanos:977017000} status:ONLINE} versions:{version:2 model_id:1 description:"YoloV4 for object detection" created_at:{seconds:1645402779 nanos:961661000} updated_at:{seconds:1645402779 nanos:961692000}}

    The response only includes the status of version 1, but no status of version 2.

    Test the model

    # Test the model
    go run test-model/main.go --model-name yolov4 --test-image dog.jpg --model-version 2
    

    Get response

    2022/02/21 00:24:15 error when triggering predict: rpc error: code = Code(400) desc = {"status":400,"title":"PredictModel","detail":"Model is offline"}
    

    Note: shouldn't we use status code 422 instead of 400 for the above scenario?

    But when GET /models/yolov4

    {
        "id": 1,
        "name": "yolov4",
        "full_Name": "local-user/yolov4",
        "cv_task": "DETECTION",
        "versions": [
            {
                "version": 1,
                "model_id": 1,
                "description": "YoloV4 for object detection",
                "created_at": "2022-02-21T00:15:47.080792Z",
                "updated_at": "2022-02-21T00:20:09.486272Z",
                "status": "ONLINE"
            },
            {
                "version": 2,
                "model_id": 1,
                "description": "YoloV4 for object detection",
                "created_at": "2022-02-21T00:19:39.961661Z",
                "updated_at": "2022-02-21T00:20:09.486272Z",
                "status": "ONLINE"
            }
        ]
    }
    

    The response shows both model versions are online.

    bug hot-fix 
    opened by xiaofei-du 2
  • docs: refactor doc structure

    docs: refactor doc structure

    Because

    • The root README.md is too long and needs to be restructured

    This commit

    • replace trigger mechanism of a data source with pipeline mode concept and add the docs in docs/pipeline-mode.md.
    • move prepare-you-own-model doc to docs/model.md
    • add check-yaml in pre-commit
    • update redoc service name to redoc_openapi

    The pipeline mode is determined by the combination of data source and destination. It describes how an end-to-end pipeline processes its workload.

    opened by xiaofei-du 2
  • chore(main): release 0.3.0-alpha

    chore(main): release 0.3.0-alpha

    :robot: I have created a release beep boop

    Product Updates

    Announcement đŸ“Ŗ

    • VDP (originally, Visual Data Preparation) is officially renamed to Versatile Data Pipeline.

    We have realised that as a general ETL infrastructure, VDP is in fact capable of processing all kinds of unstructured data. We should not limit its usage to only visual data but to more general versatile data. In addition, the term, Data Preparation, has been misleading for users often thinking it has to do with only data labelling, cleaning, or wrangling. In our vision, while VDP should involve data preparation in its MLOps practice, it should not be conceptually confined to only data preparation. VDP does more than that and is focused on the overall effectiveness of the unstructured data ETL with a data-centric paradigm. The end form of a VDP infrastructure is nothing but a data pipeline. The term Data Pipeline is more precise to capture the core concept of VDP hence renamed as Versatile Data Pipeline.

    Features ✨

    VDP (0.3.0-alpha)

    Features

    Bug Fixes

    • fix wrong triton environment when deploying HuggingFace models (#150) (b2fda36)
    • use COCO RLE format for instance segmentation (4d10e46)
    • update model output protocol (e6ea88d)

    Pipeline-backend (0.9.3-alpha)

    Bug Fixes

    • fix pipeline trigger model hanging (https://github.com/instill-ai/pipeline-backend/issues/80) (7ba58e5)

    Connector-backend (0.7.2-alpha)

    Bug Fixes

    • fix connector empty description update (0bc3086)

    Model-backend (0.10.0-alpha)

    Features

    • support instance segmentation task (https://github.com/instill-ai/model-backend/issues/183) (d28cfdc)
    • support async deploy and undeploy model instance (https://github.com/instill-ai/model-backend/issues/192) (ed36dc7)
    • support semantic segmentation (https://github.com/instill-ai/model-backend/issues/203) (f22262c)

    Bug Fixes

    • allow updating emtpy description for a model (https://github.com/instill-ai/model-backend/issues/177) (100ec84)
    • HuggingFace batching bug in preprocess model (b1582e8)
    • model instance state update to unspecified state (https://github.com/instill-ai/model-backend/issues/206) (14c87d5)
    • panic error with nil object (https://github.com/instill-ai/model-backend/issues/208) (a342113)

    Console

    Features

    • extend the time span of our user cookie (https://github.com/instill-ai/console/issues/289) (76a6f99)
    • finish integration test and make it stable (https://github.com/instill-ai/console/issues/281) (3fd8d21)
    • replace prism.js with code-hike (https://github.com/instill-ai/console/issues/292) (cb61708)
    • unify the gap between elements in every table (https://github.com/instill-ai/console/issues/291) (e743820)
    • update console request URL according to new protobuf (https://github.com/instill-ai/console/issues/287) (fa7ecc3)
    • add hg model id field at model_instance page (https://github.com/instill-ai/console/issues/300) (31a6eab)
    • cleanup connector after test (https://github.com/instill-ai/console/issues/295) (f9c8e4c)
    • disable html report (https://github.com/instill-ai/console/issues/297) (689f50d)
    • enhance the warning of the resource id field (https://github.com/instill-ai/console/issues/303) (6c4aa4f)
    • make playwright output dot on CI (https://github.com/instill-ai/console/issues/293) (e5c2958)
    • support model-backend async long run operation (https://github.com/instill-ai/console/issues/309) (f795ce8)
    • update e2e test (https://github.com/instill-ai/console/issues/313) (88bf0cd) update how we test model detail page (https://github.com/instill-ai/console/issues/310) (04c83a1)
    • wipe out all data after test (https://github.com/instill-ai/console/issues/296) (e4085dd)

    Bug Fixes

    • fix pipeline e2e not stable (https://github.com/instill-ai/console/issues/285) (a26e599)
    • fix set-cookie api route issue due to wrong domain name (https://github.com/instill-ai/console/issues/284) (c3efcdd)

    This PR was generated with Release Please. See documentation.

    autorelease: tagged 
    opened by droplet-bot 1
  • chore(main): release 0.3.1-alpha

    chore(main): release 0.3.1-alpha

    opened by droplet-bot 0
  • Support async model inference

    Support async model inference

    At the moment, a pipeline in async mode still relies on sync model inference and only asynchronously write to the destination. We should make the model-backend trigger endpoints use Temporal workflow. The pipeline-backend should implement Temporal workflow to request model-backend async inference as well.

    enhancement 
    opened by pinglin 0
Releases(v0.3.0-alpha)
  • v0.3.0-alpha(Dec 25, 2022)

    0.3.0-alpha (2022-12-25)

    Product Updates

    Announcement đŸ“Ŗ

    • VDP (originally, Visual Data Preparation) is officially renamed to Versatile Data Pipeline

    We have realised that as a general ETL infrastructure, VDP is in fact capable of processing all kinds of unstructured data. We should not limit its usage to only visual data but to more versatile data. In addition, the term, Data Preparation, has been misleading for users often thinking it has to do with only data labelling, cleaning, or wrangling. In our vision, while VDP should involve data preparation in its MLOps practice, it should not be conceptually confined to only data preparation. VDP does more than that and is focused on the overall effectiveness of the unstructured data ETL with a data-centric paradigm. In spite of the high complexity of the underneath data processing, the end form of a VDP infrastructure is nothing but a data pipeline. The term Data Pipeline is more precise to capture the core concept of VDP, hence renamed as Versatile Data Pipeline.

    Features ✨

    • support new task Instance segmentation. Check out the example here

    VDP (0.3.0-alpha)

    Features

    Bug Fixes

    • fix wrong triton environment when deploying HuggingFace models (#150) (b2fda36)
    • use COCO RLE format for instance segmentation (4d10e46)
    • update model output protocol (e6ea88d)

    Pipeline-backend (0.9.3-alpha)

    Bug Fixes

    • fix pipeline trigger model hanging (https://github.com/instill-ai/pipeline-backend/issues/80) (7ba58e5)

    Connector-backend (0.7.2-alpha)

    Bug Fixes

    • fix connector empty description update (0bc3086)

    Model-backend (0.10.0-alpha)

    Features

    • support instance segmentation task (https://github.com/instill-ai/model-backend/issues/183) (d28cfdc)
    • support async deploy and undeploy model instance (https://github.com/instill-ai/model-backend/issues/192) (ed36dc7)
    • support semantic segmentation (https://github.com/instill-ai/model-backend/issues/203) (f22262c)

    Bug Fixes

    • allow updating emtpy description for a model (https://github.com/instill-ai/model-backend/issues/177) (100ec84)
    • HuggingFace batching bug in preprocess model (b1582e8)
    • model instance state update to unspecified state (https://github.com/instill-ai/model-backend/issues/206) (14c87d5)
    • panic error with nil object (https://github.com/instill-ai/model-backend/issues/208) (a342113)

    Console

    Features

    • extend the time span of our user cookie (https://github.com/instill-ai/console/issues/289) (76a6f99)
    • finish integration test and make it stable (https://github.com/instill-ai/console/issues/281) (3fd8d21)
    • replace prism.js with code-hike (https://github.com/instill-ai/console/issues/292) (cb61708)
    • unify the gap between elements in every table (https://github.com/instill-ai/console/issues/291) (e743820)
    • update console request URL according to new protobuf (https://github.com/instill-ai/console/issues/287) (fa7ecc3)
    • add hg model id field at model_instance page (https://github.com/instill-ai/console/issues/300) (31a6eab)
    • cleanup connector after test (https://github.com/instill-ai/console/issues/295) (f9c8e4c)
    • disable html report (https://github.com/instill-ai/console/issues/297) (689f50d)
    • enhance the warning of the resource id field (https://github.com/instill-ai/console/issues/303) (6c4aa4f)
    • make playwright output dot on CI (https://github.com/instill-ai/console/issues/293) (e5c2958)
    • support model-backend async long run operation (https://github.com/instill-ai/console/issues/309) (f795ce8)
    • update e2e test (https://github.com/instill-ai/console/issues/313) (88bf0cd) update how we test model detail page (https://github.com/instill-ai/console/issues/310) (04c83a1)
    • wipe out all data after test (https://github.com/instill-ai/console/issues/296) (e4085dd)

    Bug Fixes

    • fix pipeline e2e not stable (https://github.com/instill-ai/console/issues/285) (a26e599)
    • fix set-cookie api route issue due to wrong domain name (https://github.com/instill-ai/console/issues/284) (c3efcdd)

    This PR was generated with Release Please. See documentation.

    Source code(tar.gz)
    Source code(zip)
  • v0.2.6-alpha(Sep 20, 2022)

    0.2.6-alpha (2022-09-20)

    Product Updates

    BREAKING changes đŸ…ąī¸

    • updated VDP protocol

    Features ✨

    • add data mapping field data_mapping_indices in the data to be sent to the pipeline destination. The field includes unique IDs that can be mapped to CV task output for each processed image
    • support all CV tasks when building ASYNC pipelines
    • add confidence score for OCR task

    Fix bugs 🐛

    • fix ASYNC trigger block issue, close https://github.com/instill-ai/pipeline-backend/issues/67
    • allow user-defined description for GitHub model

    VDP

    Bug Fixes

    • update examples to be compatible with the latest protocol (36a847b)

    Miscellaneous Chores

    Pipeline-backend (0.8.0-alpha)

    Features

    Bug Fixes

    • fix async trigger block issue (59f0fb8), closes #67
    • fix multipart trigger data_mapping_indices empty (d3160b4)

    Connector-backend (0.6.0-alpha)

    Features

    • add data mapping (5596716)
    • support all cv tasks defined in vdp protocol (c38f9e8), closes #33

    Bug Fixes

    • add http/grpc release stage in connector definitions (f2648c0), closes #30
    • fix destination write for multiple models (860f62e)
    • fix update not using name field (f8e98c3)
    • fix zero-value field dropped (31e297c)

    Model-backend (v0.8.1-alpha)

    Features

    Bug Fixes

    • update description for GitHub model from user input (#173) (821dab3)

    Miscellaneous Chores

    Console (v0.12.0-alpha)

    Features

    • add buffer into calculation of instance cell more button (#268) (19d1470)
    • add destination configuration form (#255) (0164946)
    • Implement onboarding integration test (#271) (f9c494e)
    • Implement source integration test (#272) (b85a285)
    • use env var in pipeline snippet (#275) (3bc4d7a)

    Bug Fixes

    Source code(tar.gz)
    Source code(zip)
  • v0.2.5-alpha(Aug 18, 2022)

    0.2.5-alpha (2022-08-18)

    Product Updates

    Features ✨

    • add new CV Task Optical Character Recognition (OCR)
    • adopt new standard CV Task output, check the VDP Protocol
    • add code snippets about how to trigger a pipeline in the Console

    Fix bugs 🐛

    • fix HTTP and gRPC connector state change logic
    • make the response consistent across all triggering pipeline operations

    VDP

    Miscellaneous Chores

    Console (v0.10.0-alpha)

    Features

    Bug Fixes

    Mgmt-backend (v0.2.5-alpha)

    Miscellaneous Chores

    Pipeline-backend (v0.7.1-alpha)

    Bug Fixes

    • fix multipart trigger resp fmt inconsist (0663542)

    Connector-backend (v0.5.2-alpha)

    Bug Fixes

    • fix http and grpc state change logic (9e7b7eb)
    • fix worker container naming (b0ad69e)

    Miscellaneous Chores

    • update instill connector doc url (34560f6)
    • release v0.5.2-alpha (d2daad6)

    Model-backend (v0.7.0-alpha)

    Features

    Source code(tar.gz)
    Source code(zip)
  • v0.2.4-alpha(Aug 1, 2022)

    0.2.4-alpha (2022-08-01)

    Product Updates

    fix console bugs 🐛

    • fix keep redirecting back to onboarding form issue
    • fix Airbyte form default value issue
    • fix Toggle Field and Upload file Field issue
    • adapt new design-system component props structure
    • fix the wrong connector state
    • remove default scroll behaviour on type=number input

    VDP

    Miscellaneous Chores

    Console (0.7.0-alpha)

    Features

    Bug Fixes

    Source code(tar.gz)
    Source code(zip)
  • v0.2.3-alpha(Jul 29, 2022)

    0.2.3-alpha (2022-07-29)

    Product Updates

    features ✨

    • annotate destination data name with batch index and timestamp

    fix bugs 🐛

    • fix ArtiVC model import, destination creation and pipeline display issue
    • improve management backend error message
    • fix pipeline destination data loss issue

    VDP

    Miscellaneous Chores

    Console (0.6.3-alpha)

    Bug Fixes

    • fix get pipeline bug on model page (#174) (5e8116d)
    • fix ArtiVC model creation issue (#178) (d8ef4e0)
    • fix can't correctly create Airbyte destination (#185) (732844d)
    • fix destination cache not remove and create http & grpc issue (#184) (8c35d95)
    • fix onboarding form (#182) (a944e1d)

    Mgmt-backend (0.2.4-alpha)

    Bug Fixes

    Pipeline-backend (0.7.0-alpha)

    Features

    • add data association with dst conn (9233429)

    Connector-backend (0.5.0-alpha)

    Features

    • add data association with pipeline (c3236f7)

    Bug Fixes

    • fix dst data duplicate and lost issue (eb7ad97), closes #25
    Source code(tar.gz)
    Source code(zip)
  • v0.2.2-alpha(Jul 20, 2022)

    0.2.2-alpha (2022-07-20)

    Product Updates

    fix bugs 🐛🐛🐛

    • âšĄī¸ speed up creating model operation
    • better console form handling
    • improve output format for models that are not labelled with any task
    • handle inference for images with RGBA channels

    VDP

    Miscellaneous Chores

    Console (0.6.0-alpha)

    Console update

    • Update sidebar design
    • Refactor lots of types, make our codebase much more consistent
    • Fix tracking and collect user usage data issue
    • Fix Airbyte nested fields validation and not correct value issue

    Features

    Bug Fixes

    • fix action issue (#172) (c02e4ad)
    • fix Airbyte nested form validation issue (#171) (1101acc)
    • fix description field is duplicated when create pipeline (#165) (7ef7b1a)

    Connector-backend (0.4.5-alpha)

    Bug Fixes

    • fix airbyte emitted_at timestamp (3eca9d6)
    • fix wrong state when connector check failed (92367e1)
    • remove airbyte namespace (bd7baa4)

    Model-backend (0.6.3-alpha)

    Features

    • speed up creating model operation (#115) (6930140)
    • standardize the output of the unspecified task(7fcd867)

    Bug Fixes

    • fix client stream server recv wrong file length interval (#143) (0e06f7c)
    • post process for unspecified task output (ad88068)
    • trigger image with 4 channel (#141) (7445f5f)

    Pipeline-backend (0.6.5-alpha)

    Bug fixes

    complete trigger async binary file for REST multipart (5003e5c)

    Source code(tar.gz)
    Source code(zip)
  • v0.2.1-alpha(Jul 12, 2022)

    0.2.1-alpha (2022-07-12)

    Product Updates

    • đŸ”ĨđŸ”ĨđŸ”Ĩ New YOLOv4 vs. YOLOv7 live demo
    • Add optional description field when creating models
    • Better console page title
    • Support default model card for models without README.md

    VDP

    Miscellaneous Chores

    Console (0.5.0-alpha)

    Features

    Bug Fixes

    • use instill owned icons when interact with instill owned destination (#150) (d466a09)

    Model-backend (v0.6.0-alpha)

    Miscellaneous Chores

    Source code(tar.gz)
    Source code(zip)
  • v0.2.0-alpha(Jul 8, 2022)

    0.2.0-alpha (2022-07-08)

    Product Updates

    A lot of gooooodies đŸŦ 🍰 🧁 to share with the community

    • đŸĨŗ Add new management server mgmt-backend for user management
    • 🎊 Add new connector server connector-backend for pre-built source/destination connector management
      • new HTTP connector
      • new gRPC connector
    • đŸ”Ĩ Now you can import models from GitHub, Hugging Face, or remote storage managed by DVC or ArtiVC
    • 🎉 New vision task: besides image classification and object detection, VDP supports keypoint detection
    • 🎮 New Console UI to enable youenabling you to use VDP without writing code
    • 🚀 Build pipelines for diverse scenarios
      • SYNC pipeline for real-time inference
      • ASYNC pipeline for on-demand workload

    VDP

    Features

    Bug Fixes

    • fix connector-backend-worker in docker-compose-dev (c5d7260)
    • fix OpenAPI swagger service (#104) (2e3fd90)
    • fix temporal dynamicconfig (57edcde)

    Mgmt-backend (v0.2.3-alpha)

    Features

    • add reporter for usage collection (320d9a4)
    • add retrieve all user function (da78893)

    Bug Fixes

    • support CORS (https://github.com/instill-ai/mgmt-backend/issues/3) (5fb1439)
    • use cors package to replace naive implementation (ae49c1c)
    • add tlsenabled in usage backend configuration (dd8843a)
    • init config before logger (455ef7a)
    • init config first (4e3dea6)
    • put client connection in main (3ff4821)
    • refactor usage collection (c6a41f6)
    • specify time duration unit (27b77d2)
    • check existing user by id (dd862a4)
    • change port from 8080 to 8084 (364f155)

    Connector-backend (v0.4.3-alpha)

    Features

    Bug Fixes

    • fix det jsonschema for empty case (cd4f102)
    • fix duration configuration bug (4a4111c)
    • fix usage collection (c19cf9b)
    • fix usage disbale logic (664660d)
    • fix usage-backend non-tls dial (457a74d)
    • remove worker debug volume (acaa01c)
    • return 422 delete error msg (5498494)
    • fix error handler middleware (2fc9487)
    • skip JSON Schema snake_case convert (c19cc53), closes https://github.com/instill-ai/connector-backend/issues/17

    Model-backend (v0.6.0-alpha)

    Features

    • add credential definition (https://github.com/instill-ai/model-backend/issues/109) (92d3391)
    • support artivc (https://github.com/instill-ai/model-backend/issues/102) (b8e21a4)
    • support creating a HuggingFace model (https://github.com/instill-ai/model-backend/issues/113) (1577d87)

    Bug Fixes

    • fix config path (a8cf2c0)
    • regexp zap logger with new protobuf package (8b9c463)
    • bug usage storage (https://github.com/instill-ai/model-backend/issues/103) (975fdc1)
    • fix duration configuration bug (ee4a310)
    • init config before logger (9d3fb4a)
    • status code when deploy model error (https://github.com/instill-ai/model-backend/issues/111) ([31d3f11](https://github.com/instill-ai/model- backend/commit/31d3f11ba04ee59b12521b8e0dd724849a81b94f))
    • update model definitions and tasks in usage collection (https://github.com/instill-ai/model-backend/issues/100) ([c593087](https://github.com/instill-ai/model- backend/commit/c5930870595c5d280d7db005a711c0cc9bff802c))
    • wrong logic when checking user account and service account (7058db6)
    • model definition in list model and missing zero in output (https://github.com/instill-ai/model-backend/issues/121) (a90072d)

    Pipeline-backend (v0.6.3-alpha)

    Features

    • add cors support (c847912)
    • add async pipeline trigger (6d39b0a)
    • add filter for list pipeline (ffe8856)
    • add usage collection (7e71744)
    • support trigger multi model instances (e3d4263)

    Bug Fixes

    • fix create pipeline recipe resource name (bab3eaa)
    • fix duration configuration bug (998eafa)
    • fix usage collection (243e7a1)
    • fix usage disbale logic (962823b)
    • fix usage-backend non-tls dial (b864df3)
    • close https://github.com/instill-ai/pipeline-backend/issues/56 ([c627a53](https://github.com/instill-ai/pipeline- backend/commit/c627a539999d65bc96ac6f88e2bd203548c34825))
    • fix empty description update (2579f2e)

    Console (v0.3.3-alpha)

    Features

    • add ArtiVC Icon for model definition (#18) (b5250db)
    • add delete connector, pipeline service (#9) (526ff70)
    • add DISABLE_USAGE_COLLECTION args config (#45) (7f9352d)
    • add GitHub action related to docker hub (#11) (5e7569a)
    • add mock server and render method of react-query (#12) (80d89d7)
    • add mock source service handler (#13) (21ea1ee)
    • add onboarding e2e test (#20) (c952b73)
    • add pipeline e2e test (#44) (0acc846)
    • add playwright (#14) (01beb24)
    • add project docker-compose (#2) (14ab822)
    • add release please action (#21) (2321bab)
    • add tailwind plugin to support intelligence picking up custom style and refactor classnames (#16) (62d853f)
    • add test model instance service (#15) (a367577)
    • add unit-test action (#17) (d57d265)
    • implement statefull toggle and adapt new prop (#49) (9358247)
    • make sidebar collapsable (#8) (ecf8cd3)
    • let user add backend related env with docker-compose environment (#62) (507534a)
    • add ModelInstanceReadmeCard (#76) (28d3927)
    • add script to download airbyte's icons (#66) (96dd22e)
    • support ArtiVC (#64) (1f1fd11)

    Bug Fixes

    Source code(tar.gz)
    Source code(zip)
  • v0.1.6-alpha(Apr 3, 2022)

  • v0.1.5-alpha(Mar 22, 2022)

  • v0.1.4-alpha(Mar 22, 2022)

  • v0.1.3-alpha(Feb 24, 2022)

  • v0.1.2-alpha(Feb 21, 2022)

    0.1.2-alpha (2022-02-21)

    Project Updates

    • A full local development tooling with docker-compose
    • Preliminary model serving for multiple Deep Learning frameworks
    • Pipeline with sync mode to handle blocking data flow
    • Go example codes

    Bug Fixes

    • add name in update model api (#41) (35b7e31)
    • docker-compose: change correct dependency for temporal (#60) (eb98049)
    • update model backend without specify version when creating model (#63) (f4cf161)
    • wrong type in sample model prediction (#39) (523d5f8)
    Source code(tar.gz)
    Source code(zip)
  • v0.1.1-alpha(Feb 13, 2022)

  • v0.1.0-alpha(Feb 11, 2022)

    0.1.0-alpha (2022-02-11)

    Features

    • add provisioned visual-data-pipeline from docker-compose (0ec790b)
    • initial commit (2916d75)
    • refactor the name to visual data preparation and add model-backend into docker-compose (#7) (d24c0ef)
    • refactor to adopt koanf configuration management (51eba89)

    Bug Fixes

    Source code(tar.gz)
    Source code(zip)
Owner
Instill AI
Visual data preparation made for all - Empower modern data stack, tapping the value of unstructured visual data with our open source community.
Instill AI
An open-source, distributed, cloud-native CD (Continuous Delivery) product designed for developersAn open-source, distributed, cloud-native CD (Continuous Delivery) product designed for developers

Developer-oriented Continuous Delivery Product âŖ English | įŽ€äŊ“中文 Table of Contents Zadig Table of Contents What is Zadig Quick start How to use? How to

null 0 Oct 19, 2021
Kstone is an etcd management platform, providing cluster management, monitoring, backup, inspection, data migration, visual viewing of etcd data, and intelligent diagnosis.

Kstone 中文 Kstone is an etcd management platform, providing cluster management, monitoring, backup, inspection, data migration, visual viewing of etcd

TKEStack 596 Dec 27, 2022
go-opa-validate is an open-source lib that evaluates OPA (open policy agent) policy against JSON or YAML data.

go-opa-validate go-opa-validate is an open-source lib that evaluates OPA (open policy agent) policy against JSON or YAML data. Installation Usage Cont

chenk 6 Nov 17, 2022
The Elastalert Operator is an implementation of a Kubernetes Operator, to easily integrate elastalert with gitops.

Elastalert Operator for Kubernetes The Elastalert Operator is an implementation of a Kubernetes Operator. Getting started Firstly, learn How to use el

null 20 Jun 28, 2022
Plugin for Helm to integrate the sigstore ecosystem

helm-sigstore Plugin for Helm to integrate the sigstore ecosystem. Search, upload and verify signed Helm Charts in the Rekor Transparency Log. Info he

sigstore 47 Dec 21, 2022
🔮 âœˆī¸ to integrate OPA Gatekeeper's new ExternalData feature with cosign to determine whether the images are valid by verifying their signatures

cosign-gatekeeper-provider To integrate OPA Gatekeeper's new ExternalData feature with cosign to determine whether the images are valid by verifying i

sigstore 50 Dec 8, 2022
Vilicus is an open source tool that orchestrates security scans of container images(docker/oci) and centralizes all results into a database for further analysis and metrics.

Vilicus Table of Contents Overview How does it work? Architecture Development Run deployment manually Usage Example of analysis Overview Vilicus is an

Ederson Brilhante 80 Dec 6, 2022
Open Source runtime tool which help to detect malware code execution and run time mis-configuration change on a kubernetes cluster

Kube-Knark Project Trace your kubernetes runtime !! Kube-Knark is an open source tracer uses pcap & ebpf technology to perform runtime tracing on a de

Chen Keinan 32 Sep 19, 2022
Horusec is an open source tool that improves identification of vulnerabilities in your project with just one command.

Table of contents 1. About 2. Getting started 2.1. Requirements 2.2. Installation 3. Usage 3.1. CLI Usage 3.2. Using Docker 3.3. Older versions 3.4. U

ZUP IT INNOVATION 839 Jan 7, 2023
:rocket: Modern cross-platform HTTP load-testing tool written in Go

English | 中文 Cassowary is a modern HTTP/S, intuitive & cross-platform load testing tool built in Go for developers, testers and sysadmins. Cassowary d

Roger Welin 643 Dec 29, 2022
k6 is a modern load testing tool for developers and testers in the DevOps era.

k6 is a modern load testing tool, building on our years of experience in the load and performance testing industry. It provides a clean, approachable scripting API, local and cloud execution, and flexible configuration.

k6 19k Jan 8, 2023
Bubbly is an open-source platform that gives you confidence in your continuous release process.

Bubbly Bubbly - Release Readiness in a Bubble Bubbly emerged from a need that many lean software teams practicing Continuous Integration and Delivery

Valocode 35 Nov 29, 2022
Open Source runtime scanner for Linux containers (LXD), It performs security audit checks based on CIS Linux containers Benchmark specification

lxd-probe Scan your Linux container runtime !! Lxd-Probe is an open source audit scanner who perform audit check on a linux container manager and outp

Chen Keinan 16 Dec 26, 2022
KubeCube is an open source enterprise-level container platform

KubeCube English | ä¸­æ–‡æ–‡æĄŖ KubeCube is an open source enterprise-level container platform that provides enterprises with visualized management of Kuberne

KubeCube IO 339 Jan 4, 2023
Devtron is an open source software delivery workflow for kubernetes written in go.

Devtron is an open source software delivery workflow for kubernetes written in go.

Devtron Labs 2.8k Jan 8, 2023
TriggerMesh open source event-driven integration platform powered by Kubernetes and Knative.

TriggerMesh open source event-driven integration platform powered by Kubernetes and Knative. TriggerMesh allows you to declaratively define event flows between sources and targets as well as add even filter, splitting and processing using functions.

TriggerMesh 373 Dec 30, 2022
Fleet - Open source device management, built on osquery.

Fleet - Open source device management, built on osquery.

Fleet Device Management 1k Dec 30, 2022