Prophecis is a one-stop machine learning platform developed by WeBank

Overview

Prophecis

License

English | 中文

Introduction

Prophecis is a one-stop machine learning platform developed by WeBank. It integrates multiple open-source machine learning frameworks, has the multi tenant management capability of machine learning compute cluster, and provides full stack container deployment and management services for production environment.

Architecture

  • Overall Structure

    Prophecis

    Five key services in Prophecis:

  • Prophecis Machine Learning Flow:Distributed machine learning modeling tool, with stand-alone and distributed mode model training functions, supports Tensorflow, Python, XGBoost and other machine learning frameworks, and supports the complete pipeline from machine learning modeling to deployment;

  • Prophecis MLLabis:Machine learning development and exploration tools, providing development and exploration services. MLLabis is an online IDE based on Jupyter Lab. It also supports machine learning tasks for GPU and Hadoop clusters, supports Python, R, and Julia, and integrates Debug and TensorBoard plugins;

  • Prophecis Model Factory: MF provides machine learning model storage, model deployment, AB Test, model management and other services;

  • Prophecis Data Factory: DF provides feature engineering tools, data labeling tools and material management services;

  • Prophecis Application Factory:AF is jointly developed by the big data platform team and AI Department of Webank. It is based on QingCloud's open source system KubeSphere custom development, providing CI/CD and DevOps tools, GPU cluster monitoring and warning capabilities.

  • Features

    Prophecis功能特色

  • Whole Machine Learning Life Cycle Support:Prophecis's MLFlow can be nested into the workflow of DataSphere Stdudio through AppJoint. Support the entire machine learning process from data upload, data preprocessing, feature engineering, model training, model evaluation, model release to model deployment.

    DSS-Prophecis

  • One-Click Model Deployment Service:Prophecis MF supports deploying models generated by Prophecis Machine Learning Flow and Propheics MLLabis as restful API or RPC interface with one click, so as to realize seamless connection between model and business system.

  • Comprehensive Management Platform:Based on the community open source program customization, Prophecis provides complete, reliable, and highly flexible enterprise-level machine learning application release, monitoring, service management, log collection and query management tools, and realizes the full control of machine learning applications to meet the needs of enterprises Learn to apply all the work requirements of the online production environment.

Quick Start Guide

Developing

  • Read the Develop Guide guide to quickly get how to develop Prophecis.

Roadmap

  • See our Roadmap for what's coming soon in Prophecis.

Contributing

Contributions are warmly welcomed and greatly appreciated.

Communication

If you desire immediate response, please kindly raise issues to us or scan the below QR code by WeChat and QQ to join our group: :

Communication

License

Prophecis is under the Apache 2.0 license. See the LICENSE file for details.

Issues
  • Add Prophecis-DI Storage Module

    Add Prophecis-DI Storage Module

    Add Prophecis-DI Storage Module, which is Responsible for the operation of storage module, such as Minio, ES, Mongo.

    component=Prophecis/DI/Storage type=NewFeature 
    opened by SelfImpr001 1
  • which version of k8s supported?

    which version of k8s supported?

    I had test kubernetes v1.20.0 and v1.18.6 and it shows errors, and below is the detail information:

    run helm install notebook-controller . in folder Prophecis/helm-charts/k8s 1.18.6/notebook-controller it shows: Error: template: MLSS/templates/notebook-controller-0.5.1.yaml:115:24: executing "MLSS/templates/notebook-controller-0.5.1.yaml" at <.Values.aide.controller.notebook.repository>: nil pointer evaluating interface {}.controller

    run helm install notebook-controller . in folder Prophecis/helm-charts/notebook-controller it shows: Error: unable to build kubernetes objects from release manifest: [unable to recognize "": no matches for kind "Deployment" in version "apps/v1beta1", unable to recognize "": no matches for kind "StatefulSet" in version "apps/v1beta2"

    opened by xueyoo 1
  • Add Prophecis-DI Storage Module

    Add Prophecis-DI Storage Module

    Add Prophecis-DI Storage Module, which is Responsible for the operation of storage module, such as Minio, ES, Mongo.

    component=Prophecis/DI/Storage type=NewFeature 
    opened by SelfImpr001 0
  • Update helm chart & Doc

    Update helm chart & Doc

    1. Update User Guide basic operations: MLLabis Guide: namespace, user information, and project group information. DI Guide: distributed modeling and CLI use. CC Controller Guide: namespace, user information, and project group information. Development Guide: CI/CD construction and Remote debugging.
    2. Update helm chart, and move the LogCollectorDS component to the Prophecis component.
    3. Update README.
    opened by finalTestin 0
  • fix image tags

    fix image tags

    fix image tag misstakes and change cc/aide/di to service name

    opened by ykfq 0
  • Helm Chart Image Version Fix & Update Doc

    Helm Chart Image Version Fix & Update Doc

    Helm Chart Image Version Fix & Update Doc

    opened by alexzyWu 0
  • update prophecis-data.sql

    update prophecis-data.sql

    insert admin user record.

    opened by alexzyWu 0
  • fix helm chart error & update doc

    fix helm chart error & update doc

    Fix Helm Chart & Update Doc

    opened by alexzyWu 0
  • Fix Helm Chart & Update Doc

    Fix Helm Chart & Update Doc

    Fix Helm Chart & Update Doc

    opened by alexzyWu 0
  • Add Remote Debug Guide

    Add Remote Debug Guide

    Add Remote Debug Guide.

    documentation 
    opened by alexzyWu 0
  • 镜像无法拉取!!!

    镜像无法拉取!!!

    您好! wedatasphere/prophecis:metrics-0.2.0 wedatasphere/prophecis:jobmonitor-0.2.0 wedataspere/prophecis:minio-2020-06-14 wedatasphere/prophecis:lcm-0.2.0 wedatasphere/prophecis:trainer-0.2.0 这些镜像都无法拉取,请问仓库里有这些镜像吗?

    opened by gugumituo 2
  • 搭建过程中,镜像无法拉取

    搭建过程中,镜像无法拉取

    有2个镜像无法拉取,一个是wedataspere/prophecis:minio-2020-06-14的镜像无法拉取;一个是wedatasphere/prophecis:fluent-bit-1.2.1的镜像 ,请问是否仓库当前是否存在该镜像?

    opened by liunick2021 2
  • 什么时候发布0.3.x

    什么时候发布0.3.x

    一直关注Prophecis,请问v0.2.x和v0.3.x这两个版本有大概的发布时间点么?

    opened by ixiejun 2
  • expect v0.2.x and  v0.3.x

    expect v0.2.x and v0.3.x

    opened by DreamerCather 0
  • fix label name does not match

    fix label name does not match

    The matchLabels has a redundant -deployment suffix, remove it to fix the error below:

    $ helm install prophecis . --namespace prophecis
    
    Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: ValidationError(Deployment.spec): missing required field "selector" in io.k8s.api.apps.v1.DeploymentSpec
    
    opened by ykfq 0
Releases(v0.2.0)
  • v0.2.0(Mar 15, 2021)

    Prophecis 0.2.0 release

    Prophecis v0.2.0 mainly publish distributed modeling module(DI). This module is based on FfDL and mainly provides single machine modeling and distributed tensorflow tasks.

    Enhancement

    [1] Add Prophecis-DI Rest Module. #12 [2] Add Prophecis-DI Trainer & JobMonitor Module, which is responsible for managing the task lifecycle. #13 [3] Add Prophecis-DI LCM Module, which is responsible task scheduling, building single machine and distributed tasks. #8 [4] Add Prophecis-DI Storage Module, which is Responsible for the operation of storage module, such as Minio, ES, Mongo, etc. #14 [5] ADD Log in CLI Program, a command-line interface tool. #11

    Bugfix

    [1] Fix Helm Chart Setting Error. #16

    Prophecis-DI module is built based on the FfDL. The main modifications are as follows:

    [1] Integrate Kubeflow Arena,Provide distributed tensorflow task ability. [2] Modify the creation mode of single machine modeling task:remove helper and job jobmonitor in task, and change deploy pod to deploy job. [3] The log collection service is changed to daemonset, and the collection tool is changed to fluent bit. [4] The task status update mode is changed to an independent service job monitor. [5] Add user GUID control in container data directory. [6] Enhance CLI, added parameter replacement of yaml template, and the train command was modified to websocket connect, providing log and state. [7] The code file storage server is changed to Minio.

    Source code(tar.gz)
    Source code(zip)
  • v0.1.1(Dec 21, 2020)

    Prophecis 0.1.1 release

    Prophecis v0.1.1 is mainly a bug fix version, this version aim to fix some login error and adds helm chart for notebook controller deployment.

    Enhancement

    [1] User login add password RSA encryption verification. #1 [2] Add configurable administrator users to the system. #1 [3] Add Notebook Controller Deploy Helm Chart. #1

    Bugfix

    [1] Fix LDAP BUG:Add BaseDN config. #1 [2] Fix Helm Chart BUG:Wrong repository and variable. #1

    Source code(tar.gz)
    Source code(zip)
Gorgonia is a library that helps facilitate machine learning in Go.

Gorgonia is a library that helps facilitate machine learning in Go. Write and evaluate mathematical equations involving multidimensional arrays easily

Gorgonia 4.1k Jul 27, 2021
Gorgonia is a library that helps facilitate machine learning in Go.

Gorgonia is a library that helps facilitate machine learning in Go. Write and evaluate mathematical equations involving multidimensional arrays easily

Gorgonia 4.1k Jul 19, 2021
Deploy, manage, and scale machine learning models in production

Deploy, manage, and scale machine learning models in production. Cortex is a cloud native model serving platform for machine learning engineering teams.

Cortex Labs 7.6k Jul 21, 2021
Go Machine Learning Benchmarks

Benchmarks of machine learning inference for Go

Nikolay Dubina 15 Jun 17, 2021
Bigmachine is a library for self-managing serverless computing in Go

Bigmachine Bigmachine is a toolkit for building self-managing serverless applications in Go. Bigmachine provides an API that lets a driver process for

GRAIL 170 Jun 18, 2021
On-line Machine Learning in Go (and so much more)

goml Golang Machine Learning, On The Wire goml is a machine learning library written entirely in Golang which lets the average developer include machi

Conner DiPaolo 1.2k Jul 20, 2021
Standard machine learning models

Cog: Standard machine learning models Define your models in a standard format, store them in a central place, run them anywhere. Standard interface fo

Replicate 70 Jul 27, 2021
Reinforcement Learning in Go

Overview Gold is a reinforcement learning library for Go. It provides a set of agents that can be used to solve challenges in various environments. Th

AUNUM 228 Jul 21, 2021
Machine Learning libraries for Go Lang - Linear regression, Logistic regression, etc.

package ml - Machine Learning Libraries ###import "github.com/alonsovidales/go_ml" Package ml provides some implementations of usefull machine learnin

Alonso Vidales 191 Apr 17, 2021
Ensembles of decision trees in go/golang.

CloudForest Google Group Fast, flexible, multi-threaded ensembles of decision trees for machine learning in pure Go (golang). CloudForest allows for a

Ryan Bressler 687 Jul 17, 2021
Generative Adversarial Network in Go via Gorgonia

Generative adversarial networks Recipe for simple GAN in Golang ecosystem via Gorgonia library Table of Contents About Why Instruments Usage Code expl

Dimitrii Lopanov 59 Jul 19, 2021
A Go idiomatic binding to the C++ core of PyTorch

GoTorch GoTorch reimplements PyTorch high-level APIs, including modules and functionals, in idiomatic Go. Thus enables deep learning programming in Go

Yi Wang 53 Jul 17, 2021
A High-level Machine Learning Library for Go

Overview Goro is a high-level machine learning library for Go built on Gorgonia. It aims to have the same feel as Keras. Usage import ( . "github.

AUNUM 282 Jul 10, 2021
A Kubernetes Native Batch System (Project under CNCF)

Volcano is a batch system built on Kubernetes. It provides a suite of mechanisms that are commonly required by many classes of batch & elastic workloa

Volcano 1.8k Jul 27, 2021