KubeGene : dedicated to making genome sequencing process simple, portable and scalable

Overview

KubeGene

Go Report Card

What is KubeGene

The KubeGene is dedicated to making genome sequencing process simple, portable and scalable. It provides a complete solution for genome sequencing on the kubernetes and supports mainstream biological genome sequencing scenarios such as DNA, RNA, and liquid biopsy. KubeGene is based on lightweight container technology and official standard algorithms. You can make a flexible and customizable genome sequencing process by using KubeGene.

Advantages

Universal workflow design grammar

KubeGene provides a complete set of genome sequencing workflow grammars which decouples with specific analysis tools. It requires a very low learning cost to learn how to write and use the workflow. You can easily migrate the genome sequencing business to KubeGene.

Tailor-made workflow for the biosequencing industry

The workflow grammar is designed by comparing different genome sequencing scenarios. It also keeps the user’s traditional usage habit as much as possible and is closer to user scenarios.

More efficient resource usage

KubeGene uses container to run the genome sequencing business. Compared to traditional genome sequencing solutions using virtual machines, KubeGene makes resource usage more efficient and avoiding resource idle.

Scaling based on demand

Kubernetes can automatically scale your cluster based on your genome sequencing workload by using KubeGene. Also you can easily scale the kubernetes cluster manually. It can save your production costs.

Prerequisites

  • Kubernetes 1.10+
  • golang 1.10+

If you have an older version of Kubernetes, we recommend upgrading Kubernetes first and then use the kubeGene.
If you use 1.10 kubernetes, you should make sure that the feature gate CustomResourceSubresources is open.
You can use minikube to build a local kubernetes cluster quickly. For more information, you can use minikube.

Deploy KubeGene

KubeGene has two main componments.

  • Kubedag. Kubedag is a DAG workflow engine on Kubernetes platform. It is dedicated to making Gene Sequencing workflow execute in container easily.

  • genectl. genectl is a command line interface for running commands against KubeGene.

Clone code

First, make work directory:

$ mkdir -p $GOPATH/src/kubegene.io

Enter the work directory and clone the code:

$ git clone https://github.com/kubegene/kubegene.git

Install kubedag

We have provide the YAML file that contains all API objects that are necessary to run kubedag, you can easily deploy the kubedag using the follow commands:

$ ./hack/local-up-kubedag.sh

kubedag will automatically create a Kubernetes Custom Resource Definition (CRD):

$ kubectl get customresourcedefinitions
NAME                                    AGE
executions.execution.kubegene.io        2s 

For more information, you can see Kugedag.

Install genectl

Compile the genectl

$ make genectl

Enter bin directory and make the genectl binary executable.

$ chmod +x ./genectl

Move the binary in to your PATH.

$ sudo mv ./genectl /usr/local/bin/genectl

For how to use genectl, you can see genectl.

Build your tool repo

A tool is a mirrored package of bioinformatics software that genome sequencing use. When you use genectl to submit your workflow, you should specify your tool repo address,it can be a directory or URL, if it is a URL, it must point to a specify tool file. The default tool repo is local directory “/${home}/kubegene/tools”). You can write your own tools and put them in the tool repo. For how to write your tool yaml, you can see tool.

Write and submit your workflow.

We have defined a complete set of gene sequencing workflow grammars. It keeps the user’s traditional usage habit as much as possible and requires a very low learning cost to learn how to write and use the workflow. For how to write your workflow, you can see workflow grammar.

We have several example workflow yaml in the example directory. A simple sample you can see in the example/simple-sample directory.

For submit the workflow, you need create the storage volume first:

$ kubectl create -f example/simple-sample/sample-pv.yaml
persistentvolume "sample-pv" created
$ kubectl create -f example/simple-sample/sample-pvc.yaml
persistentvolumeclaim "sample-pvc" created

Then you can submit the workflow by the following command:

$ genectl sub workflow example/simple-sample/simple-sample.yaml
The workflow has been submitted successfully! And the execution has been created.
Your can use the follow command to query the status of workflow execution.

        genectl get execution execution-bf0dc -n default

or use the follow command to query the detail info for workflow execution.

        genectl describe execution execution-bf0dc -n default

or use the follow command to delete the workflow execution.

        genectl delete execution execution-bf0dc -n default

Query the detail execution status of workflow:

$ genectl describe execution execution-bf0dc -n default
Namespace:    default
Labels:       
   
    
Annotations:  
    
     
Phase:        Succeeded
Message:      execution has run successfully
workflow:
  jobprepare(total: 1; success: 1; failed: 0; running: 0; error: 0):
    Subtask                       Phase      Message
    -------                       -----      -------
    execution-bf0dc.jobprepare.0  Succeeded  success
  joba(total: 2; success: 2; failed: 0; running: 0; error: 0):
    Subtask                 Phase      Message
    -------                 -----      -------
    execution-bf0dc.joba.0  Succeeded  success
    execution-bf0dc.joba.1  Succeeded  success
  jobb(total: 3; success: 3; failed: 0; running: 0; error: 0):
    Subtask                 Phase      Message
    -------                 -----      -------
    execution-bf0dc.jobb.0  Succeeded  success
    execution-bf0dc.jobb.1  Succeeded  success
    execution-bf0dc.jobb.2  Succeeded  success
  jobc(total: 2; success: 2; failed: 0; running: 0; error: 0):
    Subtask                 Phase      Message
    -------                 -----      -------
    execution-bf0dc.jobc.1  Succeeded  success
    execution-bf0dc.jobc.0  Succeeded  success
  jobd(total: 2; success: 2; failed: 0; running: 0; error: 0):
    Subtask                 Phase      Message
    -------                 -----      -------
    execution-bf0dc.jobd.1  Succeeded  success
    execution-bf0dc.jobd.0  Succeeded  success
  jobfinish(total: 1; success: 1; failed: 0; running: 0; error: 0):
    Subtask                      Phase      Message
    -------                      -----      -------
    execution-bf0dc.jobfinish.0  Succeeded  success       

    
   

Feature on the road

KubeGene has provide the basic functionalities for running the genome sequencing workflow. More feature will be added:

  • Support other workflow engine, such as argo and so on.
  • User-defined workflow grammar plugin support.
  • Support for hybrid cloud deployment.

Support

If you need support, start with the [troubleshooting guide], and work your way through the process that we've outlined.

That said, if you have questions, reach out to us, feel free to reach out to these folks:

  • @kevin-wangzefeng
  • @lichuqiang
  • @wackxu
Comments
  • submit execution error:the server could not find the requested resource

    submit execution error:the server could not find the requested resource

    When I submited the workflow, I met a mistake: Error: submit execution error: the server could not find the requested resource (post executions.execution.kubegene.io) the command I used is as follows: genectl sub workflow example/simple-sample/simple-sample.yaml --tool-repo example/tools/ what does the error means and how can I solve it?

    opened by xllbit 16
  • Hi

    Hi

    Hi folks,

    Very cool project!

    Jeremy Lewi (Google Kubeflow) mentioned your project to us. We collaborate with Jeremy and the Kubeflow team because they use Argo (https://github.com/argoproj/argo) as their workflow engine for Kubernetes.

    Please let us know if you may find Argo useful for your project as well.

    Regards, Ed

    opened by edlee2121 4
  • Validation in controller

    Validation in controller

    What this PR does / why we need it: This PR adds validations for condition and get result in the controller and related UT test cases This PR has to be merged after merging the #52 because this branch is on top of that PR branch

    opened by SrinivasChilveri 2
  • Example and E2E test case for dynamic concurrency

    Example and E2E test case for dynamic concurrency

    What this PR does / why we need it: This PR adds dynamic concurrency feature e2e test case and sample example

    added some validation in cmd/genectl/parser/parser.go

    Which issue(s) this PR fixes (optional, in fixes #(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged): Fixes #45

    opened by SrinivasChilveri 2
  • indicate explicit resource type for genectl describe

    indicate explicit resource type for genectl describe

    genectl describe command is now implemented with implict and fixed resource type: genectl describe <resource name> actually means genectl describe execution <resource name>.

    To address this issue, the command format shall be

    genectl describe <resource type> <resource name>
    

    Or

    genectl describe <resource type>/<resource name>
    
    good first issue 
    opened by kevin-wangzefeng 2
  • Job with condition and getresult

    Job with condition and getresult

    What this PR does / why we need it: This PR adds example & e2e test case for the job which has condition and getresult This PR has to be merged after merging the https://github.com/kubegene/kubegene/pull/50 because this branch is on top of that PR branch

    Fixes #49

    opened by SrinivasChilveri 1
  •  Conditional Concurrency Requirement Implementation

    Conditional Concurrency Requirement Implementation

    What this PR does / why we need it: This PR adds new requirement Conditional Concurrency. Conditional Concurrency Requirement Implementation Conditional Concurrency Example Conditional Concurrency E2E test case

    Which issue(s) this PR fixes (optional, in fixes #(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):

    Fixes #44

    opened by SrinivasChilveri 1
  • Changes in describe and del commands

    Changes in describe and del commands

    Fixes: #32

    Fix:

    1. from genectl describe <resource name> to genectl describe <resource type> <resource name>

    2. from genectl del <resource name> to genectl delete <resource type> <resource name>

    opened by SrinivasChilveri 1
  • Samples and e2e test cases and framework changes to support exec crd creation/validation

    Samples and e2e test cases and framework changes to support exec crd creation/validation

    What this PR does / why we need it: This PR adds samples & e2e test cases for the new features like Generic Condition dynamic handling, get result,condition direct creation using execution crd

    framework is changed to support the automation & validation for the execution crd creation/validations etc

    opened by SrinivasChilveri 0
  • Example and E2E test case for combination of condition and get_result

    Example and E2E test case for combination of condition and get_result

    What this PR does / why we need it: This PR adds example & e2e test case for the combination of condition & get_result

    Related to https://github.com/kubegene/kubegene/issues/49

    opened by SrinivasChilveri 0
  • Dynamic Concurrency feature

    Dynamic Concurrency feature

    What this PR does / why we need it: This PR adds new requirement Dynamic Concurrency.

    Which issue(s) this PR fixes (optional, in fixes #(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged): Fixes #40

    Its just in Initial steps, so just concentrate on the logical review not on other silly mistakes. just compiled but yet to test & verify the same.

    opened by SrinivasChilveri 0
  • Seperate kubedag to standalone repo

    Seperate kubedag to standalone repo

    Hi team,

    There's a kubedag in kubegene for worklfow; as workflow/dag is a common requirement for batch workload, prefer to seperate kubedage to a standalone repo of volcano-sh.

    If any comments, please let me know :)

    -- k

    opened by k82cn 7
  • Request to migrate kubegene to volcano.sh as a subproject

    Request to migrate kubegene to volcano.sh as a subproject

    As what https://github.com/volcano-sh/volcano/issues/1560 mentions, we request to migrate kubegene to volcano.sh as subproject and work on provide better integration.

    opened by Thor-wl 1
  • support var_step when container concurrence

    support var_step when container concurrence

    it's already support var_iter when job concurrence.

    var1: [A,B,C]
    var2:[1,2,3]
    it get : A1, A2, A3, B1, B2, B3, C1, C2, C3
    

    but, when container job concurrence,it need all $vars iterater both add index like:

    var1: [A,B,C]
    var2:[1,2,3]
    it get : A1, B2, C3
    
    opened by tsjsdbd 0
  • Upgrade dependency

    Upgrade dependency

    We haven't upgrade dependencies for long. It's time to do that, basically as k8s 1.18 has released. We must do that.

    • [X] K8s version, we should support k8s 1.15+ #63

    • [X] go version, support go 1.13 #63

    • [X] replace dep with go module #62

    opened by hzxuzhonghu 0
Releases(0.1.0)
Owner
Volcano
A Kubernetes Native Batch System
Volcano
A simple database application that I was asked to implement as part of a job application process

This is a simple database application that I was asked to implement as part of a job application process. They told me I could choose any languages an

null 0 Nov 24, 2021
rpipe relays message between child process and redis pubsub channel.

rpipe rpipe relays message between child process and redis pubsub channel. rpipe subscribes Redis channel named HOSTNAME. rpipe spawns child process a

sng2c 5 Jan 27, 2022
The source code for workshop Scalable architecture using Redis as backend database using Golang + Redis

The source code for workshop Scalable architecture using Redis as backend database using Golang + Redis

3DS INTERACTIVE 6 Sep 23, 2022
The 'Scalable' URL shortener

Chinnaswamy The 'Scalable' URL shortener Running tests To run end to end tests: CHINNASWAMY_TEST_HOST="<host:port>" go test -count=1 ./integration_tes

nilenso 2 May 21, 2022
Simple Bank is a simple REST API that allows users to perform transferences with each other.

Simple Bank is a simple REST API that allows users to perform transferences with each other. ?? Technologies Golang Docker PostgreSQ

Matheus Mosca 12 Feb 15, 2022
A Golang REST API to handle users and posts for a simple instagram backend. Uses MongoDB as the database. Tested using golang-testing and Postman.

A Golang REST API to handle users and posts for a simple instagram backend. Uses MongoDB as the database. Tested using golang-testing and Postman.

Nitin Narayanan 1 Oct 10, 2021
Remark42 is a self-hosted, lightweight, and simple comment engine

Remark42 is a self-hosted, lightweight, and simple (yet functional) comment engine, which doesn't spy on users. It can be embedded into blogs, articles or any other place where readers add comments.

Umputun 4k Nov 28, 2022
The cider is a simple tool of building GitHub pages. It's fast and easy to use.

The cider is a simple tool of building GitHub pages. It's fast and easy to use. See example: https://www.leyafo.com Install Compiling from source code

李亚夫 49 Feb 13, 2022
Analytics box a simple and privacy focused analytics tool written in go like google analytics

Analytics box is analytics tool like google analytics but instead of ripping user's privacy off them like google it respects their privacy and doesn't collect any unnecessary information.

Ketan Iralepatil 26 Nov 27, 2022
A simple and privacy focused analytics tool written in go.

Analytics Box Hello guys, this is privacy friendly analytics tool, Analytics Box for web written in go. It ensures user privacy but at the same time a

Ketan Iralepatil 26 Nov 27, 2022
Simple vanilla Go CRUD application with mongoDB database with its mflix dataset that I use for my thesis about benchmarking REST API and GraphQL.

Go GraphQL Simple vanilla Go CRUD application with mongoDB database with its mflix dataset that I use for my thesis about benchmarking REST API and Gr

Adrian Edy Pratama 0 Oct 15, 2021
Simple web app using Go and Gin framework

go-gin-app Simple web app using Go and Gin framework Golang 과 Gin 프레임워크를 사용한 간단한 웹 앱 How to get Started Install Gin and have Go installed on your syst

Sean Hong(홍성민) 0 Oct 18, 2021
A simple demo written in Golang to shorten URL and redirect

url-shortener A simple demo written in Golang to shorten URL and redirect . ├── go.mod ├── go.sum ├── handler │ └── handlers.go ├── main.go ├── shor

lunarwhite 1 Jun 5, 2022
A simple RESTful API for storing and retrieving data during MRI image reconstructions

MRD Storage Server The MRD Storage Server provides a simple RESTful API for storing and retrieving data during MRI image reconstructions. It supports:

null 0 Mar 18, 2022
Simple control panel for Golang based on Gin framework and MongoDB

Summer panel Simple control panel for Golang based on Gin framework and MongoDB How To Install go install github.com/night-codes/summer/[email protected]

Oleksiy Chechel 6 Aug 1, 2022
A simple app that reads NBMiner status REST API data and sends it to InfluxDB

NBMiner Reporter A simple Go app that reads NBMiner status REST API data and sends it to InfluxDB. Usage Using the reporter is quite easy, specially i

null 3 Feb 8, 2022
Simple Todolist using Golang IO and Mysql

Simple Todolist using Golang IO and Mysql HOW TO USE create 2 database with name : todolist for product todolist_test for test create table in both da

Raden Mohamad Rishwan 0 Nov 22, 2021
Go-service-gin - Simple Web api application developed in Golang and Gin

Simple Web api application developed in Golang and Gin Initial Tutorial URL http

Nurul Huda Robin 0 Jan 4, 2022
A simple CRUD todo list application that is powered by Docker and Go

A simple CRUD todo list application that is powered by Docker and Go. Docker compose will run two containers todolist and mysql.

Marco Canchola 0 Jan 18, 2022