Modular Kubernetes operator to manage the lifecycle of databases

Overview

Ensemble

Ensemble is a simple and modular Kubernetes Operator to manage the lifecycle of a wide range of databases.

  • Infrastructure as code with Kubernetes CRDs to define clusters (number of nodes, segments...) and their resources (users, permissions).

  • Modular design to support a wide range of databases. Currently available: Cassandra, Rabbitmq and Zookeeper.

  • Provision and dynamic scaling of the clusters.

Read the documentation to learn more https://ensembleoss.io

Issues
  • Refactor state

    Refactor state

    This PR introduces a big refactor change for the state. Before, K8s api server was the state where Ensemble would store internal information like for example Nodes in a cluster. However, it was a bit limiting in terms of how complex the state could be. Now, the Operator uses a boltdb file to store the state and it is the main interface that the Operator uses to reconcile. Then, the K8s provider updates that state with changes in the CRDs.

    opened by ferranbt 2
  • Load instance from state by id

    Load instance from state by id

    Before, the operator would need both the deployment id and the instance id to get the reference of the instance. Then, there was always the need to keep both values together available at the same time on the different async ops (i.e. Provider). Now, it only needs the id to get the reference of the instance.

    opened by ferranbt 1
  • Fix too many evals

    Fix too many evals

    The K8s provider was updating the instance every time there was a Pod event update but we only need to do it for state changes that might create a reconciliation. This PR filters the watch events and only updates when the state changes either to Running or Stopped.

    opened by ferranbt 1
  • Add single stopped state + desiredState

    Add single stopped state + desiredState

    This PR does two things:

    • Closes #72 and only uses one state to represent that the instance is not running anymore.
    • Uses a Pod watcher instead of an Event watcher to track pod changes.
    opened by ferranbt 1
  • Add control plane

    Add control plane

    This PR introduces the control plane interface for the Operator. Both Providers like Kubernetes/Docker and backends (databases) use the interface to listen for instance updates and act accordingly. Thus, we can safely remove the synchronous Create/Delete Resource methods in the Provider which were not that flexible.

    opened by ferranbt 1
  • Add readiness hook

    Add readiness hook

    This PR replaces the current hardcoded readiness probe with a generic readiness probe using hooks. This is the first iteration of the hooks controller. Later on we will expand the concept to include not just post-hooks like readiness but also pre-hooks before removing a node.

    opened by ferranbt 1
  • Feature backend test

    Feature backend test

    This PR does two things:

    • Moves some of the logic to schedule clusters to his own method Scheduler.
    • Uses the new scheduler to test the backends without a running docker instance. Closes #40.
    opened by ferranbt 1
  • Use Sqlite3 as a database

    Use Sqlite3 as a database

    Right now Ensemble uses boltdb as a backend. It is nice because it is embedded and does not require any external service to run it. However, it is hard to build complex database layouts on top without reinventing indexes every time.

    This issue proposes to replace boltdb with sqlite3 which is also embedded and it gives us more flexibility since it uses SQL. I think the only problem is that it requires cgo to run.

    opened by ferranbt 0
  • Add plan command

    Add plan command

    With the new scheduler we can generate Plans for the specs without executing them. We can use that feature to include a plan Grpc and cli endpoint to retrieve information and annotations about the execution plan.

    cli 
    opened by ferranbt 0
  • Hook support

    Hook support

    There are some use cases that require generic pre/post hook support:

    • Redis "join" command to create the cluster.
    • Elasticsearch drain the nodes during scale down.
    • Clickhouse replica distributed tables during scale up.

    Even the generic concept of Readiness (after the start) could be modeled as a post hook. A hook should be a one-off operation. Long running jobs like monitoring are not applied here.

    core 
    opened by ferranbt 1
  • Use default value in all the schema configs

    Use default value in all the schema configs

    In #45 we included support for a default value in the schema. Right now, the default value is used only for the default params, we should apply the default value to all the schema config items (resources, storage, node and database config).

    opened by ferranbt 0
  • Add command to return the list of specs per deployment

    Add command to return the list of specs per deployment

    There are two types of CRD specs that belong to a deployment: cluster and resource. We should be able to return for a specific deployment his specs alongside information of their state (completed, pending, canceled...).

    This change requires the following changes:

    • State: We need to add an index to keep track an index of spec -> deployment.
    • Grpc: Add a Grpc endpoint.
    • Cli: Add the command (name TBD).
    cli 
    opened by ferranbt 0
  • Cluster entrypoint

    Cluster entrypoint

    Once we know the liveness of the nodes in the cluster we need to create a load balancer entrypoint to access the cluster. We might need to use envoy since I do not know if we can change with an API the status of the nodes in the builtin Service K8s stanza.

    enhancement 
    opened by ferranbt 0
Owner
Tesera
Data Platform as a Service
Tesera
Kubegres is a Kubernetes operator allowing to create a cluster of PostgreSql instances and manage databases replication, failover and backup.

Kubegres is a Kubernetes operator allowing to deploy a cluster of PostgreSql pods with data replication enabled out-of-the box. It brings simplicity w

Reactive Tech Ltd 1k Aug 11, 2022
Basic Kubernetes operator that have multiple versions in CRD. This operator can be used to experiment and understand Operator/CRD behaviors.

add-operator Basic Kubernetes operator that have multiple versions in CRD. This operator can be used to experiment and understand Operator/CRD behavio

Dinesh Parvathaneni 0 Dec 15, 2021
Helm Operator is designed to managed the full lifecycle of Helm charts with Kubernetes CRD resource.

Helm Operator Helm Operator is designed to install and manage Helm charts with Kubernetes CRD resource. Helm Operator does not create the Helm release

Chen Zhiwei 5 Jul 9, 2022
An operator which complements grafana-operator for custom features which are not feasible to be merged into core operator

Grafana Complementary Operator A grafana which complements grafana-operator for custom features which are not feasible to be merged into core operator

Snapp Cab Incubators 6 Jul 18, 2022
kube-champ 39 Aug 6, 2022
Addon Operator coordinates the lifecycle of Add-ons in managed OpenShift

Addon Operator Addon Operator coordinates the lifecycle of Addons in managed OpenShift. dev tools setup pre-commit hooks: make pre-commit-install glob

OpenShift 13 Aug 1, 2022
The OCI Service Operator for Kubernetes (OSOK) makes it easy to connect and manage OCI services from a cloud native application running in a Kubernetes environment.

OCI Service Operator for Kubernetes Introduction The OCI Service Operator for Kubernetes (OSOK) makes it easy to create, manage, and connect to Oracle

Oracle 22 Jun 17, 2022
PolarDB-X Operator is a Kubernetes extension that aims to create and manage PolarDB-X cluster on Kubernetes.

GalaxyKube -- PolarDB-X Operator PolarDB-X Operator is a Kubernetes extension that aims to create and manage PolarDB-X cluster on Kubernetes. It follo

null 63 Jul 27, 2022
The Oracle Database Operator for Kubernetes (a.k.a. OraOperator) helps developers, DBAs, DevOps and GitOps teams reduce the time and complexity of deploying and managing Oracle Databases

The Oracle Database Operator for Kubernetes (a.k.a. OraOperator) helps developers, DBAs, DevOps and GitOps teams reduce the time and complexity of deploying and managing Oracle Databases. It eliminates the dependency on a human operator or administrator for the majority of database operations.

Oracle 74 Aug 5, 2022
The Elastalert Operator is an implementation of a Kubernetes Operator, to easily integrate elastalert with gitops.

Elastalert Operator for Kubernetes The Elastalert Operator is an implementation of a Kubernetes Operator. Getting started Firstly, learn How to use el

null 20 Jun 28, 2022
Minecraft-operator - A Kubernetes operator for Minecraft Java Edition servers

Minecraft Operator A Kubernetes operator for dedicated servers of the video game

James Laverack 7 Jul 20, 2022
K8s-network-config-operator - Kubernetes network config operator to push network config to switches

Kubernetes Network operator Will add more to the readme later :D Operations The

Daniel Hertzberg 6 May 16, 2022
Pulumi-k8s-operator-example - OpenGitOps Compliant Pulumi Kubernetes Operator Example

Pulumi GitOps Example OpenGitOps Compliant Pulumi Kubernetes Operator Example Pr

Christian Hernandez 3 May 6, 2022
Kubernetes Operator Samples using Go, the Operator SDK and OLM

Kubernetes Operator Patterns and Best Practises This project contains Kubernetes operator samples that demonstrate best practices how to develop opera

International Business Machines 19 Jul 28, 2022
A Kubernetes operator to manage ThousandEyes tests

ThousandEyes Kubernetes Operator ThousandEyes Kubernetes Operator is a Kubernetes operator used to manage ThousandEyes Tests deployed via Kubernetes c

Cisco DevNet 29 Jul 18, 2022
Cloudflare-operator - Manage Cloudflare DNS records with Kubernetes objects

cloudflare-operator Documentation The goal of cloudflare-operator is to manage C

containeroo 12 Jun 15, 2022
Kubernetes Admission Controller Demo: Validating Webhook for Namespace lifecycle events

Kubernetes Admission Controller Based on How to build a Kubernetes Webhook | Admission controllers Local Kuberbetes cluster # create kubernetes cluste

Marco Lehmann 2 Feb 27, 2022
Test Operator using operator-sdk 1.15

test-operator Test Operator using operator-sdk 1.15 operator-sdk init --domain rbt.com --repo github.com/ravitri/test-operator Writing kustomize manif

Ravi Trivedi 0 Dec 28, 2021
a k8s operator 、operator-sdk

helloworld-operator a k8s operator 、operator-sdk Operator 参考 https://jicki.cn/kubernetes-operator/ https://learnku.com/articles/60683 https://opensour

Mark YiL 0 Jan 27, 2022