Go library to create resilient feedback loop/control controllers.

Overview

gontroller

Gontroller Build Status Go Report Card GoDoc

A Go library to create feedback loop/control controllers, or in other words... a Go library to create controllers without Kubernetes resources.

Current state

Alpha state

Introduction

Kubernetes controllers/operators are based on controller pattern a.k.a reconciliation loop, this pattern is based on maintaining the desired state and self-heal if it is required. It's a very resilient and robust pattern to maintain and automate tasks.

Kubernetes operators and controllers are awesome but sometimes we want to create them with resources that are not Kubernetes objects/resources (example: Github repositories, Vault secrets, Slack channels...) or we want to convert these objects to Kubernetes CRDs automatically... or we don't use Kubernetes at all but we want to create controllers and operators in a similar way...

Gontroller let's you apply this pattern on non-kubernetes applications. You don't need to use a Kubernetes apiserver to subscribe for events, CRDs... to create an operator/controller, you can use simple structs as objects and you will only need to implement a few interfaces to use this pattern.

Features

Gontroller is mainly inspired by Kubernetes controllers design and client-go internal library implementation with a simplified design and some changes to be more flexible and easy to use. The main features are:

  • No Kubernetes dependency.
  • Easy to create controllers and operators.
  • Automatic retries.
  • Ensure only one worker is handling a same object (based on ID) at the same time.
  • An object to be processed will be only once on the processing queue.
  • Handle all objects at regular intervals (for reconciliation loop) and updated objects on real-time.
  • Metrics and Prometheus/Openmetrics implementation.
  • Extensible, all is based on behavior (go interfaces) and not concrete types.
  • Easy to test, business logic not coupled with infrastructure code (controller implementation/library)

Getting started

Run the example...

go run github.com/spotahome/gontroller/examples/stub-funcs-controller

And check the examples folder to get an idea of how you could create a controller.

How does it work

The controller is composed by 3 main components:

  • ListerWatcher: This piece is the one that will provide the object IDs to the controller queue. Its composed by two internal pieces, the List, that will list all object IDs in constant intervals (reconciliation) and the Watch that will receive object events (create, modify, delete...).
  • Storage: The storage is the one that know how to get the object based on the ListerWatcher enqueued ID and the controller will call this store just before calling the Handler.
  • Handler: The handler will handle the Add (exists) and Delete (doesn't exist) objects queued by the ListerWatcher.

The controller will call the ListerWatcher.List method every T interval (e.g. 30s) to enqueue the IDs to process and the ListerWatcher.Watch will enqueue real time events to be processed (so there is no need to wait for next List iteration).

The controller will be dequeueing from the queue the IDs to process them but before passing to the workers it will get the object to process from the Storage, after getting the object it will call one of the workers to handle it using the Handler.

Internal architecture

  1. The ListerWatcher will return the object IDs to handle.
    1. The List will be called in interval loop and call the enqueuer with all the listed IDs.
    2. The Watch will return through the channel real-time events of the IDs and the enqueuer will be listening on the channel.
  2. The enqueuer will enqueue the object if necessary
    1. The enqueue will check if the ID is already on the internal state cache. If is there it will not send the ID to the queue, if its not there it will send to the controller queue.
    2. It will save new state (present or deleted) of the ID on the internal state cache.
  3. The queue has FIFO priority.
  4. The worker will process the queued IDs
    1. Will try acquiring the lock using the ObjectLocker. If not acquired means is already being handled by another worker and it will ignore the ID and return to the worker pool. On the contrary, if acquired, it will continue with the processing.
    2. Will get the latest state of the object form the State cache.
    3. If state is not deleted it will get the object data from the Storage based on the ID.
    4. Will call the worker handler passing the object data obtained from the Storage if the latest state is present using Handler.Add, on the contrary if the latest state is missing, it will call Handler.Delete.
    5. The Handler will execute the business logic on the object.
    6. When the Handler finishes it will release the lock of the ID.

FAQ

Why not use a simple loop?

A loop is simple and in lot of cases is good enough, the problem is that is not reliable, on the contrary the reconciliation loop gives us a robust state by handling all objects in regular intervals and it acts on real time events, gontroller has also retries, apart from that it ensures one object is only being handled concurrently by one worker and the state that is handling is the latest state of that object (in case it has been updated in the meantime it was waiting in the queue to be processed).

Why only enqueue IDs?

In the end the controller only needs a reference or a unique ID to work (queue, lock...), the business logic is on the handler and is the one that needs the object data, that's why on the step before calling the handler, the store is called to get all the data.

We could delete the Storage component and let the handler get whatever it wants based on the ID, but having a storage component it makes easier the controller to be structured and understandable. And you could make the storage be transparent and always return the received ID string in case of not wanting to use this component).

How do you ensure a object is handled by a single worker?

The controller uses an ObjectLocker interface to do this, is an optional component, by default gontroller uses a memory based locker that is only available to that controller, this locks the IDs once they are being handled and unlocks once finished.

In case you want a shared lock between instances you can implement a custom ObjectLocker, check the question in the FAQ about locking multiple instances for more information.

How do you ensure a same object is not repeated N times on the queue?

The controller uses a small cache of what's queued and if is already there it will ignore it.

Why do we need the Storage component?

We could create the ListerWatcher to return the whole object (like Kubernetes) instead of the IDs and let the controller maintain the state, but this couples us the input of the queue with the output, and not always are the same.

On the contrary Gontroller takes another design approach making the ListerWatcher only return the IDs, this way delegating the retrieval of object data to the last part of the controller (the Storage) so listing the objects to be handled don't require all the data.

ListerWatcher -> Controller -> Handler
  ^                    ^
  │                    │
  └─ Get IDs           └── Storage (Get data from ID)

Also with this design you can use the same cache for the ListerWatcher and the Storage if you want and you could obtain the same result as a kubernetes style controller/operator by getting the data, storing the data on the Storage and returning the IDs (be aware of concurrency):

ListerWatcher ----------------------------------------> Controller ----> Handler
     ^                                                       ^
     │                                                       │
     └── (Get IDs from data) ───> Storage ── (Get data) ────-┘

Can I interact with Kubernetes?

Although you have read on this readme "without Kubernetes", Gontroller handler is not coupled with anything so you could interact with Kubernetes. A very powerful kind of controller could be to convert no CRDs into CRDs.

For example, an organization in Github has Git repositories with a file named info.json, this file has info about the repository source code and we have a CRD on Kubernetes that is filled with this info (and some Kubernetes operators that do actions based on this CRD), previously the CRD was made manually, now we will create a controller using Gontroller to convert Git repositories to CRDs based on the content of that file:

  • ListerWatcher: Lists the Github organization repositories.
  • Store: Gets a Github repo file content using the ID (eg github.com/spotahome/gontroller) and returns an object with all the data.
  • Handler: Receives the Storage returned object (has the data of the required repo file) and converts to CRD and applies then on Kubernetes.

Does the lock of object handling work with independent instances?

Gontroller by default will ensure only a same object (same == same ID) is processed by a worker at the same time, but this is guaranteed only on the same controller instance.

But this can be achieved implement with a custom controller.ObjectLocker interface:

type ObjectLocker interface {
  Acquire(id string) bool
  Release(id string)
}

An example to allow sharing this lock by multiple instances at the same time, could be implementing a lock that uses a shared Redis by all the instances that is used as the locker.

Check this simple example.

Issues
  • Allow ObjectLocker configuration on controller

    Allow ObjectLocker configuration on controller

    This PR Allows configuring a custom ObjectLocker on the controller, to do this we did:

    • Expose ObjectLocker as a public interface.
    • Create an example of how to use it.
    • Added to Readme and Changelog.
    opened by slok 0
  • Add retries to the controller logic

    Add retries to the controller logic

    This PR adds retries to the controller (and the option to set on the controller configuration the number of max retries).

    • Having to refactor the internal caches, the lock object has decoupled in a private interface (in the future we could make it public so users could implement an own lock of objects, for example using a Redis, this way multiple instances of an operator could share the lock of handling objects).
    • Fixes the race conditions on tests.
    • Improve tests.

    Question... What do you see of adding brigade or wait until we release this as OSS and add Travis? That's the main reason why CI is not set.

    opened by slok 0
Releases(v0.1.0)
  • v0.1.0(Sep 6, 2019)

    0.1.0 - 2019-09-06

    Added

    • Allow configuring ObjectLocker on controller.
    • Add retries to the controller.
    • Add Prometheus recorder implementation.
    • Add Metrics recorder.
    • Add Controller.
    • Add ListerWatcher.
    • Add Storage.
    • Add Handler.

    Changed

    • Store and ListerWatcher now receive a context.
    Source code(tar.gz)
    Source code(zip)
Write controller-runtime based k8s controllers that read/write to git, not k8s

Git Backed Controller The basic idea is to write a k8s controller that runs against git and not k8s apiserver. So the controller is reading and writin

Darren Shepherd 50 Dec 10, 2021
ControllerMesh is a solution that helps developers manage their controllers/operators better.

ControllerMesh ControllerMesh is a solution that helps developers manage their controllers/operators better. Key Features Canary update: the controlle

OpenKruise 20 Jun 1, 2022
Golang Integration Testing Framework For Kong Kubernetes APIs and Controllers.

Kong Kubernetes Testing Framework (KTF) Testing framework used by the Kong Kubernetes Team for the Kong Kubernetes Ingress Controller (KIC). Requireme

Kong 14 Jun 7, 2022
Controller-check - Run checks against K8s controllers to verify if they meets certain conventions

controller-check Run checks against K8s controllers to verify if they meets cert

Sunny 2 Jan 4, 2022
In this repository, the development of the gardener extension, which deploys the flux controllers automatically to shoot clusters, takes place.

Gardener Extension for Flux Project Gardener implements the automated management and operation of Kubernetes clusters as a service. Its main principle

23 Technologies GmbH 11 May 19, 2022
Resilient, scalable Brainf*ck, in the spirit of modern systems design

Brainf*ck-as-a-Service A little BF interpreter, inspired by modern systems design trends. How to run it? docker-compose up -d bash hello.sh # Should p

Serge Zaitsev 138 Apr 22, 2022
Go binding for rpi-rgb-led-matrix an excellent C++ library to control RGB LED displays with Raspberry Pi GPIO.

go-rpi-rgb-led-matrix Go binding for rpi-rgb-led-matrix an excellent C++ library to control RGB LED displays with Raspberry Pi GPIO. This library incl

Máximo Cuadros 65 Jun 6, 2022
Lightweight, CRD based envoy control plane for kubernetes

Lighweight, CRD based Envoy control plane for Kubernetes: Implemented as a Kubernetes Operator Deploy and manage an Envoy xDS server using the Discove

null 48 Mar 19, 2022
This project for solving the problem of chaincode being free from k8s control

Peitho - Hyperledger Fabric chaincode Cloud-native managed system The chaincode of Hyperledger Fabric can be handed over to k8s for management, which

null 5 Dec 16, 2021
Asynchronously control the different roles available in the kubernetes cluster

RBAC audit Introduction This tool allows you to asynchronously control the different roles available in the kubernetes cluster. These audits are enter

null 0 Oct 19, 2021
Working towards a control plane for the MiCo Tool and the MiCoProxy

A simple control plane for MiCo This is still largely a work in progress The overall idea is to build a kubernetes DaemonSet that watches kubernetes s

null 0 May 4, 2022
L3AFD kernel function control plane

L3AFD: Lightweight eBPF Application Foundation Daemon L3AFD is a crucial part of the L3AF ecosystem. For more information on L3AF see https://l3af.io/

L3AF 107 Jun 16, 2022
⚡️ Control plane management agent for FD.io's VPP

VPP Agent The VPP Agent is a Go implementation of a control/management plane for VPP based cloud-native Virtual Network Functions (VNFs). The VPP Agen

EMnify 0 Aug 3, 2020
Pulumi-awscontroltower - A Pulumi provider for AWS Control Tower

Terraform Bridge Provider Boilerplate This repository contains boilerplate code

Lee Briggs 1 Apr 16, 2022
Sapfun - Utility that takes control over your video card coolers to keep it cool and steady

What? sapfun - Utility that takes control over your video card coolers to keep i

Vadim Kuznetsov 1 Feb 18, 2022
A binary to control the Z-Cam line of cameras via API

The Z-Cam flagship line has an API of sorts. This can be used to control the camera--via a StreamDeck, say. This seems like a good enough reason to me

Corey Quinn 10 Jun 22, 2022
Testcontainers is a Golang library that providing a friendly API to run Docker container. It is designed to create runtime environment to use during your automatic tests.

When I was working on a Zipkin PR I discovered a nice Java library called Testcontainers. It provides an easy and clean API over the go docker sdk to

null 1.4k Jun 26, 2022
Create changelogs for Helm Charts, based on git history

helm-changelog Create changelogs for Helm Charts, based on git history. The application depends on the assumption that the helm chart is released on t

Frederik Mogensen 35 Jun 3, 2022