Cheiron is a Kubernetes Operator made with OperatorSDK for reconciling service account and attaching imagePullSecrets to service accounts automatically

Overview

anny-co/cheiron

NOTE: Cheiron is currently in very early stages of development and and far from anything usable. Feel free to contribute if you want to, an(n)y contributions are welcome!

Cheiron is a Kubernetes Operator built from the Operator SDK and written in Go. It maintains imagePullSecrets, both on ServiceAccounts and PodSpecs, cluster-wide or only namespaced.

Introduction

Why build an Operator in the first place?

Becase adding the same imagePullSecret to all Pods or ServiceAccounts added to possibly multiple clusters is tedious for our engineers and we want to automate this task as much as possible. Additionally, we wanted to experiment with Kubernetes Operators and this seemed like a reasonable idea to realize as an operator.

Why is Cheiron necessary?

Because still, many deployments, Helm Charts, or static manifests use Docker Hub as the container image source. However, due to several licence changes, Docker Hub has become heavily rate-limited, which is a particular problem if the cluster is located behind a singular NAT, i.e., all kubelets share the same external IP address. In our tests, you exceed this rate-limit already by only deploying a basic set of tools into a fresh cluster and this hinders progress massively.

When you cannot simply assign public IP addresses to the cluster nodes, or you want to entirely omit the rate-limit by using credentials to authenticate the node to Docker Hub (or any other registry for that matter!), you're required to provide ImagePullSecrets for each pod you deploy. This can either be done directly on the PodSpec or on the ServiceAccount that is attached to the Pod by Kubernetes.

By separation of concern, many Helm charts deploy their own service account for the release. This then requires repeated addition of the imagePullSecret to the ServiceAccount or the Pod (we think that the former one is more scalable, though). Doing this by hand however is massively time-consuming, even when using tools such as Kustomize to apply patches to resources automatically.

Note that it is also possible to attach the credentials directly to the node running the kubelet. While for high-maintenance clusters or self-provisioned nodes this might be feasible, in our scenario, where we have (potentially a lot of) dynamically provisioned nodes without tools such as Ansible in the process, we want to perform this inclusion of credentials as little invasive into the node as possible. Hence we do it directly in Kubernetes.

But is this actually good practice?

The quick disclaimer: Probably not!. When using the namespaced version, things are slightly less anti-pattern prone. The operator runs on a opt-out basis, i.e., when not explicitly flagged, each Pod or ServiceAccount will get augmented with all secrets that the CR creator specified. Kubernetes handles the secrets, but inlining the secret into the CR will store the credentials in plain-text in etcd when not encrypting all data in etcd at rest.

When using a cluster-scoped version, the operator will create secrets for all specified registry credentials in ALL namespaces, i.e., will inject likely private data into all namespaces. This is definitly not feasible in clusters with multiple tenants, and should only be used in scenarios where the entire cluster is a trusted environment.

In our scenario, this is the case, hence we wrote this operator. We needed to automate as much of the tedious work of adding explicit pull secrets to public registries as possible. You should probably also make a careful evaluation before going ahead and adding the operator to your cluster.

Usage

Cheiron follows the latest Operator SDK design, i.e., it is scaffolded from the CLI and automatically build and deployed using the SDK's CLI.

Additionally, we ship static deployment manifests:

kubectl apply -f config/manifests/

After that, the cluster knows the CRDs that Cheiron manages; you can then go ahead and add your image registry credentials in a CR such as

apiVersion: cheiron.anny.co/v1alpha1
kind: ImagePullSecretManager
metadata:
  name: my-registry
spec:
  mode: ServiceAccount # or Pod
  secrets:
    - name: docker-hub
      registry: https://index.docker.io/v1
      username: <my-docker-username>
      password: <my-docker-password>
      email: <my-docker-email>
    - name: github-container-registry
      registry: ghcr.io
      username: <my-github-username>
      password: <my-github-access-token>
      email: <my-github-email>
    - existingSecretRef:
        name: gitlab-registry
    - existingSecretRef:
        name: <my-registry>

The controller for those resources reconciles the resource spec by either creating a kubernetes.io/dockerconfigjson secret or updating an existing one to contain the specified credentials. Then, it collects all the names of the created/updated secrets, and marks either all ServiceAccounts or Pods in its scope (either namespaced or cluster-scoped) reconcilable using annotations:

metadata:
  annotations:
    cheiron.anny.co/reconcilable: "true"
    cheiron.anny.co/reconcile-with: "secret-name-a,secret-name-b,..."

Note that this is only done on ressources that do not yet have the annotation or those who don't have the annotation set to false or do not have the specific annotation cheiron.anny.co/ignore: "true". The controller also adds an annotation that contains all the names of the fresh secrets, s.t. we can construct a list of LocalObjectReferences on either the PodSpec.ImagePullSecrets or ServiceAccount.ImagePullSecrets.

The PodController and ServiceAccountController perform similar operations on their respective resources: As the first controller adds the annotations from above, their (the pod controller's and service account controller's) reconciliation loop picks up the annotated resource and adds the inlined image pull secrets to the specs. They finish their operations by annotating their ressource with cheiron.anny.co/reconciled: "true". This allows the controllers to filter what resources they already worked.

You might also like...
Kubernetes Operator to sync secrets between different secret backends and Kubernetes

Vals-Operator Here at Digitalis we love vals, it's a tool we use daily to keep secrets stored securely. We also use secrets-manager on the Kubernetes

Test Operator using operator-sdk 1.15

test-operator Test Operator using operator-sdk 1.15 operator-sdk init --domain rbt.com --repo github.com/ravitri/test-operator Writing kustomize manif

a k8s operator 、operator-sdk

helloworld-operator a k8s operator 、operator-sdk Operator 参考 https://jicki.cn/kubernetes-operator/ https://learnku.com/articles/60683 https://opensour

The NiFiKop NiFi Kubernetes operator makes it easy to run Apache NiFi on Kubernetes.
The NiFiKop NiFi Kubernetes operator makes it easy to run Apache NiFi on Kubernetes.

The NiFiKop NiFi Kubernetes operator makes it easy to run Apache NiFi on Kubernetes. Apache NiFI is a free, open-source solution that support powerful and scalable directed graphs of data routing, transformation, and system mediation logic.

A simple program to automatically backup a database using git. Err handling by Sentry, Reporting by Betteruptime. Made with 🩸 , 😓 & 😭

backup What is this? A Simple program to automatically backup a database using git. Err handling by Sentry, Uses heartbeats by Betteruptime Made with

Just a dummy Kubernetes Operator, playing with another dummy service

My first operator Just playing/learning to create a K8S operator in go. I will create a dummy operator that creates pods to open a shell inside It is

The AWS Enumerator was created for service enumeration and info dumping for investigations of penetration testers during Black-Box testing. The tool is intended to speed up the process of Cloud review in case the security researcher compromised AWS Account Credentials.
The Oracle Database Operator for Kubernetes (a.k.a. OraOperator) helps developers, DBAs, DevOps and GitOps teams reduce the time and complexity of deploying and managing Oracle Databases

The Oracle Database Operator for Kubernetes (a.k.a. OraOperator) helps developers, DBAs, DevOps and GitOps teams reduce the time and complexity of deploying and managing Oracle Databases. It eliminates the dependency on a human operator or administrator for the majority of database operations.

Kubegres is a Kubernetes operator allowing to create a cluster of PostgreSql instances and manage databases replication, failover and backup.

Kubegres is a Kubernetes operator allowing to deploy a cluster of PostgreSql pods with data replication enabled out-of-the box. It brings simplicity w

Owner
anny
anny
An operator which complements grafana-operator for custom features which are not feasible to be merged into core operator

Grafana Complementary Operator A grafana which complements grafana-operator for custom features which are not feasible to be merged into core operator

Snapp Cab Incubators 6 Aug 16, 2022
kube-champ 43 Oct 19, 2022
Kubernetes Operator Samples using Go, the Operator SDK and OLM

Kubernetes Operator Patterns and Best Practises This project contains Kubernetes operator samples that demonstrate best practices how to develop opera

International Business Machines 26 Oct 31, 2022
The Elastalert Operator is an implementation of a Kubernetes Operator, to easily integrate elastalert with gitops.

Elastalert Operator for Kubernetes The Elastalert Operator is an implementation of a Kubernetes Operator. Getting started Firstly, learn How to use el

null 20 Jun 28, 2022
Minecraft-operator - A Kubernetes operator for Minecraft Java Edition servers

Minecraft Operator A Kubernetes operator for dedicated servers of the video game

James Laverack 8 Oct 3, 2022
K8s-network-config-operator - Kubernetes network config operator to push network config to switches

Kubernetes Network operator Will add more to the readme later :D Operations The

Daniel Hertzberg 6 May 16, 2022
Pulumi-k8s-operator-example - OpenGitOps Compliant Pulumi Kubernetes Operator Example

Pulumi GitOps Example OpenGitOps Compliant Pulumi Kubernetes Operator Example Pr

Christian Hernandez 3 May 6, 2022
The OCI Service Operator for Kubernetes (OSOK) makes it easy to connect and manage OCI services from a cloud native application running in a Kubernetes environment.

OCI Service Operator for Kubernetes Introduction The OCI Service Operator for Kubernetes (OSOK) makes it easy to create, manage, and connect to Oracle

Oracle 24 Sep 27, 2022
Operator Permissions Advisor is a CLI tool that will take a catalog image and statically parse it to determine what permissions an Operator will request of OLM during an install

Operator Permissions Advisor is a CLI tool that will take a catalog image and statically parse it to determine what permissions an Operator will request of OLM during an install. The permissions are aggregated from the following sources:

International Business Machines 2 Apr 22, 2022
PolarDB-X Operator is a Kubernetes extension that aims to create and manage PolarDB-X cluster on Kubernetes.

GalaxyKube -- PolarDB-X Operator PolarDB-X Operator is a Kubernetes extension that aims to create and manage PolarDB-X cluster on Kubernetes. It follo

null 63 Nov 3, 2022