Kubernetes Operator to sync secrets between different secret backends and Kubernetes

Overview

Vals-Operator

Here at Digitalis we love vals, it's a tool we use daily to keep secrets stored securely. We also use secrets-manager on the Kubernetes deployment we manage. Inspired by these two wonderful tools we have created this operator.

vals-operator syncs secrets from any secrets store supported by vals into Kubernetes. It works very similarly to secrets-manager and the code is actually based on it. Where they differ is that it not just supports HashiCorp Vault but many other secrets stores.

Installation

You can use the helm chart to install vals-operator. You will need to provide the configuration to access the secrets store you decided on via either environment variables pre existing secrets.

# Example for Vault
helm upgrade --install vals-operator --create-namespace -n vals-operator \
  --set "env[0].name=VAULT_ROLE_ID,env[0].value="vals-operator"" \
  --set "env[1].name=VAULT_SECRET_ID,env[1].value="my-secret-id"" \
  --set "env[2].name=VAULT_ADDR,env[2].value=https://vault:8200"
  charts/vals-operator

# Example for AWS using a secret
kubectl create secret generic -n vals-operator aws-creds \
  --from-literal=AWS_ACCESS_KEY_ID=foo \
  --from-literal=AWS_SECRET_ACCESS_KEY=bar \
  --from-literal=AWS_DEFAULT_REGION=us-west-2

helm upgrade --install vals-operator --create-namespace -n vals-operator \
  --set "secretEnv[0].secretRef.name=aws-creds"  \
  charts/vals-operator

Usage

apiVersion: digitalis.io/v1
kind: ValsSecret
metadata:
  name: vals-secret-sample
  labels:
    owner: digitalis.io
spec:
  name: my-secret # Optional, default is the resource name
  ttl: 3600       # Optional, default is 0. The secret will be checked at every "reconcile period". See below.
  type: Opaque    # Default type, others supported
  data:
    username:
      ref: ref+vault://secret/database/username
      encoding: text
    password:
      ref: ref+vault://secret/database/password
      encoding: text
    ssh:
      ref: ref+vault://secret/database/ssh-private-key
      encoding: base64
    aws-user:
      ref: ref+awssecrets://kube/test#username
    aws-pass:
      ref: ref+awssecrets://kube/test#password

The example above will create a secret named my-secret and get the values from the different sources. The secret will be kept in sync against the backed secrets store.

The TTL is optional and used to decrease the number of times the operator calls the backend secrets store as some of them such as AWS Secrets Manager will incur a cost.

The default encoding is text but you can change it to base64 per secret reference. This way you can, for example, base64 encode large configuration files.

Options

The following options are available. See the helm chart documentation for more information on adding them to your deployment configuration.

  -exclude-namespaces string
    	Comma separated list of namespaces to ignore.
  -health-probe-bind-address string
    	The address the probe endpoint binds to. (default ":8081")
  -kubeconfig string
    	Paths to a kubeconfig. Only required if out-of-cluster.
  -leader-elect
    	Enable leader election for controller manager. Enabling this will ensure there is only one active controller manager.
  -metrics-bind-address string
    	The address the metric endpoint binds to. (default ":8080")
  -reconcile-period duration
    	How often the controller will re-queue secretdefinition events (default 5s)
  -record-changes
    	Records every time a secret has been updated. You can view them with kubectl describe (default true)
  -watch-namespaces string
    	Comma separated list of namespaces that vals-operator will watch.
  -zap-devel
    	Development Mode defaults(encoder=consoleEncoder,logLevel=Debug,stackTraceLevel=Warn). Production Mode defaults(encoder=jsonEncoder,logLevel=Info,stackTraceLevel=Error) (default true)
  -zap-encoder value
    	Zap log encoding (one of 'json' or 'console')
  -zap-log-level value
    	Zap Level to configure the verbosity of logging. Can be one of 'debug', 'info', 'error', or any integer value > 0 which corresponds to custom debug levels of increasing verbosity
  -zap-stacktrace-level value
    	Zap Level at and above which stacktraces are captured (one of 'info', 'error', 'panic').
Issues
  • Fix Helm chart kubeVersion comparison error

    Fix Helm chart kubeVersion comparison error

    Fixes this error when trying to install the chart:

    $ helm upgrade --install vals-operator ./vals-operator-0.5.0.tgz Release "vals-operator" does not exist. Installing it now. Error: chart requires kubeVersion: >= 1.19 which is incompatible with Kubernetes v1.21.5-eks-bc4871b

    The issue and solution is described here: https://github.com/helm/helm/issues/3810

    In addition, the duplicate kubeVersion field was removed.

    opened by irasnyd 3
  • Permit combing secrets with raw data

    Permit combing secrets with raw data

    For example,

    ---
    apiVersion: digitalis.io/v1
    kind: ValsSecret
    metadata:
      name: my-secret
    spec:
      ttl: 3600       # Optional, default is 0. The secret will be checked at every "reconcile period". See below.
      type: Opaque    # Default type, others supported
      data:
        username:
          ref: ref+vault://secret/username
          encoding: text
        password:
          ref: ref+vault://secret/password
          encoding: text
        server:
          plain: my-server.com
    
    documentation enhancement 
    opened by digiserg 2
  • Don't log

    Don't log "updated" messages or create Events if the secret has not changed

    Currently whenever we check a secret we log the message "Secret created or updated" and we write an Event to K8s even if the secret did not change. We should stop logging that message (possibly replaced with a "secret has not changed" message) and only create the Event if the secret was changed.

    bug 
    opened by rgooding 2
  • Issue with digitalisdocker/vals-operator:v0.6.0

    Issue with digitalisdocker/vals-operator:v0.6.0

    Hi, trying install latest version with helm get an issue bellow:

    Failed to pull image "digitalisdocker/vals-operator:v0.6.0": rpc error: code = Unknown desc = Error response from daemon: manifest for digitalisdocker/vals-operator:v0.6.0 not found: manifest unknown: manifest unknown

    opened by alexander-dragun 1
  • Bugfix: kubernetes login creates too many tokens

    Bugfix: kubernetes login creates too many tokens

    The environment variables VAULT_AUTH_METHOD conflicts with the underlying vals library and we were doing login all the time instead of login once and renewing the token when its about to expire.

    bug 
    opened by digiserg 0
  • Implement a better Vault token management

    Implement a better Vault token management

    Following this example, there is a better way of managing tokens:

    https://www.vaultproject.io/docs/concepts/auth#code-example

    Edit: reclassified as bug as it appears it creates new vault tokens all the time when using Kubernetes login.

    bug 
    opened by digiserg 0
  • Adds password rotation for databases

    Adds password rotation for databases

    This feature implements a new option to add a list of databases (MySQL, Cassandra or Postgres) to sync down credentials. This is intended for password rotation:

    • Admin changes password on secrets store backend (Vault, AWS, Google, etc)
    • Credentials are synced down and converted into a Kubernetes secret used by the applications
    • Database is updated with the new password
    • Applications re-read secret and continue operations

    Please be aware your application must be coded in a way it will support password rotation.

    enhancement 
    opened by digiserg 0
  • Adds helm chart repo to README

    Adds helm chart repo to README

    I've created a helm chart repo using GitHub pages where I've published the latest helm chart. This PR is to update the readm with the installation instructions.

    opened by digiserg 0
  • Allow a secret to exist on multiple namespaces

    Allow a secret to exist on multiple namespaces

    The change would allow the same secret to being synced to multiple namespaces. This can be very useful for secrets such as private docker registries. The new schema could look like the example below:

    ---
    apiVersion: digitalis.io/v1
    kind: ValsSecret
    metadata:
      name: private-registry
    spec:
      type: kubernetes.io/dockerconfigjson
      namespaces:
        - one
        - two
        - three
      data:
        auth:
          ref: "ref+vault://secret/registry/dockerconfigjson"
          encoding: text
    
    enhancement 
    opened by digiserg 0
Releases(v0.6.0)
  • v0.6.0(Mar 2, 2022)

    There is a conflict between vals and the vals-operator when setting up the VAULT_AUTH_METHOD variable causing both vals-operator and vals to perform login every time the controller refreshes a secret. This release rewrites the HaschiCorp Vault code to avoid conflict.

    What's Changed

    • Correct chart version number by @digiserg in https://github.com/digitalis-io/vals-operator/pull/29
    • Bugfix: kubernetes login creates too many tokens by @digiserg in https://github.com/digitalis-io/vals-operator/pull/32

    Full Changelog: https://github.com/digitalis-io/vals-operator/compare/v0.5.0...v0.6.0

    Source code(tar.gz)
    Source code(zip)
  • v0.5.0(Feb 16, 2022)

  • v0.4.0(Dec 15, 2021)

    This release adds password rotation to MySQL, Postgres and Cassandra. Once the password has been synced from the Secrets Store, it can be configured to update the password for a selected username.

    Source code(tar.gz)
    Source code(zip)
  • v0.3.0(Nov 16, 2021)

    vals does not yet support Kubernetes Authentication for Vault. We're adding this support to the controller. Also, it now checks whether the vault token is about to expire and renews it.

    Source code(tar.gz)
    Source code(zip)
  • v0.2.0(Nov 5, 2021)

    This release adds a new reference syntax of ref+k8s://namespace/secret#key which will allow you to grab a secret from a different namespace and keep it in sync as per defined TTL.

    Source code(tar.gz)
    Source code(zip)
  • v0.1.0(Nov 1, 2021)

Owner
digitalis.io
digitalis.io
Basic Kubernetes operator that have multiple versions in CRD. This operator can be used to experiment and understand Operator/CRD behaviors.

add-operator Basic Kubernetes operator that have multiple versions in CRD. This operator can be used to experiment and understand Operator/CRD behavio

Dinesh Parvathaneni 0 Dec 15, 2021
A k8s vault webhook is a Kubernetes webhook that can inject secrets into Kubernetes resources by connecting to multiple secret managers

k8s-vault-webhook is a Kubernetes admission webhook which listen for the events related to Kubernetes resources for injecting secret directly from sec

Opstree Container Kit 111 Apr 28, 2022
An operator which complements grafana-operator for custom features which are not feasible to be merged into core operator

Grafana Complementary Operator A grafana which complements grafana-operator for custom features which are not feasible to be merged into core operator

Snapp Cab Incubators 4 May 24, 2022
A CLI to sync configmaps and secrets in a kubernetes cluster

kube-sync Kube Sync is a CLI application to copy/sync configmaps and secrets from one namespace to another. Motivation While working with kubernetes,

Vivek Kumar Singh 5 Jun 21, 2022
Secret - Prevent your secrets from leaking into logs and std*

secret - Prevent your secrets from leaking into logs and std* The package provid

Ravi Shekhar Jethani 11 Jun 19, 2022
This repo contains example on how to consume secrets from Google Secret Manager from GKE

GKE Secret Manager. Environment setup This repo contains examples of how to consume secrets from Google Secret Manager (GSM) from Google Kubernetes En

Abdel SGHIOUAR 8 May 30, 2022
To copy a secret to another namespace and sync it up-to-date

Secret Mirror Operator This kubebuilder-based Kubernetes operator copies a Secret to another namespace and synchronizes it with the custom resource Se

 beBit, Inc. 1 Apr 4, 2022
kube-champ 37 Jun 6, 2022
A component for sync services between Nacos and Kubernetes.

简介 该项目用于同步Kubernetes和Nacos之间的服务信息。 目前该项目仅支持 Kubernetes Service -> Nacos Service 的同步 TODO 增加高性能zap的logger 增加 Nacos Service -> Kubernetes Service 的同步 监听

Nacos Group 6 May 16, 2022
Kubernetes Operator Samples using Go, the Operator SDK and OLM

Kubernetes Operator Patterns and Best Practises This project contains Kubernetes operator samples that demonstrate best practices how to develop opera

International Business Machines 16 Jun 22, 2022
grafana-sync Keep your grafana dashboards in sync.

grafana-sync Keep your grafana dashboards in sync. Table of Contents grafana-sync Table of Contents Installing Getting Started Pull Save all dashboard

Maksym Postument 151 Jun 14, 2022
The Elastalert Operator is an implementation of a Kubernetes Operator, to easily integrate elastalert with gitops.

Elastalert Operator for Kubernetes The Elastalert Operator is an implementation of a Kubernetes Operator. Getting started Firstly, learn How to use el

null 18 Apr 13, 2022
Minecraft-operator - A Kubernetes operator for Minecraft Java Edition servers

Minecraft Operator A Kubernetes operator for dedicated servers of the video game

James Laverack 6 May 10, 2022
K8s-network-config-operator - Kubernetes network config operator to push network config to switches

Kubernetes Network operator Will add more to the readme later :D Operations The

Daniel Hertzberg 6 May 16, 2022
Pulumi-k8s-operator-example - OpenGitOps Compliant Pulumi Kubernetes Operator Example

Pulumi GitOps Example OpenGitOps Compliant Pulumi Kubernetes Operator Example Pr

Christian Hernandez 3 May 6, 2022
Help developer to sync between local file and remote apollo portal web since portal web is so messy to use

apollo-synchronizer Help developer to sync between local file and remote apollo portal web since portal web is so messy to use Features download names

yeqown 3 May 16, 2022
Operator Permissions Advisor is a CLI tool that will take a catalog image and statically parse it to determine what permissions an Operator will request of OLM during an install

Operator Permissions Advisor is a CLI tool that will take a catalog image and statically parse it to determine what permissions an Operator will request of OLM during an install. The permissions are aggregated from the following sources:

International Business Machines 2 Apr 22, 2022
Test Operator using operator-sdk 1.15

test-operator Test Operator using operator-sdk 1.15 operator-sdk init --domain rbt.com --repo github.com/ravitri/test-operator Writing kustomize manif

Ravi Trivedi 0 Dec 28, 2021
a k8s operator 、operator-sdk

helloworld-operator a k8s operator 、operator-sdk Operator 参考 https://jicki.cn/kubernetes-operator/ https://learnku.com/articles/60683 https://opensour

Mark YiL 0 Jan 27, 2022