The Elastalert Operator is an implementation of a Kubernetes Operator, to easily integrate elastalert with gitops.

Overview

Go 1.16 codecov CI Workflow Go Report Card

Elastalert Operator for Kubernetes

The Elastalert Operator is an implementation of a Kubernetes Operator.

Getting started

Firstly, learn How to use elastalert, exactly how to setup a config.yaml and rule. The default command to start elastalert container is elastalert --config /etc/elastalert/config.yaml --verbose.

To install the operator, run:

kubectl create namespace alert
kubectl create -n alert -f https://raw.githubusercontent.com/toughnoah/elastalert-operator/master/deploy/es.noah.domain_elastalerts.yaml
kubectl create -n alert -f https://raw.githubusercontent.com/toughnoah/elastalert-operator/master/deploy/role.yaml
kubectl create -n alert -f https://raw.githubusercontent.com/toughnoah/elastalert-operator/master/deploy/role_binding.yaml
kubectl create -n alert -f https://raw.githubusercontent.com/toughnoah/elastalert-operator/master/deploy/service_account.yaml
kubectl create -n alert -f https://raw.githubusercontent.com/toughnoah/elastalert-operator/master/deploy/deployment.yaml

Args for Operator:

# /manager -h
-health-probe-bind-address string
The address the probe endpoint binds to. (default ":8081")

-leader-elect bool
Enable leader election for controller manager. Enabling this will ensure there is only one active controller manager. (default false)

-metrics-bind-address string
The address the metric endpoint binds to. (default ":8080")

-zap-log-level string
Zap Level to configure the verbosity of logging. Can be one of debug, info, error. (default info)

Once the elastalert-operator deployment in the namespace alert is ready, create an Elastalert instance in any namespace, like:

kubectl apply -n alert -f - <<EOF
apiVersion: es.noah.domain/v1alpha1
kind: Elastalert
metadata:
  name: elastalert
spec:
  rule:
  - name: error-messages
    type: any
    index: your-index-here
    filter:
    - query:
        query_string:
          query: "message: error"
    min_threshold: 1
  - name: error-status-code
    type: any
    index: your-index-here
    filter:
    - query:
        query_string:
          query: "http_status_code: 500"
    min_threshold: 1
  config:
    es_host: es.domain
    es_port: 9200
    es_username: username
    es_password: password
    writeback_index: your-elastalert-index
    run_every: 
      minutes: 5
    buffer_time: 
      minutes: 5
  overall:
    alert:
      - "post"
      http_post_url: "test.com"
      http_post_headers:
        Content-Type: |-
          "application/json"
      http_post_static_payload:
          ...
      
EOF

This will create an elastalert instance named elastalert, and the operator will create a deployment named the same, as:

# kubectl get -n alert deployment
NAME              READY   UP-TO-DATE   AVAILABLE   AGE
elastalert        1/1     1            1           10m

Next you can check your elastalert pod.

# kubectl get -n alert pod
NAME                               READY   STATUS    RESTARTS   AGE
elastalert-76c95597f8-zmbzz         1/1    Running      0       10m

And check your configmaps that -config is created from config. -rule is created from rule.

# kubectl get configmap
NAME                     DATA   AGE
elastalert-config         1     10m
elastalert-rule           1     10m

Above mentioned configmaps will be mounted to /etc/elastalert and /etc/elastalert/rule as config.yaml and rule yaml named by its ["name"]

# /etc/elastalert  ls
config.yaml  rules
# /etc/elastalert/rules ls
error-message.yaml      error-status-code.yaml

What's more

Elasticsearch Cert

kubectl apply -n alert -f - <<EOF
apiVersion: es.noah.domain/v1alpha1
kind: Elastalert
metadata:
  name: elastalert
spec:
  rule:
  - name: error-messages
    ...
  - name: error-status-code
    ...
  config:
    use_ssl: True
  overall:
    alert:
      ...
  cert: |-
        -----BEGIN CERTIFICATE-----
      MIIDYjCCAkqgAwIBAgIRAPUiB7MQDUsQlH5qOvwzaeQwDQYJKoZIhvcNAQELBQAw
      OzEZMBcGA1UECxMQc2ctZWxhc3RpY3NlYXJjaDEeMBwGA1UEAxMVc2ctZWxhc3Rp
      Y3NlYXJjaC1odHRwMB4XDTIwMTIxNjA2MTYyMloXDTIxMTIxNjA2MjYyMlowOzEZ
      MBcGA1UECxMQc2ctZWxhc3RpY3NlYXJjaDEeMBwGA1UEAxMVc2ctZWxhc3RpY3Nl
      YXJjaC1odHRwMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAv6wgW0Al
      6txmJN0/ihVZkqxtmY9owIskB3CxOaaXva27LGqj0Rp5b/OXJIem2h9UKuYqTQCp
      xwToEVw7rmIuyGXRtC2qSKHDFDSFHSDJFHSDKFSDHKCTi6O6uqMcrgjWwATfT76i
      dzeceg4Ly8FINTd2MBi5IbB+UybT/V0T89CeRssjQHEjAX5qEJYW7iTG01nAvaVl
      qfmsMvsZVzU+T8K7ZWrfhkRI2y6ln3hfsE1rKeAzX788RwU8o3GA41Jk5md0yE3a
      y5h0odVoanM1GLH29nswl28t0UAvq/K7kg38V0Kzuocc09mStCXKTf+I74YG7LBu
      V8IHPfCliPSvCwSKDHFJSDHFJKSDNMXCWER72sfSDFETVNFQAheqoQwHQYDVR0lB
      KwYBBQUHAwEGCCsGAQUFBwMCMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFGVU
      OaiBC+86H7lAHu5xM95vSO+VMA0GCSqGSIb3DQEBCwUAA4IBAQCM8mvNRjDOj/kn
      7Fni7FVp6v7Oa7yXiK0knzRX9GoHkniA/a5rZN3Fau+i2y6g2vaUs4BtdsAAdyC7
      GHIImn2M91nJXxcCFD0sfSKDJHFJSKDHFsdfsfeuWwMZJpBHSIw4aNmfX3lsde4c
      6D4+FzlWBKg/JTCv/63HVJ+m+HMKDE8h12aUM1n2rTHiMtnuRkIBa9uoXydN+QCM
      trjLH8AHqXNEPpraKFPRsCVHHhlBlDfpTSkvFgQLAKCh+heXDLyG1e+NEeU+sw9X
      sVz/o0pQkfwUNQjKsPwxQCHxjjw0qZn02wwX8fHgvmCwNIOK/WgsUlNDpny8CLiC
      pxb0oEG7
      -----END CERTIFICATE-----
EOF

This action will create a new secret mounted as /ssl/elasticCA.crt, and operator will automatically create a secret.

# kubectl get -n alert secret
NAME                               TYPE                                  DATA   AGE
elastalert-es-cert                 Opaque                                1      10m

Overall

overall is used to config global alert settings. If you defined alert in a rule, it will override overall settings.

kubectl apply -n alert -f - <<EOF
apiVersion: es.noah.domain/v1alpha1
kind: Elastalert
metadata:
  name: elastalert
spec:
  rule:
  - name: error-messages
    ...
  - name: error-status-code
    ...
  config:
    ...
  overall:
    alert:
      - "post"
      http_post_url: "test.com"
      http_post_headers:
        Content-Type: |-
          "application/json"
      http_post_static_payload:
          ...
  cert: |-
    ...
EOF

Pod Template

Define customized podTemplate

kubectl apply -n alert -f - <<EOF
apiVersion: es.noah.domain/v1alpha1
kind: Elastalert
metadata:
  name: elastalert
spec:
  rule:
    ...
  config:
    ...
  overall:
    ...
  cert: |-
    ...
  podTemplate:
      spec:
        containers:
        - name: elastalert
          imagePullPolicy: Always
          resources:
            limits:
              cpu: "2000m"
              memory: "4Gi"
            requests:
              cpu: "500m"
              memory: "1Gi" 
EOF

You can override the default command here.

Build Your Own Elastalert Dockerfile.

FROM docker.io/library/python:3.7.4-alpine

ENV TZ=Asia/Shanghai

ENV CRYPTOGRAPHY_DONT_BUILD_RUST=1

RUN apk add gcc g++ make libffi-dev openssl-dev

RUN apk add --no-cache tzdata

RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone

RUN pip install --upgrade pip

RUN pip install elastalert

ENTRYPOINT ["elastalert", "--config", "/etc/elastalert/config.yaml", "--verbose"]

Notice

You don't have to specify rules_folder in config section, because operator will auto patch rules_folder: /etc/elastalert/rules/..data/ for your config. The reason why have to be ..data/ is the workaround when the configmap is mounted as file(such as /etc/elastalert/rules/test.yaml) in a pod, it will create a soft-link to /etc/elastalert/rules/..data/test.yaml. That is to say, you will receive duplicated rules name error that both files in rules and ..data would be loaded if you specify merely rules_folder: /etc/elastalert/rules

Contact Me

Any advice is welcome! Please email to [email protected]

You might also like...
ORBOS - GitOps everything
ORBOS - GitOps everything

ORBOS - GitOps everything ORBOS explained ORBITER BOOM Getting Started on Google Compute Engine In the following example we will create a kubernetes c

ArgoCD is widely used for enabling CD GitOps. ArgoCD internally builds manifest from source data in Git repository, and auto-sync it with target clusters.
ArgoCD is widely used for enabling CD GitOps. ArgoCD internally builds manifest from source data in Git repository, and auto-sync it with target clusters.

ArgoCD Interlace ArgoCD is widely used for enabling CD GitOps. ArgoCD internally builds manifest from source data in Git repository, and auto-sync it

Cluster bootstraps for GitOps
Cluster bootstraps for GitOps

Introduction Documentation Site Cluster bootstraps for Crossplane GitOps based on argocd, see main doc site for details PreRequisites K8 cluster eg ki

The OCI Service Operator for Kubernetes (OSOK) makes it easy to connect and manage OCI services from a cloud native application running in a Kubernetes environment.

OCI Service Operator for Kubernetes Introduction The OCI Service Operator for Kubernetes (OSOK) makes it easy to create, manage, and connect to Oracle

PolarDB-X Operator is a Kubernetes extension that aims to create and manage PolarDB-X cluster on Kubernetes.

GalaxyKube -- PolarDB-X Operator PolarDB-X Operator is a Kubernetes extension that aims to create and manage PolarDB-X cluster on Kubernetes. It follo

Kubernetes Operator to sync secrets between different secret backends and Kubernetes

Vals-Operator Here at Digitalis we love vals, it's a tool we use daily to keep secrets stored securely. We also use secrets-manager on the Kubernetes

The NiFiKop NiFi Kubernetes operator makes it easy to run Apache NiFi on Kubernetes.
The NiFiKop NiFi Kubernetes operator makes it easy to run Apache NiFi on Kubernetes.

The NiFiKop NiFi Kubernetes operator makes it easy to run Apache NiFi on Kubernetes. Apache NiFI is a free, open-source solution that support powerful and scalable directed graphs of data routing, transformation, and system mediation logic.

Plugin for Helm to integrate the sigstore ecosystem

helm-sigstore Plugin for Helm to integrate the sigstore ecosystem. Search, upload and verify signed Helm Charts in the Rekor Transparency Log. Info he

🔮 ✈️ to integrate OPA Gatekeeper's new ExternalData feature with cosign to determine whether the images are valid by verifying their signatures

cosign-gatekeeper-provider To integrate OPA Gatekeeper's new ExternalData feature with cosign to determine whether the images are valid by verifying i

Comments
  • Set ElastAlert custom resource apiVersion to v1 in examples

    Set ElastAlert custom resource apiVersion to v1 in examples

    First of all thank you for this project, I think it may be very useful. I installed the CRDs and RBAC as told in the documentation, but it seems the CRD is v1, and the CR examples are still in v1alpha1. This PR changes the CR examples. Feel free to reject or make changes. Cheers 🍻

    opened by RoeyPrat 2
  • Add liveness and readiness probes

    Add liveness and readiness probes

    How can I configure liveness and readiness probes for elastalert-operator. Since we do not have any rest APIs to query for and also the port at which to listen. How can this be done ?

    opened by karthiknagraj 1
Releases(v1.0.2)
Owner
null
Basic Kubernetes operator that have multiple versions in CRD. This operator can be used to experiment and understand Operator/CRD behaviors.

add-operator Basic Kubernetes operator that have multiple versions in CRD. This operator can be used to experiment and understand Operator/CRD behavio

Dinesh Parvathaneni 0 Dec 15, 2021
Digitalocean-kubernetes-challenge - Deploy a GitOps CI/CD implementation

DigitalOcean Kubernetes Challenge 2021 I chose to participate in the DigitalOcean Kubernetes Challenge in order to learn more about Kubernetes and to

Engin Diri 2 Nov 9, 2022
kube-champ 43 Oct 19, 2022
Bootstrap curated Kubernetes stacks. Logging, metrics, ingress and more - delivered with gitops.

Gimlet Stack Bootstrap curated Kubernetes stacks. Logging, metrics, ingress and more - delivered with gitops. You can install logging aggregators, met

null 12 Dec 1, 2021
gokp aims to install a GitOps Native Kubernetes Platform

gokp gokp aims to install a GitOps Native Kubernetes Platform. This project is a Proof of Concept centered around getting a GitOps aware Kubernetes Pl

Christian Hernandez 25 Nov 4, 2022
An operator which complements grafana-operator for custom features which are not feasible to be merged into core operator

Grafana Complementary Operator A grafana which complements grafana-operator for custom features which are not feasible to be merged into core operator

Snapp Cab Incubators 6 Aug 16, 2022
Minecraft-operator - A Kubernetes operator for Minecraft Java Edition servers

Minecraft Operator A Kubernetes operator for dedicated servers of the video game

James Laverack 8 Dec 15, 2022
K8s-network-config-operator - Kubernetes network config operator to push network config to switches

Kubernetes Network operator Will add more to the readme later :D Operations The

Daniel Hertzberg 6 May 16, 2022
Pulumi-k8s-operator-example - OpenGitOps Compliant Pulumi Kubernetes Operator Example

Pulumi GitOps Example OpenGitOps Compliant Pulumi Kubernetes Operator Example Pr

Christian Hernandez 3 May 6, 2022
Kubernetes Operator Samples using Go, the Operator SDK and OLM

Kubernetes Operator Patterns and Best Practises This project contains Kubernetes operator samples that demonstrate best practices how to develop opera

International Business Machines 27 Nov 24, 2022