๐Ÿ‘€ A Kubernetes cluster resource sanitizer

Overview

Popeye - A Kubernetes Cluster Sanitizer

Popeye is a utility that scans live Kubernetes cluster and reports potential issues with deployed resources and configurations. It sanitizes your cluster based on what's deployed and not what's sitting on disk. By scanning your cluster, it detects misconfigurations and helps you to ensure that best practices are in place, thus preventing future headaches. It aims at reducing the cognitive overload one faces when operating a Kubernetes cluster in the wild. Furthermore, if your cluster employs a metric-server, it reports potential resources over/under allocations and attempts to warn you should your cluster run out of capacity.

Popeye is a readonly tool, it does not alter any of your Kubernetes resources in any way!




Go Report Card codebeat badge Build Status release License Docker Repository on Quay GitHub stars Releases


Installation

Popeye is available on Linux, OSX and Windows platforms.

  • Binaries for Linux, Windows and Mac are available as tarballs in the release page.

  • For OSX/Unit using Homebrew/LinuxBrew

    brew install derailed/popeye/popeye
  • Building from source Popeye was built using go 1.12+. In order to build Popeye from source you must:

    1. Clone the repo

    2. Add the following command in your go.mod file

      replace (
        github.com/derailed/popeye => MY_POPEYE_CLONED_GIT_REPO
      )
      
    3. Build and run the executable

      go run main.go

    Quick recipe for the impatient:

    # Clone outside of GOPATH
    git clone https://github.com/derailed/popeye
    cd popeye
    # Build and install
    go install
    # Run
    popeye

PreFlight Checks

  • Popeye uses 256 colors terminal mode. On `Nix system make sure TERM is set accordingly.

    export TERM=xterm-256color

Sanitizers

Popeye scans your cluster for best practices and potential issues. Currently, Popeye only looks at nodes, namespaces, pods and services. More will come soon! We are hoping Kubernetes friends will pitch'in to make Popeye even better.

The aim of the sanitizers is to pick up on misconfigurations, i.e. things like port mismatches, dead or unused resources, metrics utilization, probes, container images, RBAC rules, naked resources, etc...

Popeye is not another static analysis tool. It runs and inspect Kubernetes resources on live clusters and sanitize resources as they are in the wild!

Here is a list of some of the available sanitizers:

Resource Sanitizers Aliases
๐Ÿ›€ Node no
Conditions ie not ready, out of mem/disk, network, pids, etc
Pod tolerations referencing node taints
CPU/MEM utilization metrics, trips if over limits (default 80% CPU/MEM)
๐Ÿ›€ Namespace ns
Inactive
Dead namespaces
๐Ÿ›€ Pod po
Pod status
Containers statuses
ServiceAccount presence
CPU/MEM on containers over a set CPU/MEM limit (default 80% CPU/MEM)
Container image with no tags
Container image using latest tag
Resources request/limits presence
Probes liveness/readiness presence
Named ports and their references
๐Ÿ›€ Service svc
Endpoints presence
Matching pods labels
Named ports and their references
๐Ÿ›€ ServiceAccount sa
Unused, detects potentially unused SAs
๐Ÿ›€ Secrets sec
Unused, detects potentially unused secrets or associated keys
๐Ÿ›€ ConfigMap cm
Unused, detects potentially unused cm or associated keys
๐Ÿ›€ Deployment dp, deploy
Unused, pod template validation, resource utilization
๐Ÿ›€ StatefulSet sts
Unsed, pod template validation, resource utilization
๐Ÿ›€ DaemonSet ds
Unsed, pod template validation, resource utilization
๐Ÿ›€ PersistentVolume pv
Unused, check volume bound or volume error
๐Ÿ›€ PersistentVolumeClaim pvc
Unused, check bounded or volume mount error
๐Ÿ›€ HorizontalPodAutoscaler hpa
Unused, Utilization, Max burst checks
๐Ÿ›€ PodDisruptionBudget
Unused, Check minAvailable configuration pdb
๐Ÿ›€ ClusterRole
Unused cr
๐Ÿ›€ ClusterRoleBinding
Unused crb
๐Ÿ›€ Role
Unused ro
๐Ÿ›€ RoleBinding
Unused rb
๐Ÿ›€ Ingress
Valid ing
๐Ÿ›€ NetworkPolicy
Valid np
๐Ÿ›€ PodSecurityPolicy
Valid psp

You can also see the full list of codes

Save the report

To save the Popeye report to a file pass the --save flag to the command. By default it will create a temp directory and will store the report there, the path of the temp directory will be printed out on STDOUT. If you have the need to specify the output directory for the report, you can use the environment variable POPEYE_REPORT_DIR. By default, the name of the output file follow the following format : sanitizer_<cluster-name>_<time-UnixNano>.<output-extension> (e.g. : "sanitizer-mycluster-1594019782530851873.html"). If you have the need to specify the output file name for the report, you can pass the --output-file flag with the filename you want as parameter.

Example to save report in working directory:

  $ POPEYE_REPORT_DIR=$(pwd) popeye --save

Example to save report in working directory in HTML format under the name "report.html" :

  $ POPEYE_REPORT_DIR=$(pwd) popeye --save --out html --output-file report.html

Save the report to S3

You can also save the generated report to an AWS S3 bucket with providing the flag --s3-bucket. As parameter you need to provide the name of the S3 bucket where you want to store the report. To save the report in a bucket subdirectory provide the bucket parameter as bucket/path/to/report.

Underlying the AWS Go lib is used which is handling the credential loading. For more information check out the official documentation.

Example to save report to S3:

popeye --s3-bucket=NAME-OF-YOUR-S3-BUCKET/OPTIONAL/SUBDIRECTORY --out=json

Run public docker image locally

You don't have to build and/or install the binary to run popeye: you can just run it directly from the official docker repo on quay.io. The default command when you run the docker container is popeye, so you just need to pass whatever cli args are normally passed to popeye. To access your clusters, map your local kube config directory into the container with -v :

  docker run --rm -it \
    -v $HOME/.kube:/root/.kube \
    quay.io/derailed/popeye --context foo -n bar

Running the above docker command with --rm means that the container gets deleted when popeye exits. When you use --save, it will write it to /tmp in the container and then delete the container when popeye exits, which means you lose the output. To get around this, map /tmp to the container's /tmp. NOTE: You can override the default output directory location by setting POPEYE_REPORT_DIR env variable.

  docker run --rm -it \
    -v $HOME/.kube:/root/.kube \
    -e POPEYE_REPORT_DIR=/tmp/popeye \
    -v /tmp:/tmp \
    quay.io/derailed/popeye --context foo -n bar --save --output-file my_report.txt

  # Docker has exited, and the container has been deleted, but the file
  # is in your /tmp directory because you mapped it into the container
  $ cat /tmp/popeye/my_report.txt
    <snip>

The Command Line

You can use Popeye standalone or using a spinach yaml config to tune the sanitizer. Details about the Popeye configuration file are below.

# Dump version info
popeye version
# Popeye a cluster using your current kubeconfig environment.
popeye
# Popeye uses a spinach config file of course! aka spinachyaml!
popeye -f spinach.yml
# Popeye a cluster using a kubeconfig context.
popeye --context olive
# Stuck?
popeye help

Output Formats

Popeye can generate sanitizer reports in a variety of formats. You can use the -o cli option and pick your poison from there.

Format Description Default Credits
standard The full monty output iconized and colorized yes
jurassic No icons or color like it's 1979
yaml As YAML
html As HTML
json As JSON
junit For the Java melancholic
prometheus Dumps report a prometheus scrappable metrics dardanel
score Returns a single cluster sanitizer score value (0-100) kabute

The SpinachYAML Configuration

A spinach.yml configuration file can be specified via the -f option to further configure the sanitizers. This file may specify the container utilization threshold and specific sanitizer configurations as well as resources that will be excluded from the sanitization.

NOTE: This file will change as Popeye matures!

Under the excludes key you can configure to skip certain resources, or certain checks by code. Here, resource types are indicated in a group/version/resource notation. Example: to exclude PodDisruptionBugdets, use the notation policy/v1/poddisruptionbudgets. Note that the resource name is written in the plural form and everything is spelled in lowercase. For resources without an API group, the group part is omitted (Examples: v1/pods, v1/services, v1/configmaps).

A resource is identified by a resource kind and a fully qualified resource name, i.e. namespace/resource_name.

For example, the FQN of a pod named fred-1234 in the namespace blee will be blee/fred-1234. This provides for differentiating fred/p1 and blee/p1. For cluster wide resources, the FQN is equivalent to the name. Exclude rules can have either a straight string match or a regular expression. In the latter case the regular expression must be indicated using the rx: prefix.

NOTE! Please be careful with your regex as more resources than expected may get excluded from the report with a loose regex rule. When your cluster resources change, this could lead to a sub-optimal sanitization. Once in a while it might be a good idea to run Popeye โ€žconfiglessโ€œ to make sure you will recognize any new issues that may have arisen in your clustersโ€ฆ

Here is an example spinach file as it stands in this release. There is a fuller eks and aks based spinach file in this repo under spinach. (BTW: for new comers into the project, might be a great way to contribute by adding cluster specific spinach file PRs...)

# A Popeye sample configuration file
popeye:
  # Checks resources against reported metrics usage.
  # If over/under these thresholds a sanitization warning will be issued.
  # Your cluster must run a metrics-server for these to take place!
  allocations:
    cpu:
      underPercUtilization: 200 # Checks if cpu is under allocated by more than 200% at current load.
      overPercUtilization: 50   # Checks if cpu is over allocated by more than 50% at current load.
    memory:
      underPercUtilization: 200 # Checks if mem is under allocated by more than 200% at current load.
      overPercUtilization: 50   # Checks if mem is over allocated by more than 50% usage at current load.

  # Excludes excludes certain resources from Popeye scans
  excludes:
    v1/pods:
    # In the monitoring namespace excludes all probes check on pod's containers.
    - name: rx:monitoring
      codes:
      - 102
    # Excludes all istio-proxy container scans for pods in the icx namespace.
    - name: rx:icx/.*
      containers:
        # Excludes istio init/sidecar container from scan!
        - istio-proxy
        - istio-init
    # ConfigMap sanitizer exclusions...
    v1/configmaps:
      # Excludes key must match the singular form of the resource.
      # For instance this rule will exclude all configmaps named fred.v2.3 and fred.v2.4
      - name: rx:fred.+\.v\d+
    # Namespace sanitizer exclusions...
    v1/namespaces:
      # Exclude all fred* namespaces if the namespaces are not found (404), other error codes will be reported!
      - name: rx:kube
        codes:
          - 404
      # Exclude all istio* namespaces from being scanned.
      - name: rx:istio
    # Completely exclude horizontal pod autoscalers.
    autoscaling/v1/horizontalpodautoscalers:
      - name: rx:.*

  # Configure node resources.
  node:
    # Limits set a cpu/mem threshold in % ie if cpu|mem > limit a lint warning is triggered.
    limits:
      # CPU checks if current CPU utilization on a node is greater than 90%.
      cpu:    90
      # Memory checks if current Memory utilization on a node is greater than 80%.
      memory: 80

  # Configure pod resources
  pod:
    # Restarts check the restarts count and triggers a lint warning if above threshold.
    restarts:
      3
    # Check container resource utilization in percent.
    # Issues a lint warning if about these threshold.
    limits:
      cpu:    80
      memory: 75

Popeye In Your Clusters!

Alternatively, Popeye is containerized and can be run directly in your Kubernetes clusters as a one-off or CronJob.

Here is a sample setup, please modify per your needs/wants. The manifests for this are in the k8s directory in this repo.

kubectl apply -f k8s/popeye/ns.yml && kubectl apply -f k8s/popeye
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name:      popeye
  namespace: popeye
spec:
  schedule: "* */1 * * *" # Fire off Popeye once an hour
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: popeye
          restartPolicy: Never
          containers:
            - name: popeye
              image: quay.io/derailed/popeye
              imagePullPolicy: IfNotPresent
              args:
                - -o
                - yaml
                - --force-exit-zero
                - true
              resources:
                limits:
                  cpu:    500m
                  memory: 100Mi

The --force-exit-zero should be set to true. Otherwise, the pods will end up in an error state. Note that popeye exits with a non-zero error code if the report has any errors.

Popeye got your RBAC!

In order for Popeye to do his work, the signed-in user must have enough RBAC oomph to get/list the resources mentioned above.

Sample Popeye RBAC Rules (please note that those are subject to change.)

---
# Popeye ServiceAccount.
apiVersion: v1
kind:       ServiceAccount
metadata:
  name:      popeye
  namespace: popeye

---
# Popeye needs get/list access on the following Kubernetes resources.
apiVersion: rbac.authorization.k8s.io/v1
kind:       ClusterRole
metadata:
  name: popeye
rules:
- apiGroups: [""]
  resources:
   - configmaps
   - deployments
   - endpoints
   - horizontalpodautoscalers
   - namespaces
   - nodes
   - persistentvolumes
   - persistentvolumeclaims
   - pods
   - secrets
   - serviceaccounts
   - services
   - statefulsets
  verbs:     ["get", "list"]
- apiGroups: ["rbac.authorization.k8s.io"]
  resources:
  - clusterroles
  - clusterrolebindings
  - roles
  - rolebindings
  verbs:     ["get", "list"]
- apiGroups: ["metrics.k8s.io"]
  resources:
  - pods
  - nodes
  verbs:     ["get", "list"]

---
# Binds Popeye to this ClusterRole.
apiVersion: rbac.authorization.k8s.io/v1
kind:       ClusterRoleBinding
metadata:
  name: popeye
subjects:
- kind:     ServiceAccount
  name:     popeye
  namespace: popeye
roleRef:
  kind:     ClusterRole
  name:     popeye
  apiGroup: rbac.authorization.k8s.io

Screenshots

Cluster D Score

Cluster A Score

Report Morphology

The sanitizer report outputs each resource group scanned and their potential issues. The report is color/emoji coded in term of Sanitizer severity levels:

Level Icon Jurassic Color Description
Ok โœ… OK Green Happy!
Info ๐Ÿ”Š I BlueGreen FYI
Warn ๐Ÿ˜ฑ W Yellow Potential Issue
Error ๐Ÿ’ฅ E Red Action required

The heading section for each scanned Kubernetes resource provides a summary count for each of the categories above.

The Summary section provides a Popeye Score based on the sanitization pass on the given cluster.

Known Issues

This initial drop is brittle. Popeye will most likely blow up whenโ€ฆ

  • You're running older versions of Kubernetes. Popeye works best with Kubernetes 1.13+.
  • You don't have enough RBAC oomph to manage your cluster (see RBAC section)

Disclaimer

This is work in progress! If there is enough interest in the Kubernetes community, we will enhance per your recommendations/contributions. Also if you dig this effort, please let us know that too!

ATTA Girls/Boys!

Popeye sits on top of many of open source projects and libraries. Our sincere appreciations to all the OSS contributors that work nights and weekends to make this project a reality!

Contact Info

  1. Email: [email protected]
  2. Twitter: @kitesurfer

 ยฉ 2020 Imhotep Software LLC. All materials licensed under Apache v2.0

Issues
  • pushgateway support

    pushgateway support

    Hi , we are running popeye as cronjob on Kubernetes , and added pushgateway support in so basically it converts popeye sanitizer to prometheus metrics as below

    popeye_errors{instance="",job="popeye"} 1
    # HELP popeye_grade Grade of kubernetes cluster. (1: A, 2: B, 3: C, 4: D, 5: E, 6: F)
    # TYPE popeye_grade gauge
    popeye_grade{instance="",job="popeye"} 5
    # HELP popeye_sanitizers_error Sanitizer error level results for resource groups.
    # TYPE popeye_sanitizers_error gauge
    popeye_sanitizers_error{instance="",job="popeye",title="configmap"} 0
    popeye_sanitizers_error{instance="",job="popeye",title="daemonset"} 0
    popeye_sanitizers_error{instance="",job="popeye",title="deployment"} 1
    popeye_sanitizers_error{instance="",job="popeye",title="horizontalpodautoscaler"} 0
    popeye_sanitizers_error{instance="",job="popeye",title="ing"} 0
    popeye_sanitizers_error{instance="",job="popeye",title="namespace"} 0
    popeye_sanitizers_error{instance="",job="popeye",title="networkpolicy"} 0
    popeye_sanitizers_error{instance="",job="popeye",title="node"} 0
    popeye_sanitizers_error{instance="",job="popeye",title="persistentvolume"} 0
    popeye_sanitizers_error{instance="",job="popeye",title="persistentvolumeclaim"} 0
    popeye_sanitizers_error{instance="",job="popeye",title="pod"} 6
    popeye_sanitizers_error{instance="",job="popeye",title="poddisruptionbudget"} 1
    popeye_sanitizers_error{instance="",job="popeye",title="podsecuritypolicy"} 0
    popeye_sanitizers_error{instance="",job="popeye",title="replicaset"} 0
    popeye_sanitizers_error{instance="",job="popeye",title="secret"} 0
    popeye_sanitizers_error{instance="",job="popeye",title="service"} 1
    popeye_sanitizers_error{instance="",job="popeye",title="serviceaccount"} 0
    popeye_sanitizers_error{instance="",job="popeye",title="statefulset"} 0
    # HELP popeye_sanitizers_info Sanitizer info level results for resource groups.
    # TYPE popeye_sanitizers_info gauge
    popeye_sanitizers_info{instance="",job="popeye",title="configmap"} 83
    popeye_sanitizers_info{instance="",job="popeye",title="daemonset"} 0
    popeye_sanitizers_info{instance="",job="popeye",title="deployment"} 1
    popeye_sanitizers_info{instance="",job="popeye",title="horizontalpodautoscaler"} 0
    popeye_sanitizers_info{instance="",job="popeye",title="ing"} 0
    popeye_sanitizers_info{instance="",job="popeye",title="namespace"} 3
    popeye_sanitizers_info{instance="",job="popeye",title="networkpolicy"} 0
    popeye_sanitizers_info{instance="",job="popeye",title="node"} 0
    popeye_sanitizers_info{instance="",job="popeye",title="persistentvolume"} 0
    popeye_sanitizers_info{instance="",job="popeye",title="persistentvolumeclaim"} 0
    popeye_sanitizers_info{instance="",job="popeye",title="pod"} 0
    popeye_sanitizers_info{instance="",job="popeye",title="poddisruptionbudget"} 0
    popeye_sanitizers_info{instance="",job="popeye",title="podsecuritypolicy"} 0
    popeye_sanitizers_info{instance="",job="popeye",title="replicaset"} 0
    popeye_sanitizers_info{instance="",job="popeye",title="secret"} 36
    popeye_sanitizers_info{instance="",job="popeye",title="service"} 25
    popeye_sanitizers_info{instance="",job="popeye",title="serviceaccount"} 4
    popeye_sanitizers_info{instance="",job="popeye",title="statefulset"} 0
    # HELP popeye_sanitizers_ok Sanitizer ok level results for resource groups.
    # TYPE popeye_sanitizers_ok gauge
    popeye_sanitizers_ok{instance="",job="popeye",title="configmap"} 16
    popeye_sanitizers_ok{instance="",job="popeye",title="daemonset"} 5
    popeye_sanitizers_ok{instance="",job="popeye",title="deployment"} 4
    popeye_sanitizers_ok{instance="",job="popeye",title="horizontalpodautoscaler"} 0
    popeye_sanitizers_ok{instance="",job="popeye",title="ing"} 0
    popeye_sanitizers_ok{instance="",job="popeye",title="namespace"} 2
    popeye_sanitizers_ok{instance="",job="popeye",title="networkpolicy"} 0
    popeye_sanitizers_ok{instance="",job="popeye",title="node"} 8
    popeye_sanitizers_ok{instance="",job="popeye",title="persistentvolume"} 0
    popeye_sanitizers_ok{instance="",job="popeye",title="persistentvolumeclaim"} 0
    popeye_sanitizers_ok{instance="",job="popeye",title="pod"} 0
    popeye_sanitizers_ok{instance="",job="popeye",title="poddisruptionbudget"} 0
    popeye_sanitizers_ok{instance="",job="popeye",title="podsecuritypolicy"} 0
    popeye_sanitizers_ok{instance="",job="popeye",title="replicaset"} 68
    popeye_sanitizers_ok{instance="",job="popeye",title="secret"} 41
    popeye_sanitizers_ok{instance="",job="popeye",title="service"} 5
    popeye_sanitizers_ok{instance="",job="popeye",title="serviceaccount"} 52
    popeye_sanitizers_ok{instance="",job="popeye",title="statefulset"} 1
    # HELP popeye_sanitizers_score Sanitizer score results for resource groups.
    # TYPE popeye_sanitizers_score gauge
    popeye_sanitizers_score{instance="",job="popeye",title="configmap"} 100
    popeye_sanitizers_score{instance="",job="popeye",title="daemonset"} 50
    popeye_sanitizers_score{instance="",job="popeye",title="deployment"} 19
    popeye_sanitizers_score{instance="",job="popeye",title="horizontalpodautoscaler"} 0
    popeye_sanitizers_score{instance="",job="popeye",title="ing"} 0
    popeye_sanitizers_score{instance="",job="popeye",title="namespace"} 100
    popeye_sanitizers_score{instance="",job="popeye",title="networkpolicy"} 0
    popeye_sanitizers_score{instance="",job="popeye",title="node"} 100
    popeye_sanitizers_score{instance="",job="popeye",title="persistentvolume"} 0
    popeye_sanitizers_score{instance="",job="popeye",title="persistentvolumeclaim"} 0
    popeye_sanitizers_score{instance="",job="popeye",title="pod"} 0
    popeye_sanitizers_score{instance="",job="popeye",title="poddisruptionbudget"} 0
    popeye_sanitizers_score{instance="",job="popeye",title="podsecuritypolicy"} 0
    popeye_sanitizers_score{instance="",job="popeye",title="replicaset"} 100
    popeye_sanitizers_score{instance="",job="popeye",title="secret"} 100
    popeye_sanitizers_score{instance="",job="popeye",title="service"} 96
    popeye_sanitizers_score{instance="",job="popeye",title="serviceaccount"} 100
    popeye_sanitizers_score{instance="",job="popeye",title="statefulset"} 50
    # HELP popeye_sanitizers_warning Sanitizer warning level results for resource groups.
    # TYPE popeye_sanitizers_warning gauge
    popeye_sanitizers_warning{instance="",job="popeye",title="configmap"} 0
    popeye_sanitizers_warning{instance="",job="popeye",title="daemonset"} 5
    popeye_sanitizers_warning{instance="",job="popeye",title="deployment"} 20
    popeye_sanitizers_warning{instance="",job="popeye",title="horizontalpodautoscaler"} 2
    popeye_sanitizers_warning{instance="",job="popeye",title="ing"} 12
    popeye_sanitizers_warning{instance="",job="popeye",title="namespace"} 0
    popeye_sanitizers_warning{instance="",job="popeye",title="networkpolicy"} 0
    popeye_sanitizers_warning{instance="",job="popeye",title="node"} 0
    popeye_sanitizers_warning{instance="",job="popeye",title="persistentvolume"} 0
    popeye_sanitizers_warning{instance="",job="popeye",title="persistentvolumeclaim"} 0
    popeye_sanitizers_warning{instance="",job="popeye",title="pod"} 114
    popeye_sanitizers_warning{instance="",job="popeye",title="poddisruptionbudget"} 0
    popeye_sanitizers_warning{instance="",job="popeye",title="podsecuritypolicy"} 0
    popeye_sanitizers_warning{instance="",job="popeye",title="replicaset"} 0
    popeye_sanitizers_warning{instance="",job="popeye",title="secret"} 0
    popeye_sanitizers_warning{instance="",job="popeye",title="service"} 0
    popeye_sanitizers_warning{instance="",job="popeye",title="serviceaccount"} 0
    popeye_sanitizers_warning{instance="",job="popeye",title="statefulset"} 1
    # HELP popeye_score Score of kubernetes cluster.
    # TYPE popeye_score gauge
    popeye_score{instance="",job="popeye"} 58
    

    Run command is like below

    popeye --out prometheus --pushgateway-address http://pushgateway.address

    opened by eminugurkenar 18
  • v0.3.0 crashes at start: runtime error: invalid memory address or nil pointer dereference

    v0.3.0 crashes at start: runtime error: invalid memory address or nil pointer dereference




    Describe the bug Running popeye by just typing "popeye" on the shell leads to the following error: runtime error: invalid memory address or nil pointer dereference.

    The logfile is empty.

    To Reproduce Steps to reproduce the behavior:

    1. Open shell
    2. type popeye
    3. press enter
    4. See error

    Expected behavior See popeye work as desired

    Screenshots image

    image

    Versions (please complete the following information):

    • OS: macOS
    • Popeye 0.3.0
    • K8s 1.13.5 and 1.12.6

    Additional context ... my sincere condolences! ...

    Log

    8:01AM ERR runtime error: invalid memory address or nil pointer dereference
    8:01AM ERR goroutine 1 [running]:
    runtime/debug.Stack(0x2e9aae0, 0x20caa03, 0x0)
    	/usr/local/Cellar/go/1.12.3/libexec/src/runtime/debug/stack.go:24 +0x9d
    github.com/derailed/popeye/cmd.doIt.func1()
    	/Users/fernand/go_wk/derailed/src/github.com/derailed/popeye/cmd/root.go:50 +0x15b
    panic(0x1f3e620, 0x2e83250)
    	/usr/local/Cellar/go/1.12.3/libexec/src/runtime/panic.go:522 +0x1b5
    github.com/derailed/popeye/internal/linter.(*Secret).checkContainerRefs(0xc00043a0b0, 0xc0001270e0, 0x25, 0xc000566140, 0x1, 0x1, 0xc000616ea0)
    	/Users/fernand/go_wk/derailed/src/github.com/derailed/popeye/internal/linter/secret.go:178 +0x26a
    github.com/derailed/popeye/internal/linter.(*Secret).lint(0xc00043a0b0, 0xc0004b6ba0, 0xc00039f410, 0xc0003dc480)
    	/Users/fernand/go_wk/derailed/src/github.com/derailed/popeye/internal/linter/secret.go:85 +0x2ce
    github.com/derailed/popeye/internal/linter.(*Secret).Lint(0xc00043a0b0, 0x22e15e0, 0xc0000f8540, 0x20cb19c, 0x3)
    	/Users/fernand/go_wk/derailed/src/github.com/derailed/popeye/internal/linter/secret.go:51 +0xec
    github.com/derailed/popeye/pkg.(*Popeye).sanitize(0xc0000f8500)
    	/Users/fernand/go_wk/derailed/src/github.com/derailed/popeye/pkg/popeye.go:140 +0x1ef
    github.com/derailed/popeye/pkg.(*Popeye).Sanitize(0xc0000f8500)
    	/Users/fernand/go_wk/derailed/src/github.com/derailed/popeye/pkg/popeye.go:106 +0x2b
    github.com/derailed/popeye/cmd.doIt(0x2e8d460, 0x2eb87c0, 0x0, 0x0)
    	/Users/fernand/go_wk/derailed/src/github.com/derailed/popeye/cmd/root.go:60 +0x11b
    github.com/spf13/cobra.(*Command).execute(0x2e8d460, 0xc00003a1b0, 0x0, 0x0, 0x2e8d460, 0xc00003a1b0)
    	/Users/fernand/go_wk/derailed/pkg/mod/github.com/spf13/[email protected]/command.go:766 +0x2ae
    github.com/spf13/cobra.(*Command).ExecuteC(0x2e8d460, 0x0, 0x0, 0xc0003d9f88)
    	/Users/fernand/go_wk/derailed/pkg/mod/github.com/spf13/[email protected]/command.go:852 +0x2ec
    github.com/spf13/cobra.(*Command).Execute(...)
    	/Users/fernand/go_wk/derailed/pkg/mod/github.com/spf13/[email protected]/command.go:800
    github.com/derailed/popeye/cmd.Execute()
    	/Users/fernand/go_wk/derailed/src/github.com/derailed/popeye/cmd/root.go:37 +0x32
    main.main()
    	/Users/fernand/go_wk/derailed/src/github.com/derailed/popeye/main.go:25 +0x20
    
    bug 
    opened by swe-covis 16
  • can't run popeye when kubernetes has networking.k8s.io/v1/ingresses

    can't run popeye when kubernetes has networking.k8s.io/v1/ingresses




    Describe the bug When i download the released binary file, and run "./popeye", it shows "Boom! No resource meta found for networking.k8s.io/v1/ingresses". Can I skip the resource?

    To Reproduce just run "./popeye"

    Expected behavior popeye can run as normal

    Screenshots If applicable, add screenshots to help explain your problem.

    Versions (please complete the following information): image

    Additional context Add any other context about the problem here.

    bug 
    opened by woodliu 12
  • Add S3 support

    Add S3 support

    This PR will add support to upload reports directly to AWS S3.

    Current state, already working ๐ŸŽ‰

    ~There is still documentation & some refactoring missing.~

    Use case: Our current workflow is currently to run Popeye on a regular base via a CronJob (once a week) and afterwards post the report to AWS, this will be easier when it's possible to persist the report to S3.

    opened by guusvw 11
  • Allow suppressing specific checks / reports for specific instances

    Allow suppressing specific checks / reports for specific instances




    Is your feature request related to a problem? Please describe. Some sanitizers are producing warnings which make sense most of time, but on some objects, I would like to be able to accept those "issues". For example, the "container runs as root" is very helpful 98% of the times, but some very infrastructure-related pods require this.

    Describe the solution you'd like It would be nice if there was a way to skip specific aspects of a sanitizer by given names or regex for objects on which they should not be run, or at least on which they should not be reported (and maybe even not considered for the overall score)

    Describe alternatives you've considered Just excluding the objects (e.g. pods) is not enough, since this will ignore all checks on them.

    Additional context No additional context.

    enhancement question 
    opened by djablonski-moia 9
  • Port name missing considered an error

    Port name missing considered an error

    Hi, just a brainstorming here.

    Why do you consider an unnamed port in a deployment/pod as an error? Do you have a source on the documentation that supports this decision?

    Thanks

    enhancement question 
    opened by whites11 9
  • Add extra labels to prometheus report (warn and error level only)

    Add extra labels to prometheus report (warn and error level only)

    Basic details about issues for prometheus report. Afaik it can be largely improved.

    Capabilities:

    • Expose detail messages for the outcome issues
    • Fine tune the output via lint level flag

    Atm the final result on the Prometheus POV would be (example for service): $popeye -o prometheus --pushgateway-address http://localhost:9091 -l error -n testns --out-detail verbose

    popeye_sanitizer_reports_count{cluster="testcluster",issues="testns/kafka [POP-1100] No pods match service selector,testns/kafka [POP-1105] No associated endpoints",job="popeye",level="error",namespace="testns/kafka",resource="service"} | 1

    opened by sky-philipalmeida 7
  • Add check for Service with a single endpoint

    Add check for Service with a single endpoint




    Is your feature request related to a problem? Please describe. No. The feature request is about adding a check for something that is currently not covered.

    Describe the solution you'd like

    Services backed by only a single endpoint can fail once the endpoitn goes down, e. g. due to a failure of the node running the endpoint pod. Services should ideally be backed by more than one endpoint for high availability.

    popeye should inform/warn about services with only a single endpoint.

    Describe alternatives you've considered None

    Additional context

    PR: https://github.com/derailed/popeye/pull/135

    opened by marians 7
  • ADD Support to return just the score value as output

    ADD Support to return just the score value as output

    Add support to just return the score value. This change is introduced to simplify the usage of Popeye for general instrumentation (e.g. automated testing/metrics).

    opened by kabute 6
  • Junit shows all tests as failed in jenkins

    Junit shows all tests as failed in jenkins

    This is super exciting there is now junit support! Our Jenkins job is showing all of the tests as failed in the results. I don't have any output to paste right now. If you are unable to see what's going on I'll be happy to try and recreate in my private profile.

    • run tests and pipe the output to a results file (the --save flag was storing the file in the Jenkins workspace in a location our job does not have access to.)
    • use junit <path to results file> to chart the results
    question 
    opened by gabeduke 6
  • Calling bottle :unneeded is deprecated! There is no replacement

    Calling bottle :unneeded is deprecated! There is no replacement




    Describe the bug When running brew commands it outputs that deailed/popeye is calling :unneeded which is deprecated

    To Reproduce Steps to reproduce the behavior:

    1. run "brew update; brew upgrade; brew upgrade --cask; brew cleanup;"
    2. check output
    3. should say something like: "Warning: Calling bottle :unneeded is deprecated! There is no replacement. Please report this issue to the derailed/popeye tap (not Homebrew/brew or Homebrew/core): /usr/local/Homebrew/Library/Taps/derailed/homebrew-popeye/Formula/popeye.rb:9"

    Expected behavior no answer to this. The error should be gone

    Screenshots Screenshot 2021-10-22 at 08 45 12

    Versions (please complete the following information):

    • OS: MacOS 11.6 with latest brew
    • popeye 0.9.7
    • kubernetes-cli 1.22.2
    opened by robspekschoor 0
  • Incorrect counting of scanned resources or non-intuitive scoring

    Incorrect counting of scanned resources or non-intuitive scoring




    Describe the bug When I exclude codes from being scanned, the end result is that my score is lower whereas I'd expect the opposite.

    To Reproduce Steps to reproduce the behavior:

    1. On my cluster and with the current excludes I have in place (105 & 108), under the report for this namespace I have a score of 42 for pods: 14 scanned, 0 error, 8 warning, 6 info, 0 OK.
    2. If I add 206 and 301 to excluded the results become: score 0, scanned 2, error 0, warning 2, info 0, OK 0.
    3. For completeness, this is the relevant spinach config:
    popeye:
      excludes:
        v1/pods:
        - name: rx:.*
          codes:
          - 105 # named ports for liveness/readiness probes
          - 108 # unnamed ports
          - 206 # pod disruption budget
          - 301 # service account mounted
    ...
    

    Expected behavior If I exclude a code that appears on all of my pods (and many of these have only this particular code), the report ends up giving a score of zero along with a count of zero for 'OK'. This seems like unusual behavior to me; if I filter out codes I would expect the pods that have only this code would be listed in the 'OK' state and thus the score would be higher, not lower.

    Screenshots If applicable, add screenshots to help explain your problem.

    Versions (please complete the following information):

    • OS: Ubuntu 20.04.3
    • Popeye latest master (f150b2a)
    • K8s microk8s v1.21.5-3

    Additional context I think this might be related or a repro of #111 but not entirely sure - more then happy to continue the discussion there if necessary.

    opened by taintedkernel 0
  • False positive on NetworkPolicy using a catch all namespaceSelector

    False positive on NetworkPolicy using a catch all namespaceSelector




    Describe the bug Popeye reports the following NetworkPolicy with a false positive on POP-1201.

    To Reproduce

    apiVersion: networking.k8s.io/v1
    kind: NetworkPolicy
    metadata:
      name: foo
      namespace: bar
    spec:
      podSelector:
        matchLabels:
          app: foo
      ingress:
        - from:
            - namespaceSelector: {}
    

    Expected behavior Popeye should support NetworkPolicies that use empty namespaceSelectors and not consider this a warning.

    Versions (please complete the following information):

    • OS: windows
    • Popeye: 0.9.7
    • K8s: 1.19
    opened by meier-christoph 0
  • Can't filter containers in spinach.yaml for deployments

    Can't filter containers in spinach.yaml for deployments




    Describe the bug Can't filter containers in spinach.yaml for deployments. Filtering by deployment FQN or code works as expected.

    To Reproduce

    1. Have a report with a deployment warning for no resources/limits on a container

    image

    1. Create a spinach.yaml file with the following content
    popeye:
      excludes:
        apps/v1/deployments:
          - name: linkerd/linkerd-controller
            codes:
              - 106
    
    1. Run with the spinach.yaml passed in .\popeye.exe --save --out html -l warn -f .\spinach.yaml

    As expected, the warn isn't there

    image

    1. Create a spinach.yaml file with the following content (adding the container)
    popeye:
      excludes:
        apps/v1/deployments:
          - name: linkerd/linkerd-controller
            containers:
              - public-api
            codes:
              - 106
    
    1. Run with the spinach.yaml passed in .\popeye.exe --save --out html -l warn -f .\spinach.yaml

    2. Container isn't picked up by configuration, fails back to showing non-matched deployment

    image

    Expected behavior (I think) it should allow filtering to containers, unless this is a limitation of the deployments API and the way they're referenced in its spec.

    kubectl get deployment -n linkerd linkerd-controller -o yaml
    apiVersion: apps/v1
    kind: Deployment
    metadata:
    ...
      name: linkerd-controller
      namespace: linkerd
    ...
    spec:
    ...
      template:
    ...
        spec:
          containers:
          - args:
            - public-api
    ...
    ...
            name: public-api
    
    

    Versions (please complete the following information):

    • OS: Windows 10
    • Popeye: 0.9.7 (also on 0.9.0)
    • K8s 1.20.0

    Additional context Ideally we don't want to filter the whole deployment or popeye code, as any other containers would require limits/requests

    opened by 1stewart 0
  • DRY up exclusions

    DRY up exclusions

    I have a few charts for services that have warnings that I can't fix, because there's no configuration for them in the chart's values.

    In order to quiet the noise from these charts, I need to specify every resource type they create, and then exclude them via name, a la:

    popeye:
      excludes:
        v1/pods:
        - name: rx:infrastructure/aws-load-balancer-controller.*
        - name: rx:infrastructure/metrics-server.*
        - name: rx:istio-system/istiod.*
        - name: rx:istio-system/istio-ingressgateway.*
        apps/v1/daemonsets:
        - name: rx:infrastructure/aws-load-balancer-controller.*
        - name: rx:infrastructure/metrics-server.*
        - name: rx:istio-system/istiod.*
        - name: rx:istio-system/istio-ingressgateway.*
        apps/v1/deployments:
        - name: rx:infrastructure/aws-load-balancer-controller.*
        - name: rx:infrastructure/metrics-server.*
        - name: rx:istio-system/istiod.*
        - name: rx:istio-system/istio-ingressgateway.*
        v1/serviceaccounts:
        - name: rx:infrastructure/aws-load-balancer-controller.*
        - name: rx:infrastructure/metrics-server.*
        - name: rx:istio-system/istiod.*
        - name: rx:istio-system/istio-ingressgateway.*
        v1/services:
        - name: rx:infrastructure/aws-load-balancer-controller.*
        - name: rx:infrastructure/metrics-server.*
        - name: rx:istio-system/istiod.*
        - name: rx:istio-system/istio-ingressgateway.*
        ...
    

    This is pretty bulky.

    I can think of two ways to support this:

    1. Support a method of reference in sequences. .gitlab-ci.yml does this by making their sequences support 1 extra layer of sequences, and then spreads them, so that you can reference an anchor, a la:
    duplicateNames: &duplicateNames
    - name: rx:infrastructure/aws-load-balancer-controller.*
    - name: rx:infrastructure/metrics-server.*
    - name: rx:istio-system/istiod.*
    - name: rx:istio-system/istio-ingressgateway.*
    
    popeye:
      excludes:
        v1/pods:
        - *duplicateNames
        apps/v1/daemonsets:
        - *duplicateNames
        apps/v1/deployments:
        - *duplicateNames
        v1/serviceaccounts:
        - *duplicateNames
        v1/services:
        - *duplicateNames
    

    which would be unwrapped as:

    # just showing the one example
    popeye:
      excludes:
        v1/pods:
        - - name: rx:infrastructure/aws-load-balancer-controller.*
          - name: rx:infrastructure/metrics-server.*
          - name: rx:istio-system/istiod.*
          - name: rx:istio-system/istio-ingressgateway.*
    

    The sequence containing sequences would then simply be spread into one sequence, so it only supports 2-dimensional lists, but is converted to a 1-dimensional list at runtime. I like this, because it's a pretty clean workaround for what I consider to be a missing feature in yaml.

    They also have an includes key on a number of maps for imports like this.

    1. Making resource types to scan support wildcards, like names.

    I don't see any documentation on this feature, so I assume it doesn't exist.

    opened by kevin-lindsay-1 0
  • [Docs] Description, reasoning, and examples for codes

    [Docs] Description, reasoning, and examples for codes




    Is your feature request related to a problem? Please describe. I would call it a documentation problem more than anything.

    Describe the solution you'd like When I use popeye via k9s, I get the POP- error/warning codes and their messages, similar to something like eslint, however generally the purpose of these codes it to look them up, and from what I've seen the only "documentation" on these codes is simply an index of all the codes.

    It would be good to describe the code in greater detail, and preferably include good and bad examples that would trigger such a code, similar to how eslint does in their docs. For example, https://eslint.org/docs/rules/no-shadow.

    Maybe including a link in the output code directly to the docs would be a good idea, too.

    Describe alternatives you've considered Can't really think of any. In my opinion, eslint sets a really good bar for quality documentation regarding error codes like this.

    opened by kevin-lindsay-1 0
  • Run popeye on kube behind a http proxy

    Run popeye on kube behind a http proxy

    My cluster isn't accessible directly but through a proxy (tinyproxy) and I run all kubectl command like this: HTTP_PROXY=host:port kubectl get pods


    Is your feature request related to a problem? Please describe. Without a proxy configuration, popeye can't connect to the cluster and I get a connection timeout.

    Describe the solution you'd like I'd like to run popeye through my proxy too so I'd want to define the proxy address with a cmd parameter and if possible with docker

    opened by lineberty-internal 0
  • How to fix POP-300 : Using โ€œdefaultโ€ ServiceAccount ?

    How to fix POP-300 : Using โ€œdefaultโ€ ServiceAccount ?

    Hi,

    Sorry for this question, if it's not the correct place to ask, etc...

    But I have been trying to fix POP-300 ("Using "default" ServiceAccount") but I am not really sure how to do that. I know I can set a different serviceAccountName but I don't have any service account I wish to use.

    I am just running a simple Java pod/deployment, what should I do in this case?

    If I should do nothing and ignore POP-300 by filtering it out, then I wonder why this is set to severity 2 (warning).

    question 
    opened by michelzanini 2
  • use of Chocolatey package manager on Windows

    use of Chocolatey package manager on Windows

    for Windows installation, there should be a Chocolatey package that is push for each release.

    it's easier to update with Chocolatey and downloading the release manually.

    opened by survivant 0
  • Rule to check a list of allowed registries

    Rule to check a list of allowed registries

    Adds rule 113 which makes possible to add a list of allowed registries to pull images from. Otherwise, all images are allowed. Images which do not contain an explicit registry are checked as docker.io

    This PR would fix #177

    opened by danibaeyens 3
Releases(v0.9.7)
Owner
Fernand Galiana
Owner of Imhotep Software a consultancy specializing in architecture, software development and corporate training for GO and Kubernetes
Fernand Galiana
Managing your Kubernetes clusters (including public, private, edge, etc) as easily as visiting the Internet

Clusternet Managing Your Clusters (including public, private, hybrid, edge, etc) as easily as Visiting the Internet. Clusternet (Cluster Internet) is

Clusternet 235 Oct 13, 2021
Lightweight, CRD based envoy control plane for kubernetes

Lighweight, CRD based Envoy control plane for Kubernetes: Implemented as a Kubernetes Operator Deploy and manage an Envoy xDS server using the Discove

null 36 Oct 20, 2021
Kubernetes Cluster API Provider AWS

Kubernetes Cluster API Provider AWS Kubernetes-native declarative infrastructure for AWS. What is the Cluster API Provider AWS The Cluster API brings

null 0 Oct 23, 2021
๐Ÿถ Kubernetes CLI To Manage Your Clusters In Style!

K9s - Kubernetes CLI To Manage Your Clusters In Style! K9s provides a terminal UI to interact with your Kubernetes clusters. The aim of this project i

Fernand Galiana 13.8k Oct 20, 2021
An operator for managing ephemeral clusters in GKE

Test Cluster Operator for GKE This operator provides an API-driven cluster provisioning for integration and performance testing of software that integ

Isovalent 28 Mar 19, 2021
An operator to support Haschicorp Vault configuration workflows from within Kubernetes

Vault Config Operator This operator helps set up Vault Configurations. The main intent is to do so such that subsequently pods can consume the secrets

Red Hat Communities of Practice 2 Oct 14, 2021
An operator to support Haschicorp Vault configuration workflows from within Kubernetes

Vault Config Operator This operator helps set up Vault Configurations. The main intent is to do so such that subsequently pods can consume the secrets

null 0 Oct 18, 2021
Kubedd โ€“ Check migration issues of Kubernetes Objects while K8s upgrade

Kubedd โ€“ Check migration issues of Kubernetes Objects while K8s upgrade

Devtron Labs 95 Oct 20, 2021
Kubernetes Operator for MySQL NDB Cluster.

MySQL NDB Operator The MySQL NDB Operator is a Kubernetes operator for managing a MySQL NDB Cluster setup inside a Kubernetes Cluster. This is in prev

MySQL 8 Oct 16, 2021
KinK is a helper CLI that facilitates to manage KinD clusters as Kubernetes pods. Designed to ease clusters up for fast testing with batteries included in mind.

kink A helper CLI that facilitates to manage KinD clusters as Kubernetes pods. Table of Contents kink (KinD in Kubernetes) Introduction How it works ?

Trendyol Open Source 298 Oct 16, 2021
Simple Kubernetes real-time dashboard and management.

Skooner - Kubernetes Dashboard We are changing our name from k8dash to Skooner! Please bear with us as we update our documentation and codebase to ref

null 859 Oct 20, 2021
This process installs onto kubernetes cluster(s) and provisions workloads designated by the uffizzi interface

Uffizzi Cloud Resource Controller This application connects to a Kubernetes (k8s) Cluster to provision Uffizzi users' workloads on their behalf. While

Uffizzi 3 Oct 20, 2021
Kubernetes IN Docker - local clusters for testing Kubernetes

kind is a tool for running local Kubernetes clusters using Docker container "nodes".

Kubernetes SIGs 8.6k Oct 15, 2021
the simplest testing framework for Kubernetes controller.

KET(Kind E2e Test framework) KET is the simplest testing framework for Kubernetes controller. KET is available as open source software, and we look fo

Riita 15 Sep 18, 2021
Kilo is a multi-cloud network overlay built on WireGuard and designed for Kubernetes (k8s + wg = kg)

Kilo Kilo is a multi-cloud network overlay built on WireGuard and designed for Kubernetes. Overview Kilo connects nodes in a cluster by providing an e

Lucas Servรฉn Marรญn 1.1k Oct 15, 2021
Enterprise-grade container platform tailored for multicloud and multi-cluster management

KubeSphere Container Platform What is KubeSphere English | ไธญๆ–‡ KubeSphere is a distributed operating system providing cloud native stack with Kubernete

KubeSphere 7.1k Oct 21, 2021
Large-scale Kubernetes cluster diagnostic tool.

English | ็ฎ€ไฝ“ไธญๆ–‡ KubeProber What is KubeProber? KubeProber is a diagnostic tool designed for large-scale Kubernetes clusters. It is used to perform diag

Erda 35 Oct 20, 2021
kubetnl tunnels TCP connections from within a Kubernetes cluster to a cluster-external endpoint, e.g. to your local machine. (the perfect complement to kubectl port-forward)

kubetnl kubetnl (kube tunnel) is a command line utility to tunnel TCP connections from within a Kubernetes to a cluster-external endpoint, e.g. to you

null 3 Sep 7, 2021
General Pod Autoscaler(GPA) is a extension for K8s HPA, which can be used not only for serving, also for game.

Introduction General Pod Autoscaler(GPA) is a extension for K8s HPA, which can be used not only for serving, also for game. Features Compatible with a

Open Cloud-native Game-application Initiative 11 Oct 12, 2021