Report
Consider below scaledobject, and a dummy k8s deployment called "demo"
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: demo-scaledobject
namespace: default
spec:
scaleTargetRef:
name: demo
minReplicaCount: 0
cooldownPeriod: 30
triggers:
- type: prometheus
metadata:
serverAddress: http://192.168.1.165:9090
metricName: visit_counter_total
threshold: '2'
query: visit_counter_total
disableScaleToZero: 'true'
Issue:
Prometheus reported visit_counter_total=38, everything good 19 pods got created.
Prometheus reported visit_counter_total=39, pods stayed 19 (which is fine)
Prometheus reported visit_counter_total=40, ISSUE: NUMBER OF PODS STUCK AT 19
% k get pods --no-headers | wc -l
19
LOGS: WHEN I INCREASED visit_counter_total FROM 38 TO 39 (WAITED FOR 3 MINS) TO 40 (WAITED FOR 10 MINS)
1.6563435558217254e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "demo-scaledobject", "namespace": "default"}
1.6563435558218324e+09 DEBUG controller.scaledobject Parsed Group, Version, Kind, Resource {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "demo-scaledobject", "namespace": "default", "GVK": "apps/v1.Deployment", "Resource": "deployments"}
1.656343555832207e+09 DEBUG controller.scaledobject ScaledObject is defined correctly and is ready for scaling {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "demo-scaledobject", "namespace": "default"}
1.6563435708244784e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "demo-scaledobject", "namespace": "default"}
1.6563435708245707e+09 DEBUG controller.scaledobject Parsed Group, Version, Kind, Resource {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "demo-scaledobject", "namespace": "default", "GVK": "apps/v1.Deployment", "Resource": "deployments"}
1.6563435708350515e+09 DEBUG controller.scaledobject ScaledObject is defined correctly and is ready for scaling {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "demo-scaledobject", "namespace": "default"}
1.6563435791322286e+09 DEBUG scalehandler Scaler for scaledObject is active {"scaledobject.Name": "demo-scaledobject", "scaledObject.Namespace": "default", "scaleTarget.Name": "demo", "Metrics Name": "s0-prometheus-visit_counter_total"}
1.6563436090996041e+09 DEBUG scalehandler Scaler for scaledObject is active {"scaledobject.Name": "demo-scaledobject", "scaledObject.Namespace": "default", "scaleTarget.Name": "demo", "Metrics Name": "s0-prometheus-visit_counter_total"}
1.6563436390634325e+09 DEBUG scalehandler Scaler for scaledObject is active {"scaledobject.Name": "demo-scaledobject", "scaledObject.Namespace": "default", "scaleTarget.Name": "demo", "Metrics Name": "s0-prometheus-visit_counter_total"}
1.65634366903128e+09 DEBUG scalehandler Scaler for scaledObject is active {"scaledobject.Name": "demo-scaledobject", "scaledObject.Namespace": "default", "scaleTarget.Name": "demo", "Metrics Name": "s0-prometheus-visit_counter_total"}
1.6563436989966836e+09 DEBUG scalehandler Scaler for scaledObject is active {"scaledobject.Name": "demo-scaledobject", "scaledObject.Namespace": "default", "scaleTarget.Name": "demo", "Metrics Name": "s0-prometheus-visit_counter_total"}
1.6563437289631693e+09 DEBUG scalehandler Scaler for scaledObject is active {"scaledobject.Name": "demo-scaledobject", "scaledObject.Namespace": "default", "scaleTarget.Name": "demo", "Metrics Name": "s0-prometheus-visit_counter_total"}
1.6563437589283757e+09 DEBUG scalehandler Scaler for scaledObject is active {"scaledobject.Name": "demo-scaledobject", "scaledObject.Namespace": "default", "scaleTarget.Name": "demo", "Metrics Name": "s0-prometheus-visit_counter_total"}
1.6563
Expected Behavior
POD COUNT SHOULD HAVE GONE UP TO 20.
Actual Behavior
POD COUNT STUCK AT 19
Steps to Reproduce the Problem
- INSTALL PROMETHEUS LOCALLY
- RUN THIS SAMPLE MICROSERVICE https://tanzu.vmware.com/developer/guides/spring-prometheus/
- INCREASE visit_counter_total FROM 38 TO 39 (WAITED FOR 3 MINS) TO 40 (WAITED FOR 10 MINS)
Logs from KEDA operator
1.6563435558217254e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "demo-scaledobject", "namespace": "default"}
1.6563435558218324e+09 DEBUG controller.scaledobject Parsed Group, Version, Kind, Resource {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "demo-scaledobject", "namespace": "default", "GVK": "apps/v1.Deployment", "Resource": "deployments"}
1.656343555832207e+09 DEBUG controller.scaledobject ScaledObject is defined correctly and is ready for scaling {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "demo-scaledobject", "namespace": "default"}
1.6563435708244784e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "demo-scaledobject", "namespace": "default"}
1.6563435708245707e+09 DEBUG controller.scaledobject Parsed Group, Version, Kind, Resource {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "demo-scaledobject", "namespace": "default", "GVK": "apps/v1.Deployment", "Resource": "deployments"}
1.6563435708350515e+09 DEBUG controller.scaledobject ScaledObject is defined correctly and is ready for scaling {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "demo-scaledobject", "namespace": "default"}
1.6563435791322286e+09 DEBUG scalehandler Scaler for scaledObject is active {"scaledobject.Name": "demo-scaledobject", "scaledObject.Namespace": "default", "scaleTarget.Name": "demo", "Metrics Name": "s0-prometheus-visit_counter_total"}
1.6563436090996041e+09 DEBUG scalehandler Scaler for scaledObject is active {"scaledobject.Name": "demo-scaledobject", "scaledObject.Namespace": "default", "scaleTarget.Name": "demo", "Metrics Name": "s0-prometheus-visit_counter_total"}
1.6563436390634325e+09 DEBUG scalehandler Scaler for scaledObject is active {"scaledobject.Name": "demo-scaledobject", "scaledObject.Namespace": "default", "scaleTarget.Name": "demo", "Metrics Name": "s0-prometheus-visit_counter_total"}
1.65634366903128e+09 DEBUG scalehandler Scaler for scaledObject is active {"scaledobject.Name": "demo-scaledobject", "scaledObject.Namespace": "default", "scaleTarget.Name": "demo", "Metrics Name": "s0-prometheus-visit_counter_total"}
1.6563436989966836e+09 DEBUG scalehandler Scaler for scaledObject is active {"scaledobject.Name": "demo-scaledobject", "scaledObject.Namespace": "default", "scaleTarget.Name": "demo", "Metrics Name": "s0-prometheus-visit_counter_total"}
1.6563437289631693e+09 DEBUG scalehandler Scaler for scaledObject is active {"scaledobject.Name": "demo-scaledobject", "scaledObject.Namespace": "default", "scaleTarget.Name": "demo", "Metrics Name": "s0-prometheus-visit_counter_total"}
1.6563437589283757e+09 DEBUG scalehandler Scaler for scaledObject is active {"scaledobject.Name": "demo-scaledobject", "scaledObject.Namespace": "default", "scaleTarget.Name": "demo", "Metrics Name": "s0-prometheus-visit_counter_total"}
1.6563
KEDA Version
2.7.1
Kubernetes Version
1.23
Platform
Other
Scaler Details
Prometheus
Anything else?
ALL THIS IN THE LOCAL ENVIRONMENT USING KIND
bug