Report
I reported similar issue 3448 before, the root cause was null latest data point to cause the exception. Recently I finally have time to test the query again. This time, I make sure all data points have data and with the same number of data points within the period
Query 1: sum:trace.express.request.hits{service:foo,env:bar}.as_rate() , keda can get the metrics of this query without any problem
# curl result of query 1
{
"status": "ok",
"resp_version": 1,
"series": [
{
"end": 1669106969000,
"attributes": {},
"metric": "trace.express.request.hits",
"interval": 10,
"tag_set": [],
"start": 1669106910000,
"length": 6,
"query_index": 0,
"aggr": "sum",
"scope": "env:bar,service:foo",
"pointlist": [
[
1669106910000,
417.3
],
[
1669106920000,
465.4
],
[
1669106930000,
447.1
],
[
1669106940000,
440.9
],
[
1669106950000,
440.8
],
[
1669106960000,
748.5000000000001
]
],
"expression": "sum:trace.express.request.hits{env:bar,service:foo}.as_rate()",
"unit": [
{
"family": "cache",
"scale_factor": 1,
"name": "hit",
"short_name": null,
"plural": "hits",
"id": 39
},
{
"family": "time",
"scale_factor": 1,
"name": "second",
"short_name": "s",
"plural": "seconds",
"id": 11
}
],
"display_name": "trace.express.request.hits"
}
],
"to_date": 1669106966000,
"query": "sum:trace.express.request.hits{service:foo,env:bar}.as_rate()",
"message": "",
"res_type": "time_series",
"times": [],
"from_date": 1669106906000,
"group_by": [],
"values": []
}
Query 2: avg:kubernetes.cpu.requests{service:foo,env:bar}.rollup(10) , keda can get the metrics of this query without any problem```
# curl result of query 2
{
"status": "ok",
"resp_version": 1,
"series": [
{
"end": 1669106969000,
"attributes": {},
"metric": "kubernetes.cpu.requests",
"interval": 10,
"tag_set": [],
"start": 1669106910000,
"length": 6,
"query_index": 0,
"aggr": "avg",
"scope": "env:bar,service:foo",
"pointlist": [
[
1669106910000,
0.11999999731779099
],
[
1669106920000,
0.11999999731779099
],
[
1669106930000,
0.11999999731779099
],
[
1669106940000,
0.11999999731779099
],
[
1669106950000,
0.11999999731779099
],
[
1669106960000,
0.11999999731779099
]
],
"expression": "avg:kubernetes.cpu.requests{env:bar,service:foo}.rollup(10)",
"unit": [
{
"family": "cpu",
"scale_factor": 1,
"name": "core",
"short_name": null,
"plural": "cores",
"id": 31
},
null
],
"display_name": "kubernetes.cpu.requests"
}
],
"to_date": 1669106966000,
"query": "avg:kubernetes.cpu.requests{service:foo,env:bar}.rollup(10)",
"message": "",
"res_type": "time_series",
"times": [],
"from_date": 1669106906000,
"group_by": [],
"values": []
}
If I combine above two queries into "query 1 / query 2"
Query 3: sum:trace.express.request.hits{service:foo,env:bar}.as_rate()/avg:kubernetes.cpu.requests{service:foo,env:bar}.rollup(10)
The curl result looks ok but KEDA is failed to get the metrics
# curl result if query 3
{
"status": "ok",
"resp_version": 1,
"series": [
{
"end": 1669106969000,
"attributes": {},
"metric": "(trace.express.request.hits / kubernetes.cpu.requests)",
"interval": 10,
"tag_set": [],
"start": 1669106910000,
"length": 6,
"query_index": 0,
"aggr": "sum",
"scope": "env:bar,service:foo",
"pointlist": [
[
1669106910000,
3477.500077728184
],
[
1669106920000,
3878.3334200208405
],
[
1669106930000,
3725.8334166122
],
[
1669106940000,
3674.166748790693
],
[
1669106950000,
3673.3334154387335
],
[
1669106960000,
6237.500139418993
]
],
"expression": "(sum:trace.express.request.hits{env:bar,service:foo}.as_rate() / avg:kubernetes.cpu.requests{env:bar,service:foo}.rollup(10))",
"unit": null,
"display_name": "(trace.express.request.hits / kubernetes.cpu.requests)"
}
],
"to_date": 1669106966000,
"query": "sum:trace.express.request.hits{service:foo,env:bar}.as_rate()/avg:kubernetes.cpu.requests{service:foo,env:bar}.rollup(10)",
"message": "",
"res_type": "time_series",
"times": [],
"from_date": 1669106906000,
"group_by": [],
"values": []
}
HPA events
Warning FailedGetExternalMetric 31m (x8 over 23h) horizontal-pod-autoscaler unable to get external metric development/s0-datadog-sum-trace-express-request-hits/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: foo,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: no matching metrics found for s0-datadog-sum-trace-express-request-hits
Warning FailedComputeMetricsReplicas 31m (x8 over 23h) horizontal-pod-autoscaler invalid metrics (1 invalid out of 1), first error is: failed to get s0-datadog-sum-trace-express-request-hits external metric: unable to get external metric development/s0-datadog-sum-trace-express-request-hits/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: foo,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: no matching metrics found for s0-datadog-sum-trace-express-request-hits
If I remove as_rate() from Query 3
Query 4: sum:trace.express.request.hits{service:foo,env:bar}/avg:kubernetes.cpu.requests{service:foo,env:bar}.rollup(10)
KEDA can get the metrics without any problem
# curl result of query 4
"status": "ok",
"resp_version": 1,
"series": [
{
"end": 1669106960000,
"attributes": {},
"metric": "(trace.express.request.hits / kubernetes.cpu.requests)",
"interval": 1,
"tag_set": [],
"start": 1669106910000,
"length": 6,
"query_index": 0,
"aggr": "sum",
"scope": "service:foo,env:bar",
"pointlist": [
[
1669106910000,
64426.34634984409
],
[
1669106920000,
66898.41915361531
],
[
1669106930000,
65685.75394920415
],
[
1669106940000,
66132.75977946246
],
[
1669106950000,
66287.59952733
],
[
1669106960000,
66472.42117646785
]
],
"expression": "(sum:trace.express.request.hits{service:foo,env:bar} / avg:kubernetes.cpu.requests{service:foo,env:bar}.rollup(10))",
"unit": null,
"display_name": "(trace.express.request.hits / kubernetes.cpu.requests)"
}
],
"to_date": 1669106966000,
"query": "sum:trace.express.request.hits{service:foo,env:bar}/avg:kubernetes.cpu.requests{service:foo,env:bar}.rollup(10)",
"message": "",
"res_type": "time_series",
"times": [],
"from_date": 1669106906000,
"group_by": [],
"values": []
}
Expected Behavior
KEDA can get the metrics without any problem as long as it is working with curl for same query
Actual Behavior
Explained above
Steps to Reproduce the Problem
Explained above
Logs from KEDA operator
E1123 09:05:17.876484 1 datadog_scaler.go:318] keda_metrics_adapter/datadog_scaler "msg"="error getting metrics from Datadog" "error"="error when retrieving Datadog metrics: Get \"https://api.datadoghq.com/api/v1/query?from=1669194254&query=sum%3Atrace.express.request.hits%7Bservice%3Afoo%2Cenv%3Abar%7D.as_rate%28%29%2Favg%3Akubernetes.cpu.requests%7Bservice%3Afoo%2Cenv%3Abar%7D.rollup%2810%29&to=1669194314\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" "name"="foo" "namespace"="bar" "type"="ScaledObject"
KEDA Version
2.8.1
Kubernetes Version
< 1.23
Platform
Amazon Web Services
Scaler Details
Datadog
Anything else?
No response
bug
![[Datadog scaler] Failed to get the metrics for some specific datadog query](https://avatars.githubusercontent.com/u/23068316?v=4)
0 Dec 28, 2021
201 Nov 21, 2022
0 Nov 28, 2021
64.6k Nov 23, 2022
1 Nov 23, 2021
345 Nov 23, 2022
5 Sep 5, 2022
106 Nov 24, 2022
6 May 16, 2022
10 Nov 15, 2022
1 Nov 22, 2021
13 Oct 17, 2022
8 Sep 20, 2022
5 Dec 13, 2021
0 Jan 23, 2022
0 Dec 7, 2021
4.6k Nov 27, 2022
30 Jul 28, 2022