I have an AKS test-cluster, with a few sample apps deployed. With Istio 1.0.2 both the app container and the istio-proxy container start as expected. When removing Istio and the sample app, installing Istio daily build istio-release-1.1-20181021-09-15 from scratch, and then redeploying the sample apps, the sidecar proxy fails to start, while logging the following:

<*snip* start>

2018-10-24T12:29:06.122324Z	info	Envoy command: [-c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster nginx-1-pod.default --service-node sidecar~10.244.0.17~nginx-1-pod-6d86955d8d-xhj79.default~default.svc.cluster.local --max-obj-name-len 189 --allow-unknown-fields -l warning --v2-config-only]
[2018-10-24 12:29:06.144][19][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:243] gRPC config stream closed: 14, no healthy upstream
[2018-10-24 12:29:06.144][19][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:43] Unable to establish new stream
[2018-10-24 12:29:07.431][19][warning][config] src/envoy/utils/mixer_control.cc:171] ExtractInfo  metadata missing:
[2018-10-24 12:29:07.432][19][warning][config] src/envoy/utils/mixer_control.cc:171] ExtractInfo  metadata missing:

<*snip* multiple identical log lines>

[2018-10-24 12:29:07.498][48][warning][config] src/envoy/utils/mixer_control.cc:171] ExtractInfo  metadata missing:
[2018-10-24 12:29:07.498][48][warning][config] src/envoy/utils/mixer_control.cc:171] ExtractInfo  metadata missing:
2018-10-24T12:29:07.997196Z	info	Envoy proxy is NOT ready: 3 errors occurred:

* failed checking application ports. listeners="0.0.0.0:15090","10.244.0.17:8081","10.0.253.20:443","10.0.253.20:31400","10.0.184.140:15011","10.0.69.228:42422","10.0.178.7:443","10.0.0.1:443","10.0.132.238:80","10.0.165.166:443","10.0.225.39:443","0.0.0.0:15004","0.0.0.0:80","0.0.0.0:15030","0.0.0.0:8080","0.0.0.0:9093","0.0.0.0:8060","0.0.0.0:8081","0.0.0.0:9091","0.0.0.0:15029","0.0.0.0:15031","0.0.0.0:15010","0.0.0.0:9901","0.0.0.0:15032","0.0.0.0:9090","10.244.0.17:15020","0.0.0.0:15001"
* envoy missing listener for inbound application port: 0
* envoy missing listener for inbound application port: 80
2018-10-24T12:29:09.997207Z	info	Envoy proxy is NOT ready: 3 errors occurred:

* failed checking application ports. listeners="0.0.0.0:15090","10.244.0.17:8081","10.0.253.20:443","10.0.253.20:31400","10.0.184.140:15011","10.0.69.228:42422","10.0.178.7:443","10.0.0.1:443","10.0.132.238:80","10.0.165.166:443","10.0.225.39:443","0.0.0.0:15004","0.0.0.0:80","0.0.0.0:15030","0.0.0.0:8080","0.0.0.0:9093","0.0.0.0:8060","0.0.0.0:8081","0.0.0.0:9091","0.0.0.0:15029","0.0.0.0:15031","0.0.0.0:15010","0.0.0.0:9901","0.0.0.0:15032","0.0.0.0:9090","10.244.0.17:15020","0.0.0.0:15001"
* envoy missing listener for inbound application port: 0
* envoy missing listener for inbound application port: 80

<etc., repeating every two seconds>

In the listener list, 10.244.0.17 is the pod IP. According to https://istio.io/help/ops/traffic-management/proxy-cmd/#deep-dive-into-envoy-configuration, I should see A virtual listener on the pod IP for each exposed port for inbound traffic., but I can only find entries for 10.244.0.17:15020 and 10.244.0.17:8081, the latter being the service port... What could have gone wrong here?

Sample app configuration: sample_app.yaml.txt

Version Kubernetes: 1.11.2 Istio: Version:"release-1.1-20181021-09-15", GitRevision:"bd24a62648c07e24ca655c39727aeb0e4761919a"

Installation Using Helm 2.9.1:

helm install ${ISTIO_HOME}/install/kubernetes/helm/istio --name istio --namespace istio-system --tls --wait \
    --set global.configValidation=true \
    --set sidecarInjectorWebhook.enabled=true \
    --set gateways.istio-ingressgateway.loadBalancerIP=${PUBLIC_IP}

Environment MS Azure, AKS

Describe the bug I have a web application in a Pod running with an Istio sidecar and I get random 503 errors from the sidecar itself (Envoy) not even reaching the web application. This is happening with a very low demand. In the logs below you will see a request to an endpoint named /logout which is throwing the 503 error and not reaching the web application whatsoever.

Expected behavior Everything is forwarded to the web application.

Steps to reproduce the bug Simple test invoking several endpoints in the web application, with a very small load.

Version Istio --> 1.0.1 Kubernetes --> 1.11.3

Installation istio.yaml - use this generated file for installation without authentication enabled

Environment Bare metal servers, on premise, OL7.5, Linux 4.17.5-1.el7.elrepo.x86_64 #1 SMP Sun Jul 8 10:40:01 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux

Logs

preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.958][95][debug][http] external/envoy/source/common/http/conn_manager_impl.cc:190] [C32] new stream
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.958][95][trace][http] external/envoy/source/common/http/http1/codec_impl.cc:309] [C32] completed header: key=host value=pp-helpers.test.oami.eu
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.958][95][trace][http] external/envoy/source/common/http/http1/codec_impl.cc:309] [C32] completed header: key=cookie value=CLASSIFSESSIONID=MjM2N2MyOWUtODFhOS00ZGVjLTkwNjktOTQ5NzhlZTc2ZDk1
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.958][95][trace][http] external/envoy/source/common/http/http1/codec_impl.cc:309] [C32] completed header: key=content-type value=application/x-www-form-urlencoded
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.958][95][trace][http] external/envoy/source/common/http/http1/codec_impl.cc:309] [C32] completed header: key=content-length value=0
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.958][95][trace][http] external/envoy/source/common/http/http1/codec_impl.cc:309] [C32] completed header: key=user-agent value=Apache-HttpClient/4.5.5 (Java/1.8.0_151)
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.958][95][trace][http] external/envoy/source/common/http/http1/codec_impl.cc:309] [C32] completed header: key=x-forwarded-for value=10.133.0.44, 174.16.212.0
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.958][95][trace][http] external/envoy/source/common/http/http1/codec_impl.cc:309] [C32] completed header: key=x-forwarded-proto value=http
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.958][95][trace][http] external/envoy/source/common/http/http1/codec_impl.cc:309] [C32] completed header: key=x-envoy-external-address value=174.16.212.0
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.958][95][trace][http] external/envoy/source/common/http/http1/codec_impl.cc:309] [C32] completed header: key=x-request-id value=82aeb468-9f91-9b30-9482-2aa3591e1979
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.958][95][trace][http] external/envoy/source/common/http/http1/codec_impl.cc:309] [C32] completed header: key=x-envoy-decorator-operation value=classification-helper-ui.preprod-cb.svc.cluster.local:8080/api/classification*
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.958][95][trace][http] external/envoy/source/common/http/http1/codec_impl.cc:309] [C32] completed header: key=x-b3-traceid value=e4ece8f07a5a7481
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.958][95][trace][http] external/envoy/source/common/http/http1/codec_impl.cc:309] [C32] completed header: key=x-b3-spanid value=e4ece8f07a5a7481
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.958][95][trace][http] external/envoy/source/common/http/http1/codec_impl.cc:309] [C32] completed header: key=x-b3-sampled value=1
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.958][95][trace][http] external/envoy/source/common/http/http1/codec_impl.cc:309] [C32] completed header: key=x-istio-attributes value=Ck8KCnNvdXJjZS51aWQSQRI/a3ViZXJuZXRlczovL2lzdGlvLWluZ3Jlc3NnYXRld2F5LTVkOTk5Yzg3NTctZHY5cnMuaXN0aW8tc3lzdGVtCk4KE2Rlc3RpbmF0aW9uLnNlcnZpY2USNxI1Y2xhc3NpZmljYXRpb24taGVscGVyLXVpLnByZXByb2QtY2Iuc3ZjLmNsdXN0ZXIubG9jYWwKUwoYZGVzdGluYXRpb24uc2VydmljZS5ob3N0EjcSNWNsYXNzaWZpY2F0aW9uLWhlbHBlci11aS5wcmVwcm9kLWNiLnN2Yy5jbHVzdGVyLmxvY2FsClEKF2Rlc3RpbmF0aW9uLnNlcnZpY2UudWlkEjYSNGlzdGlvOi8vcHJlcHJvZC1jYi9zZXJ2aWNlcy9jbGFzc2lmaWNhdGlvbi1oZWxwZXItdWkKLQodZGVzdGluYXRpb24uc2VydmljZS5uYW1lc3BhY2USDBIKcHJlcHJvZC1jYgo2ChhkZXN0aW5hdGlvbi5zZXJ2aWNlLm5hbWUSGhIYY2xhc3NpZmljYXRpb24taGVscGVyLXVp
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.958][95][trace][http] external/envoy/source/common/http/http1/codec_impl.cc:405] [C32] headers complete
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.958][95][trace][http] external/envoy/source/common/http/http1/codec_impl.cc:309] [C32] completed header: key=x-envoy-original-path value=/api/classification/logout
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.958][95][trace][http] external/envoy/source/common/http/http1/codec_impl.cc:426] [C32] message complete
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.958][95][debug][filter] src/envoy/http/mixer/filter.cc:60] Called Mixer::Filter : Filter
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.958][95][debug][filter] src/envoy/http/mixer/filter.cc:204] Called Mixer::Filter : setDecoderFilterCallbacks
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.958][95][debug][http] external/envoy/source/common/http/conn_manager_impl.cc:889] [C32][S16141632285256672176] request end stream
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.958][95][debug][http] external/envoy/source/common/http/conn_manager_impl.cc:490] [C32][S16141632285256672176] request headers complete (end_stream=true):
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: ':authority', 'pp-helpers.test.oami.eu'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: ':path', '//logout'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: ':method', 'POST'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'cookie', 'CLASSIFSESSIONID=MjM2N2MyOWUtODFhOS00ZGVjLTkwNjktOTQ5NzhlZTc2ZDk1'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'content-type', 'application/x-www-form-urlencoded'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'content-length', '0'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'user-agent', 'Apache-HttpClient/4.5.5 (Java/1.8.0_151)'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'x-forwarded-for', '10.133.0.44, 174.16.212.0'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'x-forwarded-proto', 'http'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'x-envoy-external-address', '174.16.212.0'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'x-request-id', '82aeb468-9f91-9b30-9482-2aa3591e1979'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'x-envoy-decorator-operation', 'classification-helper-ui.preprod-cb.svc.cluster.local:8080/api/classification*'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'x-b3-traceid', 'e4ece8f07a5a7481'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'x-b3-spanid', 'e4ece8f07a5a7481'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'x-b3-sampled', '1'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'x-istio-attributes', 'Ck8KCnNvdXJjZS51aWQSQRI/a3ViZXJuZXRlczovL2lzdGlvLWluZ3Jlc3NnYXRld2F5LTVkOTk5Yzg3NTctZHY5cnMuaXN0aW8tc3lzdGVtCk4KE2Rlc3RpbmF0aW9uLnNlcnZpY2USNxI1Y2xhc3NpZmljYXRpb24taGVscGVyLXVpLnByZXByb2QtY2Iuc3ZjLmNsdXN0ZXIubG9jYWwKUwoYZGVzdGluYXRpb24uc2VydmljZS5ob3N0EjcSNWNsYXNzaWZpY2F0aW9uLWhlbHBlci11aS5wcmVwcm9kLWNiLnN2Yy5jbHVzdGVyLmxvY2FsClEKF2Rlc3RpbmF0aW9uLnNlcnZpY2UudWlkEjYSNGlzdGlvOi8vcHJlcHJvZC1jYi9zZXJ2aWNlcy9jbGFzc2lmaWNhdGlvbi1oZWxwZXItdWkKLQodZGVzdGluYXRpb24uc2VydmljZS5uYW1lc3BhY2USDBIKcHJlcHJvZC1jYgo2ChhkZXN0aW5hdGlvbi5zZXJ2aWNlLm5hbWUSGhIYY2xhc3NpZmljYXRpb24taGVscGVyLXVp'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'x-envoy-original-path', '/api/classification/logout'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.958][95][debug][filter] src/envoy/http/mixer/filter.cc:122] Called Mixer::Filter : decodeHeaders
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.958][95][debug][filter] ./src/envoy/utils/header_update.h:46] Mixer forward attributes set: ClEKCnNvdXJjZS51aWQSQxJBa3ViZXJuZXRlczovL2NsYXNzaWZpY2F0aW9uLWhlbHBlci11aS03NmY5Y2RjZDRkLXJ6Nm45LnByZXByb2QtY2I=
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.958][95][debug][router] external/envoy/source/common/router/router.cc:252] [C0][S14382728320977988265] cluster 'outbound|9091||istio-policy.istio-system.svc.cluster.local' match for URL '/istio.mixer.v1.Mixer/Check'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.958][95][debug][router] external/envoy/source/common/router/router.cc:303] [C0][S14382728320977988265] router decoding headers:
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: ':method', 'POST'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: ':path', '/istio.mixer.v1.Mixer/Check'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: ':authority', 'mixer'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: ':scheme', 'http'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'te', 'trailers'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'grpc-timeout', '5000m'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'content-type', 'application/grpc'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'x-b3-traceid', 'e4ece8f07a5a7481'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'x-b3-spanid', 'df547f7d4992bf4e'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'x-b3-parentspanid', 'e4ece8f07a5a7481'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'x-b3-sampled', '1'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'x-istio-attributes', 'ClEKCnNvdXJjZS51aWQSQxJBa3ViZXJuZXRlczovL2NsYXNzaWZpY2F0aW9uLWhlbHBlci11aS03NmY5Y2RjZDRkLXJ6Nm45LnByZXByb2QtY2I='
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'x-envoy-internal', 'true'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'x-forwarded-for', '174.16.213.27'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'x-envoy-expected-rq-timeout-ms', '5000'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.958][95][debug][pool] external/envoy/source/common/http/http2/conn_pool.cc:97] [C236] creating stream
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.958][95][debug][router] external/envoy/source/common/router/router.cc:971] [C0][S14382728320977988265] pool ready
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.958][95][trace][http2] external/envoy/source/common/http/http2/codec_impl.cc:492] [C236] send data: bytes=85
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.958][95][trace][connection] external/envoy/source/common/network/connection_impl.cc:326] [C236] writing 85 bytes, end_stream false
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.958][95][trace][http2] external/envoy/source/common/http/http2/codec_impl.cc:446] [C236] sent frame type=1
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.958][95][trace][router] external/envoy/source/common/router/router.cc:871] [C0][S14382728320977988265] proxying 985 bytes
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.958][95][trace][connection] external/envoy/source/common/network/connection_impl.cc:326] [C236] writing 994 bytes, end_stream false
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.958][95][trace][http2] external/envoy/source/common/http/http2/codec_impl.cc:446] [C236] sent frame type=0
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.959][95][debug][grpc] src/envoy/utils/grpc_transport.cc:46] Sending Check request: attributes {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   words: "inbound"
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   words: "x-envoy-original-path"
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   words: "/api/classification/logout"
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   words: "Apache-HttpClient/4.5.5 (Java/1.8.0_151)"
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   words: "e4ece8f07a5a7481"
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   words: "//logout"
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   words: "x-envoy-external-address"
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   words: "174.16.212.0"
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   words: "application/x-www-form-urlencoded"
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   words: "CLASSIFSESSIONID=MjM2N2MyOWUtODFhOS00ZGVjLTkwNjktOTQ5NzhlZTc2ZDk1"
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   words: "pp-helpers.test.oami.eu"
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   words: "82aeb468-9f91-9b30-9482-2aa3591e1979"
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   words: "10.133.0.44, 174.16.212.0"
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   words: "classification-helper-ui.preprod-cb.svc.cluster.local"
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   words: "preprod-cb"
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   words: "classification-helper-ui"
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   words: "kubernetes://classification-helper-ui-76f9cdcd4d-rz6n9.preprod-cb"
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   words: "origin.ip"
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   words: "kubernetes://istio-ingressgateway-5d999c8757-dv9rs.istio-system"
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   words: "istio://preprod-cb/services/classification-helper-ui"
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   strings {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     key: 3
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     value: -19
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   strings {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     key: 17
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     value: -6
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   strings {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     key: 18
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     value: -11
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   strings {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     key: 19
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     value: 91
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   strings {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     key: 22
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     value: 92
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   strings {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     key: 25
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     value: -4
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   strings {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     key: 131
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     value: 92
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   strings {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     key: 152
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     value: -14
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   strings {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     key: 154
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     value: -17
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   strings {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     key: 155
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     value: -15
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   strings {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     key: 190
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     value: -20
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   strings {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     key: 191
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     value: -16
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   strings {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     key: 192
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     value: -15
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   strings {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     key: 193
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     value: -14
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   strings {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     key: 197
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     value: -17
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   strings {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     key: 201
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     value: -1
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   int64s {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     key: 151
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     value: 8080
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   bools {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     key: 177
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     value: false
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   timestamps {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     key: 24
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     value {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       seconds: 1538556462
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       nanos: 958551761
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   bytes {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     key: -18
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     value: "\256\020=8"
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   bytes {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     key: 150
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     value: "\000\000\000\000\000\000\000\000\000\000\377\377\256\020\325\033"
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   string_maps {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     key: 15
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     value {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       entries {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:         key: -7
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:         value: -8
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       entries {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:         key: -2
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:         value: -3
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       entries {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:         key: 31
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:         value: -11
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       entries {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:         key: 32
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:         value: 91
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       entries {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:         key: 33
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:         value: -6
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       entries {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:         key: 55
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:         value: 134
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       entries {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:         key: 58
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:         value: -9
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       entries {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:         key: 59
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:         value: -10
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       entries {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:         key: 86
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:         value: -4
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       entries {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:         key: 98
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:         value: -13
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       entries {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:         key: 100
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:         value: 92
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       entries {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:         key: 102
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:         value: -12
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       entries {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:         key: 121
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:         value: -5
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       entries {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:         key: 122
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:         value: 135
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       entries {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:         key: 123
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:         value: -5
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: global_word_count: 203
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: deduplication_id: "231e89c0-6dcb-4ada-a813-375cedb8a64427"
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.959][95][debug][filter] src/envoy/http/mixer/filter.cc:146] Called Mixer::Filter : decodeHeaders Stop
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.959][95][trace][http] external/envoy/source/common/http/conn_manager_impl.cc:719] [C32][S16141632285256672176] decode headers called: filter=0x115c9040 status=1
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.959][95][trace][http] external/envoy/source/common/http/http1/codec_impl.cc:362] [C32] parsed 1253 bytes
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.959][95][trace][connection] external/envoy/source/common/network/connection_impl.cc:232] [C32] readDisable: enabled=true disable=true
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.959][95][trace][connection] external/envoy/source/common/network/connection_impl.cc:389] [C236] socket event: 2
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.959][95][trace][connection] external/envoy/source/common/network/connection_impl.cc:457] [C236] write ready
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.959][95][trace][connection] external/envoy/source/common/network/raw_buffer_socket.cc:62] [C236] write returns: 1079
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.959][95][trace][connection] external/envoy/source/common/network/connection_impl.cc:389] [C32] socket event: 2
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.959][95][trace][connection] external/envoy/source/common/network/connection_impl.cc:457] [C32] write ready
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.960][95][trace][connection] external/envoy/source/common/network/connection_impl.cc:389] [C236] socket event: 3
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.960][95][trace][connection] external/envoy/source/common/network/connection_impl.cc:457] [C236] write ready
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.960][95][trace][connection] external/envoy/source/common/network/connection_impl.cc:427] [C236] read ready
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.960][95][trace][connection] external/envoy/source/common/network/raw_buffer_socket.cc:21] [C236] read returns: 103
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.960][95][trace][connection] external/envoy/source/common/network/raw_buffer_socket.cc:21] [C236] read returns: -1
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.960][95][trace][connection] external/envoy/source/common/network/raw_buffer_socket.cc:29] [C236] read error: 11
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.960][95][trace][http2] external/envoy/source/common/http/http2/codec_impl.cc:277] [C236] dispatching 103 bytes
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.960][95][trace][http2] external/envoy/source/common/http/http2/codec_impl.cc:335] [C236] recv frame type=1
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.960][95][debug][router] external/envoy/source/common/router/router.cc:583] [C0][S14382728320977988265] upstream headers complete: end_stream=false
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.960][95][debug][http] external/envoy/source/common/http/async_client_impl.cc:93] async http request response headers (end_stream=false):
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: ':status', '200'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'content-type', 'application/grpc'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'x-envoy-upstream-service-time', '1'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'date', 'Wed, 03 Oct 2018 08:47:42 GMT'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'server', 'envoy'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'x-envoy-decorator-operation', 'Check'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.960][95][trace][http2] external/envoy/source/common/http/http2/codec_impl.cc:335] [C236] recv frame type=0
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.960][95][trace][http] external/envoy/source/common/http/async_client_impl.cc:101] async http request response data (length=45 end_stream=false)
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.960][95][trace][http2] external/envoy/source/common/http/http2/codec_impl.cc:335] [C236] recv frame type=1
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.960][95][debug][client] external/envoy/source/common/http/codec_client.cc:94] [C236] response complete
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.960][95][trace][main] external/envoy/source/common/event/dispatcher_impl.cc:126] item added to deferred deletion list (size=1)
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.960][95][debug][pool] external/envoy/source/common/http/http2/conn_pool.cc:189] [C236] destroying stream: 0 remaining
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.960][95][debug][http] external/envoy/source/common/http/async_client_impl.cc:108] async http request response trailers:
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'grpc-status', '0'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'grpc-message', ''
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.960][95][debug][grpc] src/envoy/utils/grpc_transport.cc:67] Check response: precondition {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   status {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   valid_duration {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     seconds: 46
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     nanos: 386564935
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   valid_use_count: 10000
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   referenced_attributes {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     attribute_matches {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       name: 155
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       condition: EXACT
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     attribute_matches {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       name: 201
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       condition: EXACT
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     attribute_matches {
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       name: 152
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:       condition: EXACT
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:     }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:   }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: }
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.960][95][debug][filter] src/envoy/http/mixer/filter.cc:211] Called Mixer::Filter : check complete OK
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.960][95][trace][http] external/envoy/source/common/http/conn_manager_impl.cc:1296] [C32][S16141632285256672176] continuing filter chain: filter=0x115c9040
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.960][95][trace][http] external/envoy/source/common/http/conn_manager_impl.cc:719] [C32][S16141632285256672176] decode headers called: filter=0x12d6f310 status=0
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.960][95][trace][http] external/envoy/source/common/http/conn_manager_impl.cc:719] [C32][S16141632285256672176] decode headers called: filter=0x12d6fcc0 status=0
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.960][95][debug][router] external/envoy/source/common/router/router.cc:252] [C32][S16141632285256672176] cluster 'inbound|8080||classification-helper-ui.preprod-cb.svc.cluster.local' match for URL '//logout'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.960][95][debug][router] external/envoy/source/common/router/router.cc:303] [C32][S16141632285256672176] router decoding headers:
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: ':authority', 'pp-helpers.test.oami.eu'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: ':path', '//logout'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: ':method', 'POST'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: ':scheme', 'http'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'cookie', 'CLASSIFSESSIONID=MjM2N2MyOWUtODFhOS00ZGVjLTkwNjktOTQ5NzhlZTc2ZDk1'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'content-type', 'application/x-www-form-urlencoded'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'content-length', '0'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'user-agent', 'Apache-HttpClient/4.5.5 (Java/1.8.0_151)'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'x-forwarded-for', '10.133.0.44, 174.16.212.0'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'x-forwarded-proto', 'http'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'x-envoy-external-address', '174.16.212.0'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'x-request-id', '82aeb468-9f91-9b30-9482-2aa3591e1979'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'x-b3-traceid', 'e4ece8f07a5a7481'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'x-b3-spanid', 'e4ece8f07a5a7481'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'x-b3-sampled', '1'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'x-envoy-original-path', '/api/classification/logout'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.960][95][debug][pool] external/envoy/source/common/http/http1/conn_pool.cc:89] [C502] using existing connection
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.960][95][trace][connection] external/envoy/source/common/network/connection_impl.cc:232] [C502] readDisable: enabled=false disable=false
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.960][95][debug][router] external/envoy/source/common/router/router.cc:971] [C32][S16141632285256672176] pool ready
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][trace][connection] external/envoy/source/common/network/connection_impl.cc:326] [C502] writing 546 bytes, end_stream false
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][trace][http] external/envoy/source/common/http/conn_manager_impl.cc:719] [C32][S16141632285256672176] decode headers called: filter=0x111d3c20 status=1
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][trace][main] external/envoy/source/common/event/dispatcher_impl.cc:126] item added to deferred deletion list (size=2)
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][trace][main] external/envoy/source/common/event/dispatcher_impl.cc:126] item added to deferred deletion list (size=3)
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][debug][http2] external/envoy/source/common/http/http2/codec_impl.cc:501] [C236] stream closed: 0
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][trace][main] external/envoy/source/common/event/dispatcher_impl.cc:126] item added to deferred deletion list (size=4)
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][trace][http2] external/envoy/source/common/http/http2/codec_impl.cc:292] [C236] dispatched 103 bytes
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][trace][main] external/envoy/source/common/event/dispatcher_impl.cc:52] clearing deferred deletion list (size=4)
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][trace][connection] external/envoy/source/common/network/connection_impl.cc:389] [C502] socket event: 2
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][trace][connection] external/envoy/source/common/network/connection_impl.cc:457] [C502] write ready
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][trace][connection] external/envoy/source/common/network/raw_buffer_socket.cc:62] [C502] write returns: 546
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][trace][connection] external/envoy/source/common/network/connection_impl.cc:389] [C502] socket event: 3
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][trace][connection] external/envoy/source/common/network/connection_impl.cc:457] [C502] write ready
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][trace][connection] external/envoy/source/common/network/connection_impl.cc:427] [C502] read ready
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][trace][connection] external/envoy/source/common/network/raw_buffer_socket.cc:21] [C502] read returns: 0
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][debug][connection] external/envoy/source/common/network/connection_impl.cc:451] [C502] remote close
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][debug][connection] external/envoy/source/common/network/connection_impl.cc:133] [C502] closing socket: 0
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][trace][http] external/envoy/source/common/http/http1/codec_impl.cc:341] [C502] parsing 0 bytes
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][trace][http] external/envoy/source/common/http/http1/codec_impl.cc:362] [C502] parsed 0 bytes
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][debug][client] external/envoy/source/common/http/codec_client.cc:81] [C502] disconnect. resetting 1 pending requests
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][debug][client] external/envoy/source/common/http/codec_client.cc:104] [C502] request reset
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][trace][main] external/envoy/source/common/event/dispatcher_impl.cc:126] item added to deferred deletion list (size=1)
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][debug][router] external/envoy/source/common/router/router.cc:457] [C32][S16141632285256672176] upstream reset
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][debug][filter] src/envoy/http/mixer/filter.cc:191] Called Mixer::Filter : encodeHeaders 2
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][trace][http] external/envoy/source/common/http/conn_manager_impl.cc:997] [C32][S16141632285256672176] encode headers called: filter=0x11d85140 status=0
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][trace][http] external/envoy/source/common/http/conn_manager_impl.cc:997] [C32][S16141632285256672176] encode headers called: filter=0x11305dc0 status=0
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][debug][http] external/envoy/source/common/http/conn_manager_impl.cc:1083] [C32][S16141632285256672176] encoding headers via codec (end_stream=false):
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: ':status', '503'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'content-length', '57'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'content-type', 'text/plain'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'date', 'Wed, 03 Oct 2018 08:47:42 GMT'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: 'server', 'envoy'
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]:
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][trace][connection] external/envoy/source/common/network/connection_impl.cc:326] [C32] writing 134 bytes, end_stream false
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][trace][http] external/envoy/source/common/http/conn_manager_impl.cc:1157] [C32][S16141632285256672176] encode data called: filter=0x11d85140 status=0
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][trace][http] external/envoy/source/common/http/conn_manager_impl.cc:1157] [C32][S16141632285256672176] encode data called: filter=0x11305dc0 status=0
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][trace][http] external/envoy/source/common/http/conn_manager_impl.cc:1170] [C32][S16141632285256672176] encoding data via codec (size=57 end_stream=true)
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][trace][connection] external/envoy/source/common/network/connection_impl.cc:326] [C32] writing 57 bytes, end_stream false
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][debug][filter] src/envoy/http/mixer/filter.cc:257] Called Mixer::Filter : onDestroy state: 2
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][trace][main] external/envoy/source/common/event/dispatcher_impl.cc:126] item added to deferred deletion list (size=2)
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][trace][connection] external/envoy/source/common/network/connection_impl.cc:232] [C32] readDisable: enabled=false disable=false
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][debug][pool] external/envoy/source/common/http/http1/conn_pool.cc:122] [C502] client disconnected
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][trace][main] external/envoy/source/common/event/dispatcher_impl.cc:126] item added to deferred deletion list (size=3)
preprod-cb/classification-helper-ui-76f9cdcd4d-rz6n9[istio-proxy]: [2018-10-03 08:47:42.961][95][trace][main] external/envoy/source/common/event/dispatcher_impl.cc:52] clearing deferred deletion list (size=3)

The purpose of this change is:

• consistency with other modules (e.g. mixer)
• better separation of library code from other directories
• more clearly indicating the points of extension
• more clearly indicating those things that are k8s-specific

Overview of the changes:

• Adding pkg at the top-level to house all library code.
• Removed adapter and moved config and serviceregistry to the top level under pkg
• Moved all service registry implementations under serviceregistry (kube, consul, eureka, cloudfoundry)
• Removed platform and moved up the remaining kube package to the top level under pkg
• Moved cmd/pilot-discovery/server to the top-level under pkg and renamed to bootstrap
• Moved cmd/pilot-discovery/mock to pkg/proxy/envoy

The resulting layout for pilot is:

• bin
• cmd
• doc
• docker
• pkg
  • bootstrap
  • config
    • aggregate
    • memory
    • kube
      • crd
        • file
      • ingress
  • dataplane
  • kube
  • model
  • proxy
    • envoy
      • mock
  • serviceregistry
    • aggregate
    • kube
    • consul
    • eureka
    • cloudfoundry
• test
• tools

Istio currently does not support statefulset. There are already many issues related, so here is a umbrella issue.

#10053 #1277 #10490 #10586 #9666 #19280

Anyone can add missing issues below.

I've noticed pretty much every application periodically gets this, a 503 with the envoy code UC.

It's retried as you can see, but it's bothering me that I cannot get to the bottom of why it is happening. My destinationrule should deal with any connection timeouts etc:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: sauron-graphql-server
spec:
  host: app
  trafficPolicy:
    connectionPool:
      http:
        http1MaxPendingRequests: 1024
        http2MaxRequests: 10240
        idleTimeout: 5m
        maxRequestsPerConnection: 1024
      tcp:
        maxConnections: 10240
        tcpKeepalive:
          interval: 1m
          time: 3m
    loadBalancer:
      simple: LEAST_CONN
    outlierDetection:
      baseEjectionTime: 5s
      consecutiveErrors: 5
      interval: 5s
      maxEjectionPercent: 50
      minHealthPercent: 50
    tls:
      mode: ISTIO_MUTUAL

The frequency of these bothers me, because across a distributed service graph, we can get spikes in latency on client requests as a result.

Hey, On a fresh install of istio 0.5.0 on k8s 1.8.6 in GKE with mTLS enabled, i'm seeing errors in mixer:

2018-02-02T11:56:49.051875Z	info	getting kubeconfig from: ""	{"adapter": "handler.kubernetesenv.istio-system"}
W0202 11:56:49.051913       1 client_config.go:529] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2018-02-02T11:56:49.052169Z	info	getting k8s client from config	{"adapter": "handler.kubernetesenv.istio-system"}
2018-02-02T11:56:49.052923Z	info	Waiting for kubernetes cache sync...	{"adapter": "handler.kubernetesenv.istio-system"}
2018-02-02T11:56:49.153124Z	info	Cache sync successful.	{"adapter": "handler.kubernetesenv.istio-system"}
2018-02-02T11:56:49.153867Z	info	Published snapshot[2] with 4 rules, 3 handlers, previously 0 rules
2018-02-02T11:56:49.153929Z	info	serving prometheus metrics on :42422	{"adapter": "handler.prometheus.istio-system"}
2018-02-02T11:57:22.360156Z	info	grpc: Server.Serve failed to create ServerTransport: connection error: desc = "transport: http2Server.HandleStreams failed to receive the preface from client: EOF"
2018-02-02T11:57:23.364267Z	info	grpc: Server.Serve failed to create ServerTransport: connection error: desc = "transport: http2Server.HandleStreams failed to receive the preface from client: EOF"

and errors in pilot:

2018-02-02T11:56:26.720600Z	info	Unable to retrieve availability zone from pilot: Get http://istio-pilot:15003/v1/az/istio-proxy/pilot~~.~.svc.cluster.local: read tcp 10.194.7.22:50296->10.192.11.232:15003: read: connection reset by peer
2018-02-02T11:56:56.721834Z	info	Unable to retrieve availability zone from pilot: Get http://istio-pilot:15003/v1/az/istio-proxy/pilot~~.~.svc.cluster.local: read tcp 10.194.7.22:44674->10.192.11.232:15003: read: connection reset by peer
2018-02-02T11:57:26.725496Z	info	Unable to retrieve availability zone from pilot: Get http://istio-pilot:15003/v1/az/istio-proxy/pilot~~.~.svc.cluster.local: read tcp 10.194.7.22:39132->10.192.11.232:15003: read: connection reset by peer
2018-02-02T11:57:56.726764Z	info	Unable to retrieve availability zone from pilot: Get http://istio-pilot:15003/v1/az/istio-proxy/pilot~~.~.svc.cluster.local: read tcp 10.194.7.22:33110->10.192.11.232:15003: read: connection reset by peer
2018-02-02T11:58:26.728359Z	info	Unable to retrieve availability zone from pilot: Get http://istio-pilot:15003/v1/az/istio-proxy/pilot~~.~.svc.cluster.local: read tcp 10.194.7.22:54060->10.192.11.232:15003: read: connection reset by peer
2018-02-02T11:58:56.730797Z	info	Unable to retrieve availability zone from pilot: Get http://istio-pilot:15003/v1/az/istio-proxy/pilot~~.~.svc.cluster.local: read tcp 10.194.7.22:41572->10.192.11.232:15003: read: connection reset by peer
2018-02-02T11:59:26.733221Z	info	Unable to retrieve availability zone from pilot: Get http://istio-pilot:15003/v1/az/istio-proxy/pilot~~.~.svc.cluster.local: read tcp 10.194.7.22:50644->10.192.11.232:15003: read: connection reset by peer
2018-02-02T11:59:56.735592Z	info	Unable to retrieve availability zone from pilot: Get http://istio-pilot:15003/v1/az/istio-proxy/pilot~~.~.svc.cluster.local: read tcp 10.194.7.22:51056->10.192.11.232:15003: read: connection reset by peer
2018-02-02T12:00:26.738272Z	info	Unable to retrieve availability zone from pilot: Get http://istio-pilot:15003/v1/az/istio-proxy/pilot~~.~.svc.cluster.local: read tcp 10.194.7.22:51146->10.192.11.232:15003: read: connection reset by peer
2018-02-02T12:00:56.740580Z	info	Unable to retrieve availability zone from pilot: Get http://istio-pilot:15003/v1/az/istio-proxy/pilot~~.~.svc.cluster.local: read tcp 10.194.7.22:51246->10.192.11.232:15003: read: connection reset by peer

Describe the bug Hey, We are noticing blips in services under load during kubernetes rollouts. We observe a handful of 503 errors from istio-proxy on the pod being removed (either because of a rollout, or a scale down). This screenshot is from three separate "scale downs":

When scaling down, this is the sequence of events we observe:

• pod goes into TERMINATING state and is removed from kubernetes endpoints
• A handful of the last requests to the pod are reported by istio-proxy as 503
• those requests are also logged in the upstream calling service as 503
• application exits
• istio-proxy exits

As you can see here:

At the moment, our only saving grace is that we have configured a retry policy which means our end users experience a bit of a slow request, but not a failure - however relying on a retry mechanism in this scenario doesn't feel right.

Expected behavior The isito-proxy on the application being scaled down should not receive any requests after it has entered a TERMINATING state.

Steps to reproduce the bug As above, but I can get on hangouts and show you this in detail.

The application itself gracefully handles sigterms and drains and have confirmed this with load tests without istio-proxy in play. I have also added a preStop hook to the application with istio, to ensure the app doesn't receive a SIGTERM until well after istio-proxy shuts down.

Version gke 1.10.5, istio 1.0

Is Istio Auth enabled or not? Yes

Environment GKE

Bug Description Istiod is not responding to new namespaces. Created a new namespace and labelled istio-injectio=enabled. I deployed one app to the namespace and the pod is stuck in init stage. Istio proxy is not able to mount configmap "istio-ca-root-cert"

Unable to mount volumes for pod "azure-vote-front-5bc759676c-7hg5t_am-dev(a45f7c3a-d546-4461-af15-c5442ae39de9)": timeout expired waiting for volumes to attach or mount for pod "am-dev"/"azure-vote-front-5bc759676c-7hg5t".

Last log from istiod -

2020-03-25T10:04:33.132573Z warn k8s.io/[email protected]/tools/cache/reflector.go:105: watch of *v1.ConfigMap ended with: too old resource version: 2764317 (2764517)

list of unmounted volumes=[istiod-ca-cert]. list of unattached volumes=[default-token-58r9k istio-envoy podinfo istio-token istiod-ca-cert]

Warning FailedMount 3m57s (x33 over 54m) kubelet, aks-linuxpool02-21909056-vmss000000 MountVolume.SetUp failed for volume "istiod-ca-cert" : configmap "istio-ca-root-cert" not found

[ x] Configuration Infrastructure [ ] Docs [ ] Installation [x ] Networking [ ] Performance and Scalability [ ] Policies and Telemetry [ ] Security [ ] Test and Release [ ] User Experience [x ] Developer Infrastructure

New namespace should have configmap "istio-ca-root-cert" and should deploy the app

Steps to reproduce the bug This happened intermittently. In one of the namespace I was able to attach the proxy and once the problem started, further no namespace is working.

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm) Istioctl - client version: 1.5.0 kubectl - server: v1.15.7, client: v1.17.0

How was Istio installed? istioctl

Environment where bug was observed (cloud vendor, OS, etc) Azure Kubernetes Engine

running 100qps through ingress in the perf setup getting a small % of 503s (only on multi hops - maybe a timeout issue?) - it also only happens with >=20 connections or so

35.199.14.248/fortio2/fortio/fetch/echosrv1:8080/echo

Starting at 100 qps with 30 thread(s) [gomax 2] : exactly 3000, 100 calls each (total 3000 + 0)
Ended after 30.071664869s : 3000 calls. qps=99.762
Sleep times : count 2970 avg 0.26524946 +/- 0.01577 min 0.192739205 max 0.292777981 sum 787.790887
Aggregated Function Time : count 3000 avg 0.037497644 +/- 0.01582 min 0.009672528 max 0.10991789 sum 112.492931
# range, mid point, percentile, count
>= 0.00967253 <= 0.01 , 0.00983626 , 0.03, 1
> 0.01 <= 0.012 , 0.011 , 0.17, 4
> 0.012 <= 0.014 , 0.013 , 0.70, 16
> 0.014 <= 0.016 , 0.015 , 2.63, 58
> 0.016 <= 0.018 , 0.017 , 5.33, 81
> 0.018 <= 0.02 , 0.019 , 8.17, 85
> 0.02 <= 0.025 , 0.0225 , 19.70, 346
> 0.025 <= 0.03 , 0.0275 , 33.83, 424
> 0.03 <= 0.035 , 0.0325 , 52.87, 571
> 0.035 <= 0.04 , 0.0375 , 67.87, 450
> 0.04 <= 0.045 , 0.0425 , 76.67, 264
> 0.045 <= 0.05 , 0.0475 , 82.83, 185
> 0.05 <= 0.06 , 0.055 , 91.00, 245
> 0.06 <= 0.07 , 0.065 , 94.97, 119
> 0.07 <= 0.08 , 0.075 , 98.10, 94
> 0.08 <= 0.09 , 0.085 , 98.87, 23
> 0.09 <= 0.1 , 0.095 , 99.60, 22
> 0.1 <= 0.109918 , 0.104959 , 100.00, 12
# target 50% 0.0342469
# target 75% 0.044053
# target 99% 0.0918182
# target 99.9% 0.107438
Code 200 : 2995 (99.8 %)
Code 503 : 5 (0.2 %)
Response Header Sizes : count 3000 avg 165.72233 +/- 6.772 min 0 max 167 sum 497167
Response Body/Total Sizes : count 3000 avg 166.09833 +/- 2.432 min 165 max 226 sum 498295
Saved result to data/2018-02-08-055310_Fortio.json
All done 3000 calls 37.498 ms avg, 99.8 qps

Bug description

Following the guide Install Multi-Primary on different networks, everything seems to install as expected without errors and is running in the cluster. For secret/cacerts I am using the example certificate material from samples/certs/*.pem in both cluster1 and cluster2.

When I attempt to verify the installation using the guide Verify the installation the requests are not getting routed to the remote cluster as expected. I am only getting responses from the service on the local cluster:

# From Cluster 1 [where helloworld v1 is deployed]
$ while true; do kubectl exec --context="${CTX_CLUSTER1}" -n sample -c sleep     "$(kubectl get pod --context="${CTX_CLUSTER1}" -n sample -l \
    app=sleep -o jsonpath='{.items[0].metadata.name}')"     -- curl -s helloworld.sample:5000/hello; done
Hello version: v1, instance: helloworld-v1-578dd69f69-r9lkz
Hello version: v1, instance: helloworld-v1-578dd69f69-r9lkz
Hello version: v1, instance: helloworld-v1-578dd69f69-r9lkz
Hello version: v1, instance: helloworld-v1-578dd69f69-r9lkz
Hello version: v1, instance: helloworld-v1-578dd69f69-r9lkz
...
# From Cluster 2 [where helloworld v2 is deployed]
$ while true; do kubectl exec --context="${CTX_CLUSTER2}" -n sample -c sleep     "$(kubectl get pod --context="${CTX_CLUSTER2}" -n sample -l \
    app=sleep -o jsonpath='{.items[0].metadata.name}')"     -- curl -s helloworld.sample:5000/hello; done
Hello version: v2, instance: helloworld-v2-776f74c475-h5j2q
Hello version: v2, instance: helloworld-v2-776f74c475-h5j2q
Hello version: v2, instance: helloworld-v2-776f74c475-h5j2q
Hello version: v2, instance: helloworld-v2-776f74c475-h5j2q
Hello version: v2, instance: helloworld-v2-776f74c475-h5j2q
...

istioctl proxy-config endpoint for the sleep pod in cluster1 and cluster2 to helloworld destination service:

# Cluster 1
$ istioctl -n sample --context=${CTX_CLUSTER1} proxy-config endpoint "$(kubectl get pod --context="${CTX_CLUSTER1}" -n sample -l app=sleep -o jsonpath='{.items[0].metadata.name}')" | grep helloworld
10.100.1.12:5000                 HEALTHY     OK                outbound|5000||helloworld.sample.svc.cluster.local

# Cluster 2
$ istioctl -n sample --context=${CTX_CLUSTER2} proxy-config endpoint "$(kubectl get pod --context="${CTX_CLUSTER2}" -n sample -l app=sleep -o jsonpath='{.items[0].metadata.name}')" | grep helloworld
10.100.2.188:5000                HEALTHY     OK                outbound|5000||helloworld.sample.svc.cluster.local

It seems like maybe something is missing from the docs or example configs, but I understand that there are tests for the docs/examples, which is why I’ve been troubleshooting my own cluster… but just seems like something small is missing

[X] Docs [X] Installation [X] Networking [ ] Performance and Scalability [ ] Extensions and Telemetry [ ] Security [ ] Test and Release [ ] User Experience [ ] Developer Infrastructure [ ] Upgrade

Expected behavior I expected to be able to follow the guide and get the same behavior that the guide expects

Steps to reproduce the bug Following the guides for multi-primary, multi-network install and verify the install

Version (include the output of istioctl version --remote and kubectl version --short and helm version --short if you used Helm)

$ istioctl version
client version: 1.8.0
control plane version: 1.8.0
data plane version: 1.8.0 (4 proxies)

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.4", GitCommit:"d360454c9bcd1634cf4cc52d1867af5491dc9c5f", GitTreeState:"clean", BuildDate:"2020-11-12T01:09:16Z", GoVersion:"go1.15.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.12-eks-7684af", GitCommit:"7684af4ac41370dd109ac13817023cb8063e3d45", GitTreeState:"clean", BuildDate:"2020-10-20T22:57:40Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

How was Istio installed? Istio Operator installed with istioctl operator init and the rest of the installation of istio in istio-system is done as steps in the guide [using example manifests & scripts to compile example manifests]

Environment where the bug was observed (cloud vendor, OS, etc) AWS EKS with Kubernetes v1.17 Istio 1.8.0 on Mac

$ uname -a
Darwin OAK-MAC-HLJHD2 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_6

Additionally, please consider running istioctl bug-report and attach the generated cluster-state tarball to this issue. Refer cluster state archive for more details.

This PR addresses https://github.com/istio/istio/issues/5384, whereby users can enable pilot to publish https/http listeners with the proxy protocol enabled.

Manual cherrypick required.

#41930 failed to apply on top of branch "experimental-ambient":

Applying: add gosec linter
Using index info to reconstruct a base tree...
M	common/config/.golangci.yml
M	pkg/hbone/dialer.go
M	pkg/test/echo/server/endpoint/http.go
M	security/pkg/nodeagent/caclient/providers/citadel/client_test.go
Falling back to patching base and 3-way merge...
Auto-merging security/pkg/nodeagent/caclient/providers/citadel/client_test.go
Auto-merging pkg/test/echo/server/endpoint/http.go
Auto-merging pkg/hbone/dialer.go
Auto-merging common/config/.golangci.yml
CONFLICT (content): Merge conflict in common/config/.golangci.yml
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 add gosec linter
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

Please provide a description of this PR:

Update the common-files to the latest. Will create a new automated job to keep this up to date in the future.

Bug Description

Hi All, running into some odd behavior with DestinationRule workloadSelector and was hoping you could shed some light. I am attempting to have a hybrid ServiceEntry which references pods and WorkloadEntries and balances between them as described in the docs.

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: external-svc
spec:
  hosts:
  - failover-istio-lb.testing.mydomain.com
  location: MESH_INTERNAL
  ports:
  - number: 80
    name: http
    protocol: HTTP
    targetPort: 8080
  resolution: DNS
  workloadSelector:
    labels:
      app: echo-edge-istio

This is functioning well and traffic can be directed to both destinations. The issue comes into play with the WorkloadEntrys requiring a TLS traffic policy of SIMPLE and the internal pods not requiring this.

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: external-svc
spec:
  host: failover-istio-lb.testing.mydomain.com
  # workloadSelector:
  #   matchLabels:
  #     app: echo-edge-istio
  trafficPolicy:
    tls:
      mode: SIMPLE

So I figured adding a workloadSelector matchLabel to the destination rule would be the solution here - unfortunately I cant seem to get it to work. If I do not specify the selector, then all traffic has the rule applied and half the connections are successful (as expected). When I add a selector it doen’t appear to take affect at all (even when I set the selector to the same selector label used in the ServiceEntry to direct traffic to both destinations it does not apply to either end). I had expected this to perform the same as not having a selector given that the DR should apply to both destinations in the Service Entry.

The workload entry is labeled with:

labels:
  app: echo-edge-istio
  class: cloud

and the pods are labeled with:

labels:
  app: echo-edge-istio
  class: pod

and I have tried using match labels of app: echo-edge-istio, class: cloud and class: pod with no success (DR does not apply to either).

regardless of the workloadselector on the destination rule, the command

./istioctl pc cluster echo-edge-istio-688b469d74-5msxx

still shows the destination rule applied however the behavior is as if the DR does not exist

failover-istio-lb.testing.mydomain.com                                  80        -          outbound      STRICT_DNS       external-svc.edge-istio

Also when trying to verify rules with experimental describe it doesnt show any DRs applied regardless of selector or not, however changing the DR impacts the traffic as expected

./istioctl x describe pod echo-edge-istio-688b469d74-5msxx
Pod: echo-edge-istio-688b469d74-5msxx
   Pod Revision: default
   Pod Ports: 8080 (echo), 15090 (istio-proxy)
Suggestion: add 'version' label to pod for Istio telemetry.
--------------------
Service: echo-edge-istio
   Port: low 80/UnsupportedProtocol targets pod port 8080
--------------------
Effective PeerAuthentication:
   Workload mTLS mode: PERMISSIVE

Ideally what im looking for is a method to have a blanket destination rule for a given host (loadbalancing/outlier detection/etc) and then have a specific DR for the workloadentries via selector to set the TLS trafficpolicy on just those objects. Not sure where I can go about debugging from here, have even tried setting the inverse thinking the selector only applied to pods (base rule of TLS simple with a DR + selector to disable on pods but no luck). Would be happy to dump configs/more info if its helpful

Version

$ istioctl version
client version: 1.16.1
control plane version: 1.16.1
data plane version: 1.16.1 (2 proxies)
$ kubectl version --short
Client Version: v1.24.0
Kustomize Version: v4.5.4
Server Version: v1.23.14+rke2r1

Additional Information

./istioctl bug-report

Target cluster context: testing

Running with the following config:

istio-namespace: istio-system full-secrets: false timeout (mins): 30 include: { } exclude: { Namespaces: kube-node-lease,kube-public,kube-system,local-path-storage } end-time: 2023-01-04 15:11:48.388494 -0500 EST

Cluster endpoint: redacted CLI version: version.BuildInfo{Version:"1.16.1", GitRevision:"f6d7bf648e571a6a523210d97bde8b489250354b", GolangVersion:"go1.19.4", BuildStatus:"Clean", GitTag:"1.16.1"}

The following Istio control plane revisions/versions were found in the cluster: Revision default: &version.MeshInfo{ { Component: "pilot", Info: version.BuildInfo{Version:"1.16.1", GitRevision:"f6d7bf648e571a6a523210d97bde8b489250354b", GolangVersion:"", BuildStatus:"Clean", GitTag:"1.16.1"}, }, }

The following proxy revisions/versions were found in the cluster: Revision default: Versions {1.16.1}

Fetching proxy logs for the following containers:

calico-system//calico-node-bnrln/calico-node calico-system//calico-node-z6spp/calico-node calico-system//calico-node-zxwqj/calico-node calico-system/calico-kube-controllers/calico-kube-controllers-7897f79447-xd2vn/calico-kube-controllers calico-system/calico-typha/calico-typha-79f569f5f7-tkgnb/calico-typha calico-system/calico-typha/calico-typha-79f569f5f7-xbql6/calico-typha cattle-fleet-system/fleet-agent/fleet-agent-5779d486dc-xq74z/fleet-agent cattle-system/cattle-cluster-agent/cattle-cluster-agent-6cb6596fb8-jvf8x/cluster-register cattle-system/cattle-cluster-agent/cattle-cluster-agent-6cb6596fb8-xl6k2/cluster-register cattle-system/system-upgrade-controller/system-upgrade-controller-7f9f559b4f-2lslv/system-upgrade-controller edge-istio/echo-edge-istio/echo-edge-istio-688b469d74-5msxx/echo edge-istio/echo-edge-istio/echo-edge-istio-688b469d74-5msxx/istio-proxy istio-system/istio-ingressgateway/istio-ingressgateway-598595fcf6-r5bts/istio-proxy istio-system/istiod/istiod-5b86c45f48-7bxl2/discovery metallb-system//speaker-fcjb2/speaker metallb-system//speaker-hwjnq/speaker metallb-system//speaker-w9pk9/speaker metallb-system/controller/controller-7597dd4f7b-c249v/controller upstream/echo-upstream/echo-upstream-588c888c78-8nbxc/echo upstream/echo-upstream/echo-upstream-588c888c78-s9g8x/echo upstream/echo-upstream/echo-upstream-588c888c78-ws9z5/echo

Fetching Istio control plane information from cluster.

Running istio analyze on all namespaces and report as below: Analysis Report: Error [IST0128] (DestinationRule edge-istio/external-svc-dns) DestinationRule edge-istio/external-svc in namespace edge-istio has TLS mode set to SIMPLE but no caCertificates are set to validate server identity for host: failover-istio-lb.testing.mydomain.com Info [IST0102] (Namespace calico-system) The namespace is not enabled for Istio injection. Run 'kubectl label namespace calico-system istio-injection=enabled' to enable it, or 'kubectl label namespace calico-system istio-injection=disabled' to explicitly mark it as not needing injection. Info [IST0102] (Namespace cattle-fleet-system) The namespace is not enabled for Istio injection. Run 'kubectl label namespace cattle-fleet-system istio-injection=enabled' to enable it, or 'kubectl label namespace cattle-fleet-system istio-injection=disabled' to explicitly mark it as not needing injection. Info [IST0102] (Namespace cattle-impersonation-system) The namespace is not enabled for Istio injection. Run 'kubectl label namespace cattle-impersonation-system istio-injection=enabled' to enable it, or 'kubectl label namespace cattle-impersonation-system istio-injection=disabled' to explicitly mark it as not needing injection. Info [IST0102] (Namespace cattle-system) The namespace is not enabled for Istio injection. Run 'kubectl label namespace cattle-system istio-injection=enabled' to enable it, or 'kubectl label namespace cattle-system istio-injection=disabled' to explicitly mark it as not needing injection. Info [IST0102] (Namespace default) The namespace is not enabled for Istio injection. Run 'kubectl label namespace default istio-injection=enabled' to enable it, or 'kubectl label namespace default istio-injection=disabled' to explicitly mark it as not needing injection. Info [IST0118] (Service calico-system/calico-kube-controllers-metrics) Port name metrics-port (port: 9094, targetPort: 9094) doesn't follow the naming convention of Istio port. Info [IST0118] (Service calico-system/calico-typha) Port name calico-typha (port: 5473, targetPort: calico-typha) doesn't follow the naming convention of Istio port. Info [IST0118] (Service edge-istio/echo-edge-istio) Port name low (port: 80, targetPort: 8080) doesn't follow the naming convention of Istio port. Info [IST0118] (Service upstream/echo-upstream) Port name high (port: 8080, targetPort: 8080) doesn't follow the naming convention of Istio port. Info [IST0118] (Service upstream/echo-upstream) Port name low (port: 80, targetPort: 8080) doesn't follow the naming convention of Istio port. Info [IST0118] (Service metallb-system/webhook-service) Port name (port: 443, targetPort: 9443) doesn't follow the naming convention of Istio port.