Local development against a remote Kubernetes or OpenShift cluster

Overview

Documentation - start here!

Build Status Join the chat at https://d6e.co/slack CII Best Practices

** Note: Telepresence 1 is being replaced by our even better Telepresence 2. Please try Telepresence 2 first and report any issues as we expect this will be the default by Q2 2021. **

Demo

asciicast

Telepresence: fast, efficient local development for Kubernetes microservices

Telepresence gives developers infinite scale development environments for Kubernetes. With Telepresence:

  • You run one service locally, using your favorite IDE and other tools
  • You run the rest of your application in the cloud, where there is unlimited memory and compute

This gives developers:

  • a fast local dev loop, with no waiting for a container build / push / deploy
  • ability to use their favorite local tools (IDE, debugger, etc.)
  • ability to run large-scale applications that can't run locally

Quick Start

  1. Install locally with Homebrew, apt, or dnf.

  2. Run telepresence.

  3. You now have a shell that proxies connections to Kubernetes.

For more about Telepresence, and the various options, read the documentation.

Usage Reporting

Telepresence collects some basic information about its users so it can send important client notices, such as new version availability and security bulletins. We also use the information to aggregate basic usage analytics anonymously. To disable this behavior set the environment variable SCOUT_DISABLE:

export SCOUT_DISABLE=1

To know more, check the documentation on usage reporting.

Get Involved

About Telepresence

Telepresence is an open source project hosted by the Cloud Native Computing Foundation and originally created by Ambassador Labs. Telepresence is licensed under the Apache 2.0 License. For information about recent releases, see https://www.telepresence.io/reference/changelog. Ambassador Labs also provides commercial support for a version of Telepresence that is designed for teams.

Issues
  • Kubernetes DNS Resolution Failing Spuriously on Mac

    Kubernetes DNS Resolution Failing Spuriously on Mac

    I'm opening a telepresence session to our K8s cluster using method vpn-tcp without swapping out any deployments just to access K8s resources. Roughly half of the time I do this, it works perfectly. The other times, I get errors complaining that DNS resolution of the K8s addresses failed. However, when I run dig <K8s-IP>, it reports status NOERROR, so I know that there is no real issue with the addresses themselves. The only workaround I've found thus far is to restart my machine entirely. I've also tried the same setup on Linux and have not seen the issue after many runs. Specifically, this is on macOS Sierra 10.12.6. This seems like a particularly hairy issue that may be solved by the plans to run DNS locally, but I still wanted to document it.

    Thank you!

    opened by esorey 30
  • Homebrew Install fails: Error: sshfs has been disabled because it requires FUSE!

    Homebrew Install fails: Error: sshfs has been disabled because it requires FUSE!

    Hi everyone,

    I tried installing Telepresence following the official installation guide at https://www.telepresence.io/reference/install and the brew command does not work for me. It fails with this output:

    brew install datawire/blackbird/telepresence redacted Updating Homebrew... ==> Auto-updated Homebrew! Updated 1 tap (homebrew/core). ==> Updated Formulae Updated 4 formulae.

    ==> Installing telepresence from datawire/blackbird ==> Downloading https://homebrew.bintray.com/bottles/gdbm-1.19.big_sur.bottle.tar.gz Already downloaded: /Users/redacted/Library/Caches/Homebrew/downloads/9d8f2b865b1f004ad8a1b27b468833da402e5feb31e88557175a25209660d595--gdbm-1.19.big_sur.bottle.tar.gz ==> Downloading https://homebrew.bintray.com/bottles/mpdecimal-2.5.1.big_sur.bottle.tar.gz Already downloaded: /Users/redacted/Library/Caches/Homebrew/downloads/29dd7202ebb6142202c80dec9080c1a24ef37a7039f92356a122061477efce21--mpdecimal-2.5.1.big_sur.bottle.tar.gz ==> Downloading https://homebrew.bintray.com/bottles/openssl%401.1-1.1.1k.big_sur.bottle.tar.gz Already downloaded: /Users/redacted/Library/Caches/Homebrew/downloads/fae83f7618[email protected]1.1-1.1.1k.big_sur.bottle.tar.gz ==> Downloading https://homebrew.bintray.com/bottles/readline-8.1.big_sur.bottle.tar.gz Already downloaded: /Users/redacted/Library/Caches/Homebrew/downloads/b03c5b80e59c91f05f4327bf3cb7a4dbab63a8902b76c0c53c6d36eb2e4331e9--readline-8.1.big_sur.bottle.tar.gz ==> Downloading https://homebrew.bintray.com/bottles/sqlite-3.35.4.big_sur.bottle.tar.gz Already downloaded: /Users/redacted/Library/Caches/Homebrew/downloads/c9a360f163a962e10c1beb105f0478500624eedbec866bda679b0347558375a0--sqlite-3.35.4.big_sur.bottle.tar.gz ==> Downloading https://homebrew.bintray.com/bottles/python%403.9-3.9.4.big_sur.bottle.tar.gz Already downloaded: /Users/redacted/Library/Caches/Homebrew/downloads/e85b23594[email protected]3.9-3.9.4.big_sur.bottle.tar.gz ==> Downloading https://homebrew.bintray.com/bottles/torsocks-2.3.0.big_sur.bottle.1.tar.gz Already downloaded: /Users/redacted/Library/Caches/Homebrew/downloads/5da81126b71bef8bd6ce670c223f822dd875b993234c02ce45db3c778d3f0efa--torsocks-2.3.0.big_sur.bottle.1.tar.gz ==> Downloading https://homebrew.bintray.com/bottles/libffi-3.3_3.big_sur.bottle.tar.gz Already downloaded: /Users/redacted/Library/Caches/Homebrew/downloads/60b45c0f23d19cde24cfc8e6834288901010f39f7733d9b3312e759a58229193--libffi-3.3_3.big_sur.bottle.tar.gz ==> Downloading https://homebrew.bintray.com/bottles/python%403.9-3.9.4.big_sur.bottle.tar.gz Already downloaded: /Users/redacted/Library/Caches/Homebrew/downloads/e85b23594[email protected]3.9-3.9.4.big_sur.bottle.tar.gz ==> Downloading https://homebrew.bintray.com/bottles/glib-2.68.0.big_sur.bottle.tar.gz Already downloaded: /Users/redacted/Library/Caches/Homebrew/downloads/34a93ed75ca6c68aa1dc16c348870beef563edd005994b4c0ccb88b8e3628fa0--glib-2.68.0.big_sur.bottle.tar.gz Error: sshfs has been disabled because it requires FUSE!

    This happens even after running "brew cask install osxfuse" before and even if I install sshfs and MacFuse manually. Could you please help? I am using MacOS BigSur 11.2.3 (Intel, not Apple Silicon)

    Thanks and best regards Tarkleigh

    opened by Tarkleigh 28
  • Support for Non-Deployment Type Pods

    Support for Non-Deployment Type Pods

    Are there any plans to implement support for pods launched via statefulsets? The majority of our workloads are statefulsets, so being able to run e.g: telepresence --swap-pod cassandra-2 --docker-run --hostname cassandra-2 -it cassandra:local-v1 would be very useful.

    t:feature 
    opened by jordanwilson230 24
  • Telepresence doesn't work - SSH isn't starting

    Telepresence doesn't work - SSH isn't starting

    What were you trying to do?

    Run the $ telepresence command. This was freshly installed on a new machine.

    What did you expect to happen?

    It would create a connection into the cluster

    What happened instead?

    I get the traceback below.

    Automatically included information

    Command line: ['/usr/local/bin/telepresence', '--swap-deployment', 'one-service', '--namespace=develop', '--docker-run', '--rm', '-e', 'MAVEN_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005', '-v/Users/eugene/kuber/one:/build', '-v', '/Users/eugene/.m2/repository:/m2', '-p', '9001:9001', '-p', '5005:5005', 'maven-build:jdk8', 'mvn', '-Dmaven.repo.local=/m2', '-f', '/build', 'spring-boot:run'] Version: 0.95 Python version: 3.7.1 (default, Nov 28 2018, 11:51:54) [Clang 10.0.0 (clang-1000.11.45.5)] kubectl version: Client Version: v1.13.0 // Server Version: v1.10.0 oc version: (error: [Errno 2] No such file or directory: 'oc': 'oc') OS: Darwin home.local 17.7.0 Darwin Kernel Version 17.7.0: Wed Oct 10 23:06:14 PDT 2018; root:xnu-4570.71.13~1/RELEASE_X86_64 x86_64 Traceback:

    Traceback (most recent call last):
      File "/usr/local/bin/telepresence/telepresence/cli.py", line 130, in crash_reporting
        yield
      File "/usr/local/bin/telepresence/telepresence/main.py", line 84, in main
        runner, remote_info, env, socks_port, ssh, mount_dir
      File "/usr/local/bin/telepresence/telepresence/outbound/setup.py", line 75, in launch
        args.also_proxy, env, ssh, mount_dir
      File "/usr/local/bin/telepresence/telepresence/outbound/container.py", line 122, in run_docker_command
        local_ssh.wait()
      File "/usr/local/bin/telepresence/telepresence/connect/ssh.py", line 82, in wait
        raise RuntimeError("SSH isn't starting.")
    RuntimeError: SSH isn't starting.
    
    

    Logs:

    xit 255 in 0.00 secs.
      32.9 TEL | [139] Running: ssh -F /dev/null -q -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -p 52795 [email protected] /bin/true
      32.9 TEL | [139] exit 255 in 0.02 secs.
      33.1  28 |   28.8 TEL | [116] Running: ssh -F /dev/null -q -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -p 38023 [email protected] /bin/true
      33.1  28 |   28.8 TEL | [116] exit 255 in 0.00 secs.
      33.2 TEL | [140] Running: ssh -F /dev/null -q -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -p 52795 [email protected] /bin/true
      33.2 TEL | [140] exit 255 in 0.02 secs.
      33.4  28 |   29.0 TEL | [117] Running: ssh -F /dev/null -q -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -p 38023 [email protected] /bin/true
      33.4  28 |   29.0 TEL | [117] exit 255 in 0.00 secs.
      33.4 TEL | [141] Running: ssh -F /dev/null -q -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -p 52795 [email protected] /bin/true
      33.5 TEL | [141] exit 255 in 0.02 secs.
    
    
    stale 
    opened by dveoz 20
  • Telepresence breaks ec2 machine's internet

    Telepresence breaks ec2 machine's internet

    Describe the bug

    Hi, I am using telepresence to connect to my staging and CI EKS clusters via EC2 machines. Connecting to CI clusters works amazingly well, but connecting to staging clusters don't. Specifically, when I connect to telepresence via sudo -E telepresence connect, the root/user daemon starts but then it somehow interferes with my EC2's internet. Worst case is I am not able to SSH in my machine, the best case - my EC2 machine can't even ping google.com unless I do telepresence quit.

    Orthogonal question: Is there a way to run telepresence in a more isolated way such that it doesn't interfere with my machine's internet?


    To Reproduce Steps to reproduce the behavior:

    1. When I run sudo -E telepresence connect
    2. My ec2's internet vanishes until I reboot/disconnect telepresence.

    Expected behavior Telepresence to not mess with my machine's internet.

    Versions

    • 2.4.10

    Additional context Due to PII, I can't share all the logs gathered from telepresence gather-logs but if there's any specific log which is required then I can share after redacting.

    Seeing logs like: Connector

    2022-02-13 01:26:39.3373 info    connector/background-manager : Existing Traffic Manager 2.4.11 not owned by cli or does not need upgrade, will not modify
    2022-02-13 01:29:06.6510 error   connector/server-grpc/conn=2 : Tunnel manager.Send() failed: EOF
    
    2022-02-13 13:59:48.8730 error   connector/server-grpc/conn=9/Uninstall-13 : Unable to look for existing helm release: Kubernetes cluster unreachable: Get "<cluster>": dial tcp: lookup <cluster> on 127.0.0.53:53: read udp 127.0.0.1:52514->127.0.0.53:53: i/o timeout. Assuming it's already gone...
    

    Daemon

    2022-02-13 13:23:35.0659 info    Logging at this level "info"
    2022-02-13 13:23:35.0661 info    ---
    2022-02-13 13:23:35.0661 info    Telepresence daemon v2.4.10 (api v3) starting...
    2022-02-13 13:23:35.0661 info    PID is 193
    2022-02-13 13:23:35.0661 info    
    2022-02-13 13:23:35.0869 info    daemon/server-grpc : gRPC server started
    2022-02-13 13:23:36.7385 info    daemon/server-grpc/conn=2 : Adding never-proxy subnet 50.18.23.135/32
    2022-02-13 13:23:36.7491 info    daemon/server-grpc/conn=2 : Adding never-proxy subnet 52.8.73.143/32
    2022-02-13 13:23:36.7535 info    daemon/watch-cluster-info : Adding service subnet 10.100.0.0/16
    2022-02-13 13:23:36.7536 info    daemon/watch-cluster-info : Adding pod subnet 172.31.0.0/18
    2022-02-13 13:23:36.7540 info    daemon/watch-cluster-info : started command ["ip" "a" "add" "10.100.0.0/16" "dev" "tel0"] : dexec.pid="228"
    2022-02-13 13:23:36.7541 info    daemon/watch-cluster-info :  : dexec.pid="228" dexec.stream="stdin" dexec.err="EOF"
    2022-02-13 13:23:36.7548 info    daemon/watch-cluster-info : finished successfully: exit status 0 : dexec.pid="228"
    2022-02-13 13:23:36.7551 info    daemon/watch-cluster-info : started command ["ip" "a" "add" "172.31.0.0/18" "dev" "tel0"] : dexec.pid="229"
    2022-02-13 13:23:36.7552 info    daemon/watch-cluster-info :  : dexec.pid="229" dexec.stream="stdin" dexec.err="EOF"
    2022-02-13 13:23:36.7561 info    daemon/watch-cluster-info : finished successfully: exit status 0 : dexec.pid="229"
    2022-02-13 13:23:36.7562 info    daemon/watch-cluster-info : Setting cluster DNS to 10.100.0.10
    2022-02-13 13:23:36.7562 info    daemon/watch-cluster-info : Setting cluster domain to "cluster.local."
    2022-02-13 13:23:36.7588 info    daemon/server-router/MGR stream : Connected to Manager 2.4.11
    2022-02-13 13:23:36.8048 info    daemon/server-dns/docker : Automatically set -dns=127.0.0.53
    2022-02-13 13:23:36.8085 info    daemon/server-dns/docker/NAT-redirect : started command ["iptables" "-t" "nat" "-D" "OUTPUT" "-j" "telepresence-dns"] : dexec.pid="230"
    2022-02-13 13:23:36.8086 info    daemon/server-dns/docker/NAT-redirect :  : dexec.pid="230" dexec.stream="stdin" dexec.err="EOF"
    2022-02-13 13:23:36.8134 info    daemon/server-dns/docker/NAT-redirect :  : dexec.pid="230" dexec.stream="stdout+stderr" dexec.data="iptables v1.8.4 (legacy): Couldn't load target `telepresence-dns':No such file or directory\n"
    2022-02-13 13:23:36.8135 info    daemon/server-dns/docker/NAT-redirect :  : dexec.pid="230" dexec.stream="stdout+stderr" dexec.data="\n"
    2022-02-13 13:23:36.8135 info    daemon/server-dns/docker/NAT-redirect :  : dexec.pid="230" dexec.stream="stdout+stderr" dexec.data="Try `iptables -h' or 'iptables --help' for more information.\n"
    2022-02-13 13:23:36.8136 info    daemon/server-dns/docker/NAT-redirect : finished with error: exit status 2 : dexec.pid="230"
    2022-02-13 13:23:36.8138 info    daemon/server-dns/docker/NAT-redirect : started command ["iptables" "-t" "nat" "-F" "telepresence-dns"] : dexec.pid="231"
    2022-02-13 13:23:36.8139 info    daemon/server-dns/docker/NAT-redirect :  : dexec.pid="231" dexec.stream="stdin" dexec.err="EOF"
    2022-02-13 13:23:36.8144 info    daemon/server-dns/docker/NAT-redirect :  : dexec.pid="231" dexec.stream="stdout+stderr" dexec.data="iptables: No chain/target/match by that name.\n"
    2022-02-13 13:23:36.8145 info    daemon/server-dns/docker/NAT-redirect : finished with error: exit status 1 : dexec.pid="231"
    2022-02-13 13:23:36.8147 info    daemon/server-dns/docker/NAT-redirect : started command ["iptables" "-t" "nat" "-X" "telepresence-dns"] : dexec.pid="232"
    2022-02-13 13:23:36.8148 info    daemon/server-dns/docker/NAT-redirect :  : dexec.pid="232" dexec.stream="stdin" dexec.err="EOF"
    2022-02-13 13:23:36.8153 info    daemon/server-dns/docker/NAT-redirect :  : dexec.pid="232" dexec.stream="stdout+stderr" dexec.data="iptables: No chain/target/match by that name.\n"
    2022-02-13 13:23:36.8154 info    daemon/server-dns/docker/NAT-redirect : finished with error: exit status 1 : dexec.pid="232"
    2022-02-13 13:23:36.8156 info    daemon/server-dns/docker/NAT-redirect : started command ["iptables" "-t" "nat" "-N" "telepresence-dns"] : dexec.pid="233"
    2022-02-13 13:23:36.8157 info    daemon/server-dns/docker/NAT-redirect :  : dexec.pid="233" dexec.stream="stdin" dexec.err="EOF"
    2022-02-13 13:23:36.8163 info    daemon/server-dns/docker/NAT-redirect : finished successfully: exit status 0 : dexec.pid="233"
    2022-02-13 13:23:36.8165 info    daemon/server-dns/docker/NAT-redirect : started command ["iptables" "-t" "nat" "-I" "OUTPUT" "1" "-j" "telepresence-dns"] : dexec.pid="234"
    2022-02-13 13:23:36.8166 info    daemon/server-dns/docker/NAT-redirect :  : dexec.pid="234" dexec.stream="stdin" dexec.err="EOF"
    2022-02-13 13:23:36.8180 info    daemon/server-dns/docker/NAT-redirect : finished successfully: exit status 0 : dexec.pid="234"
    2022-02-13 13:23:36.8184 info    daemon/server-dns/docker/NAT-redirect : started command ["iptables" "-t" "nat" "-A" "telepresence-dns" "-p" "udp" "--source" "127.0.0.1" "--sport" "39891" "-j" "RETURN"] : dexec.pid="235"
    2022-02-13 13:23:36.8185 info    daemon/server-dns/docker/NAT-redirect :  : dexec.pid="235" dexec.stream="stdin" dexec.err="EOF"
    2022-02-13 13:23:36.8211 info    daemon/server-dns/docker/NAT-redirect : finished successfully: exit status 0 : dexec.pid="235"
    2022-02-13 13:23:36.8213 info    daemon/server-dns/docker/NAT-redirect : started command ["iptables" "-t" "nat" "-A" "telepresence-dns" "-p" "udp" "--dest" "127.0.0.53/32" "--dport" "53" "-j" "REDIRECT" "--to-ports" "52912"] : dexec.pid="236"
    2022-02-13 13:23:36.8213 info    daemon/server-dns/docker/NAT-redirect :  : dexec.pid="236" dexec.stream="stdin" dexec.err="EOF"
    2022-02-13 13:23:36.8265 info    daemon/server-dns/docker/NAT-redirect : finished successfully: exit status 0 : dexec.pid="236"
    2022-02-13 13:24:21.0625 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:21.0626 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:21.0626 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:21.0626 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:21.0626 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:26.0630 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:26.0630 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:26.0631 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:26.0631 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:26.0631 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:31.3219 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:31.3219 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:31.3219 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:31.3220 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:31.3220 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:31.3220 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:36.3224 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:36.3225 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:36.3225 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:36.3225 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:36.3226 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:36.3226 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:39.0648 info    daemon/background-metriton : scout report "incluster_dns_query" failed: Post "https://metriton.datawire.io/scout": dial tcp: lookup metriton.datawire.io on 127.0.0.53:53: read udp 127.0.0.1:55046->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:41.3406 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:41.3406 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:41.3407 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:41.3407 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:46.3398 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:46.3398 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:46.3398 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:46.3398 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:51.3261 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:51.3261 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:51.3261 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:51.3261 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:56.3256 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:56.3257 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:56.3256 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:56.3256 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:24:59.0668 info    daemon/background-metriton : scout report "incluster_dns_query" failed: Post "https://metriton.datawire.io/scout": dial tcp: lookup metriton.datawire.io on 127.0.0.53:53: read udp 127.0.0.1:35353->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:01.3395 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:01.3395 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:01.3396 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:01.3396 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:06.3353 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:06.3354 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:06.3353 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:06.3354 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:11.3286 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:11.3286 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:11.3287 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:11.3287 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:16.3285 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:16.3286 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:16.3286 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:16.3285 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:19.0698 info    daemon/background-metriton : scout report "incluster_dns_query" failed: Post "https://metriton.datawire.io/scout": dial tcp: lookup metriton.datawire.io on 127.0.0.53:53: read udp 127.0.0.1:60155->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:21.3429 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:21.3429 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:21.3428 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:21.3429 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:26.3392 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:26.3393 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:26.3393 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:26.3393 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:31.0924 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:31.0924 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:36.0878 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:36.0879 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:39.0740 info    daemon/background-metriton : scout report "incluster_dns_query" failed: Post "https://metriton.datawire.io/scout": dial tcp: lookup metriton.datawire.io on 127.0.0.53:53: read udp 127.0.0.1:38120->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:41.0752 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:41.0753 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:46.0755 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:46.0756 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:46.0756 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:46.0756 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:51.0898 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:51.0898 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:51.0899 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:51.0899 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:56.9214 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:56.9215 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:56.9215 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:56.9215 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:56.9215 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:56.9215 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:25:59.0764 info    daemon/background-metriton : scout report "incluster_dns_query" failed: Post "https://metriton.datawire.io/scout": dial tcp: lookup metriton.datawire.io on 127.0.0.53:53: read udp 127.0.0.1:44467->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:01.9262 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:01.9262 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:01.9263 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:01.9263 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:01.9263 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:01.9263 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:06.9900 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:06.9901 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:06.9900 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:06.9901 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:06.9901 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:06.9901 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:11.9472 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:11.9472 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:11.9473 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:11.9473 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:11.9473 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:11.9473 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:19.0796 info    daemon/background-metriton : scout report "incluster_dns_query" failed: Post "https://metriton.datawire.io/scout": dial tcp: lookup metriton.datawire.io on 127.0.0.53:53: read udp 127.0.0.1:33979->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:36.0958 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:36.0958 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:36.0958 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:36.0958 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:36.0959 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:36.0959 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:36.0959 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:36.0959 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:36.0960 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:36.0960 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:36.0960 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:36.0960 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:36.0960 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:36.0960 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:36.0961 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:36.0961 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:36.0961 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:36.0961 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:36.0961 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:36.0961 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:36.0961 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:36.0961 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:36.0962 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:36.0962 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:36.0962 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:36.0962 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:36.0962 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:36.0962 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:39.0825 info    daemon/background-metriton : scout report "incluster_dns_query" failed: Post "https://metriton.datawire.io/scout": dial tcp: lookup metriton.datawire.io on 127.0.0.53:53: read udp 127.0.0.1:34883->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:41.0835 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:41.0835 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:41.0836 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:41.0836 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:41.0836 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:41.0836 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:46.0845 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:46.0846 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:46.0846 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:46.0846 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:46.0846 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:46.0846 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:51.0955 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:51.0955 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:51.0956 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:51.0956 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:51.0956 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:51.0956 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:56.0942 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:56.0943 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:56.0943 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:56.0943 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:56.0944 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:56.0944 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:26:59.0865 info    daemon/background-metriton : scout report "incluster_dns_query" failed: Post "https://metriton.datawire.io/scout": dial tcp: lookup metriton.datawire.io on 127.0.0.53:53: read udp 127.0.0.1:37320->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:01.0872 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:01.0872 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:01.0873 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:01.0873 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:01.0873 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:01.0873 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:06.0884 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:06.0884 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:06.0884 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:06.0885 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:11.1045 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:11.1046 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:11.1046 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:11.1046 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:16.1006 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:16.1007 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:16.1007 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:16.1007 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:19.0896 info    daemon/background-metriton : scout report "incluster_dns_query" failed: Post "https://metriton.datawire.io/scout": dial tcp: lookup metriton.datawire.io on 127.0.0.53:53: read udp 127.0.0.1:58771->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:21.0903 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:21.0904 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:26.0906 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:26.0906 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:26.0907 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:26.0907 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:31.1043 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:31.1044 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:31.1044 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:31.1044 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:36.1120 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:36.1121 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:36.1121 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:36.1121 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:39.0918 info    daemon/background-metriton : scout report "incluster_dns_query" failed: Post "https://metriton.datawire.io/scout": dial tcp: lookup metriton.datawire.io on 127.0.0.53:53: read udp 127.0.0.1:43301->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:41.0927 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:41.0927 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:41.0928 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:41.0928 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:46.0925 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:46.0925 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:51.1020 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:51.1020 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:56.1050 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:56.1050 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:27:59.0941 info    daemon/background-metriton : scout report "incluster_dns_query" failed: Post "https://metriton.datawire.io/scout": dial tcp: lookup metriton.datawire.io on 127.0.0.53:53: read udp 127.0.0.1:33507->127.0.0.53:53: i/o timeout
    2022-02-13 13:28:01.0946 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:28:01.0946 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:28:06.0951 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:28:06.0951 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:28:11.1069 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:28:11.1069 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:28:16.1049 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:28:16.1049 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:28:19.0963 info    daemon/background-metriton : scout report "incluster_dns_query" failed: Post "https://metriton.datawire.io/scout": dial tcp: lookup metriton.datawire.io on 127.0.0.53:53: read udp 127.0.0.1:34977->127.0.0.53:53: i/o timeout
    2022-02-13 13:28:21.0973 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:28:21.0973 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:28:26.0979 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:28:26.0979 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:28:31.1118 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:28:31.1118 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:28:34.1052 info    daemon/server-dns/docker/Server : scout report "incluster_dns_query" discarded. Output buffer is full (or closed)
    2022-02-13 13:28:36.1062 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:28:36.1062 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:28:39.0997 info    daemon/background-metriton : scout report "incluster_dns_query" failed: Post "https://metriton.datawire.io/scout": dial tcp: lookup metriton.datawire.io on 127.0.0.53:53: read udp 127.0.0.1:39160->127.0.0.53:53: i/o timeout
    2022-02-13 13:28:41.1012 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:28:41.1013 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:28:46.1008 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:28:46.1009 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:28:51.1156 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:28:51.1156 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:28:54.1177 info    daemon/server-dns/docker/Server : scout report "incluster_dns_query" discarded. Output buffer is full (or closed)
    2022-02-13 13:28:56.1181 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:28:56.1181 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:28:59.1024 info    daemon/background-metriton : scout report "incluster_dns_query" failed: Post "https://metriton.datawire.io/scout": dial tcp: lookup metriton.datawire.io on 127.0.0.53:53: read udp 127.0.0.1:52227->127.0.0.53:53: i/o timeout
    2022-02-13 13:29:01.1032 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:29:01.1032 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:29:06.1046 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:29:06.1047 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:29:11.1174 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:29:11.1175 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:29:14.1135 info    daemon/server-dns/docker/Server : scout report "incluster_dns_query" discarded. Output buffer is full (or closed)
    2022-02-13 13:29:16.1146 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:29:16.1146 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:29:19.1049 info    daemon/background-metriton : scout report "incluster_dns_query" failed: Post "https://metriton.datawire.io/scout": dial tcp: lookup metriton.datawire.io on 127.0.0.53:53: read udp 127.0.0.1:35760->127.0.0.53:53: i/o timeout
    2022-02-13 13:29:21.1056 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:29:21.1056 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:29:26.1071 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:29:26.1072 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:29:31.1215 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:29:31.1216 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:29:34.1222 info    daemon/server-dns/docker/Server : scout report "incluster_dns_query" discarded. Output buffer is full (or closed)
    2022-02-13 13:29:36.1230 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:29:36.1230 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:29:39.1083 info    daemon/background-metriton : scout report "incluster_dns_query" failed: Post "https://metriton.datawire.io/scout": dial tcp: lookup metriton.datawire.io on 127.0.0.53:53: read udp 127.0.0.1:48514->127.0.0.53:53: i/o timeout
    2022-02-13 13:29:41.1094 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:29:41.1095 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:29:46.1108 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:29:46.1109 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:29:46.1110 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:29:46.1110 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:29:51.1184 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:29:51.1185 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:29:51.1185 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:29:51.1185 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:29:54.1337 info    daemon/server-dns/docker/Server : scout report "incluster_dns_query" discarded. Output buffer is full (or closed)
    2022-02-13 13:29:56.1344 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:29:56.1344 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:29:56.1345 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:29:56.1345 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:29:59.1127 info    daemon/background-metriton : scout report "incluster_dns_query" failed: Post "https://metriton.datawire.io/scout": dial tcp: lookup metriton.datawire.io on 127.0.0.53:53: read udp 127.0.0.1:52449->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:01.1137 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:01.1138 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:01.1139 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:01.1139 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:06.1142 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:06.1142 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:06.1143 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:06.1143 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:08.5349 info    daemon/server-dns/docker/Server : scout report "incluster_dns_query" discarded. Output buffer is full (or closed)
    2022-02-13 13:30:09.1652 info    daemon/server-dns/docker/Server : scout report "incluster_dns_query" discarded. Output buffer is full (or closed)
    2022-02-13 13:30:11.1659 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:11.1660 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:11.1659 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:11.1660 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:14.1222 info    daemon/server-dns/docker/Server : scout report "incluster_dns_query" discarded. Output buffer is full (or closed)
    2022-02-13 13:30:16.2579 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:16.2580 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:16.2580 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:16.2580 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:19.1155 info    daemon/background-metriton : scout report "incluster_dns_query" failed: Post "https://metriton.datawire.io/scout": dial tcp: lookup metriton.datawire.io on 127.0.0.53:53: read udp 127.0.0.1:48125->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:21.2628 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:21.2628 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:21.2629 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:21.2627 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:26.3170 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:26.3171 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:26.3171 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:26.3170 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:29.1292 info    daemon/server-dns/docker/Server : scout report "incluster_dns_query" discarded. Output buffer is full (or closed)
    2022-02-13 13:30:29.2814 info    daemon/server-dns/docker/Server : scout report "incluster_dns_query" discarded. Output buffer is full (or closed)
    2022-02-13 13:30:31.2816 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:31.2817 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:31.2817 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:31.2817 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:34.1354 info    daemon/server-dns/docker/Server : scout report "incluster_dns_query" discarded. Output buffer is full (or closed)
    2022-02-13 13:30:36.1359 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:36.1359 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:38.2081 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:38.2081 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:39.1178 info    daemon/background-metriton : scout report "incluster_dns_query" failed: Post "https://metriton.datawire.io/scout": dial tcp: lookup metriton.datawire.io on 127.0.0.53:53: read udp 127.0.0.1:50489->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:41.1182 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:41.1183 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:43.2130 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:43.2130 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:46.1186 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:46.1187 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:48.2351 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:48.2351 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:49.1290 info    daemon/server-dns/docker/Server : scout report "incluster_dns_query" discarded. Output buffer is full (or closed)
    2022-02-13 13:30:51.1296 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:51.1296 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:51.2304 info    daemon/server-dns/docker/Server : scout report "incluster_dns_query" discarded. Output buffer is full (or closed)
    2022-02-13 13:30:53.2308 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:53.2308 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:54.1388 info    daemon/server-dns/docker/Server : scout report "incluster_dns_query" discarded. Output buffer is full (or closed)
    2022-02-13 13:30:56.1391 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:56.1392 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:30:59.1206 info    daemon/background-metriton : scout report "incluster_dns_query" failed: Post "https://metriton.datawire.io/scout": dial tcp: lookup metriton.datawire.io on 127.0.0.53:53: read udp 127.0.0.1:59145->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:01.9039 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:01.9039 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:01.9040 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:01.9040 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:06.9086 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:06.9086 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:06.9087 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:06.9087 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:09.9299 info    daemon/server-dns/docker/Server : scout report "incluster_dns_query" discarded. Output buffer is full (or closed)
    2022-02-13 13:31:11.9301 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:11.9302 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:11.9302 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:11.9302 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:14.1294 info    daemon/server-dns/docker/Server : scout report "incluster_dns_query" discarded. Output buffer is full (or closed)
    2022-02-13 13:31:14.9307 info    daemon/server-dns/docker/Server : scout report "incluster_dns_query" discarded. Output buffer is full (or closed)
    2022-02-13 13:31:16.9311 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:16.9311 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:16.9311 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:16.9311 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:19.1237 info    daemon/background-metriton : scout report "incluster_dns_query" failed: Post "https://metriton.datawire.io/scout": dial tcp: lookup metriton.datawire.io on 127.0.0.53:53: read udp 127.0.0.1:59946->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:21.1243 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:21.1244 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:26.1253 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:26.1254 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:31.1540 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:31.1541 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:31.1541 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:31.1541 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:34.1429 info    daemon/server-dns/docker/Server : scout report "incluster_dns_query" discarded. Output buffer is full (or closed)
    2022-02-13 13:31:36.1438 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:36.1438 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:36.1439 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:36.1439 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:37.7106 info    daemon/server-dns/docker/Server : scout report "incluster_dns_query" discarded. Output buffer is full (or closed)
    2022-02-13 13:31:39.1265 info    daemon/background-metriton : scout report "incluster_dns_query" failed: Post "https://metriton.datawire.io/scout": dial tcp: lookup metriton.datawire.io on 127.0.0.53:53: read udp 127.0.0.1:53171->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:41.1273 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:41.1273 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:41.1273 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:41.1273 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:46.1281 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:46.1281 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:46.1282 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:46.1282 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:49.1424 info    daemon/server-dns/docker/Server : scout report "incluster_dns_query" discarded. Output buffer is full (or closed)
    2022-02-13 13:31:50.9147 info    daemon/daemon-quit : Shutting down connector
    2022-02-13 13:31:50.9147 info    daemon/server-dns/docker:shutdown_logger : shutting down (gracefully)...
    2022-02-13 13:31:50.9147 info    daemon/server-dns/docker/Server:shutdown_logger : shutting down (gracefully)...
    2022-02-13 13:31:50.9147 info    daemon:shutdown_logger : shutting down (gracefully)...
    2022-02-13 13:31:51.1436 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    2022-02-13 13:31:51.1436 error   daemon/server-dns/docker/Server : read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout
    time="2022-02-13T13:31:51Z" level=info dexec.pid=280 msg="started command [\"iptables\" \"-t\" \"nat\" \"-D\" \"OUTPUT\" \"-j\" \"telepresence-dns\"]"
    time="2022-02-13T13:31:51Z" level=info dexec.pid=280 dexec.stream=stdin dexec.err=EOF
    time="2022-02-13T13:31:51Z" level=info dexec.pid=280 msg="finished successfully: exit status 0"
    time="2022-02-13T13:31:51Z" level=info dexec.pid=281 msg="started command [\"iptables\" \"-t\" \"nat\" \"-F\" \"telepresence-dns\"]"
    time="2022-02-13T13:31:51Z" level=info dexec.pid=281 dexec.stream=stdin dexec.err=EOF
    time="2022-02-13T13:31:51Z" level=info dexec.pid=281 msg="finished successfully: exit status 0"
    time="2022-02-13T13:31:51Z" level=info dexec.pid=282 msg="started command [\"iptables\" \"-t\" \"nat\" \"-X\" \"telepresence-dns\"]"
    time="2022-02-13T13:31:51Z" level=info dexec.pid=282 dexec.stream=stdin dexec.err=EOF
    time="2022-02-13T13:31:51Z" level=info dexec.pid=282 msg="finished successfully: exit status 0"
    2022-02-13 13:31:52.9155 info    daemon:shutdown_logger : shutting down (not-so-gracefully)...
    
    opened by bhavitsharma 19
  • Regression: Unable to download large files via intercept after telepresence 2.3

    Regression: Unable to download large files via intercept after telepresence 2.3

    Hello,

    I have been using telepresence v2.2.2 with a GKE cluster and an nginx ingress for a few weeks, using telepresence to develop frontend against backend already in the cluster.

    The frontend is using react and TypeScript, using yarn and yarn.lock file to lock all the dependencies explicitly. We're using webpack dev server with hot reload for development and that is how telepresence was used - original deployment of the prod version of the app gets intercepted and all requests for UI are routed to the webpack dev server on developer's computer.

    The application in GKE is behind nginx-ingress (version 0.32.0) and ingress is exposing the application over https.

    Things have worked fine for a while, including the hot reload feature. Recently I have updated to telepresence v2.3.2 and things stopped working. Specifically the UI never finishes loading. I can see some of webpack messages, but nothing finishes loading.

    After a lot of trial and error with things that could possibly go wrong, I have ruled out:

    • updates to the backend code - by rolling back to version that I am confident worked with telepresence and ensuring all backend components use a specific container image and settings (the project has a separate config repository for Kubernetes so it is easy to roll back to a specific version)
    • updates to the frontend code - I have rolled back to commit I know to work with telepresence ; since all the npm packages are yarn.lock-ed, it should be a reproducible setup
    • browser issues - I have run an older version of Chrome for a while to ensure the problem is not related to an update in the browser
    • generic networking issues - I have replaced webpack dev server with nginx running on my machine and hosting a static directory, I was able to download small and large files without issues

    After ruling out all the issues that could be caused by changes on my end, I have consistently tried all versions of telepresence and came up with v2.2.2 working as expected and v2.3.0 not working. I have also tried v2.3.1 and v2.3.2 and all of them fail to work and with telepresence intercept enabled.

    I suspect this has to be some edge case related to many features webpack dev server is using - such as websockets it uses to inform the browser of updates in the remote code and need to reload.

    Could anyone help determine what could be the issue? Is this a known problem with v2.3?

    Are there some settings, additional configuration or changes to traffic-manager deployment I could make to get additional logs / information?

    t:bug v2 w:6 
    opened by wojciechka 19
  • Unable to intercept even though it says it was successful in creating the intercept

    Unable to intercept even though it says it was successful in creating the intercept

    Describe the bug Unable to intercept service. The local environment is MACOS-Catalina

    telepresence intercept demo-app-v1 --namespace=dev --port 9082:9082 --env-file config.env
    Using Deployment demo-app-v1
    intercepted
        Intercept name         : demo-app-v1-dev
        State                  : ACTIVE
        Workload kind          : Deployment
        Destination            : 127.0.0.1:9082
        Service Port Identifier: 9082
        Volume Mount Error     : macFUSE 4.0.5 or higher is required on your local machine
        Intercepting           : all TCP connections
    
    telepresence list
    No Workloads (Deployments, StatefulSets, or ReplicaSets)
    

    On the traffic agent side car logs I find this

    "2022-02-03 20:09:24.9697 debug   client : Lookup response for macc02zv6ttmd6t.dev.svc.cluster.local -> NOT FOUND
    "
    

    The intercept is just not working. WOnder what needs to be done.

    opened by Sabyasachi2k 18
  • Downgrade Telepresence to version 0.99

    Downgrade Telepresence to version 0.99

    HI, I have telepresence version 0.101 and i want to downgrade to version 0.99. I tried to downgrade using a pre built copy using the link below but looks like its still takes the latest version. Is there a way for me to keep both versions but let me choose which version i want to use? If not then should i just delete my existing telepresence and use the link below? Thanks!!

    Pre-built link: https://s3.amazonaws.com/datawire-static-files/telepresence/telepresence-0.99.tar.gz

    opened by Newbie112222 17
  • dns not work on windows10

    dns not work on windows10

    Still failed to parse the k8s service name after executing "telepresenece connect" successfully:

    io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Search domain query failed. Original hostname: 'redis-cs.svc.cluster.local' failed to resolve 'redis-cs.svc.cluster.local' after 7 queries at io.netty.resolver.dns.DnsResolveContext.finishResolve(DnsResolveContext.java:718)

    While "telnet redis-cs.svc.cluster.local 6379" is OK in the command.

    Windows10 64bit telepresence v2.4.4-nightly-5cae5bfc(api v3)

    opened by ytfrank 16
  • ssh keys too permissive in docker image

    ssh keys too permissive in docker image

    What were you trying to do?

    swap out deployment, like this:

    telepresence --swap-deployment goalkeeper --run-shell
    

    What did you expect to happen?

    deployment is successfully replaced in the k8s cluster.

    What happened instead?

    it seems like the proxy deployment failed to come up because the docker image's ssh key files (such as /etc/ssh/ssh_host_rsa_key) have linux permissions that are too permissive. chmod 0600 is desirable, but 0640 is found, and although it says "warning" a number of times, it ends up being fatal, it seems, because it can't find any keys. the docker image it was using was:

    datawire/telepresence-k8s:0.99
    

    Automatically included information

    Command line: ['/usr/local/bin/telepresence', '--swap-deployment', 'goalkeeper-qa8', '--run-shell'] Version: 0.99 Python version: 3.7.3 (default, Mar 29 2019, 07:52:52) [Clang 10.0.1 (clang-1001.0.46.3)] kubectl version: Client Version: v1.11.0 // Server Version: v1.13.5 oc version: (error: [Errno 2] No such file or directory: 'oc': 'oc') OS: Darwin rslmac14810.example.com 18.5.0 Darwin Kernel Version 18.5.0: Mon Mar 11 20:40:32 PDT 2019; root:xnu-4903.251.3~3/RELEASE_X86_64 x86_64

    Traceback (most recent call last):
      File "/usr/local/bin/telepresence/telepresence/cli.py", line 136, in crash_reporting
        yield
      File "/usr/local/bin/telepresence/telepresence/main.py", line 60, in main
        remote_info = start_proxy(runner)
      File "/usr/local/bin/telepresence/telepresence/proxy/__init__.py", line 95, in start_proxy
        run_id=run_id,
      File "/usr/local/bin/telepresence/telepresence/proxy/remote.py", line 202, in get_remote_info
        wait_for_pod(runner, remote_info)
      File "/usr/local/bin/telepresence/telepresence/proxy/remote.py", line 134, in wait_for_pod
        "Pod isn't starting or can't be found: {}".format(pod["status"])
    RuntimeError: Pod isn't starting or can't be found: {'conditions': [{'lastProbeTime': None, 'lastTransitionTime': '2019-04-26T20:08:06Z', 'status': 'True', 'type': 'Initialized'}, {'lastProbeTime': None, 'lastTransitionTime': '2019-04-26T20:08:06Z', 'message': 'containers with unready status: [goalkeeper-qa8]', 'reason': 'ContainersNotReady', 'status': 'False', 'type': 'Ready'}, {'lastProbeTime': None, 'lastTransitionTime': '2019-04-26T20:08:06Z', 'message': 'containers with unready status: [goalkeeper-qa8]', 'reason': 'ContainersNotReady', 'status': 'False', 'type': 'ContainersReady'}, {'lastProbeTime': None, 'lastTransitionTime': '2019-04-26T20:08:06Z', 'status': 'True', 'type': 'PodScheduled'}], 'containerStatuses': [{'containerID': 'docker://aa93b526f3065f567f1d18b813df07b0db56c8711d4bd4ecc43b037604d73c65', 'image': 'datawire/telepresence-k8s:0.99', 'imageID': 'docker-pullable://datawire/[email protected]:f9b640fa6640a0437cbafb0a3a238b10daf86e117e5a3ca4d4cd4b07eee76f9c', 'lastState': {'terminated': {'containerID': 'docker://aa93b526f3065f567f1d18b813df07b0db56c8711d4bd4ecc43b037604d73c65', 'exitCode': 1, 'finishedAt': '2019-04-26T20:09:51Z', 'message': "@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    Permissions 0640 for '/etc/ssh/ssh_host_rsa_key' are too open.
    It is required that your private key files are NOT accessible by others.
    This private key will be ignored.
    key_load_private: bad permissions
    Could not load host key: /etc/ssh/ssh_host_rsa_key
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    Permissions 0640 for '/etc/ssh/ssh_host_dsa_key' are too open.
    It is required that your private key files are NOT accessible by others.
    This private key will be ignored.
    key_load_private: bad permissions
    Could not load host key: /etc/ssh/ssh_host_dsa_key
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    Permissions 0640 for '/etc/ssh/ssh_host_ecdsa_key' are too open.
    It is required that your private key files are NOT accessible by others.
    This private key will be ignored.
    key_load_private: bad permissions
    Could not load host key: /etc/ssh/ssh_host_ecdsa_key
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    Permissions 0640 for '/etc/ssh/ssh_host_ed25519_key' are too open.
    It is required that your private key files are NOT accessible by others.
    This private key will be ignored.
    key_load_private: bad permissions
    Could not load host key: /etc/ssh/ssh_host_ed25519_key
    sshd: no hostkeys available -- exiting.
    
    opened by burnettk 16
  • Error when attempting to use telepresence.

    Error when attempting to use telepresence.

    I got a message saying there was a bug and I should submit a bug report:

    
    19:28:18.297 [ERROR] [system.err] Mounting remote volumes failed, they will be unavailable in this session. If you are running on Windows Subystem for Linux then see https://github.com/datawire/telepresence/issues/115, otherwise please report a bug, attaching telepresence.log to the bug report: https://github.com/datawire/telepresence/issues/new
    19:28:18.297 [QUIET] [system.out]
    19:28:18.299 [QUIET] [system.out] Looks like there's a bug in our code. Sorry about that!
    19:28:18.300 [QUIET] [system.out]
    19:28:18.300 [QUIET] [system.out] Here's the traceback:
    19:28:18.300 [QUIET] [system.out]
    19:28:18.300 [QUIET] [system.out] Traceback (most recent call last):
    19:28:18.301 [QUIET] [system.out]   File "/usr/local/Cellar/telepresence/0.75/libexec/lib/python3.6/site-packages/telepresence/cli.py", line 73, in call_f
    19:28:18.301 [QUIET] [system.out]     return f(*args, **kwargs)
    19:28:18.301 [QUIET] [system.out]   File "/usr/local/Cellar/telepresence/0.75/libexec/lib/python3.6/site-packages/telepresence/main.py", line 493, in go
    19:28:18.301 [QUIET] [system.out]     runner, remote_info, args, env, subprocesses, socks_port, ssh
    19:28:18.301 [QUIET] [system.out]   File "/usr/local/Cellar/telepresence/0.75/libexec/lib/python3.6/site-packages/telepresence/local.py", line 154, in run_local_command
    19:28:18.301 [QUIET] [system.out]     connect_sshuttle(runner, remote_info, args, subprocesses, env, ssh)
    19:28:18.301 [QUIET] [system.out]   File "/usr/local/Cellar/telepresence/0.75/libexec/lib/python3.6/site-packages/telepresence/vpn.py", line 241, in connect_sshuttle
    19:28:18.302 [QUIET] [system.out]     get_hellotelepresence()
    19:28:18.302 [QUIET] [system.out]   File "/usr/local/Cellar/telepresence/0.75/libexec/lib/python3.6/site-packages/telepresence/vpn.py", line 230, in get_hellotelepresence
    19:28:18.303 [QUIET] [system.out]     format(next(counter))
    19:28:18.303 [QUIET] [system.out]   File "/usr/local/Cellar/telepresence/0.75/libexec/lib/python3.6/site-packages/telepresence/runner.py", line 95, in get_output
    19:28:18.304 [QUIET] [system.out]     result = str(check_output(*args, **kwargs).strip(), "utf-8")
    19:28:18.304 [QUIET] [system.out]   File "/usr/local/opt/python/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 336, in check_output
    19:28:18.304 [QUIET] [system.out]     **kwargs).stdout
    19:28:18.304 [QUIET] [system.out]   File "/usr/local/opt/python/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 418, in run
    19:28:18.304 [QUIET] [system.out]     output=stdout, stderr=stderr)
    19:28:18.304 [QUIET] [system.out] subprocess.CalledProcessError: Command '['python3', '-c', "import socket; socket.gethostbyname('hellotelepresence6')"]' returned non-zero exit status 1.
    19:28:18.305 [QUIET] [system.out]
    19:28:18.305 [QUIET] [system.out]
    19:28:18.305 [QUIET] [system.out] And here are the last few lines of the logfile (see ./telepresence.log for the complete logs):
    19:28:18.305 [QUIET] [system.out]
    19:28:18.306 [QUIET] [system.out]  194.3 145 | c : DNS request from ('10.55.55.220', 64570) to None: 50 bytes
    19:28:18.306 [QUIET] [system.out]  194.3 133 | 2018-03-10T01:28:16+0000 [stdout#info] A query: b'www.us-east-1.prodaa.netflix.com'
    19:28:18.306 [QUIET] [system.out]  195.0 133 | 2018-03-10T01:28:16+0000 [-] DNSDatagramProtocol starting on 30481
    19:28:18.306 [QUIET] [system.out]  195.0 133 | 2018-03-10T01:28:16+0000 [-] Starting protocol <twisted.names.dns.DNSDatagramProtocol object at 0x7f11cc36f780>
    19:28:18.307 [QUIET] [system.out]  195.0 133 | 2018-03-10T01:28:16+0000 [-] (UDP Port 31572 Closed)
    19:28:18.307 [QUIET] [system.out]  195.0 133 | 2018-03-10T01:28:16+0000 [-] Stopping protocol <twisted.names.dns.DNSDatagramProtocol object at 0x7f11cc36fb38>
    19:28:18.307 [QUIET] [system.out]  195.1 139 | debug1: client_input_global_request: rtype [email protected] want_reply 1
    19:28:18.307 [QUIET] [system.out]  195.4 137 | debug1: client_input_global_request: rtype [email protected] want_reply 1
    19:28:18.310 [QUIET] [system.out]  196.0 133 | 2018-03-10T01:28:17+0000 [-] DNSDatagramProtocol starting on 40604
    19:28:18.316 [QUIET] [system.out]  196.0 133 | 2018-03-10T01:28:17+0000 [-] Starting protocol <twisted.names.dns.DNSDatagramProtocol object at 0x7f11cc36fba8>
    19:28:18.318 [QUIET] [system.out]  196.0 133 | 2018-03-10T01:28:17+0000 [-] (UDP Port 2664 Closed)
    19:28:18.319 [QUIET] [system.out]  196.0 133 | 2018-03-10T01:28:17+0000 [-] Stopping protocol <twisted.names.dns.DNSDatagramProtocol object at 0x7f11cc359550>
    19:28:18.320 [QUIET] [system.out]  196.0 138 | debug1: client_input_global_request: rtype [email protected] want_reply 1
    19:28:18.321 [QUIET] [system.out]  196.1 139 | debug1: client_input_global_request: rtype [email protected] want_reply 1
    19:28:18.323 [QUIET] [system.out]  196.2 145 | c : DNS request from ('10.55.55.220', 56001) to None: 30 bytes
    19:28:18.326 [QUIET] [system.out]  196.2 133 | 2018-03-10T01:28:18+0000 [stdout#info] A query: b'www.hulu.com'
    19:28:18.326 [QUIET] [system.out]  196.4 137 | debug1: client_input_global_request: rtype [email protected] want_reply 1
    19:28:18.328 [QUIET] [system.out] Traceback (most recent call last):
    19:28:18.343 [QUIET] [system.out]   File "<string>", line 1, in <module>
    19:28:18.345 [QUIET] [system.out] socket.gaierror: [Errno 8] nodename nor servname provided, or not known
    
    

    telepresence.log

    a:docs stale 
    opened by nafeger 16
  • Intercept generates an invalid `.env` file when multiline variables are present

    Intercept generates an invalid `.env` file when multiline variables are present

    Describe the bug When intercepting a deployment with a multi-line environment variable, telepresence intercept generates an invalid .env file which results in a failure when paired with --docker-run.

    To Reproduce Steps to reproduce the behavior:

    1. When I run telepresence intercept <deployment> --docker-run -- ...
    2. I see docker: poorly formatted environment: variable '"keys": [' contains whitespaces.

    Expected behavior

    I don't really know because of https://github.com/moby/moby/issues/12997, but it would be great if we could at least have an .env file that is parseable and then document the multiline limitation.

    I was thinking about this approach in which we escape the environment variable as ENV="...": https://github.com/jfmyers9/telepresence/commit/8dfb4599ec9d192208ab5fd2fcb14ff79d5d6f90 but it doesn't work as expected as Docker treats anything after the = literally.

    Other options include:

    • Collapsing it to one line with literal \n that need to be evaluated with echo -e or equivalent inside the container.
    • Skipping multiline environment variables and logging a warning.
    • Passing multiline environment variables using -e instead of the .env file.

    Versions (please complete the following information):

    • Output of telepresence version
    Client: v2.7.0 (api v3)
    Root Daemon: v2.7.0 (api v3)
    User Daemon: v2.7.0 (api v3)
    
    • Operating system of workstation running telepresence commands MacOS
    • Kubernetes environment and Version [e.g. Minikube, bare metal, Google Kubernetes Engine] 1.20, EKS

    Additional context Just looking for ideas.

    opened by jfmyers9 0
  • Cannot resolve non-kubernetes Istio Entities

    Cannot resolve non-kubernetes Istio Entities

    Describe the bug

    I am trying to make Telepresence work with a cluster-mesh (Gloo Mesh) that uses Istio with mTLS under the hood.

    So I managed to intercept traffic in plaintext (so incoming requests hit the istio-sidecar and then the intercept) but am now struggling with DNS resolution

    From my local machine, I can access any K8S service running in the cluster. Where this fails is for any CRD that istio manages (VirtualDestination, ServiceEntry).

    To my understanding this is because of istio intercepting DNS queries. The sidecar checks if a DNS query can be resolved from the internal definitions of Istio and serves those instead.

    Is it possible to get this to work with telepresence?

    To Reproduce Steps to reproduce the behavior:

    1. Telepresence connect
    2. I can intercept traffic and send out calls to K8S services
    3. Querying eg. a ServiceEntry that has host: foobar.api cannot be called from my machine but is resolvable from within a pod that runs the sidecar. Also adding .api to the include-suffixes in the kubeconfig did not help

    Expected behavior I can also speak to non-k8s entities for outgoing and intercepted traffic.

    Versions (please complete the following information):

    • Output of telepresence version 2.6.8
    • Operating system of workstation running telepresence commands Linux denis-vms-perf-1 5.11.0-1020-gcp #22~20.04.1-Ubuntu SMP Tue Sep 21 10:54:26 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
    • Kubernetes environment and Version [e.g. Minikube, bare metal, Google Kubernetes Engine] Kind cluster mesh with Gloo Mesh 2.0
    opened by wirtsi 0
  • On linux connection fails if `fs.inotify.max_user_instances` is set too low.

    On linux connection fails if `fs.inotify.max_user_instances` is set too low.

    Describe the bug

    telepresence connect can fail on linux if fs.inotify.max_user_instances is set too low (was 128 on my Ubuntu 22.04 LTS system). This is due to the file watcher in the config-reload goroutine.

    Logs:

    2022-08-07 01:30:28.0732 debug   Logging at this level "debug"
    2022-08-07 01:30:28.0734 debug   Listener opened
    2022-08-07 01:30:28.0734 info    ---
    2022-08-07 01:30:28.0734 info    Telepresence Connector v2.6.7 (api v3) starting...
    2022-08-07 01:30:28.0734 info    PID is 1494392
    2022-08-07 01:30:28.0734 info
    2022-08-07 01:30:28.0739 error   connector/config-reload : goroutine "/connector/config-reload" exited with error: too many open files
    2022-08-07 01:30:28.0739 info    connector:shutdown_logger : shutting down (gracefully)...
    2022-08-07 01:30:28.0740 info    connector/server-grpc : gRPC server started
    2022-08-07 01:30:28.0741 debug   connector/server-grpc : gRPC server ended
    2022-08-07 01:30:28.0741 debug   connector/server-grpc : goroutine "/connector/server-grpc" exited without error
    2022-08-07 01:30:28.0743 debug   connector/session : goroutine "/connector/session" exited without error
    2022-08-07 01:30:28.0743 info    connector/background-systema:shutdown_logger : shutting down (gracefully)...
    2022-08-07 01:30:28.0743 debug   connector/background-systema/refresh : goroutine "/connector/background-systema/refresh" exited without error
    2022-08-07 01:30:28.0744 debug   connector/background-systema/server-http : goroutine "/connector/background-systema/server-http" exited without error
    2022-08-07 01:30:28.0744 debug   connector/background-systema : goroutine "/connector/background-systema" exited without error
    2022-08-07 01:30:28.0744 debug   connector/background-metriton : goroutine "/connector/background-metriton" exited without error
    2022-08-07 01:30:28.0746 info    connector:shutdown_status :   final goroutine statuses:
    2022-08-07 01:30:28.0746 info    connector:shutdown_status :     /connector/background-metriton: exited without error
    2022-08-07 01:30:28.0746 info    connector:shutdown_status :     /connector/background-systema : exited without error
    2022-08-07 01:30:28.0746 info    connector:shutdown_status :     /connector/config-reload      : exited with error
    2022-08-07 01:30:28.0746 info    connector:shutdown_status :     /connector/server-grpc        : exited without error
    2022-08-07 01:30:28.0746 info    connector:shutdown_status :     /connector/session            : exited without error
    2022-08-07 01:30:28.0746 error   too many open files
    telepresence: error: too many open files
    

    To Reproduce

    1. Get linux system
    2. Reduce size of fs.inotify.max_user_instances - sudo sysctl -w fs.inotify.max_user_instances=16
    3. Run telepresence connect.

    Expected behavior I guess this isn't really a bug, but it would be nice to have a better error message or at least documentation, rather than reading the source code.

    opened by pj 1
  • Josecv/rel 2.7.0

    Josecv/rel 2.7.0

    Description

    A few sentences describing the overall goals of the pull request's commits.

    Checklist

    • [ ] I made sure to update ./CHANGELOG.md.
    • [ ] I made sure to add any docs changes required for my change (including release notes).
    • [ ] My change is adequately tested.
    • [ ] I updated DEVELOPING.md with any any special dev tricks I had to use to work on this code efficiently.
    • [ ] I updated TELEMETRY.md if I added, changed, or removed a metric name.
    • [ ] Once my PR is ready to have integration tests ran, I posted the PR in #telepresence-dev in the datawire-oss slack so that the "ok to test" label can be applied.
    opened by josecv 0
  • Replace Telepresence network stack with the stack from gVisor

    Replace Telepresence network stack with the stack from gVisor

    Description

    This PR replaces the Telepresence network stack with the stack from gVisor. It also contains several stability and performance improvements to the gRPC tunnel mechanism and the network boot process.

    I strongly recommend reviewing it one commit at a time.

    Checklist

    • [x] I made sure to update ./CHANGELOG.md.
    • [x] My change is adequately tested.
    opened by thallgren 0
Owner
Telepresence
Fast, local development of Kubernetes services
Telepresence
Automatically exposes the remote container's listening ports back to the local machine

Auto-portforward (apf) A handy tool to automatically set up proxies that expose the remote container's listening ports back to the local machine. Just

Ruoshan Huang 277 Jun 14, 2022
oniongrok forwards ports on the local host to remote Onion addresses as Tor hidden services and vice-versa

oniongrok Onion addresses for anything. oniongrok forwards ports on the local host to remote Onion addresses as Tor hidden services and vice-versa. Wh

Casey Marshall 256 Aug 11, 2022
Use qs-forward with QuickSocket to enable easy local development and testing!

qs-forward Use qs-forward with QuickSocket to enable easy local development and testing! Getting Started Want to jump in quick? Head over to the relea

QuickSocket 3 Jul 3, 2022
K8s_dns_chaos: enables inject DNS chaos in a Kubernetes cluster for Chaos Engineering

k8s_dns_chaos Name k8s_dns_chaos - enables inject DNS chaos in a Kubernetes cluster for Chaos Engineering. Description This plugin implements the Kube

邢凯 0 Dec 12, 2021
🧪 Run common networking tests against your site.

dstp dstp, run common networking tests against your site. Usage Usage: dstp [OPTIONS] [ARGS]

Yagiz Degirmenci 904 Aug 8, 2022
Scripts and other small tools developed against TCM systems

TCM Tools This repo contains scripts and small tools developed against TCM services that do not really have a home other places but we would like to m

Twin Cities Maker 0 Mar 22, 2022
A little tool to test IP addresses quickly against a geolocation and a reputation API

iptester A little tool to test IP addresses quickly against a geolocation and a

Axel Vanzaghi 2 May 19, 2022
Command-line tool and library for Windows remote command execution in Go

WinRM for Go Note: if you're looking for the winrm command-line tool, this has been splitted from this project and is available at winrm-cli This is a

Brice Figureau 371 Aug 10, 2022
Simple HTTP tunnel using SSH remote port forwarding

Simple HTTP tunnel using SSH remote port forwarding

Skye L. 21 Aug 8, 2022
EasyAgent is an infrastructure component, applied to manage the life-cycle of services on the remote host.

Easyagent English | 中文 介绍 easyagent是在袋鼠云内部广泛使用的基础架构组件,最佳应用场景包括ELK体系beats等数据采集器的管控和配置管理、数栈体系自动化部署等 基本原理 easyagent主要有sidecar和server两个组件,sidecar部署在主机端,si

袋鼠云 20 Jul 26, 2022
V3IO Frames ("Frames") is a Golang based remote data frames access (over gRPC or HTTP stream)

V3IO Frames ("Frames") is a multi-model open-source data-access library that provides a unified high-performance DataFrame API for working with different types of data sources (backends). The library was developed by Iguazio to simplify working with data in the Iguazio Data Science Platform ("the platform"), but it can be extended to support additional backend types.

null 23 Mar 3, 2022
A major platform Remote Access Terminal Tool based by Blockchain/P2P.

NGLite A major platform Remote Access Terminal Tool based by Blockchain/P2P. No public IP address required.More anonymity Example Detection Warning!!!

null 285 Aug 9, 2022
ZheTian Powerful remote load and execute ShellCode tool

ZheTian ZheTian Powerful remote load and execute ShellCode tool 免杀shellcode加载框架 命令详解 -u:从远程服务器加载base64混淆后的字节码。 -r:从本地文件内读。 -s:读取无修改的原始文件,只能从本地加载 -o:参数

一曲成殇 471 Aug 14, 2022
Run commands on remote hosts, inspecting key indicators to manage infrastructure

inspector This is a very basic ssh helper tool to manage a smaller (few 100s up to a few 1000s) fleet of servers. The main point of inspector is to pr

null 17 Mar 3, 2022
A remote access tool & CNC

⚠️ ⚠️ Disclaimer just use this with good intentions ⚠️ ⚠️ An useless rat (remote acces tool in develop) web client you want to use it? download pairat

pai 16 Jul 13, 2022
A remote access tool & CNC

⚠️ ⚠️ Disclaimer just use this with good intentions ⚠️ ⚠️ An useless rat (remote acces tool in develop) web client you want to use it? download pairat

pai 16 Jul 13, 2022
The rest api that can manage the iptables rules of the remote host

fiewall-api firewall api是基于firewalld来远程管理iptables规则的rest-api,无需部署agent Features 指定一个主机ip,让这个主机上的iptables增加一个规则 处理单个IP或CIDR范围(xx.xx.xx.xx/mask,mac,inte

seal 0 Mar 24, 2022
Application wirtten in GO to check if the port on the remote host is open

portcheck A simple Pod that get API POST request with port type and number with a target's IP address and checks if the destination port is available

Oren Oichman 0 Nov 26, 2021
Grab your files periodically from a remote FTP or SFTP server easily

About FTPGrab is a CLI application written in Go and delivered as a single executable (and a Docker image) to grab your files from a remote FTP or SFT

CrazyMax 397 Aug 10, 2022