Lightweight Kubernetes

Related tags

kubernetes k8s
Overview

K3s - Lightweight Kubernetes

Lightweight Kubernetes. Production ready, easy to install, half the memory, all in a binary less than 100 MB.

Great for:

  • Edge
  • IoT
  • CI
  • Development
  • ARM
  • Embedding k8s
  • Situations where a PhD in k8s clusterology is infeasible

What is this?

K3s is a fully conformant production-ready Kubernetes distribution with the following changes:

  1. It is packaged as a single binary.
  2. It adds support for sqlite3 as the default storage backend. Etcd3, MySQL, and Postgres are also supported.
  3. It wraps Kubernetes and other components in a single, simple launcher.
  4. It is secure by default with reasonable defaults for lightweight environments.
  5. It has minimal to no OS dependencies (just a sane kernel and cgroup mounts needed).
  6. It eliminates the need to expose a port on Kubernetes worker nodes for the kubelet API by exposing this API to the Kubernetes control plane nodes over a websocket tunnel.

K3s bundles the following technologies together into a single cohesive distribution:

These technologies can be disabled or swapped out for technolgoies of your choice.

Additionally, K3s simplifies Kubernetes operations by maintaining functionality for:

  • Managing the TLS certificates of Kubernetes componenents
  • Managing the connection between worker and server nodes
  • Auto-deploying Kubernetes resources from local manifests, in realtime as they are changed.
  • Managing an embedded etcd cluster (work in progress)

What's with the name?

We wanted an installation of Kubernetes that was half the size in terms of memory footprint. Kubernetes is a 10 letter word stylized as k8s. So something half as big as Kubernetes would be a 5 letter word stylized as K3s. There is no long form of K3s and no official pronunciation.

Is this a fork?

No, it's a distribution. A fork implies continued divergence from the original. This is not K3s's goal or practice. K3s explicitly intends to not change any core Kubernetes functionality. We seek to remain as close to upstream Kubernetes as possible. We do maintain a small set of patches (well under 1000 lines) important to K3s's usecase and deployment model. We maintain patches for other components as well. When possible, we contribute these changes back to the upstream projects, for example with SELinux support in containerd. This is a common practice amongst software distributions.

K3s is a distribution because it packages additional components and services necessary for a fully functional cluster that go beyond vanilla Kubernetes. These are opinionated choices on technologies for components like ingress, storage class, network policy, service load balancer, and even container runtime. These choices and technologies are touched on in more detail in the What is this? section.

How is this lightweight or smaller than upstream Kubernetes?

There are two major ways that K3s is lighter weight than upstream Kubernetes:

  1. The memory footprint to run it is smaller
  2. The binary, which contains all the non-containerized components needed to run a cluster, is smaller

The memory footprint is reduced primarily by running many components inside of a single process. This eliminates significant overhead that would otherwise be duplicated for each component.

The binary is smaller by removing third-party storage drivers and cloud providers, which is explained in more detail below.

What have you removed from upstream Kubernetes?

This is a common point of confusion because it has changed over time. Early versions of K3s had much more removed than current version. K3s currently removes two things:

  1. In-tree storage drivers
  2. In-tree cloud provider

Both of these have out-of-tree alternatives in the form of CSI and CCM, which work in K3s and which upstream is moving towards.

We remove these to achieve a smaller binary size. They can be removed while remaining conformant because neither affect core Kubernetes functionality. They are also dependent on third-party cloud or data center technologies/services, which may not be available in many of K3s's usecases.

What's next?

Check out our roadmap to see what we have planned moving forward.

Release cadence

K3s maintains pace with upstream Kubernetes releases. Our goal is to release patch releases on the same day as upstream and minor releases within a few days.

Our release versioning reflects the version of upstream Kubernetes that is being released. For example, the K3s release v1.18.6+k3s1 maps to the v1.18.6 Kubernetes release. We add a postfix in the form of +k3s<number> to allow us to make additional releases using the same version of upstream Kubernetes, while remaining semver compliant. For example, if we discovered a high severity bug in v1.18.6+k3s1 and needed to release an immediate fix for it, we would release v1.18.6+k3s2.

Documentation

Please see the official docs site for complete documentation.

Quick-Start - Install Script

The install.sh script provides a convenient way to download K3s and add a service to systemd or openrc.

To install k3s as a service just run:

curl -sfL https://get.k3s.io | sh -

A kubeconfig file is written to /etc/rancher/k3s/k3s.yaml and the service is automatically started or restarted. The install script will install K3s and additional utilities, such as kubectl, crictl, k3s-killall.sh, and k3s-uninstall.sh, for example:

sudo kubectl get nodes

K3S_TOKEN is created at /var/lib/rancher/k3s/server/node-token on your server. To install on worker nodes we should pass K3S_URL along with K3S_TOKEN or K3S_CLUSTER_SECRET environment variables, for example:

curl -sfL https://get.k3s.io | K3S_URL=https://myserver:6443 K3S_TOKEN=XXX sh -

Manual Download

  1. Download k3s from latest release, x86_64, armhf, and arm64 are supported.
  2. Run server.
sudo k3s server &
# Kubeconfig is written to /etc/rancher/k3s/k3s.yaml
sudo k3s kubectl get nodes

# On a different node run the below. NODE_TOKEN comes from
# /var/lib/rancher/k3s/server/node-token on your server
sudo k3s agent --server https://myserver:6443 --token ${NODE_TOKEN}

Contributing

Please check out our contributing guide if you're interested in contributing to K3s.

Security

Security issues in K3s can be reported by sending an email to [email protected]. Please do not file issues about security issues.

Issues
  • CPU and memory usage of k3s

    CPU and memory usage of k3s

    Environmental Info: K3s Version: k3s version v1.18.8+k3s1 (6b595318)
    Running on CentOS 7.8

    Node(s) CPU architecture, OS, and Version: Linux k3s 3.10.0-1127.19.1.el7.x86_64 #1 SMP Tue Aug 25 17:23:54 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
    AMD GX-412TC SOC with 2GB RAM

    Cluster Configuration: Single node installation

    Describe the bug:

    When deploying the latest stable k3s on a single node, the CPU and memory usage may look important. I understand that Kubernetes isn't lightweight by definition, but the k3s is really interessing for creating/deploying appliances. On small (embedded) systems, the default CPU and memory usage is important (I'm not speaking here for modern servers). Is-there a way to optimize these ressources usage or at least to understand the k3s usage of ressources when nothing is deployed?

    Steps To Reproduce:

    • Installed K3s:
      curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server --no-deploy traefik" sh

    Expected behavior:

    Maybe less CPU and memory usage when nothing is deployed and running

    Actual behavior:

    500MB of memory used and 5% of CPU usage on each core (4 cores CPU) when idle

    Additional context / logs:

    opened by sraillard 72
  • Getting Real Client IP with k3s

    Getting Real Client IP with k3s

    Is your feature request related to a problem? Please describe. I am unable to obtain Real Client IP when using k3s and Traefik v2.2. I always get the cluster IP.

    Kernel version
    4.4.0-174-generic
    OS Image
    Ubuntu 16.04.6 LTS
    Container runtime version
    containerd://1.3.0-k3s.4
    kubelet version
    v1.16.3-k3s.2
    kube-proxy version
    v1.16.3-k3s.2
    Operating system
    linux
    Architecture
    amd64
    
    
    Images
    traefik:2.2.0
    

    Describe the solution you'd like I would like to obtain the client IP.

    Describe alternatives you've considered I already set externalTrafficPolicy: Local in Traefik's Service. Additional context Issue can be reproduced by deploying containous/whoami image in cluster Expected Response

    Hostname: a19d325823bb
    IP: 127.0.0.1
    IP: 10.0.0.147
    IP: 172.18.0.4
    RemoteAddr: 10.0.0.144:56246
    GET / HTTP/1.1
    Host: whoami.civo.com
    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36
    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
    Accept-Encoding: gzip, deflate, br
    Accept-Language: en-US,en;q=0.9
    Sec-Fetch-Dest: document
    Sec-Fetch-Mode: navigate
    Sec-Fetch-Site: none
    Upgrade-Insecure-Requests: 1
    X-Apache-Ip: 102.69.228.66
    X-Forwarded-For: 102.69.228.66, 102.69.228.66, 172.18.0.1
    X-Forwarded-Host: whoami.civo.com
    X-Forwarded-Port: 443
    X-Forwarded-Proto: https
    X-Forwarded-Server: bc3b51f28353
    X-Real-Ip: 102.69.228.66
    

    Current Response

    Hostname: whoami-76d6dfb846-jltlm
    IP: 127.0.0.1
    IP: ::1
    IP: 192.168.0.33
    IP: fe80::7863:88ff:fe45:2ad5
    RemoteAddr: 192.168.1.4:36146
    GET / HTTP/1.1
    Host: who.civo.com
    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36
    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
    Accept-Encoding: gzip, deflate, br
    Accept-Language: en-US,en;q=0.9
    Sec-Fetch-Dest: document
    Sec-Fetch-Mode: navigate
    Sec-Fetch-Site: none
    Upgrade-Insecure-Requests: 1
    X-Forwarded-For: 192.168.0.5
    X-Forwarded-Host: who.civo.com
    X-Forwarded-Port: 443
    X-Forwarded-Proto: https
    X-Forwarded-Server: traefik-8477c7d8f-fbhdg
    X-Real-Ip: 192.168.0.5
    

    Service LoadBalancer Logs

    + trap exit TERM INT
    /usr/bin/entry: line 6: can't create /proc/sys/net/ipv4/ip_forward: Read-only file system
    + echo 1
    + true
    + cat /proc/sys/net/ipv4/ip_forward
    + '[' 1 '!=' 1 ]
    + iptables -t nat -I PREROUTING '!' -s 192.168.183.229/32 -p TCP --dport 8080 -j DNAT --to 192.168.183.229:8080
    + iptables -t nat -I POSTROUTING -d 192.168.183.229/32 -p TCP -j MASQUERADE
    + '[' '!' -e /pause ]
    + mkfifo /pause
    

    Related https://github.com/rancher/k3s/pull/955 Related Discussion https://github.com/rancher/k3s/issues/679#issuecomment-516367437

    @erikwilson @btashton

    kind/enhancement kind/question 
    opened by jawabuu 71
  • Cadvisor not reporting Container/Image metadata

    Cadvisor not reporting Container/Image metadata

    Describe the bug When making the call to retrieve metrics via Cadvisor, the Container and Images values are empty in all values.

    container_tasks_state{container="",container_name="",id="/system.slice/lxd.socket",image="",name="",namespace="",pod="",pod_name="",state="running"} 0 1557525150119
    

    To Reproduce Install k3s via multipass https://medium.com/@zhimin.wen/running-k3s-with-multipass-on-mac-fbd559966f7c

    kubectl get --raw /api/v1/nodes/k3s/proxy/metrics/cadvisor
    

    Expected behavior container and image values should be populated

    Additional context Wondering if it might be related to https://github.com/rancher/k3s/issues/213

    kind/bug 
    opened by cfchad 62
  • Formally add support for CentOS 7

    Formally add support for CentOS 7

    We need to expand our testing and identify any issues that prevent us from formally supporting CentOS. Keep in mind K3s is expected to work fine on CentOS 7. This issue is to track the testing effort required to formally support and certify the operating system (See https://rancher.com/docs/k3s/latest/en/installation/node-requirements/#operating-systems )

    Currently there are existing issues with the os/centos label, but take care to note that these issues are not all necessarily caused just by utilizing CentOS. As such, it makes sense to review those GitHub issues, but we need to execute some testing and identify any other issues. As needed, we'll need to resolve these issues so we may fully support CentOS.

    SELinux support is also needed, which is tracked separately here: https://github.com/rancher/k3s/issues/1372

    gz#9311

    gz#9743

    internal kind/enhancement os/centos 
    opened by davidnuzik 58
  • k3s causes a high load average

    k3s causes a high load average

    Describe the bug I'm not sure if it's a bug, but I think it's not an expected behaviour. When running k3s on any computer, it causes a very high load average. To have a concrete example, I'll explain the situation of my raspberry pi3 node.

    When running k3s, I have a load average usage of:

    load average: 2.69, 1.52, 1.79
    

    Without running it, but still having the containers up, I have a load average of:

    load average: 0.24, 1.01, 1.72
    

    To Reproduce I just run it without any special arguments, just how is installed by the sh installer.

    Expected behavior The load average should be under 1.

    status/more-info 
    opened by drym3r 56
  • Traefik 2.0 integration

    Traefik 2.0 integration

    Is your feature request related to a problem? Please describe. The feature tls-passthrough is missing. Es. Installing argocd on the cluster is difficult due the missing of this feature.

    Describe the solution you'd like Substitute the actual version < 2.0 with the actual (that reach the GA with version 2.0)

    Describe alternatives you've considered Describe a reproducible way for remove the actual version in favor of the most updated version.

    Additional context The version of Traefik 2.0 seems most kubernetes friendly so, this seems to me a very natural step to do!

    kind/feature priority/important-soon 
    opened by Zikoel 44
  • Job for k3s.service failed because the control process exited with error code

    Job for k3s.service failed because the control process exited with error code

    Hello Team,

    Trying to run k3s cluster on raspberrypi using official doc but causing this issue. ● k3s.service - Lightweight Kubernetes Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Thu 2019-06-20 12:18:07 UTC; 4min 13s ago Docs: https://k3s.io Process: 1722 ExecStart=/usr/local/bin/k3s server (code=exited, status=1/FAILURE) Process: 1719 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS) Process: 1716 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS) Main PID: 1722 (code=exited, status=1/FAILURE) CPU: 2.150s

    Jun 20 12:18:06 master systemd[1]: k3s.service: Unit entered failed state. Jun 20 12:18:06 master systemd[1]: k3s.service: Failed with result 'exit-code'. Jun 20 12:18:07 master systemd[1]: k3s.service: Service hold-off time over, scheduling restart. Jun 20 12:18:07 master systemd[1]: Stopped Lightweight Kubernetes. Jun 20 12:18:07 master systemd[1]: k3s.service: Start request repeated too quickly. Jun 20 12:18:07 master systemd[1]: Failed to start Lightweight Kubernetes. Jun 20 12:18:07 master systemd[1]: k3s.service: Unit entered failed state. Jun 20 12:18:07 master systemd[1]: k3s.service: Failed with result 'exit-code'.

    status/more-info 
    opened by Aliabbask08 41
  • Pods not deleted after its Rancher App is deleted

    Pods not deleted after its Rancher App is deleted

    Environmental Info: K3s Version:

    k3s version v1.18.2+k3s1 (698e444a)

    Node(s) CPU architecture, OS, and Version:

    4.15.0-101-generic Ubuntu x86_64

    Cluster Configuration:

    1 master imported within Rancher.

    Describe the bug:

    Currently, I have a Rancher deployment with 2 clusters. 1 being a generic RKE cluster and the other one being a k3s one.

    When deleting a Rancher App on the first one, all goes fine with deploying and deleting Rancher Apps.

    However, when deploying a Rancher App and deleting it afterwards on the k3s cluster, the App's pods don't get deleted and still run under the radar, not detected by Rancher. Even worse, they obviously still suck up all the resources that I attempted to gain when deleting the App. So of course the cluster fills up over time and cannot accept new deployments, because all other ones are still running behind the scenes.

    Original Bug Description


    I just discovered that this issue is much worse than originally assumed. To delete a pod from a deployment, you have to delete the deployment, or else the pod resurrects, automatically. However, in this situation there is no deployment to be deleted. The pods are still left. If you now try to delete the pods, they revive, again. So there is no obvious way to delete the pods, since you'd need to delete their deployments which are already gone, though.


    For testing purposes, I set up a chart and deleted its deployments with

    kubectl delete --all deployments --namespace=test1 --grace-period=0 --force
    

    The pods remained and hat to be deleted with below workaround.

    Steps To Reproduce:

    1. Use Rancher and import the k3s cluster.
    2. Deploy a Rancher App through the Rancher WebUI.
    3. Delete the Rancher App.
    4. kubectl get pods --all-namespaces
    5. See that Pods from the deleted Rancher App are still in Running state and remain like that.

    Expected behavior:

    Pods of the deleted Rancher App also get deleted.

    Actual behavior:

    Pods of the deleted Rancher App do not get deleted.

    Additional context / logs:

    Cluster was imported quite a while ago. This issue wasn't discovered too early, because there are more than enough resources on the server and it is not used that frequently.

    Workaround

    Currently, the only way to delete the orphaned pods is this:

    kubectl delete all --all --namespace=failed-namespace --grace-period=0 --force
    

    This command has to be run twice per namespace.

    kind/bug 
    opened by theAkito 41
  • Insecure Registry Support

    Insecure Registry Support

    Is your feature request related to a problem? Please describe. As a user, I cannot access insecure registries from my k3s instance.

    Describe the solution you'd like Add --insecure-registry to server and/or node.

    Describe alternatives you've considered I'm specifically trying to address this with the proxy team, but it should be added regardless.

    Additional context Behind a corporate proxy, it might use a different cert. It is what it is...

    kind/enhancement 
    opened by nnordrum 39
  • cluster networking is broken?

    cluster networking is broken?

    helm install job never succeed, it seem that it is not possible to reach dns server.

    alpine:/home/alpine/k3s/dist/artifacts# ./k3s kubectl  get all -n kube-system 
    NAME                             READY   STATUS             RESTARTS   AGE
    pod/coredns-7748f7f6df-tp7fq     1/1     Running            1          104m
    pod/helm-install-traefik-g5rmk   0/1     CrashLoopBackOff   21         104m
    
    NAME               TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
    service/kube-dns   ClusterIP   10.43.0.10   <none>        53/UDP,53/TCP,9153/TCP   104m
    
    NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
    deployment.apps/coredns   1/1     1            1           104m
    
    NAME                                 DESIRED   CURRENT   READY   AGE
    replicaset.apps/coredns-7748f7f6df   1         1         1       104m
    
    NAME                             COMPLETIONS   DURATION   AGE
    job.batch/helm-install-traefik   0/1           104m       104m
    
    ./k3s kubectl   -n kube-system logs -f pod/helm-install-traefik-g5rmk
    + export HELM_HOST=127.0.0.1:44134+ 
    tiller --listen=127.0.0.1:44134 --storage=secret
    + HELM_HOST=127.0.0.1:44134
    + helm init --client-only
    [main] 2019/02/08 20:48:52 Starting Tiller v2.12.3 (tls=false)
    [main] 2019/02/08 20:48:52 GRPC listening on 127.0.0.1:44134
    [main] 2019/02/08 20:48:52 Probes listening on :44135
    [main] 2019/02/08 20:48:52 Storage driver is Secret
    [main] 2019/02/08 20:48:52 Max history per release is 0
    Creating /root/.helm 
    Creating /root/.helm/repository 
    Creating /root/.helm/repository/cache 
    Creating /root/.helm/repository/local 
    Creating /root/.helm/plugins 
    Creating /root/.helm/starters 
    Creating /root/.helm/cache/archive 
    Creating /root/.helm/repository/repositories.yaml 
    Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com 
    Error: Looks like "https://kubernetes-charts.storage.googleapis.com" is not a valid chart repository or cannot be reached: Get https://kubernetes-charts.storage.googleapis.com/index.yaml: dial tcp: lookup kubernetes-charts.storage.googleapis.com on 10.43.0.10:53: read udp 10.42.0.4:39333->10.43.0.10:53: i/o timeout
    

    Verify by running a busy box

    alpine:/home/alpine/k3s/dist/artifacts# ./k83s kubectl run -i --tty busybox --image=busybox --restart=Never -- sh
    ash: ./k83s: not found
    alpine:/home/alpine/k3s/dist/artifacts# ./k3s kubectl run -i --tty busybox --image=busybox --restart=Never -- sh
    If you don't see a command prompt, try pressing enter.
    / # 
    / # ping 10.43.0.10
    PING 10.43.0.10 (10.43.0.10): 56 data bytes
    ^C
    --- 10.43.0.10 ping statistics ---
    7 packets transmitted, 0 packets received, 100% packet loss
    / # ip addr
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
           valid_lft forever preferred_lft forever
        inet6 ::1/128 scope host 
           valid_lft forever preferred_lft forever
    3: [email protected]: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue 
        link/ether 32:03:33:52:8c:19 brd ff:ff:ff:ff:ff:ff
        inet 10.42.0.6/24 brd 10.42.0.255 scope global eth0
           valid_lft forever preferred_lft forever
        inet6 fe80::3003:33ff:fe52:8c19/64 scope link 
           valid_lft forever preferred_lft forever
    / # ping 10.43.0.10
    PING 10.43.0.10 (10.43.0.10): 56 data bytes
    ^C
    --- 10.43.0.10 ping statistics ---
    6 packets transmitted, 0 packets received, 100% packet loss
    / # ping 10.42.0.6
    PING 10.42.0.6 (10.42.0.6): 56 data bytes
    64 bytes from 10.42.0.6: seq=0 ttl=64 time=0.109 ms
    64 bytes from 10.42.0.6: seq=1 ttl=64 time=0.108 ms
    64 bytes from 10.42.0.6: seq=2 ttl=64 time=0.106 ms
    ^C
    --- 10.42.0.6 ping statistics ---
    3 packets transmitted, 3 packets received, 0% packet loss
    round-trip min/avg/max = 0.106/0.107/0.109 ms
    
    help wanted 
    opened by liyimeng 39
  • Streaming server stopped unexpectedly

    Streaming server stopped unexpectedly

    Environmental Info: K3s Version: v1.21.3+k3s1

    Node(s) CPU architecture, OS, and Version: Linux ctos 5.9.16-1-MANJARO #1 SMP PREEMPT Mon Dec 21 22:00:46 UTC 2020 x86_64 GNU/Linux, 16GB, i5 10thGen,

    Cluster Configuration: Single Node with Docker runtime

    Describe the bug: Want to run k3s with docker runtime 1st try: curl -sfL https://get.k3s.io | sh -s - --docker

    + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
    Failed to get unit file state for nm-cloud-setup.service: No such file or directory
    time="2021-07-26T18:24:31.781674773+05:30" level=info msg="Starting k3s v1.21.3+k3s1 (1d1f220f)"
    time="2021-07-26T18:24:31.783699262+05:30" level=info msg="Cluster bootstrap already complete"
    time="2021-07-26T18:24:31.796546141+05:30" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
    time="2021-07-26T18:24:31.796590717+05:30" level=info msg="Configuring database table schema and indexes, this may take a moment..."
    time="2021-07-26T18:24:31.796707207+05:30" level=info msg="Database tables and indexes are up to date"
    time="2021-07-26T18:24:31.797840342+05:30" level=info msg="Kine listening on unix://kine.sock"
    time="2021-07-26T18:24:31.797997181+05:30" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction --etcd-servers=unix://kine.sock --insecure-port=0 --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
    Flag --insecure-port has been deprecated, This flag has no effect now and will be removed in v1.24.
    I0726 18:24:31.799809  444609 server.go:656] external host was not specified, using 10.0.8.87
    I0726 18:24:31.800003  444609 server.go:195] Version: v1.21.3+k3s1
    I0726 18:24:31.803454  444609 shared_informer.go:240] Waiting for caches to sync for node_authorizer
    I0726 18:24:31.804490  444609 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
    I0726 18:24:31.804513  444609 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
    I0726 18:24:31.805735  444609 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
    I0726 18:24:31.805750  444609 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
    I0726 18:24:31.829180  444609 instance.go:283] Using reconciler: lease
    I0726 18:24:31.866028  444609 rest.go:130] the default service ipfamily for this cluster is: IPv4
    W0726 18:24:32.209699  444609 genericapiserver.go:425] Skipping API node.k8s.io/v1alpha1 because it has no resources.
    W0726 18:24:32.221974  444609 genericapiserver.go:425] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
    W0726 18:24:32.227045  444609 genericapiserver.go:425] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
    W0726 18:24:32.233763  444609 genericapiserver.go:425] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
    W0726 18:24:32.237310  444609 genericapiserver.go:425] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources.
    W0726 18:24:32.243942  444609 genericapiserver.go:425] Skipping API apps/v1beta2 because it has no resources.
    W0726 18:24:32.243967  444609 genericapiserver.go:425] Skipping API apps/v1beta1 because it has no resources.
    I0726 18:24:32.265841  444609 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
    I0726 18:24:32.265867  444609 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
    time="2021-07-26T18:24:32.279983217+05:30" level=info msg="Running kube-scheduler --address=127.0.0.1 --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --leader-elect=false --port=10251 --profiling=false --secure-port=0"
    time="2021-07-26T18:24:32.280032517+05:30" level=info msg="Waiting for API server to become available"
    time="2021-07-26T18:24:32.280542302+05:30" level=info msg="Running kube-controller-manager --address=127.0.0.1 --allocate-node-cidrs=true --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --leader-elect=false --port=10252 --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=0 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --use-service-account-credentials=true"
    time="2021-07-26T18:24:32.281109525+05:30" level=info msg="Running cloud-controller-manager --allocate-node-cidrs=true --bind-address=127.0.0.1 --cloud-provider=k3s --cluster-cidr=10.42.0.0/16 --configure-cloud-routes=false --kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --leader-elect=false --node-status-update-frequency=1m0s --port=0 --profiling=false"
    time="2021-07-26T18:24:32.282438094+05:30" level=info msg="Node token is available at /var/lib/rancher/k3s/server/token"
    time="2021-07-26T18:24:32.282471747+05:30" level=info msg="To join node to cluster: k3s agent -s https://10.0.8.87:6443 -t ${NODE_TOKEN}"
    time="2021-07-26T18:24:32.283503789+05:30" level=info msg="Wrote kubeconfig /etc/rancher/k3s/k3s.yaml"
    time="2021-07-26T18:24:32.283534390+05:30" level=info msg="Run: k3s kubectl"
    time="2021-07-26T18:24:32.339345596+05:30" level=info msg="Cluster-Http-Server 2021/07/26 18:24:32 http: TLS handshake error from 127.0.0.1:52210: remote error: tls: bad certificate"
    time="2021-07-26T18:24:32.343567560+05:30" level=info msg="Cluster-Http-Server 2021/07/26 18:24:32 http: TLS handshake error from 127.0.0.1:52216: remote error: tls: bad certificate"
    time="2021-07-26T18:24:32.351620225+05:30" level=info msg="certificate CN=ctos signed by [email protected]: notBefore=2021-07-26 12:52:30 +0000 UTC notAfter=2022-07-26 12:54:32 +0000 UTC"
    time="2021-07-26T18:24:32.354491937+05:30" level=info msg="certificate CN=system:node:ctos,O=system:nodes signed by [email protected]: notBefore=2021-07-26 12:52:30 +0000 UTC notAfter=2022-07-26 12:54:32 +0000 UTC"
    time="2021-07-26T18:24:32.358118198+05:30" level=info msg="Module overlay was already loaded"
    time="2021-07-26T18:24:32.358143447+05:30" level=info msg="Module nf_conntrack was already loaded"
    time="2021-07-26T18:24:32.358151771+05:30" level=info msg="Module br_netfilter was already loaded"
    time="2021-07-26T18:24:32.358160063+05:30" level=info msg="Module iptable_nat was already loaded"
    time="2021-07-26T18:24:32.365779524+05:30" level=info msg="Connecting to proxy" url="wss://127.0.0.1:6443/v1-k3s/connect"
    time="2021-07-26T18:24:32.367500734+05:30" level=info msg="Handling backend connection request [ctos]"
    time="2021-07-26T18:24:32.368359468+05:30" level=info msg="Running kubelet --address=0.0.0.0 --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=cgroupfs --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --cni-bin-dir=/var/lib/rancher/k3s/data/9df574741d2573cbbe6616e8624488b36b3340d077bc50da7cb167f1b08a64d1/bin --cni-conf-dir=/var/lib/rancher/k3s/agent/etc/cni/net.d --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ctos --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --network-plugin=cni --node-labels= --pod-infra-container-image=rancher/pause:3.1 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --read-only-port=0 --resolv-conf=/etc/resolv.conf --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key"
    time="2021-07-26T18:24:32.368867068+05:30" level=info msg="Running kube-proxy --cluster-cidr=10.42.0.0/16 --conntrack-max-per-core=0 --conntrack-tcp-timeout-close-wait=0s --conntrack-tcp-timeout-established=0s --healthz-bind-address=127.0.0.1 --hostname-override=ctos --kubeconfig=/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig --proxy-mode=iptables"
    Flag --cloud-provider has been deprecated, will be removed in 1.23, in favor of removing cloud provider code from Kubelet.
    Flag --cni-bin-dir has been deprecated, will be removed along with dockershim.
    Flag --cni-conf-dir has been deprecated, will be removed along with dockershim.
    Flag --network-plugin has been deprecated, will be removed along with dockershim.
    W0726 18:24:32.369161  444609 server.go:220] WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP.
    E0726 18:24:32.383711  444609 node.go:161] Failed to retrieve node info: nodes "ctos" is forbidden: User "system:kube-proxy" cannot get resource "nodes" in API group "" at the cluster scope
    I0726 18:24:32.393649  444609 server.go:436] "Kubelet version" kubeletVersion="v1.21.3+k3s1"
    I0726 18:24:32.423114  444609 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/rancher/k3s/agent/client-ca.crt
    W0726 18:24:32.423122  444609 manager.go:159] Cannot detect current cgroup on cgroup v2
    I0726 18:24:32.501501  444609 server.go:660] "--cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /"
    I0726 18:24:32.501713  444609 container_manager_linux.go:291] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
    I0726 18:24:32.501790  444609 container_manager_linux.go:296] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalTopologyManagerScope:container ExperimentalCPUManagerReconcilePeriod:10s ExperimentalMemoryManagerPolicy:None ExperimentalMemoryManagerReservedMemory:[] ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none Rootless:false}
    I0726 18:24:32.501830  444609 topology_manager.go:120] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container"
    I0726 18:24:32.501844  444609 container_manager_linux.go:327] "Initializing Topology Manager" policy="none" scope="container"
    I0726 18:24:32.501855  444609 container_manager_linux.go:332] "Creating device plugin manager" devicePluginEnabled=true
    I0726 18:24:32.501947  444609 kubelet.go:307] "Using dockershim is deprecated, please consider using a full-fledged CRI implementation"
    I0726 18:24:32.501989  444609 client.go:78] "Connecting to docker on the dockerEndpoint" endpoint="unix:///var/run/docker.sock"
    I0726 18:24:32.502013  444609 client.go:97] "Start docker client with request timeout" timeout="2m0s"
    I0726 18:24:32.509025  444609 docker_service.go:566] "Hairpin mode is set but kubenet is not enabled, falling back to HairpinVeth" hairpinMode=promiscuous-bridge
    I0726 18:24:32.509049  444609 docker_service.go:242] "Hairpin mode is set" hairpinMode=hairpin-veth
    I0726 18:24:32.515251  444609 docker_service.go:257] "Docker cri networking managed by the network plugin" networkPluginName="cni"
    I0726 18:24:32.521560  444609 docker_service.go:264] "Docker Info" dockerInfo=&{ID:G5RL:4X7T:W5YM:4BWC:LGDL:VYV7:O7QU:A425:2U33:MLLS:NFGR:T2IO Containers:22 ContainersRunning:1 ContainersPaused:0 ContainersStopped:21 Images:229 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Native Overlay Diff false] [userxattr false]] SystemStatus:[] Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:[] Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:false CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6tables:true Debug:false NFd:32 OomKillDisable:false NGoroutines:45 SystemTime:2021-07-26T18:24:32.515685567+05:30 LoggingDriver:json-file CgroupDriver:systemd CgroupVersion:2 NEventsListener:0 KernelVersion:5.9.16-1-MANJARO OperatingSystem:Manjaro Linux OSVersion: OSType:linux Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:0xc003715c00 NCPU:8 MemTotal:16432685056 GenericResources:[] DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:ctos Labels:[] ExperimentalBuild:false ServerVersion:20.10.7 ClusterStore: ClusterAdvertise: Runtimes:map[io.containerd.runc.v2:{Path:runc Args:[] Shim:<nil>} io.containerd.runtime.v1.linux:{Path:runc Args:[] Shim:<nil>} runc:{Path:runc Args:[] Shim:<nil>}] DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:[] Nodes:0 Managers:0 Cluster:<nil> Warnings:[]} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:36cc874494a56a253cd181a1a685b44b58a2e34a.m Expected:36cc874494a56a253cd181a1a685b44b58a2e34a.m} RuncCommit:{ID:b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7 Expected:b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=apparmor name=seccomp,profile=default name=cgroupns] ProductLicense: DefaultAddressPools:[] Warnings:[]}
    E0726 18:24:32.521597  444609 server.go:288] "Failed to run kubelet" err="failed to run Kubelet: misconfiguration: kubelet cgroup driver: \"cgroupfs\" is different from docker cgroup driver: \"systemd\""
    k3s.service: Main process exited, code=exited, status=1/FAILURE
    
    

    2nd try: add kubelet flag curl -sfL https://get.k3s.io | sh -s - --kubelet-arg 'cgroup-driver=systemd' --docker

    + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
    Failed to get unit file state for nm-cloud-setup.service: No such file or directory
    time="2021-07-26T19:43:45.773788090+05:30" level=info msg="Starting k3s v1.21.3+k3s1 (1d1f220f)"
    time="2021-07-26T19:43:45.773936226+05:30" level=info msg="Cluster bootstrap already complete"
    time="2021-07-26T19:43:45.784963404+05:30" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
    time="2021-07-26T19:43:45.784988646+05:30" level=info msg="Configuring database table schema and indexes, this may take a moment..."
    time="2021-07-26T19:43:45.785072907+05:30" level=info msg="Database tables and indexes are up to date"
    time="2021-07-26T19:43:45.786106017+05:30" level=info msg="Kine listening on unix://kine.sock"
    time="2021-07-26T19:43:45.786303643+05:30" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction --etcd-servers=unix://kine.sock --insecure-port=0 --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
    Flag --insecure-port has been deprecated, This flag has no effect now and will be removed in v1.24.
    I0726 19:43:45.787752  602560 server.go:656] external host was not specified, using 10.0.8.87
    I0726 19:43:45.787933  602560 server.go:195] Version: v1.21.3+k3s1
    I0726 19:43:45.792464  602560 shared_informer.go:240] Waiting for caches to sync for node_authorizer
    I0726 19:43:45.792966  602560 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
    I0726 19:43:45.792980  602560 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
    I0726 19:43:45.793822  602560 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
    I0726 19:43:45.793832  602560 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
    I0726 19:43:45.806814  602560 instance.go:283] Using reconciler: lease
    I0726 19:43:45.860030  602560 rest.go:130] the default service ipfamily for this cluster is: IPv4
    W0726 19:43:46.128420  602560 genericapiserver.go:425] Skipping API node.k8s.io/v1alpha1 because it has no resources.
    W0726 19:43:46.138016  602560 genericapiserver.go:425] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
    W0726 19:43:46.141459  602560 genericapiserver.go:425] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
    W0726 19:43:46.147181  602560 genericapiserver.go:425] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
    W0726 19:43:46.150571  602560 genericapiserver.go:425] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources.
    W0726 19:43:46.156566  602560 genericapiserver.go:425] Skipping API apps/v1beta2 because it has no resources.
    W0726 19:43:46.156587  602560 genericapiserver.go:425] Skipping API apps/v1beta1 because it has no resources.
    I0726 19:43:46.167785  602560 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
    I0726 19:43:46.167803  602560 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
    time="2021-07-26T19:43:46.176753451+05:30" level=info msg="Running kube-scheduler --address=127.0.0.1 --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --leader-elect=false --port=10251 --profiling=false --secure-port=0"
    time="2021-07-26T19:43:46.176808565+05:30" level=info msg="Waiting for API server to become available"
    time="2021-07-26T19:43:46.177179338+05:30" level=info msg="Running kube-controller-manager --address=127.0.0.1 --allocate-node-cidrs=true --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --leader-elect=false --port=10252 --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=0 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --use-service-account-credentials=true"
    time="2021-07-26T19:43:46.177559790+05:30" level=info msg="Running cloud-controller-manager --allocate-node-cidrs=true --bind-address=127.0.0.1 --cloud-provider=k3s --cluster-cidr=10.42.0.0/16 --configure-cloud-routes=false --kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --leader-elect=false --node-status-update-frequency=1m0s --port=0 --profiling=false"
    time="2021-07-26T19:43:46.178555112+05:30" level=info msg="Node token is available at /var/lib/rancher/k3s/server/token"
    time="2021-07-26T19:43:46.178581003+05:30" level=info msg="To join node to cluster: k3s agent -s https://10.0.8.87:6443 -t ${NODE_TOKEN}"
    time="2021-07-26T19:43:46.179294860+05:30" level=info msg="Wrote kubeconfig /etc/rancher/k3s/k3s.yaml"
    time="2021-07-26T19:43:46.179319588+05:30" level=info msg="Run: k3s kubectl"
    time="2021-07-26T19:43:46.219669890+05:30" level=info msg="Cluster-Http-Server 2021/07/26 19:43:46 http: TLS handshake error from 127.0.0.1:53082: remote error: tls: bad certificate"
    time="2021-07-26T19:43:46.222848739+05:30" level=info msg="Cluster-Http-Server 2021/07/26 19:43:46 http: TLS handshake error from 127.0.0.1:53088: remote error: tls: bad certificate"
    time="2021-07-26T19:43:46.230245755+05:30" level=info msg="certificate CN=ctos signed by [email protected]: notBefore=2021-07-26 12:52:30 +0000 UTC notAfter=2022-07-26 14:13:46 +0000 UTC"
    time="2021-07-26T19:43:46.232510407+05:30" level=info msg="certificate CN=system:node:ctos,O=system:nodes signed by [email protected]: notBefore=2021-07-26 12:52:30 +0000 UTC notAfter=2022-07-26 14:13:46 +0000 UTC"
    time="2021-07-26T19:43:46.236051185+05:30" level=info msg="Module overlay was already loaded"
    time="2021-07-26T19:43:46.236072017+05:30" level=info msg="Module nf_conntrack was already loaded"
    time="2021-07-26T19:43:46.236079584+05:30" level=info msg="Module br_netfilter was already loaded"
    time="2021-07-26T19:43:46.236087911+05:30" level=info msg="Module iptable_nat was already loaded"
    time="2021-07-26T19:43:46.242127052+05:30" level=info msg="Connecting to proxy" url="wss://127.0.0.1:6443/v1-k3s/connect"
    time="2021-07-26T19:43:46.243832486+05:30" level=info msg="Handling backend connection request [ctos]"
    time="2021-07-26T19:43:46.244772268+05:30" level=info msg="Running kubelet --address=0.0.0.0 --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --cni-bin-dir=/var/lib/rancher/k3s/data/9df574741d2573cbbe6616e8624488b36b3340d077bc50da7cb167f1b08a64d1/bin --cni-conf-dir=/var/lib/rancher/k3s/agent/etc/cni/net.d --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ctos --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --network-plugin=cni --node-labels= --pod-infra-container-image=rancher/pause:3.1 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --read-only-port=0 --resolv-conf=/etc/resolv.conf --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key"
    time="2021-07-26T19:43:46.245470654+05:30" level=info msg="Running kube-proxy --cluster-cidr=10.42.0.0/16 --conntrack-max-per-core=0 --conntrack-tcp-timeout-close-wait=0s --conntrack-tcp-timeout-established=0s --healthz-bind-address=127.0.0.1 --hostname-override=ctos --kubeconfig=/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig --proxy-mode=iptables"
    Flag --cloud-provider has been deprecated, will be removed in 1.23, in favor of removing cloud provider code from Kubelet.
    Flag --cni-bin-dir has been deprecated, will be removed along with dockershim.
    Flag --cni-conf-dir has been deprecated, will be removed along with dockershim.
    Flag --network-plugin has been deprecated, will be removed along with dockershim.
    W0726 19:43:46.245789  602560 server.go:220] WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP.
    E0726 19:43:46.258742  602560 node.go:161] Failed to retrieve node info: nodes "ctos" is forbidden: User "system:kube-proxy" cannot get resource "nodes" in API group "" at the cluster scope
    I0726 19:43:46.273577  602560 server.go:436] "Kubelet version" kubeletVersion="v1.21.3+k3s1"
    I0726 19:43:46.288772  602560 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/rancher/k3s/agent/client-ca.crt
    W0726 19:43:46.288783  602560 manager.go:159] Cannot detect current cgroup on cgroup v2
    I0726 19:43:46.368300  602560 server.go:660] "--cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /"
    I0726 19:43:46.368445  602560 container_manager_linux.go:291] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
    I0726 19:43:46.368497  602560 container_manager_linux.go:296] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:systemd KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalTopologyManagerScope:container ExperimentalCPUManagerReconcilePeriod:10s ExperimentalMemoryManagerPolicy:None ExperimentalMemoryManagerReservedMemory:[] ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none Rootless:false}
    I0726 19:43:46.368525  602560 topology_manager.go:120] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container"
    I0726 19:43:46.368533  602560 container_manager_linux.go:327] "Initializing Topology Manager" policy="none" scope="container"
    I0726 19:43:46.368539  602560 container_manager_linux.go:332] "Creating device plugin manager" devicePluginEnabled=true
    I0726 19:43:46.368592  602560 kubelet.go:307] "Using dockershim is deprecated, please consider using a full-fledged CRI implementation"
    I0726 19:43:46.368619  602560 client.go:78] "Connecting to docker on the dockerEndpoint" endpoint="unix:///var/run/docker.sock"
    I0726 19:43:46.368629  602560 client.go:97] "Start docker client with request timeout" timeout="2m0s"
    I0726 19:43:46.373900  602560 docker_service.go:566] "Hairpin mode is set but kubenet is not enabled, falling back to HairpinVeth" hairpinMode=promiscuous-bridge
    I0726 19:43:46.373920  602560 docker_service.go:242] "Hairpin mode is set" hairpinMode=hairpin-veth
    I0726 19:43:46.381650  602560 docker_service.go:257] "Docker cri networking managed by the network plugin" networkPluginName="cni"
    I0726 19:43:46.387693  602560 docker_service.go:264] "Docker Info" dockerInfo=&{ID:G5RL:4X7T:W5YM:4BWC:LGDL:VYV7:O7QU:A425:2U33:MLLS:NFGR:T2IO Containers:22 ContainersRunning:1 ContainersPaused:0 ContainersStopped:21 Images:229 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Native Overlay Diff false] [userxattr false]] SystemStatus:[] Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:[] Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:false CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6tables:true Debug:false NFd:32 OomKillDisable:false NGoroutines:45 SystemTime:2021-07-26T19:43:46.382166814+05:30 LoggingDriver:json-file CgroupDriver:systemd CgroupVersion:2 NEventsListener:0 KernelVersion:5.9.16-1-MANJARO OperatingSystem:Manjaro Linux OSVersion: OSType:linux Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:0xc0002cfe30 NCPU:8 MemTotal:16432685056 GenericResources:[] DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:ctos Labels:[] ExperimentalBuild:false ServerVersion:20.10.7 ClusterStore: ClusterAdvertise: Runtimes:map[io.containerd.runc.v2:{Path:runc Args:[] Shim:<nil>} io.containerd.runtime.v1.linux:{Path:runc Args:[] Shim:<nil>} runc:{Path:runc Args:[] Shim:<nil>}] DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:[] Nodes:0 Managers:0 Cluster:<nil> Warnings:[]} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:36cc874494a56a253cd181a1a685b44b58a2e34a.m Expected:36cc874494a56a253cd181a1a685b44b58a2e34a.m} RuncCommit:{ID:b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7 Expected:b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=apparmor name=seccomp,profile=default name=cgroupns] ProductLicense: DefaultAddressPools:[] Warnings:[]}
    I0726 19:43:46.387713  602560 docker_service.go:277] "Setting cgroupDriver" cgroupDriver="systemd"
    E0726 19:43:46.387880  602560 docker_service.go:416] "Streaming server stopped unexpectedly" err="listen tcp 10.43.164.71:0: bind: cannot assign requested address"
    k3s.service: Main process exited, code=exited, status=1/FAILURE
    

    3rd disable service lb and traefik curl -sfL https://get.k3s.io | sh -s - --kubelet-arg 'cgroup-driver=systemd' --docker --disable traefik --disable servicelb

    + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
    Failed to get unit file state for nm-cloud-setup.service: No such file or directory
    time="2021-07-26T19:45:13.963861889+05:30" level=info msg="Starting k3s v1.21.3+k3s1 (1d1f220f)"
    time="2021-07-26T19:45:13.964070011+05:30" level=info msg="Cluster bootstrap already complete"
    time="2021-07-26T19:45:13.975286718+05:30" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
    time="2021-07-26T19:45:13.975317362+05:30" level=info msg="Configuring database table schema and indexes, this may take a moment..."
    time="2021-07-26T19:45:13.975402264+05:30" level=info msg="Database tables and indexes are up to date"
    time="2021-07-26T19:45:13.976423984+05:30" level=info msg="Kine listening on unix://kine.sock"
    time="2021-07-26T19:45:13.976586440+05:30" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction --etcd-servers=unix://kine.sock --insecure-port=0 --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
    Flag --insecure-port has been deprecated, This flag has no effect now and will be removed in v1.24.
    I0726 19:45:13.977932  606291 server.go:656] external host was not specified, using 10.0.8.87
    I0726 19:45:13.978128  606291 server.go:195] Version: v1.21.3+k3s1
    I0726 19:45:13.980607  606291 shared_informer.go:240] Waiting for caches to sync for node_authorizer
    I0726 19:45:13.982441  606291 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
    I0726 19:45:13.982628  606291 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
    I0726 19:45:13.984569  606291 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
    I0726 19:45:13.984589  606291 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
    I0726 19:45:13.999510  606291 instance.go:283] Using reconciler: lease
    I0726 19:45:14.053824  606291 rest.go:130] the default service ipfamily for this cluster is: IPv4
    W0726 19:45:14.322164  606291 genericapiserver.go:425] Skipping API node.k8s.io/v1alpha1 because it has no resources.
    W0726 19:45:14.331918  606291 genericapiserver.go:425] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
    W0726 19:45:14.335625  606291 genericapiserver.go:425] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
    W0726 19:45:14.342944  606291 genericapiserver.go:425] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
    W0726 19:45:14.345444  606291 genericapiserver.go:425] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources.
    W0726 19:45:14.350097  606291 genericapiserver.go:425] Skipping API apps/v1beta2 because it has no resources.
    W0726 19:45:14.350111  606291 genericapiserver.go:425] Skipping API apps/v1beta1 because it has no resources.
    I0726 19:45:14.361069  606291 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
    I0726 19:45:14.361151  606291 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
    time="2021-07-26T19:45:14.371727177+05:30" level=info msg="Running kube-scheduler --address=127.0.0.1 --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --leader-elect=false --port=10251 --profiling=false --secure-port=0"
    time="2021-07-26T19:45:14.371787457+05:30" level=info msg="Waiting for API server to become available"
    time="2021-07-26T19:45:14.372280189+05:30" level=info msg="Running kube-controller-manager --address=127.0.0.1 --allocate-node-cidrs=true --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --leader-elect=false --port=10252 --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=0 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --use-service-account-credentials=true"
    time="2021-07-26T19:45:14.372726800+05:30" level=info msg="Running cloud-controller-manager --allocate-node-cidrs=true --bind-address=127.0.0.1 --cloud-provider=k3s --cluster-cidr=10.42.0.0/16 --configure-cloud-routes=false --kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --leader-elect=false --node-status-update-frequency=1m0s --port=0 --profiling=false"
    time="2021-07-26T19:45:14.373706046+05:30" level=info msg="Node token is available at /var/lib/rancher/k3s/server/token"
    time="2021-07-26T19:45:14.373731794+05:30" level=info msg="To join node to cluster: k3s agent -s https://10.0.8.87:6443 -t ${NODE_TOKEN}"
    time="2021-07-26T19:45:14.374453070+05:30" level=info msg="Wrote kubeconfig /etc/rancher/k3s/k3s.yaml"
    time="2021-07-26T19:45:14.374474378+05:30" level=info msg="Run: k3s kubectl"
    time="2021-07-26T19:45:14.414400347+05:30" level=info msg="Cluster-Http-Server 2021/07/26 19:45:14 http: TLS handshake error from 127.0.0.1:53724: remote error: tls: bad certificate"
    time="2021-07-26T19:45:14.418621169+05:30" level=info msg="Cluster-Http-Server 2021/07/26 19:45:14 http: TLS handshake error from 127.0.0.1:53730: remote error: tls: bad certificate"
    time="2021-07-26T19:45:14.426465276+05:30" level=info msg="certificate CN=ctos signed by [email protected]: notBefore=2021-07-26 12:52:30 +0000 UTC notAfter=2022-07-26 14:15:14 +0000 UTC"
    time="2021-07-26T19:45:14.428272397+05:30" level=info msg="certificate CN=system:node:ctos,O=system:nodes signed by [email protected]: notBefore=2021-07-26 12:52:30 +0000 UTC notAfter=2022-07-26 14:15:14 +0000 UTC"
    time="2021-07-26T19:45:14.431628901+05:30" level=info msg="Module overlay was already loaded"
    time="2021-07-26T19:45:14.431651641+05:30" level=info msg="Module nf_conntrack was already loaded"
    time="2021-07-26T19:45:14.431659785+05:30" level=info msg="Module br_netfilter was already loaded"
    time="2021-07-26T19:45:14.431665993+05:30" level=info msg="Module iptable_nat was already loaded"
    time="2021-07-26T19:45:14.438055331+05:30" level=info msg="Connecting to proxy" url="wss://127.0.0.1:6443/v1-k3s/connect"
    time="2021-07-26T19:45:14.439675149+05:30" level=info msg="Handling backend connection request [ctos]"
    time="2021-07-26T19:45:14.440586097+05:30" level=info msg="Running kubelet --address=0.0.0.0 --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --cni-bin-dir=/var/lib/rancher/k3s/data/9df574741d2573cbbe6616e8624488b36b3340d077bc50da7cb167f1b08a64d1/bin --cni-conf-dir=/var/lib/rancher/k3s/agent/etc/cni/net.d --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ctos --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --network-plugin=cni --node-labels= --pod-infra-container-image=rancher/pause:3.1 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --read-only-port=0 --resolv-conf=/etc/resolv.conf --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key"
    time="2021-07-26T19:45:14.441207253+05:30" level=info msg="Running kube-proxy --cluster-cidr=10.42.0.0/16 --conntrack-max-per-core=0 --conntrack-tcp-timeout-close-wait=0s --conntrack-tcp-timeout-established=0s --healthz-bind-address=127.0.0.1 --hostname-override=ctos --kubeconfig=/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig --proxy-mode=iptables"
    Flag --cloud-provider has been deprecated, will be removed in 1.23, in favor of removing cloud provider code from Kubelet.
    Flag --cni-bin-dir has been deprecated, will be removed along with dockershim.
    Flag --cni-conf-dir has been deprecated, will be removed along with dockershim.
    Flag --network-plugin has been deprecated, will be removed along with dockershim.
    W0726 19:45:14.441501  606291 server.go:220] WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP.
    E0726 19:45:14.454113  606291 node.go:161] Failed to retrieve node info: nodes "ctos" is forbidden: User "system:kube-proxy" cannot get resource "nodes" in API group "" at the cluster scope
    I0726 19:45:14.467739  606291 server.go:436] "Kubelet version" kubeletVersion="v1.21.3+k3s1"
    I0726 19:45:14.491377  606291 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/rancher/k3s/agent/client-ca.crt
    W0726 19:45:14.491452  606291 manager.go:159] Cannot detect current cgroup on cgroup v2
    I0726 19:45:14.579771  606291 server.go:660] "--cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /"
    I0726 19:45:14.579941  606291 container_manager_linux.go:291] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
    I0726 19:45:14.580010  606291 container_manager_linux.go:296] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:systemd KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalTopologyManagerScope:container ExperimentalCPUManagerReconcilePeriod:10s ExperimentalMemoryManagerPolicy:None ExperimentalMemoryManagerReservedMemory:[] ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none Rootless:false}
    I0726 19:45:14.580048  606291 topology_manager.go:120] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container"
    I0726 19:45:14.580062  606291 container_manager_linux.go:327] "Initializing Topology Manager" policy="none" scope="container"
    I0726 19:45:14.580072  606291 container_manager_linux.go:332] "Creating device plugin manager" devicePluginEnabled=true
    I0726 19:45:14.580141  606291 kubelet.go:307] "Using dockershim is deprecated, please consider using a full-fledged CRI implementation"
    I0726 19:45:14.580179  606291 client.go:78] "Connecting to docker on the dockerEndpoint" endpoint="unix:///var/run/docker.sock"
    I0726 19:45:14.580195  606291 client.go:97] "Start docker client with request timeout" timeout="2m0s"
    I0726 19:45:14.585680  606291 docker_service.go:566] "Hairpin mode is set but kubenet is not enabled, falling back to HairpinVeth" hairpinMode=promiscuous-bridge
    I0726 19:45:14.585699  606291 docker_service.go:242] "Hairpin mode is set" hairpinMode=hairpin-veth
    I0726 19:45:14.591373  606291 docker_service.go:257] "Docker cri networking managed by the network plugin" networkPluginName="cni"
    I0726 19:45:14.598480  606291 docker_service.go:264] "Docker Info" dockerInfo=&{ID:G5RL:4X7T:W5YM:4BWC:LGDL:VYV7:O7QU:A425:2U33:MLLS:NFGR:T2IO Containers:22 ContainersRunning:1 ContainersPaused:0 ContainersStopped:21 Images:229 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Native Overlay Diff false] [userxattr false]] SystemStatus:[] Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:[] Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:false CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6tables:true Debug:false NFd:32 OomKillDisable:false NGoroutines:45 SystemTime:2021-07-26T19:45:14.591902881+05:30 LoggingDriver:json-file CgroupDriver:systemd CgroupVersion:2 NEventsListener:0 KernelVersion:5.9.16-1-MANJARO OperatingSystem:Manjaro Linux OSVersion: OSType:linux Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:0xc000a320e0 NCPU:8 MemTotal:16432685056 GenericResources:[] DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:ctos Labels:[] ExperimentalBuild:false ServerVersion:20.10.7 ClusterStore: ClusterAdvertise: Runtimes:map[io.containerd.runc.v2:{Path:runc Args:[] Shim:<nil>} io.containerd.runtime.v1.linux:{Path:runc Args:[] Shim:<nil>} runc:{Path:runc Args:[] Shim:<nil>}] DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:[] Nodes:0 Managers:0 Cluster:<nil> Warnings:[]} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:36cc874494a56a253cd181a1a685b44b58a2e34a.m Expected:36cc874494a56a253cd181a1a685b44b58a2e34a.m} RuncCommit:{ID:b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7 Expected:b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=apparmor name=seccomp,profile=default name=cgroupns] ProductLicense: DefaultAddressPools:[] Warnings:[]}
    I0726 19:45:14.598501  606291 docker_service.go:277] "Setting cgroupDriver" cgroupDriver="systemd"
    E0726 19:45:14.598740  606291 docker_service.go:416] "Streaming server stopped unexpectedly" err="listen tcp 10.43.164.71:0: bind: cannot assign requested address"
    k3s.service: Main process exited, code=exited, status=1/FAILURE
    
    

    Steps To Reproduce:

    • Installed K3s:

    Expected behavior: it should run

    Backporting not sure

    • [ ] Needs backporting to older releases
    opened by pratikbalar 1
  • Auto-deploy Manifests Should Periodically Re-evaluate

    Auto-deploy Manifests Should Periodically Re-evaluate

    Environmental Info: K3s Version:

    k3s version v1.21.3+k3s1 (1d1f220f)
    go version go1.16.6
    

    Node(s) CPU architecture, OS, and Version:

    Linux ip-10-130-0-25 5.8.0-1041-aws #43~20.04.1-Ubuntu SMP Thu Jul 15 11:07:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
    

    Cluster Configuration:

    1 server

    Describe the bug:

    This could easily be a bug or a feature request. Technically the behavior 15 months ago was preferred from my perspective, but due to issues with HA and upgrades was altered.

    As it stands today, once the server does the initial auto-deploy, it will only pick up changes IF the modtime on the file has changed.

    Previously it behaved more like gitops model where the manifests were treated as a source of truth, whatever was in manifests directory was always applied against the cluster, so if you made changes directly to the cluster that contradicted the manifest files, they'd be overwritten on the next evaluation loop.

    In my opinion, the previous way was better.

    From a HelmChart use case, when it deploys a service along with a deployment and said service is deleted, the service will never come back.

    The behavior of the auto-deploy is a bit inconsistent too, the first run of the deploy controller will always evaluate all manifests but subsequent runs will not unless the modtime has changed. This means that after k3s is up if someone goes and modifies the traefic or metrics-server deployment directly on the cluster it'll be 100% ok, but as soon as the deploy controller restarts for any reason, all those changes will be overwritten, which could be 1 hour from now or 30 days from now.

    I suggest that the manifests directory should always be applied therefore being a reliable source of truth.

    I currently have a work around for this. I place an entry in /etc/cron.d with a script that literally runs touch against every file in the /var/lib/rancher/k3s/server/manifests directory every minute. This forces k3s to re-evaluate on it's 15 second interval and will see the modtime changed and then apply the changes against the cluster.

    Steps To Reproduce:

    • Install K3s
    • Place manifest file that creates a deployment
    • Delete the deployment

    Expected behavior:

    I would expect the auto-deploy to re-evaluate and have the deployment return eventually.

    Actual behavior:

    The deployment never returns unless the modtime of the file is modified.

    opened by ekristen 1
  • Update Documentation for Unit Tests

    Update Documentation for Unit Tests

    Related to #3706

    kind/documentation 
    opened by briandowns 0
  • Can /etc/rancher/k3s/config.yaml be used to init k3s-agents?

    Can /etc/rancher/k3s/config.yaml be used to init k3s-agents?

    Hello guys,

    So far I am deploying kubernetes master via config.yaml with

    write-kubeconfig-mode: "0644"
    token: xxx
    
    disable:
      - coredns
      - servicelb
      - traefik
      - local-storage
      - metrics-server
    

    and executing standard installer curl -sfL https://get.k3s.io | sh -

    I want to be able to init k3s-agent and join server node via config.yaml as well,

    however having in config.yaml :

    write-kubeconfig-mode: "0644"
    token: xxx
    server: https://172.16.15.8:6443
    
    disable:
      - coredns
      - servicelb
      - traefik
      - local-storage
      - metrics-server
    
    

    and executing curl -sfL https://get.k3s.io | sh - on agent node, is not working. Apparently install script will deploy whole k3s server anyway.

    Only solution that works for me is traditionally executing

    curl -sfL https://get.k3s.io | K3S_URL=https://172.16.15.8:6443 K3S_TOKEN=xxx sh -

    and agent gets deployed.

    I suppose there is currently no solution using config.yaml to deploy agent and join server?

    opened by hlacik 3
  • Cannot write data to local PVC

    Cannot write data to local PVC

    Environmental Info: K3s Version:

    k3s version v1.21.3+k3s1 (1d1f220f)
    go version go1.16.6
    

    Node(s) CPU architecture, OS, and Version:

    Linux debian-8gb-nbg1-1 4.19.0-17-amd64 #1 SMP Debian 4.19.194-2 (2021-06-21) x86_64 GNU/Linux
    

    Cluster Configuration: Single node

    Describe the bug: Postgres does not come up due to mkdir: cannot create directory ‘/var/lib/postgresql/data’: Permission denied

        Container ID:   containerd://fb0246e6a5aa94fe5f14c5c387a2609616d0c198d8a5c5606a41a4792b2c90aa
        Image:          postgres:12
    ...
        Mounts:
          /var/lib/postgresql/data from postgres (rw,path="data")
          /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-7jkg4 (ro)
    Conditions:
      Type              Status
      Initialized       True 
      Ready             False 
      ContainersReady   False 
      PodScheduled      True 
    Volumes:
      postgres:
        Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
        ClaimName:  postgres-awx-postgres-0
        ReadOnly:   false
      kube-api-access-7jkg4:
        Type:                    Projected (a volume that contains injected data from multiple sources)
        TokenExpirationSeconds:  3607
        ConfigMapName:           kube-root-ca.crt
        ConfigMapOptional:       <nil>
        DownwardAPI:             true
    

    Steps To Reproduce:

    • Installed K3s:
    • install this operator https://github.com/ansible/awx-operator/blob/devel/deploy/awx-operator.yaml
    • create awx instance

    Expected behavior: postgres comes up

    Actual behavior: postgres crashes

    Additional context / logs: mkdir: cannot create directory ‘/var/lib/postgresql/data’: Permission denied

    area/local-storage 
    opened by profhase 4
  • k3s issue with helm3 and pvc pending

    k3s issue with helm3 and pvc pending

    Environmental Info: K3s Version: k3s version v1.19.7+k3s1 (5a00e38)

    helm version version.BuildInfo{Version:"v3.5.2", GitCommit:"167aac70832d3a384f65f9745335e9fb40169dc2", GitTreeState:"dirty", GoVersion:"go1.15.7"}

    Node(s) CPU architecture, OS, and Version:

    [email protected]:/var/lib/rancher/k3s/agent/images# uname -a Linux cxcloudagent 5.4.0-65-generic #73-Ubuntu SMP Mon Jan 18 17:25:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

    Cluster Configuration: 1 node

    Describe the bug:

    after deploying some app using helm3 , pod is showing is k3s if I do Kubectl get po but able to see app by helm ls

    it is not happening all the time , but sometimes

    then after I restart the server , I am able to see all the pod


    and sometimes I see pod is in the pending state due to PVC creation then I restarted the pod "local-path-provisioner-**" and redeploy the pod to fix the issue

    both issues is not consistent ,, only sometimes happening

    Steps To Reproduce: Not able to reproduce the issue .. we deploy all of our app using script , sometimes we got this deploying/restart fixed the issue

    Additional context / logs: No error seen

    or maybe no idea where I can see the log regarding this

    please help

    opened by rajibul007 4
  • Update SECURITY.md

    Update SECURITY.md

    I'd consider updating your SECURITY.md

    1. update the email list to point to a CNCF project security list, something like [email protected] should do, you can file a servicedesk ticket to get the email created
    2. look at https://github.com/envoyproxy/envoy/blob/main/SECURITY.md as inspiration maybe to do something a bit more formal to ensure any folks that build products on k3s can get an announcement or if something ALSO affects kubernetes ensure that security process is followed
    opened by caniszczyk 0
  • Fedora VM Worker Not Joining

    Fedora VM Worker Not Joining

    In past, I've used k8s 1.20 in a bare metal cluster that i've established. I have a fair expertise with K8S.

    This is my first time using K3S.

    I am starting with a 2 VM (Fedora 33 on Virtualbox) test cluster.

    I've followed the quickstartinstructions.

    I am able to create a master ("wiggles") . image

    I run a command to have my worker node "giggles" join the cluster

    The command runs without error: image

    I've run this command with and without K3S_NODE_NAME (each time on a pristine VM).

    Yet the master doesnt see the worker.

    Guidance?

    P.S. I'm going to delete these machines.. and start fresh. This time I'll use the K3S_NODE_NAME on each node (rather than an IP address).

    opened by davesargrad 7
  • k3s service failed to restart after etcd error

    k3s service failed to restart after etcd error

    Environmental Info: K3s Version: k3s version v1.19.11+k3s1 (39552458)

    Node(s) CPU architecture, OS, and Version: 3.10.0-1127.19.1.el7.x86_64 - Red Hat Enterprise Linux Server release 7.8 (Maipo)

    Cluster Configuration: 3 servers, 2 agents

    Describe the bug: After running the cluster for a month a master node had drop and we saw this message in the logs:

    Jul 21 13:58:23 <server-name> k3s[3942]: E0721 13:58:23.353906    3942 controller.go:135] error syncing '<server-name>': handler managed-etcd-controller: node has been deleted from the cluster, requeuing
    Jul 21 13:58:23 <server-name> k3s[3942]: {"level":"info","ts":"2021-07-21T13:58:23.355-0700","caller":"raft/raft.go:1530","msg":"c74b97fe9d3df5f7 switched to configuration voters=(2785835851802423746 11433352202400286593)"}
    Jul 21 13:58:23 <server-name> k3s[3942]: {"level":"info","ts":"2021-07-21T13:58:23.355-0700","caller":"membership/cluster.go:422","msg":"removed member","cluster-id":"a139fea82a7faa13","local-member-id":"c74b97fe9d3df5f7","removed-remote-peer-id":"c74b97fe9d3df5f7","removed-remote-peer-urls":["https://10.16.64.50:2380"]}
    Jul 21 13:58:23 <server-name> k3s[3942]: {"level":"warn","ts":"2021-07-21T13:58:23.355-0700","caller":"rafthttp/stream.go:436","msg":"lost TCP streaming connection with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"c74b97fe9d3df5f7","remote-peer-id":"9eab6d4067ffc781","error":"EOF"}
    Jul 21 13:58:23 <server-name> k3s[3942]: {"level":"warn","ts":"2021-07-21T13:58:23.355-0700","caller":"rafthttp/stream.go:436","msg":"lost TCP streaming connection with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"c74b97fe9d3df5f7","remote-peer-id":"26a946f27d3b31c2","error":"EOF"}
    Jul 21 13:58:23 <server-name> k3s[3942]: {"level":"warn","ts":"2021-07-21T13:58:23.356-0700","caller":"rafthttp/stream.go:436","msg":"lost TCP streaming connection with remote peer","stream-reader-type":"stream Message","local-member-id":"c74b97fe9d3df5f7","remote-peer-id":"26a946f27d3b31c2","error":"EOF"}
    Jul 21 13:58:23 <server-name> k3s[3942]: {"level":"warn","ts":"2021-07-21T13:58:23.357-0700","caller":"rafthttp/stream.go:436","msg":"lost TCP streaming connection with remote peer","stream-reader-type":"stream Message","local-member-id":"c74b97fe9d3df5f7","remote-peer-id":"9eab6d4067ffc781","error":"EOF"}
    Jul 21 13:58:23 <server-name> k3s[3942]: {"level":"warn","ts":"2021-07-21T13:58:23.357-0700","caller":"rafthttp/peer_status.go:68","msg":"peer became inactive (message send to peer failed)","peer-id":"26a946f27d3b31c2","error":"failed to dial 26a946f27d3b31c2 on stream MsgApp v2 (the member has been permanently removed from the cluster)"}
    Jul 21 13:58:23 <server-name> k3s[3942]: {"level":"warn","ts":"2021-07-21T13:58:23.358-0700","caller":"etcdserver/server.go:1095","msg":"server error","error":"the member has been permanently removed from the cluster"}
    Jul 21 13:58:23 <server-name> k3s[3942]: {"level":"warn","ts":"2021-07-21T13:58:23.358-0700","caller":"etcdserver/server.go:1096","msg":"data-dir used by this member must be removed"}
    Jul 21 13:58:23 <server-name> k3s[3942]: {"level":"info","ts":"2021-07-21T13:58:23.358-0700","caller":"rafthttp/peer.go:333","msg":"stopping remote peer","remote-peer-id":"9eab6d4067ffc781"}
    Jul 21 13:58:23 <server-name> k3s[3942]: {"level":"warn","ts":"2021-07-21T13:58:23.358-0700","caller":"rafthttp/stream.go:291","msg":"closed TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","remote-peer-id":"9eab6d4067ffc781"}
    Jul 21 13:58:23 <server-name> k3s[3942]: {"level":"warn","ts":"2021-07-21T13:58:23.358-0700","caller":"rafthttp/stream.go:301","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","remote-peer-id":"9eab6d4067ffc781"}
    Jul 21 13:58:23 <server-name> k3s[3942]: {"level":"warn","ts":"2021-07-21T13:58:23.358-0700","caller":"rafthttp/peer_status.go:68","msg":"peer became inactive (message send to peer failed)","peer-id":"9eab6d4067ffc781","error":"failed to dial 9eab6d4067ffc781 on stream MsgApp v2 (the member has been permanently removed from the cluster)"}
    Jul 21 13:58:23 <server-name> k3s[3942]: {"level":"warn","ts":"2021-07-21T13:58:23.358-0700","caller":"rafthttp/stream.go:291","msg":"closed TCP streaming connection with remote peer","stream-writer-type":"stream Message","remote-peer-id":"9eab6d4067ffc781"}
    Jul 21 13:58:23 <server-name> k3s[3942]: {"level":"warn","ts":"2021-07-21T13:58:23.358-0700","caller":"rafthttp/stream.go:301","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"stream Message","remote-peer-id":"9eab6d4067ffc781"}
    Jul 21 13:58:23 <server-name> k3s[3942]: {"level":"info","ts":"2021-07-21T13:58:23.358-0700","caller":"rafthttp/pipeline.go:86","msg":"stopped HTTP pipelining with remote peer","local-member-id":"c74b97fe9d3df5f7","remote-peer-id":"9eab6d4067ffc781"}
    Jul 21 13:58:23 <server-name> k3s[3942]: {"level":"info","ts":"2021-07-21T13:58:23.358-0700","caller":"rafthttp/stream.go:459","msg":"stopped stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"c74b97fe9d3df5f7","remote-peer-id":"9eab6d4067ffc781"}
    Jul 21 13:58:23 <server-name> k3s[3942]: {"level":"info","ts":"2021-07-21T13:58:23.358-0700","caller":"rafthttp/stream.go:459","msg":"stopped stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"c74b97fe9d3df5f7","remote-peer-id":"9eab6d4067ffc781"}
    Jul 21 13:58:23 <server-name> k3s[3942]: {"level":"info","ts":"2021-07-21T13:58:23.358-0700","caller":"rafthttp/peer.go:340","msg":"stopped remote peer","remote-peer-id":"9eab6d4067ffc781"}
    Jul 21 13:58:23 <server-name> k3s[3942]: {"level":"info","ts":"2021-07-21T13:58:23.358-0700","caller":"rafthttp/peer.go:333","msg":"stopping remote peer","remote-peer-id":"26a946f27d3b31c2"}
    Jul 21 13:58:23 <server-name> k3s[3942]: {"level":"warn","ts":"2021-07-21T13:58:23.359-0700","caller":"rafthttp/stream.go:291","msg":"closed TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","remote-peer-id":"26a946f27d3b31c2"}
    Jul 21 13:58:23 <server-name> k3s[3942]: {"level":"warn","ts":"2021-07-21T13:58:23.359-0700","caller":"rafthttp/stream.go:301","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","remote-peer-id":"26a946f27d3b31c2"}
    Jul 21 13:58:23 <server-name> k3s[3942]: {"level":"warn","ts":"2021-07-21T13:58:23.359-0700","caller":"rafthttp/stream.go:291","msg":"closed TCP streaming connection with remote peer","stream-writer-type":"stream Message","remote-peer-id":"26a946f27d3b31c2"}
    Jul 21 13:58:23 <server-name> k3s[3942]: {"level":"warn","ts":"2021-07-21T13:58:23.359-0700","caller":"rafthttp/stream.go:301","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"stream Message","remote-peer-id":"26a946f27d3b31c2"}
    Jul 21 13:58:23 <server-name> k3s[3942]: {"level":"info","ts":"2021-07-21T13:58:23.359-0700","caller":"rafthttp/pipeline.go:86","msg":"stopped HTTP pipelining with remote peer","local-member-id":"c74b97fe9d3df5f7","remote-peer-id":"26a946f27d3b31c2"}
    Jul 21 13:58:23 <server-name> k3s[3942]: {"level":"info","ts":"2021-07-21T13:58:23.359-0700","caller":"rafthttp/stream.go:459","msg":"stopped stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"c74b97fe9d3df5f7","remote-peer-id":"26a946f27d3b31c2"}
    Jul 21 13:58:23 <server-name> k3s[3942]: {"level":"info","ts":"2021-07-21T13:58:23.359-0700","caller":"rafthttp/stream.go:459","msg":"stopped stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"c74b97fe9d3df5f7","remote-peer-id":"26a946f27d3b31c2"}
    Jul 21 13:58:23 <server-name> k3s[3942]: {"level":"info","ts":"2021-07-21T13:58:23.359-0700","caller":"rafthttp/peer.go:340","msg":"stopped remote peer","remote-peer-id":"26a946f27d3b31c2"}
    Jul 21 13:58:23 <server-name> k3s[3942]: time="2021-07-21T13:58:23.360600871-07:00" level=fatal msg="etcd stopped - if this node was removed from the cluster, you must backup and delete /apps/rancher/k3s/server/db/etcd before rejoining"
    Jul 21 13:58:23 <server-name> systemd[1]: k3s.service: main process exited, code=exited, status=1/FAILURE
    

    in trying to start back up the k3s service we received this message:

    Jul 21 15:51:41 <server-name> k3s[34311]: time="2021-07-21T15:51:41-07:00" level=fatal msg="exec: \"k3s-server\": executable file not found in $PATH"
    Jul 21 15:51:41 <server-name> systemd[1]: k3s.service: main process exited, code=exited, status=1/FAILURE
    Jul 21 15:51:41 <server-name> systemd[1]: Failed to start Lightweight Kubernetes.
    Jul 21 15:51:41 <server-name> systemd[1]: Unit k3s.service entered failed state.
    Jul 21 15:51:41 <server-name> systemd[1]: k3s.service failed.
    Jul 21 15:51:46 <server-name> systemd[1]: k3s.service holdoff time over, scheduling restart.
    Jul 21 15:51:46 <server-name> systemd[1]: Stopped Lightweight Kubernetes.
    Jul 21 15:51:46 <server-name> systemd[1]: Starting Lightweight Kubernetes...
    Jul 21 15:51:46 <server-name> k3s[34328]: time="2021-07-21T15:51:46-07:00" level=fatal msg="exec: \"k3s-server\": executable file not found in $PATH"
    Jul 21 15:51:46 <server-name> systemd[1]: k3s.service: main process exited, code=exited, status=1/FAILURE
    Jul 21 15:51:46 <server-name> systemd[1]: Failed to start Lightweight Kubernetes.
    Jul 21 15:51:46 <server-name> systemd[1]: Unit k3s.service entered failed state.
    Jul 21 15:51:46 <server-name> systemd[1]: k3s.service failed.
    Jul 21 15:51:51 <server-name> systemd[1]: k3s.service holdoff time over, scheduling restart.
    Jul 21 15:51:51 <server-name> systemd[1]: Stopped Lightweight Kubernetes.
    Jul 21 15:51:51 <server-name> systemd[1]: Starting Lightweight Kubernetes...
    Jul 21 15:51:51 <server-name> k3s[34384]: time="2021-07-21T15:51:51-07:00" level=fatal msg="exec: \"k3s-server\": executable file not found in $PATH"
    Jul 21 15:51:51 <server-name> systemd[1]: k3s.service: main process exited, code=exited, status=1/FAILURE
    Jul 21 15:51:51 <server-name> systemd[1]: Failed to start Lightweight Kubernetes.
    Jul 21 15:51:51 <server-name> systemd[1]: Unit k3s.service entered failed state.
    Jul 21 15:51:51 <server-name> systemd[1]: k3s.service failed.
    

    our workaround that help get the node back up was to do the following:

    systemctl stop k3s
    sudo rm -rf /var/lib/rancher/k3s/data
    systemctl start k3s
    

    notice that the data folder deleted is the default k3s location and in our install command we use another path.

    curl -sfL https://get.k3s.io | \
    INSTALL_K3S_VERSION="v1.19.7+k3s1" \
    K3S_TOKEN='********' \
    sh -s - \
    --disable=traefik \
    --cluster-cidr 100.100.0.0/18 \
    --service-cidr 100.100.64.0/18 \
    --write-kubeconfig-mode 644 \
    --data-dir /apps/rancher/k3s/ \
    --kubelet-arg='root-dir=/apps/rancher/kubelet' \
    --cluster-init \
    server
    

    Steps To Reproduce:

    • Installed K3s: already mentioned k3s setup command in the "Describe the bug" section.

    Expected behavior: On startup of the k3s process we'd hope the node would be able to be rejoined back into the cluster without giving the pathing error.

    Actual behavior: nodes reach a state where they cant be rejoined and the data directory needs to be cleared.

    Additional context / logs:

    opened by BayanAzima 5
  • failed to get CA certs (with public IP)

    failed to get CA certs (with public IP)

    Environmental Info: K3s Version: v1.21.2+k3s1

    Describe the bug:

    Failed to get CA certs, connection reset by peer. Using a public IP

    Steps To Reproduce:

    curl -sfL https://get.k3s.io | K3S_KUBECONFIG_MODE="644" sh -s curl https://get.k3s.io | K3S_TOKEN="[REDACTED]" K3S_URL="https://publicIP:6443" sh -

    Expected behavior:

    It should connect

    Actual behavior:

    It doesn't connect and throws the error failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:34572->127.0.0.1:6444: read: connection reset by peer"

    Additional context / logs:

    If I open it in my browser it doesn't work either. But I have port forwarded 6443 image image (using localhost it works, using the public IP it doesn't. If I remove the port forward rule it takes forever to make the request and then fails (connection timed out)). image above = not port forwarded (and takes like 1 minute if not more before the error shows) below = port forwarded (and get the error instantly)

    So what I think is that k3s blocks connections on the public IP, and if so, how to enable it?

    opened by Robin-floss 0
Releases(v1.21.3+k3s1)
Lightweight Kubernetes

K3s - Lightweight Kubernetes Lightweight Kubernetes. Production ready, easy to install, half the memory, all in a binary less than 100 MB. Great for:

null 17.3k Jul 23, 2021
Production-Grade Container Scheduling and Management

Kubernetes (K8s) Kubernetes, also known as K8s, is an open source system for managing containerized applications across multiple hosts. It provides ba

Kubernetes 79.2k Jul 20, 2021
Enterprise-grade container platform tailored for multicloud and multi-cluster management

KubeSphere Container Platform What is KubeSphere English | 中文 KubeSphere is a distributed operating system providing cloud native stack with Kubernete

KubeSphere 6.2k Jul 27, 2021
Managing your Kubernetes clusters (including public, private, edge, etc) as easily as visiting the Internet

Clusternet Managing Your Clusters (including public, private, hybrid, edge, etc) as easily as Visiting the Internet. Clusternet (Cluster Internet) is

Clusternet 69 Jul 23, 2021
GitHub中文排行榜,帮助你发现高分优秀中文项目、更高效地吸收国人的优秀经验成果;榜单每周更新一次,敬请关注!

榜单设立目的 ???? GitHub中文排行榜,帮助你发现高分优秀中文项目; 各位开发者伙伴可以更高效地吸收国人的优秀经验、成果; 中文项目只能满足阶段性的需求,想要有进一步提升,还请多花时间学习高分神级英文项目; 榜单设立范围 设立1个总榜(所有语言项目汇总排名)、18个分榜(单个语言项目排名);

kon9chunkit 36.7k Jul 25, 2021
Simplified network and services for edge applications

English | 简体中文 EdgeMesh Introduction EdgeMesh is a part of KubeEdge, and provides a simple network solution for the inter-communications between servi

KubeEdge 30 Jul 11, 2021
kubequery is a Osquery extension that provides SQL based analytics for Kubernetes clusters

kubequery powered by Osquery kubequery is a Osquery extension that provides SQL based analytics for Kubernetes clusters kubequery will be packaged as

Uptycs Inc 47 Jul 13, 2021
A Kubernetes Mutating Webhook to automatically re-point pod images to mirrors

kubernetes-mimic Kubernetes Mimic is a Mutating Webhook that will watch for pod creation and update events in a Kubernetes cluster and automatically a

null 5 Jun 21, 2021
Kubernetes Virtualization API and runtime in order to define and manage virtual machines.

Kubernetes Virtualization API and runtime in order to define and manage virtual machines.

KubeVirt 2.7k Jul 22, 2021
Go library to create resilient feedback loop/control controllers.

Gontroller A Go library to create feedback loop/control controllers, or in other words... a Go library to create controllers without Kubernetes resour

Spotahome 122 Jun 29, 2021
Client extension for interacting with Kubernetes clusters from your k6 tests.

⚠️ This is a proof of concept As this is a proof of concept, it won't be supported by the k6 team. It may also break in the future as xk6 evolves. USE

k6 4 Jul 15, 2021
Not another markup language. Framework for replacing Kubernetes YAML with Go.

Not another markup language. Replace Kubernetes YAML with raw Go! Say so long ?? to YAML and start using the Go ?? programming language to represent a

Kris Nóva 710 Jul 17, 2021
Interactive Cloud-Native Environment Client

Fenix-CLI:Interactive Cloud-Native Environment Client English | 简体中文 Fenix-CLI is an interactive cloud-native operating environment client. The goal i

IcyFenix 25 Jul 24, 2021
k0s - Zero Friction Kubernetes

k0s - Zero Friction Kubernetes k0s is an all-inclusive Kubernetes distribution with all the required bells and whistles preconfigured to make building

k0s - Kubernetes distribution - OSS Project 3.8k Jul 22, 2021