A Kubernetes Network Fabric for Enterprises that is Rich in Functions and Easy in Operations

Overview

kube_ovn_logo

License Build Tag Go Report Card Slack Card FOSSA Status

中文教程

Kube-OVN, a CNCF Sandbox Level Project, integrates the OVN-based Network Virtualization with Kubernetes. It offers an advanced Container Network Fabric for Enterprises with the most functions and the easiest operation.

Community

The Kube-OVN community is waiting for you participation!

  • Follow us at Twitter
  • Chat with us at Slack
  • Other issues please send email to [email protected]
  • 微信用户加 liumengxinfly 进入 "Kube-OVN 开源交流群",请注明 Kube-OVN 和个人信息

Features

  • Namespaced Subnets: Each Namespace can have a unique Subnet (backed by a Logical Switch). Pods within the Namespace will have IP addresses allocated from the Subnet. It's also possible for multiple Namespaces to share a Subnet.
  • Subnet Isolation: Can configure a Subnet to deny any traffic from source IP addresses not within the same Subnet. Can whitelist specific IP addresses and IP ranges.
  • Network Policy: Implementing networking.k8s.io/NetworkPolicy API by high performance ovn ACL.
  • Static IP Addresses for Workloads: Allocate random or static IP addresses to workloads.
  • DualStack IP Support: Pod can run in IPv4-Only/IPv6-Only/DualStack mode.
  • Pod NAT and EIP: Manage the pod external traffic and external ip like tradition VM.
  • Multi-Cluster Network: Connect different clusters into one L3 network.
  • IPAM for Multi NIC: A cluster-wide IPAM for CNI plugins other than Kube-OVN, such as macvlan/vlan/host-device to take advantage of subnet and static ip allocation functions in Kube-OVN.
  • Dynamic QoS: Configure Pod/Gateway Ingress/Egress traffic rate limits on the fly.
  • Embedded Load Balancers: Replace kube-proxy with the OVN embedded high performance distributed L2 Load Balancer.
  • Distributed Gateways: Every Node can act as a Gateway to provide external network connectivity.
  • Namespaced Gateways: Every Namespace can have a dedicated Gateway for Egress traffic.
  • Direct External Connectivity:Pod IP can be exposed to external network directly.
  • BGP Support: Pod/Subnet IP can be exposed to external by BGP router protocol.
  • Traffic Mirror: Duplicated container network traffic for monitoring, diagnosing and replay.
  • Hardware Offload: Boost network performance and save CPU resource by offloading OVS flow table to hardware.
  • Vlan/Underlay Support: Kube-OVN also support underlay and Vlan mode network for better performance and direct connectivity with physic network.
  • DPDK Support: DPDK application now can run in Pod with OVS-DPDK.
  • ARM Support: Kube-OVN can run on x86_64 and arm64 platforms.
  • VPC Support: Multi-tenant network with overlapped address spaces.
  • TroubleShooting Tools: Handy tools to diagnose, trace, monitor and dump container network traffic to help troubleshooting complicate network issues.
  • Prometheus & Grafana Integration: Exposing network quality metrics like pod/node/service/dns connectivity/latency in Prometheus format.

Planned Future Work

  • Policy-based QoS
  • More Metrics and Traffic Graph
  • More Diagnosis and Tracing Tools

Network Topology

The Switch, Router and Firewall showed in the diagram below are all distributed on all Nodes. There is no single point of failure for in-cluster network.

topology

Monitoring Dashboard

Kube-OVN offers prometheus integration with grafana dashboards to visualise network quality.

dashboard

Quick Start

Kube-OVN is easy to install with all necessary components/dependencies included. If you already have a Kubernetes cluster without any cni plugin, please refer to the Installation Guide.

If you want to install Kubernetes from scratch, you can try kubespray or for Chinese users try kubeasz to deploy a production ready Kubernetes cluster with Kube-OVN embedded.

Documents

Contribution

We are looking forwards to your PR!

FAQ

  1. Q: How about the scalability of Kube-OVN?

    A: We have simulated 200 Nodes with 10k Pods by kubemark, and it works fine. Some community users have deployed one cluster with 250+ Nodes and 3k+ Pods in production. It's still not reach the limitation, but we don't have enough resources to find the limitation.

  2. Q: What's the Addressing/IPAM? Node-specific or cluster-wide?

    A: Kube-OVN use a cluster-wide IPAM, Pod address can float to any nodes in the cluster.

  3. Q: What's the encapsulation?

    A: For overlay mode, Kube-OVN uses Geneve to encapsulate packets between nodes. For Vlan/Underlay mode there is no encapsulation.

Kube-OVN vs. Other CNI Implementation

Different CNI Implementation has different function scope and network topology. There is no single implementation that can resolve all network problems. In this section, we compare Kube-OVN to some other options to give users a better understanding to assess which network will fit into your infrastructure.

Kube-OVN vs. ovn-kubernetes

ovn-kubernetes is developed by the ovn community to integration ovn for Kubernetes. As both projects use OVN/OVS as the data plane, they have some same function sets and architecture. The main differences come from the network topology and gateway implementation.

ovn-kubernetes implements a subnet-per-node network topology. That means each node will have a fixed cidr range, and the ip allocation is fulfilled by each node when the pod has been invoked by kubelet.

Kube-OVN implements a subnet-per-namespace network topology. That means a cidr can spread the entire cluster nodes, and the ip allocation is fulfilled by kube-ovn-controller at a central place. And then kube-ovn can apply lots of network configurations at subnet level, like cidr, gw, exclude_ips, nat and so on. This topology also gives Kube-OVN more ability to control how ip should be allocated, on top of this topology, Kube-OVN can allocate static ip for workloads.

We believe the subnet-per-namespace topology will give more flexibility to evolve the network.

On the gateway side, ovn-kubernetes uses native ovn gateway concept to control the traffic. The native ovn gateway relies on a dedicated nic or needs to transfer the nic ip to another device to bind the nic to the ovs bridge. This implementation can reach better performance, however not all environments meet the network requirements especially in the cloud.

Kube-OVN uses policy-route, ipset and iptables to implement the gateway functions that all by software, which can fit more infrastructure and give more flexibility to more function.

Kube-OVN vs. Calico

Calico is an open-source networking and network security solution for containers, virtual machines, and native host-based workloads. It's known for its good performance and security policy.

The main difference from the design point is the encapsulation method. Calico use no encapsulation or lightweight IPIP encapsulation and Kube-OVN uses geneve to encapsulate packets. No encapsulation can achieve better network performance for both throughput and latency. However, as this method will expose pod network directly to the underlay network with it comes with the burden on deploy and maintain. In some managed network environment where BGP and IPIP is not allowed, encapsulation is a must.

Use encapsulation can lower the requirement on networking, and isolate containers and underlay network from logical. We can use the overlay technology to build a much complex network concept, like router, gateway, and vpc. For performance, ovs can make use of hardware offload and DPDK to enhance throughput and latency.

Kube-OVN can also work in non-encapsulation mode, that take use of underlay switches to switch the packets or use hardware offload to achieve better performance than kernel datapath.

From the function set, Kube-OVN can offer some more abilities like static ip, QoS and traffic mirror. The subnet in Kube-OVN and ippool in Calico share some same function set.

License

FOSSA Status

Issues
  • Kubeovn pod communication between different K8s cluster nodes not working when DPDK is enabled

    Kubeovn pod communication between different K8s cluster nodes not working when DPDK is enabled

    Hi,

    I have deployed a multinode Kubernetes setup with kubeovn as default cni. I installed Kubeovn with DPDK support following the link https://github.com/alauda/kube-ovn/blob/master/docs/dpdk.md on openstack VMS with OVS-DPDK and virtio type of network interfaces attached to cluster vms.

    But i am facing issue when my pods are scheduled on different nodes. They are not able to communicate with each other using Kubeovn interface also. I understand for DPDK based interfaces, this communication needs to be manually configured as userspace cni does not support this but Kubeovn interface communication must work fine. Same thing is working fine when i deploy kubeovn without DPDK support.

    My Environment Details are OS: Ubuntu18 Virtual Machines over openstack with ovs-dpdk RAM: 16GB Cores: 8 Nic: Virtio network device K8s: Version 1.134 Kubeovn with DPDK: "v1.3.0-pre"

    One thing i observed is there are no geneve ports added to br-int provided by Kubeovn in case DPDK is enabled

    Kubeovn with DPDK

    [email protected]:~# ovs-vsctl show 1d99a1eb-1d46-4016-8639-bc00ab08ca83 Bridge br-int fail_mode: secure Port br-int Interface br-int type: internal Port mirror0 Interface mirror0 type: internal Port "7cd52e4a918f_h" Interface "7cd52e4a918f_h" Port ovn0 Interface ovn0 type: internal ovs_version: "2.13.0"

    Kubeovn Without DPDK

    [email protected]:~# ovs-vsctl show 42591383-8fd4-4d44-b9c9-90be02958d71 Bridge br-int fail_mode: secure Port ovn-9f840e-0 Interface ovn-9f840e-0 type: geneve options: {csum="true", key=flow, remote_ip=<minion1_ip>} Port br-int Interface br-int type: internal Port f7aaa44c4a5c_h Interface f7aaa44c4a5c_h Port ovn0 Interface ovn0 type: internal Port ovn-b6bfb4-0 Interface ovn-b6bfb4-0 type: geneve options: {csum="true", key=flow, remote_ip=<minion2_ip>} Port ovn-7d41af-0 Interface ovn-7d41af-0 type: geneve options: {csum="true", key=flow, remote_ip=<minion3_ip>} Port mirror0 Interface mirror0 type: internal Port "92756136d181_h" Interface "92756136d181_h" ovs_version: "2.13.0"

    Even Kubernetes.deafult dns is not reachable in case DPDK is enabled for multihost K8s environment.

    Here are the trace of dnsutils container:

    ~# kubectl ko trace default/dnsutils 10.96.0.10 udp 53

    • kubectl exec ovn-central-5b86b448c8-6jb64 -n kube-system -- ovn-trace --ct=new ovn-default 'inport == "dnsutils.default" && ip.ttl == 64 && eth.src == 00:00:00:F7:E9:EE && ip4.src == 10.16.0.6 && eth.dst == 00:00:00:DD:D2:BC && ip4.dst == 10.96.0.10 && udp.src == 10000 && udp.dst == 53'

    udp,reg14=0x6,vlan_tci=0x0000,dl_src=00:00:00:f7:e9:ee,dl_dst=00:00:00:dd:d2:bc,nw_src=10.16.0.6,nw_dst=10.96.0.10,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=10000,tp_dst=53

    ingress(dp="ovn-default", inport="dnsutils.default") 0. ls_in_port_sec_l2 (ovn-northd.c:4629): inport == "dnsutils.default" && eth.src == {00:00:00:f7:e9:ee}, priority 50, uuid 06181115 next;

    1. ls_in_port_sec_ip (ovn-northd.c:4281): inport == "dnsutils.default" && eth.src == 00:00:00:f7:e9:ee && ip4.src == {10.16.0.6}, priority 90, uuid 292b42d3 next;
    2. ls_in_pre_acl (ovn-northd.c:4805): ip, priority 100, uuid 59276b34 reg0[0] = 1; next;
    3. ls_in_pre_lb (ovn-northd.c:4961): ip && ip4.dst == 10.96.0.10, priority 100, uuid b7662b98 reg0[0] = 1; next;
    4. ls_in_pre_stateful (ovn-northd.c:4992): reg0[0] == 1, priority 100, uuid 206aea47 ct_next;

    ct_next(ct_state=new|trk) 6. ls_in_acl (ovn-northd.c:5368): ip && (!ct.est || (ct.est && ct_label.blocked == 1)), priority 1, uuid 2b126df4 reg0[1] = 1; next; 10. ls_in_stateful (ovn-northd.c:5726): ct.new && ip4.dst == 10.96.0.10 && udp.dst == 53, priority 120, uuid f4ab5458 ct_lb(backends=10.16.0.2:53,10.16.0.4:53);

    ct_lb 19. ls_in_l2_lkup (ovn-northd.c:6912): eth.dst == 00:00:00:dd:d2:bc, priority 50, uuid af68a5a3 outport = "ovn-default-ovn-cluster"; output;

    egress(dp="ovn-default", inport="dnsutils.default", outport="ovn-default-ovn-cluster") 0. ls_out_pre_lb (ovn-northd.c:4977): ip, priority 100, uuid 6cca8aba reg0[0] = 1; next;

    1. ls_out_pre_acl (ovn-northd.c:4748): ip && outport == "ovn-default-ovn-cluster", priority 110, uuid 3fb2824f next;
    2. ls_out_pre_stateful (ovn-northd.c:4994): reg0[0] == 1, priority 100, uuid d7986c70 ct_next;

    ct_next(ct_state=est|trk /* default (use --ct to customize) */) 3. ls_out_lb (ovn-northd.c:5609): ct.est && !ct.rel && !ct.new && !ct.inv, priority 65535, uuid 2f537878 reg0[2] = 1; next; 7. ls_out_stateful (ovn-northd.c:5771): reg0[2] == 1, priority 100, uuid ed9ca3a1 ct_lb;

    ct_lb 9. ls_out_port_sec_l2 (ovn-northd.c:4695): outport == "ovn-default-ovn-cluster", priority 50, uuid 8bdc173c output; /* output to "ovn-default-ovn-cluster", type "patch" */

    ingress(dp="ovn-cluster", inport="ovn-cluster-ovn-default") 0. lr_in_admission (ovn-northd.c:7974): eth.dst == 00:00:00:dd:d2:bc && inport == "ovn-cluster-ovn-default", priority 50, uuid 780ff8c5 next;

    1. lr_in_lookup_neighbor (ovn-northd.c:8023): 1, priority 0, uuid 4e24c5d4 reg9[3] = 1; next;
    2. lr_in_learn_neighbor (ovn-northd.c:8029): reg9[3] == 1 || reg9[2] == 1, priority 100, uuid 8a3ad9f6 next;
    3. lr_in_ip_routing (ovn-northd.c:7598): ip4.dst == 10.16.0.0/16, priority 33, uuid 7e9728c9 ip.ttl--; reg8[0..15] = 0; reg0 = ip4.dst; reg1 = 10.16.0.1; eth.src = 00:00:00:dd:d2:bc; outport = "ovn-cluster-ovn-default"; flags.loopback = 1; next;
    4. lr_in_ip_routing_ecmp (ovn-northd.c:9593): reg8[0..15] == 0, priority 150, uuid 52ad4463 next;
    5. lr_in_arp_resolve (ovn-northd.c:9861): outport == "ovn-cluster-ovn-default" && reg0 == 10.16.0.4, priority 100, uuid 15ef66f9 eth.dst = 00:00:00:ae:ad:23; next;
    6. lr_in_arp_request (ovn-northd.c:10265): 1, priority 0, uuid 2b3d52d9 output;

    egress(dp="ovn-cluster", inport="ovn-cluster-ovn-default", outport="ovn-cluster-ovn-default") 3. lr_out_delivery (ovn-northd.c:10311): outport == "ovn-cluster-ovn-default", priority 100, uuid 8807ddda output; /* output to "ovn-cluster-ovn-default", type "patch" */

    ingress(dp="ovn-default", inport="ovn-default-ovn-cluster") 0. ls_in_port_sec_l2 (ovn-northd.c:4629): inport == "ovn-default-ovn-cluster", priority 50, uuid 9f808a14 next; 3. ls_in_pre_acl (ovn-northd.c:4745): ip && inport == "ovn-default-ovn-cluster", priority 110, uuid 1defc5dd next; 9. ls_in_lb (ovn-northd.c:5606): ct.est && !ct.rel && !ct.new && !ct.inv, priority 65535, uuid 9edb7d1e reg0[2] = 1; next; 10. ls_in_stateful (ovn-northd.c:5769): reg0[2] == 1, priority 100, uuid 02a8d618 ct_lb;

    ct_lb 19. ls_in_l2_lkup (ovn-northd.c:6912): eth.dst == 00:00:00:ae:ad:23, priority 50, uuid 63f29f2c outport = "coredns-86c58d9df4-82qgb.kube-system"; output;

    egress(dp="ovn-default", inport="ovn-default-ovn-cluster", outport="coredns-86c58d9df4-82qgb.kube-system") 0. ls_out_pre_lb (ovn-northd.c:4977): ip, priority 100, uuid 6cca8aba reg0[0] = 1; next;

    1. ls_out_pre_acl (ovn-northd.c:4807): ip, priority 100, uuid b6219115 reg0[0] = 1; next;
    2. ls_out_pre_stateful (ovn-northd.c:4994): reg0[0] == 1, priority 100, uuid d7986c70 ct_next;

    ct_next(ct_state=est|trk /* default (use --ct to customize) */) 3. ls_out_lb (ovn-northd.c:5609): ct.est && !ct.rel && !ct.new && !ct.inv, priority 65535, uuid 2f537878 reg0[2] = 1; next; 7. ls_out_stateful (ovn-northd.c:5771): reg0[2] == 1, priority 100, uuid ed9ca3a1 ct_lb;

    ct_lb 8. ls_out_port_sec_ip (ovn-northd.c:4281): outport == "coredns-86c58d9df4-82qgb.kube-system" && eth.dst == 00:00:00:ae:ad:23 && ip4.dst == {255.255.255.255, 224.0.0.0/4, 10.16.0.4, 10.16.255.255}, priority 90, uuid 662c2b5f next; 9. ls_out_port_sec_l2 (ovn-northd.c:4695): outport == "coredns-86c58d9df4-82qgb.kube-system" && eth.dst == {00:00:00:ae:ad:23}, priority 50, uuid 35ab4345 output; /* output to "coredns-86c58d9df4-82qgb.kube-system", type "" */

    • set +x

    Start OVS Tracing

    • kubectl exec ovs-ovn-k79ds -n kube-system -- ovs-appctl ofproto/trace br-int in_port=6,udp,nw_src=10.16.0.6,nw_dst=10.96.0.10,dl_src=00:00:00:F7:E9:EE,dl_dst=00:00:00:DD:D2:BC,tp_src=1000,tp_dst=53 Bad openflow flow syntax: in_port=6,udp,nw_src=10.16.0.6,nw_dst=10.96.0.10,dl_src=00:00:00:F7:E9:EE,dl_dst=00:00:00:DD:D2:BC,tp_src=1000,tp_dst=53: prerequisites not met for setting tp_src ovs-appctl: ovs-vswitchd: server returned an error command terminated with exit code 2

    kubectl ko nbctl list load_balancer

    _uuid : f2db64c0-7c90-468f-bf51-9807b93229b2 external_ids : {} health_check : [] ip_port_mappings : {} name : cluster-tcp-loadbalancer protocol : tcp selection_fields : [] vips : {"10.100.21.21:10665"="172.19.104.78:10665", "10.101.2.32:10660"="172.19.104.78:10660", "10.105.152.58:6642"="172.19.104.78:6642", "10.107.165.205:8080"="10.16.0.5:8080", "10.96.0.10:53"="10.16.0.2:53,10.16.0.4:53", "10.96.0.1:443"="172.19.104.78:6443", "10.97.248.180:6641"="172.19.104.78:6641"}

    _uuid : 2a4f9f2f-21cf-4bd6-a35e-d4575e7c9117 external_ids : {} health_check : [] ip_port_mappings : {} name : cluster-udp-loadbalancer protocol : udp selection_fields : [] vips : {"10.96.0.10:53"="10.16.0.2:53,10.16.0.4:53"}

    kubectl ko nbctl list logical_switch

    _uuid : 1e169528-a148-436f-811f-b3a83c089e04 acls : [821479e8-2151-4760-b27a-54d901ddfc70] dns_records : [] external_ids : {} forwarding_groups : [] load_balancer : [] name : join other_config : {exclude_ips="100.64.0.1", gateway="100.64.0.1", subnet="100.64.0.0/16"} ports : [1a6dcb70-084c-460e-a84f-7505f820f276, 1e4994f0-e749-4c8e-83e8-25502e11b769] qos_rules : []

    _uuid : bdc0fac5-6caf-4800-bd23-8aaf80c273ce acls : [c9ca44b2-d3a6-4b74-b668-baef0dc32d67] dns_records : [] external_ids : {} forwarding_groups : [] load_balancer : [2a4f9f2f-21cf-4bd6-a35e-d4575e7c9117, f2db64c0-7c90-468f-bf51-9807b93229b2] name : ovn-default other_config : {exclude_ips="10.16.0.1", gateway="10.16.0.1", subnet="10.16.0.0/16"} ports : [64a29d31-c9d7-4719-a09b-911d816982af, b150f67a-7e55-4fa2-ad92-63c45e74cd7a, b155c0b3-40fb-41b0-9b9a-8a163bb3497c, ca1a9e8b-78d3-42ce-804d-0e2244316e77, f17ea181-9bc8-4f5d-91e9-e79fc6ddd95c] qos_rules : []

    bug question 
    opened by tarun28jain 25
  • Connectivity issue with kube-ovn 1.5.0

    Connectivity issue with kube-ovn 1.5.0

    With a Fedora32 CI setup (nft) I can't manage to have a working fresh install of kube-ovn 1.4.0.

    Here are the only changes between the 2 jobs : https://gitlab.com/kargo-ci/kubernetes-sigs-kubespray/-/commit/1f2a4ee2de4cf37d762815fcf1c497e2790a72a8 (upgrade version and yamls)

    Before I try to check deeper, any insight of what might have change between the 2 versions ?

    opened by floryut 20
  • 无法正确分配pod ip

    无法正确分配pod ip

    默认子网设置为172.30.0.0/16,但部署了一个测试的busybox应用,分配的IP为172.17.0.2(docker的默认网段),那位指点一下问题出在哪儿里,多谢!

    [[email protected] ~]# kubectl get node -o wide
    NAME                STATUS   ROLES            AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION               CONTAINER-RUNTIME
    host-10-19-17-139   Ready    compute,master   25d   v1.15.2   10.19.17.139   <none>        CentOS Linux 7 (Core)   5.2.11-1.el7.elrepo.x86_64   docker://19.3.1
    host-10-19-17-140   Ready    compute          25d   v1.15.2   10.19.17.140   <none>        CentOS Linux 7 (Core)   5.2.11-1.el7.elrepo.x86_64   docker://19.3.1
    host-10-19-17-141   Ready    compute          25d   v1.15.2   10.19.17.141   <none>        CentOS Linux 7 (Core)   5.2.11-1.el7.elrepo.x86_64   docker://19.3.1
    
    [[email protected] ~]# kubectl get Subnet
    NAME          PROTOCOL   CIDR            PRIVATE   NAT     DEFAULT   GATEWAYTYPE   USED   AVAILABLE
    join          IPv4       100.64.0.0/16   false     false   false     distributed   3      65532
    ovn-default   IPv4       172.30.0.0/16   false     true    true      distributed   0      65535
    
    [[email protected] ~]# kubectl get pod --all-namespaces -o wide
    NAMESPACE   NAME                                   READY   STATUS    RESTARTS   AGE   IP             NODE                NOMINATED NODE   READINESS GATES
    default     test-588865b-zd5vt                     1/1     Running   1          31m   172.17.0.2     host-10-19-17-141   <none>           <none>
    kube-ovn    kube-ovn-cni-dvgzj                     1/1     Running   0          33m   10.19.17.141   host-10-19-17-141   <none>           <none>
    kube-ovn    kube-ovn-cni-krsnc                     1/1     Running   0          33m   10.19.17.140   host-10-19-17-140   <none>           <none>
    kube-ovn    kube-ovn-cni-w74td                     1/1     Running   0          33m   10.19.17.139   host-10-19-17-139   <none>           <none>
    kube-ovn    kube-ovn-controller-86d7c8d6c4-p4ndm   1/1     Running   0          33m   10.19.17.141   host-10-19-17-141   <none>           <none>
    kube-ovn    kube-ovn-controller-86d7c8d6c4-zjvbv   1/1     Running   0          33m   10.19.17.139   host-10-19-17-139   <none>           <none>
    kube-ovn    ovn-central-8ddc7dd8-ww7mf             1/1     Running   0          39m   10.19.17.139   host-10-19-17-139   <none>           <none>
    kube-ovn    ovs-ovn-dbrg2                          1/1     Running   0          39m   10.19.17.139   host-10-19-17-139   <none>           <none>
    kube-ovn    ovs-ovn-jxjc5                          1/1     Running   0          39m   10.19.17.140   host-10-19-17-140   <none>           <none>
    kube-ovn    ovs-ovn-s5rxz                          1/1     Running   0          39m   10.19.17.141   host-10-19-17-141   <none>           <none>
    
    opened by Coyote5 18
  • dpdk commands can not run inside kube-ovn container

    dpdk commands can not run inside kube-ovn container

    I install Kube-OVN with OVS-DPDK following the guide https://github.com/alauda/kube-ovn/blob/master/docs/dpdk.md and replace ovs images with kubeovn/kube-ovn-dpdk:19.11.2

    But when I'm trying to debug dpdk, and find the following command can not work

    [[email protected] dpdk-stable-19.11.2]# kubectl exec -it -n kube-ovn ovs-ovn-wmmgw bash
    [[email protected] /]# dpdk-pdump 
    Illegal instruction
    [[email protected] /]# dpdk-proc-info 
    Illegal instruction
    [[email protected] /]# 
    
    

    DPDK kernel modules compile and install following http://docs.openvswitch.org/en/latest/intro/install/dpdk/#installing with dpdk version 19.11.2. Kernel Version: 4.14.172 Host OS Version: CentOS Linux release 7.4.1708 (Core)

    bug 
    opened by wavezhang 15
  • Illegal instruction (core dumped)

    Illegal instruction (core dumped)

    Hi,

    We would like to use ACL feature and so we are deploying master branch. Monday we deployed and tested succesfully but today could not deploy master branch. So we planned to deploy Monday version and tried to create build related Monday commit and tried development guide. However we got many error related to "stack smashing detected". one of them as below.

    [line 0 : column 0] - loading files from package "cmd": err: signal: aborted (core dumped): stderr: *** stack smashing detected ***: terminated

    Could you please make a recommendation to overcome this issue?

    Thanks and regards.

    opened by mkcetinkaya 13
  • Add OVS-DPDK support, for issue 104

    Add OVS-DPDK support, for issue 104

    This commit adds OVS-DPDK support to Kube-OVN. User instructions are included in a new file docs/dpdk.md

    A new Dockerfile has been added to include OVS-DPDK along with OVN. Where DPDK is required, this image is used for the ovs-ovn pod, in place of the existing kernel-OVS “kube-ovn” image. This Dockerfile is currently based on Fedora 32 for reasons noted as comments within the file. It should later be possible to change this to CentOS when full DPDK 19 support is available.

    I recommend the above Dockerfile is built and tagged as kube-ovn-dpdk:, where the version corresponds to the DPDK version used within the image (in this case 19:11) rather than the Kube-OVN version. I recommend this as DPDK applications have a strong dependency on DPDK version. If we force an end user to always use the latest version, then we will likely break their DPDK app. I propose over time we provide images for multiple DPDK versions and let the user pick to suit their needs. I don’t see these images or Dockerfiles requiring maintenance or support. They should be completely independent of Kube-OVN versions and releases.

    The install.sh script has been modified. It now takes a flag --with-dpdk= so the user can indicate they want to install OVS-DPDK based on which version of DPDK. Version of DPDK required will determine version of OVS and this will be already built into the Docker image provided. The Kube-OVN version installed is still set at the top of the script as the VERSION variable. This should still be the case going forward, Kube-OVN and DPDK versions should operate independently of each other. However, it’s something to watch. If future versions of Kube-OVN have a strong dependency on newer versions of OVS, then the older version of OVS used for DPDK may become an issue. We may have to update the install script so a user wanting an older version of DPDK has no choice but to use an older version of Kube-OVN that’s known to be compatible. I don’t foresee this being an issue, but one to watch as I said.

    New startup and healthcheck scripts added for OVS-DPDK.

    performance 
    opened by garyloug 13
  • kubevirt使用kube-ovn固定虚机IP和热迁移相关问题

    kubevirt使用kube-ovn固定虚机IP和热迁移相关问题

    kubevirt使用kube-ovn固定虚机IP和热迁移相关问题 kubevrit版本0.49,kube-ovn版本1.9.0

    1. kubevrit使用kubeovn固定IP,测试热迁移失败 VM配置如下,给multus网卡分配kubeovn子网并固定IP,开启迁移选项allow_live_migration
      template:
        metadata:
          annotations:
            attachnet-kubeovn.kubevirt-demo.ovn.kubernetes.io/logical_switch: attachnet-kubeovn-subnet
            attachnet-kubeovn.kubevirt-demo.ovn.kubernetes.io/allow_live_migration: 'true'    # allow_live_migration 则不做地址冲突检查
            attachnet-kubeovn.kubevirt-demo.ovn.kubernetes.io/default_route: 'true'           # 为非默认网卡指定默认路由
            attachnet-kubeovn.kubevirt-demo.ovn.kubernetes.io/ip_address: 172.17.0.203
            attachnet-kubeovn.kubevirt-demo.ovn.kubernetes.io/mac_address: '00:00:00:53:6B:B6'
        spec:
          evictionStrategy: LiveMigrate
          domain:
            cpu:
              cores: 1
            resources:
              requests:
                memory: 1024M
    
            devices:
              disks:
              - disk:
                  bus: virtio
                name: disk-system
    
              - cdrom:
                  bus: sata
                  readonly: true
                name: cloudinitdisk
    
              interfaces:
              - name: attachnet-kubeovn
                bridge: {}
    
          networks:
            - multus:
                networkName: kubevirt-demo/attachnet-kubeovn
              name: attachnet-kubeovn
    

    如图,创建的虚机成功获取到了固定IP image

    执行热迁移命令,新创建的launcher Pod无法启动

    virtctl migrate img-centos-7-cloudimage-v4-vm
    

    image

    应该是在multus网卡获取IP处报错了 image

    1. 另外当虚机中只加固定MAC的注解不加固定IP的注解时,发现并没有使用指定的MAC地址,和文档第三条说明不符 https://github.com/kubeovn/kube-ovn/wiki/Pod-%E5%9B%BA%E5%AE%9A-IP-%E5%92%8C-Mac
            attachnet-kubeovn.kubevirt-demo.ovn.kubernetes.io/logical_switch: attachnet-kubeovn-subnet
            attachnet-kubeovn.kubevirt-demo.ovn.kubernetes.io/allow_live_migration: 'true'    # allow_live_migration 则不做地址冲突检查
            attachnet-kubeovn.kubevirt-demo.ovn.kubernetes.io/default_route: 'true'           # 为非默认网卡指定默认路由
            # attachnet-kubeovn.kubevirt-demo.ovn.kubernetes.io/ip_address: 172.17.0.203
            attachnet-kubeovn.kubevirt-demo.ovn.kubernetes.io/mac_address: '00:00:00:53:6B:B6'
    

    如图,VMI的上有MAC相关注解 image

    但是launcher Pod 注解和 虚机内部并没有使用该MAC地址 image

    image

    1. 请问OVN是否可以设置根据MAC地址来绑定IP地址,这样是否可以在固定IP的同时 也能够用ovn的dhcp分配的IP
    bug 
    opened by hurdonkey 12
  • NetworkPlugin cni failed to set up...json: cannot unmarshal string into Go value of type request.PodResponse

    NetworkPlugin cni failed to set up...json: cannot unmarshal string into Go value of type request.PodResponse

    刚测试了下k8s 1.14.2三节点集群使用kube-ovn作为网络插件,出现错误

    [email protected]:~# kubectl get node
    NAME           STATUS                     ROLES    AGE    VERSION
    10.100.97.41   Ready,SchedulingDisabled   master   4h4m   v1.14.2
    10.100.97.42   Ready                      node     4h4m   v1.14.2
    10.100.97.43   Ready                      node     4h4m   v1.14.2
    
    [email protected]:~# kubectl get cs
    NAME                 STATUS    MESSAGE             ERROR
    scheduler            Healthy   ok                  
    controller-manager   Healthy   ok                  
    etcd-0               Healthy   {"health":"true"}   
    
    [email protected]:~# kubectl get pod --all-namespaces 
    NAMESPACE     NAME                                    READY   STATUS              RESTARTS   AGE
    kube-ovn      kube-ovn-cni-2n5n5                      1/1     Running             2          3h9m
    kube-ovn      kube-ovn-cni-57q94                      1/1     Running             2          3h9m
    kube-ovn      kube-ovn-cni-99qzt                      1/1     Running             1          3h9m
    kube-ovn      kube-ovn-controller-db57b48cb-cvrz4     1/1     Running             1          3h9m
    kube-ovn      kube-ovn-controller-db57b48cb-n9lf5     1/1     Running             1          3h9m
    kube-ovn      ovn-central-8658697f7f-2ngrx            1/1     Running             1          3h56m
    kube-ovn      ovs-ovn-2t48n                           1/1     Running             1          3h47m
    kube-ovn      ovs-ovn-4fmh6                           1/1     Running             1          3h47m
    kube-ovn      ovs-ovn-qvp2w                           1/1     Running             1          3h47m
    kube-system   coredns-55f46dd959-d2r2l                0/1     ContainerCreating   0          20m
    kube-system   coredns-55f46dd959-ldnnw                0/1     ContainerCreating   0          20m
    kube-system   heapster-fdb7596d6-nvrld                0/1     ContainerCreating   0          20m
    kube-system   kubernetes-dashboard-68ddcc97fc-r2g66   0/1     ContainerCreating   0          20m
    kube-system   metrics-server-6c898b5b8b-kr6kd         0/1     ContainerCreating   0          20m
    
    

    查看处于 ContainerCreating 状态的pod信息

    Events:
      Type     Reason                  Age                 From                   Message
      ----     ------                  ----                ----                   -------
      Normal   Scheduled               25m                 default-scheduler      Successfully assigned kube-system/kubernetes-dashboard-68ddcc97fc-r2g66 to 10.100.97.42
      Warning  FailedCreatePodSandBox  25m                 kubelet, 10.100.97.42  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "f305fdde7222127f5d5d7540fb1d517fc29c13766a2b173e1cad9ca3a1faa0fe" network for pod "kubernetes-dashboard-68ddcc97fc-r2g66": NetworkPlugin cni failed to set up pod "kubernetes-dashboard-68ddcc97fc-r2g66_kube-system" network: json: cannot unmarshal string into Go value of type request.PodResponse
      Warning  FailedCreatePodSandBox  25m                 kubelet, 10.100.97.42  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "dda174779a31b49ef1824af0c653b7166b3d289fb56bf2cceff495471a644cd3" network for pod "kubernetes-dashboard-68ddcc97fc-r2g66": NetworkPlugin cni failed to set up pod "kubernetes-dashboard-68ddcc97fc-r2g66_kube-system" network: json: cannot unmarshal string into Go value of type request.PodResponse
      Warning  FailedCreatePodSandBox  24m                 kubelet, 10.100.97.42  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "52a4ae0fa25b54f3e502f1559883ae0369710f3c627ac7e921ef4c8937b4cd9d" network for pod "kubernetes-dashboard-68ddcc97fc-r2g66": NetworkPlugin cni failed to set up pod "kubernetes-dashboard-68ddcc97fc-r2g66_kube-system" network: json: cannot unmarshal string into Go value of type request.PodResponse
      Warning  FailedCreatePodSandBox  24m                 kubelet, 10.100.97.42  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "fbdc5017dedb438c9ce49e9aafc47cda1adc74c3693efa219fc397f0a2b5f540" network for pod "kubernetes-dashboard-68ddcc97fc-r2g66": NetworkPlugin cni failed to set up pod "kubernetes-dashboard-68ddcc97fc-r2g66_kube-system" network: json: cannot unmarshal string into Go value of type request.PodResponse
      Warning  FailedCreatePodSandBox  24m                 kubelet, 10.100.97.42  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "dea1c332e603aa5521da948a85a17913be382e8721f133ebf5af3204e0f20616" network for pod "kubernetes-dashboard-68ddcc97fc-r2g66": NetworkPlugin cni failed to set up pod "kubernetes-dashboard-68ddcc97fc-r2g66_kube-system" network: json: cannot unmarshal string into Go value of type request.PodResponse
      Warning  FailedCreatePodSandBox  23m                 kubelet, 10.100.97.42  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "cd98c6024d9f1ecdcc32f675d77f4b6da0340927302df95f030fcf91545770da" network for pod "kubernetes-dashboard-68ddcc97fc-r2g66": NetworkPlugin cni failed to set up pod "kubernetes-dashboard-68ddcc97fc-r2g66_kube-system" network: json: cannot unmarshal string into Go value of type request.PodResponse
      Warning  FailedCreatePodSandBox  23m                 kubelet, 10.100.97.42  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e541f93469d66a8d2bf17385ef69cf0118b71f8bcda9140abcee5fcdef55dfe8" network for pod "kubernetes-dashboard-68ddcc97fc-r2g66": NetworkPlugin cni failed to set up pod "kubernetes-dashboard-68ddcc97fc-r2g66_kube-system" network: json: cannot unmarshal string into Go value of type request.PodResponse
      Warning  FailedCreatePodSandBox  23m                 kubelet, 10.100.97.42  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "116fda52d69c71d118e9602177d1c44f9e9b8a353ac14c98c178622d8c28c1cd" network for pod "kubernetes-dashboard-68ddcc97fc-r2g66": NetworkPlugin cni failed to set up pod "kubernetes-dashboard-68ddcc97fc-r2g66_kube-system" network: json: cannot unmarshal string into Go value of type request.PodResponse
      Warning  FailedCreatePodSandBox  22m                 kubelet, 10.100.97.42  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "f84d2fb8be033b48c6d35b316bf9f708dde68d2935e8129df5201a33bf2a230c" network for pod "kubernetes-dashboard-68ddcc97fc-r2g66": NetworkPlugin cni failed to set up pod "kubernetes-dashboard-68ddcc97fc-r2g66_kube-system" network: json: cannot unmarshal string into Go value of type request.PodResponse
      Normal   SandboxChanged          21m (x12 over 25m)  kubelet, 10.100.97.42  Pod sandbox changed, it will be killed and re-created.
      Warning  FailedCreatePodSandBox  23s (x63 over 22m)  kubelet, 10.100.97.42  (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "9624d2476b22166ad59d3fc3466102ee00e2deda2eaf431ba2bdd26b3c02c147" network for pod "kubernetes-dashboard-68ddcc97fc-r2g66": NetworkPlugin cni failed to set up pod "kubernetes-dashboard-68ddcc97fc-r2g66_kube-system" network: json: cannot unmarshal string into Go value of type request.PodResponse
    

    查看节点 kubelet 日志也有类似的报错

    May 21 14:05:14 k8s402 kubelet[2088]: E0521 14:05:14.707791    2088 cni.go:331] Error adding kube-system_kubernetes-dashboard-68ddcc97fc-r2g66/0fbf235fa4d3d96044db023cf7435903bc765f2d5c718a75da4cbd14ea669412 to network kube-ovn/kube-ovn: json: cannot unmarshal string into Go value of type request.PodResponse
    May 21 14:05:14 k8s402 kubelet[2088]: E0521 14:05:14.890156    2088 remote_runtime.go:109] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to set up sandbox container "0fbf235fa4d3d96044db023cf7435903bc765f2d5c718a75da4cbd14ea669412" network for pod "kubernetes-dashboard-68ddcc97fc-r2g66": NetworkPlugin cni failed to set up pod "kubernetes-dashboard-68ddcc97fc-r2g66_kube-system" network: json: cannot unmarshal string into Go value of type request.PodResponse
    May 21 14:05:14 k8s402 kubelet[2088]: E0521 14:05:14.891084    2088 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "kubernetes-dashboard-68ddcc97fc-r2g66_kube-system(5d8638e0-7b8c-11e9-9145-525400cecc16)" failed: rpc error: code = Unknown desc = failed to set up sandbox container "0fbf235fa4d3d96044db023cf7435903bc765f2d5c718a75da4cbd14ea669412" network for pod "kubernetes-dashboard-68ddcc97fc-r2g66": NetworkPlugin cni failed to set up pod "kubernetes-dashboard-68ddcc97fc-r2g66_kube-system" network: json: cannot unmarshal string into Go value of type request.PodResponse
    May 21 14:05:14 k8s402 kubelet[2088]: E0521 14:05:14.892947    2088 kuberuntime_manager.go:693] createPodSandbox for pod "kubernetes-dashboard-68ddcc97fc-r2g66_kube-system(5d8638e0-7b8c-11e9-9145-525400cecc16)" failed: rpc error: code = Unknown desc = failed to set up sandbox container "0fbf235fa4d3d96044db023cf7435903bc765f2d5c718a75da4cbd14ea669412" network for pod "kubernetes-dashboard-68ddcc97fc-r2g66": NetworkPlugin cni failed to set up pod "kubernetes-dashboard-68ddcc97fc-r2g66_kube-system" network: json: cannot unmarshal string into Go value of type request.PodResponse
    May 21 14:05:14 k8s402 kubelet[2088]: E0521 14:05:14.893725    2088 pod_workers.go:190] Error syncing pod 5d8638e0-7b8c-11e9-9145-525400cecc16 ("kubernetes-dashboard-68ddcc97fc-r2g66_kube-system(5d8638e0-7b8c-11e9-9145-525400cecc16)"), skipping: failed to "CreatePodSandbox" for "kubernetes-dashboard-68ddcc97fc-r2g66_kube-system(5d8638e0-7b8c-11e9-9145-525400cecc16)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kubernetes-dashboard-68ddcc97fc-r2g66_kube-system(5d8638e0-7b8c-11e9-9145-525400cecc16)\" failed: rpc error: code = Unknown desc = failed to set up sandbox container \"0fbf235fa4d3d96044db023cf7435903bc765f2d5c718a75da4cbd14ea669412\" network for pod \"kubernetes-dashboard-68ddcc97fc-r2g66\": NetworkPlugin cni failed to set up pod \"kubernetes-dashboard-68ddcc97fc-r2g66_kube-system\" network: json: cannot unmarshal string into Go value of type request.PodResponse"
    May 21 14:05:15 k8s402 kubelet[2088]: W0521 14:05:15.458626    2088 docker_sandbox.go:384] failed to read pod IP from plugin/docker: NetworkPlugin cni failed on the status hook for pod "kubernetes-dashboard-68ddcc97fc-r2g66_kube-system": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "0fbf235fa4d3d96044db023cf7435903bc765f2d5c718a75da4cbd14ea669412"
    May 21 14:05:15 k8s402 kubelet[2088]: I0521 14:05:15.462253    2088 kubelet.go:1930] SyncLoop (PLEG): "kubernetes-dashboard-68ddcc97fc-r2g66_kube-system(5d8638e0-7b8c-11e9-9145-525400cecc16)", event: &pleg.PodLifecycleEvent{ID:"5d8638e0-7b8c-11e9-9145-525400cecc16", Type:"ContainerDied", Data:"0fbf235fa4d3d96044db023cf7435903bc765f2d5c718a75da4cbd14ea669412"}
    May 21 14:05:15 k8s402 kubelet[2088]: W0521 14:05:15.462549    2088 pod_container_deletor.go:75] Container "0fbf235fa4d3d96044db023cf7435903bc765f2d5c718a75da4cbd14ea669412" not found in pod's containers
    May 21 14:05:15 k8s402 kubelet[2088]: I0521 14:05:15.462680    2088 kuberuntime_manager.go:427] No ready sandbox for pod "kubernetes-dashboard-68ddcc97fc-r2g66_kube-system(5d8638e0-7b8c-11e9-9145-525400cecc16)" can be found. Need to start a new one
    May 21 14:05:15 k8s402 kubelet[2088]: W0521 14:05:15.465294    2088 cni.go:309] CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "0fbf235fa4d3d96044db023cf7435903bc765f2d5c718a75da4cbd14ea669412"
    May 21 14:05:16 k8s402 kubelet[2088]: I0521 14:05:16.483182    2088 kubelet.go:1930] SyncLoop (PLEG): "kubernetes-dashboard-68ddcc97fc-r2g66_kube-system(5d8638e0-7b8c-11e9-9145-525400cecc16)", event: &pleg.PodLifecycleEvent{ID:"5d8638e0-7b8c-11e9-9145-525400cecc16", Type:"ContainerStarted", Data:"c1f040e97e950c3579c1297b2843d2021043ba71841aa54be018561704b98bde"}
    May 21 14:05:21 k8s402 kubelet[2088]: E0521 14:05:21.746665    2088 cni.go:331] Error adding kube-system_coredns-55f46dd959-d2r2l/ade331de55afaacd53210011e1969bd85bfe7be7be96e6efa7cce2dae3a38330 to network kube-ovn/kube-ovn: json: cannot unmarshal string into Go value of type request.PodResponse
    May 21 14:05:21 k8s402 kubelet[2088]: E0521 14:05:21.971555    2088 remote_runtime.go:109] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to set up sandbox container "ade331de55afaacd53210011e1969bd85bfe7be7be96e6efa7cce2dae3a38330" network for pod "coredns-55f46dd959-d2r2l": NetworkPlugin cni failed to set up pod "coredns-55f46dd959-d2r2l_kube-system" network: json: cannot unmarshal string into Go value of type request.PodResponse
    May 21 14:05:21 k8s402 kubelet[2088]: E0521 14:05:21.971700    2088 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "coredns-55f46dd959-d2r2l_kube-system(5631ae47-7b8c-11e9-9145-525400cecc16)" failed: rpc error: code = Unknown desc = failed to set up sandbox container "ade331de55afaacd53210011e1969bd85bfe7be7be96e6efa7cce2dae3a38330" network for pod "coredns-55f46dd959-d2r2l": NetworkPlugin cni failed to set up pod "coredns-55f46dd959-d2r2l_kube-system" network: json: cannot unmarshal string into Go value of type request.PodResponse
    May 21 14:05:21 k8s402 kubelet[2088]: E0521 14:05:21.971740    2088 kuberuntime_manager.go:693] createPodSandbox for pod "coredns-55f46dd959-d2r2l_kube-system(5631ae47-7b8c-11e9-9145-525400cecc16)" failed: rpc error: code = Unknown desc = failed to set up sandbox container "ade331de55afaacd53210011e1969bd85bfe7be7be96e6efa7cce2dae3a38330" network for pod "coredns-55f46dd959-d2r2l": NetworkPlugin cni failed to set up pod "coredns-55f46dd959-d2r2l_kube-system" network: json: cannot unmarshal string into Go value of type request.PodResponse
    May 21 14:05:21 k8s402 kubelet[2088]: E0521 14:05:21.971873    2088 pod_workers.go:190] Error syncing pod 5631ae47-7b8c-11e9-9145-525400cecc16 ("coredns-55f46dd959-d2r2l_kube-system(5631ae47-7b8c-11e9-9145-525400cecc16)"), skipping: failed to "CreatePodSandbox" for "coredns-55f46dd959-d2r2l_kube-system(5631ae47-7b8c-11e9-9145-525400cecc16)" with CreatePodSandboxError: "CreatePodSandbox for pod \"coredns-55f46dd959-d2r2l_kube-system(5631ae47-7b8c-11e9-9145-525400cecc16)\" failed: rpc error: code = Unknown desc = failed to set up sandbox container \"ade331de55afaacd53210011e1969bd85bfe7be7be96e6efa7cce2dae3a38330\" network for pod \"coredns-55f46dd959-d2r2l\": NetworkPlugin cni failed to set up pod \"coredns-55f46dd959-d2r2l_kube-system\" network: json: cannot unmarshal string into Go value of type request.PodResponse"
    May 21 14:05:22 k8s402 kubelet[2088]: W0521 14:05:22.573513    2088 docker_sandbox.go:384] failed to read pod IP from plugin/docker: NetworkPlugin cni failed on the status hook for pod "coredns-55f46dd959-d2r2l_kube-system": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "ade331de55afaacd53210011e1969bd85bfe7be7be96e6efa7cce2dae3a38330"
    May 21 14:05:22 k8s402 kubelet[2088]: I0521 14:05:22.579087    2088 kubelet.go:1930] SyncLoop (PLEG): "coredns-55f46dd959-d2r2l_kube-system(5631ae47-7b8c-11e9-9145-525400cecc16)", event: &pleg.PodLifecycleEvent{ID:"5631ae47-7b8c-11e9-9145-525400cecc16", Type:"ContainerDied", Data:"ade331de55afaacd53210011e1969bd85bfe7be7be96e6efa7cce2dae3a38330"}
    May 21 14:05:22 k8s402 kubelet[2088]: W0521 14:05:22.579312    2088 pod_container_deletor.go:75] Container "ade331de55afaacd53210011e1969bd85bfe7be7be96e6efa7cce2dae3a38330" not found in pod's containers
    May 21 14:05:22 k8s402 kubelet[2088]: I0521 14:05:22.580823    2088 kuberuntime_manager.go:427] No ready sandbox for pod "coredns-55f46dd959-d2r2l_kube-system(5631ae47-7b8c-11e9-9145-525400cecc16)" can be found. Need to start a new one
    May 21 14:05:22 k8s402 kubelet[2088]: W0521 14:05:22.583792    2088 cni.go:309] CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "ade331de55afaacd53210011e1969bd85bfe7be7be96e6efa7cce2dae3a38330"
    May 21 14:05:23 k8s402 kubelet[2088]: I0521 14:05:23.598500    2088 kubelet.go:1930] SyncLoop (PLEG): "coredns-55f46dd959-d2r2l_kube-system(5631ae47-7b8c-11e9-9145-525400cecc16)", event: &pleg.PodLifecycleEvent{ID:"5631ae47-7b8c-11e9-9145-525400cecc16", Type:"ContainerStarted", Data:"b4d6e2c91763f28a1be902af75f5dff85b832bb72c944b02741ee375c0ffd30c"}
    
    bug 
    opened by gjmzj 11
  • 1 fix issue of  https://github.com/kubeovn/kube-ovn/issues/1151

    1 fix issue of https://github.com/kubeovn/kube-ovn/issues/1151

    What type of this PR

    Bug fixes
    

    I tried to reimplement the func of ovn-leader checking. But I do not know if it is as your expectation. would you give me some comments?

    Which issue(s) this PR fixes: https://github.com/kubeovn/kube-ovn/issues/1151 Fixes #(1151)

    opened by sxilinux 10
  • Pod ovn-pinger cannot ping the k8s cluster or other pod

    Pod ovn-pinger cannot ping the k8s cluster or other pod

    我组了个1master 3worker的K8s集群,apply kube-ovn后,只能ping通在此主机上的pinger pod,无法ping通其他主机上的pod,在kube-ovn的脚本中发现ping node部分已出现ping loss count的情况,已检查route、subnet、sb和nb等,未发现问题。

    Nodes NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME node1 Ready master 11h v1.18.5 128.5.49.50 CentOS Linux 7 (AltArch) 4.14.0-115.el7a.0.1.aarch64 docker://18.9.8 node2 Ready 11h v1.18.5 128.5.49.40 CentOS Linux 7 (AltArch) 4.14.0-115.el7a.0.1.aarch64 docker://18.9.8 node3 Ready 11h v1.18.5 128.5.49.30 CentOS Linux 7 (AltArch) 4.14.0-115.el7a.0.1.aarch64 docker://18.9.8 node4 Ready 11h v1.18.5 128.5.65.186 CentOS Linux 7 (AltArch) 4.14.0-115.el7a.0.1.aarch64 docker://18.9.8

    Pod NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system coredns-66bff467f8-92kw7 1/1 Running 0 11h 10.16.0.6 node4 kube-system coredns-66bff467f8-w8cnd 1/1 Running 0 11h 10.16.0.8 node2 kube-system etcd-node1 1/1 Running 0 11h 128.5.49.50 node1 kube-system kube-apiserver-node1 1/1 Running 0 11h 128.5.49.50 node1 kube-system kube-controller-manager-node1 1/1 Running 0 11h 128.5.49.50 node1 kube-system kube-ovn-cni-6blqb 1/1 Running 0 11h 128.5.49.40 node2 kube-system kube-ovn-cni-fnhb8 1/1 Running 0 11h 128.5.49.50 node1 kube-system kube-ovn-cni-nhdqd 1/1 Running 0 11h 128.5.49.30 node3 kube-system kube-ovn-cni-q9cg2 1/1 Running 0 11h 128.5.65.186 node4 kube-system kube-ovn-controller-5646ccff47-v722d 1/1 Running 0 11h 128.5.65.186 node4 kube-system kube-ovn-pinger-5xb7c 1/1 Running 0 11h 10.16.0.12 node3 kube-system kube-ovn-pinger-72rm7 1/1 Running 0 11h 10.16.0.9 node2 kube-system kube-ovn-pinger-bvdbk 1/1 Running 0 11h 10.16.0.10 node1 kube-system kube-ovn-pinger-zm4w9 1/1 Running 0 11h 10.16.0.11 node4 kube-system kube-proxy-7h2qs 1/1 Running 0 11h 128.5.49.50 node1 kube-system kube-proxy-b6fvb 1/1 Running 0 11h 128.5.65.186 node4 kube-system kube-proxy-mk96g 1/1 Running 0 11h 128.5.49.30 node3 kube-system kube-proxy-txqc6 1/1 Running 0 11h 128.5.49.40 node2 kube-system kube-scheduler-node1 1/1 Running 0 11h 128.5.49.50 node1 kube-system ovn-central-744b69cf89-kjdph 1/1 Running 0 11h 128.5.49.50 node1 kube-system ovs-ovn-5l4x7 1/1 Running 0 11h 128.5.49.40 node2 kube-system ovs-ovn-nsvwl 1/1 Running 0 11h 128.5.65.186 node4 kube-system ovs-ovn-tt78r 1/1 Running 0 11h 128.5.49.50 node1 kube-system ovs-ovn-zr9kv 1/1 Running 0 11h 128.5.49.30 node3

    Pinger LOG I0818 01:49:59.357840 97493 ping.go:112] ping pod: kube-ovn-pinger-5xb7c 10.16.0.12, count: 3, loss count 0, average rtt 15.01ms I0818 01:50:00.358645 97493 ping.go:112] ping pod: kube-ovn-pinger-72rm7 10.16.0.9, count: 3, loss count 3, average rtt 0.00ms I0818 01:50:01.359383 97493 ping.go:112] ping pod: kube-ovn-pinger-bvdbk 10.16.0.10, count: 3, loss count 3, average rtt 0.00ms I0818 01:50:02.360241 97493 ping.go:112] ping pod: kube-ovn-pinger-zm4w9 10.16.0.11, count: 3, loss count 3, average rtt 0.00ms I0818 01:50:02.360266 97493 ping.go:157] start to check dns connectivity E0818 01:50:12.360642 97493 ping.go:165] failed to resolve dns kubernetes.default, lookup kubernetes.default on 10.96.0.10:53: dial udp 10.96.0.10:53: i/o timeout I0818 01:50:12.360690 97493 ping.go:174] start to check dns connectivity E0818 01:50:22.361119 97493 ping.go:182] failed to resolve dns alauda.cn, lookup alauda.cn on 10.96.0.10:53: dial udp 10.96.0.10:53: i/o timeout I0818 01:50:22.361166 97493 ping.go:132] start to check ping external to 114.114.114.114 I0818 01:50:22.481130 97493 ping.go:145] ping external address: 114.114.114.114, total count: 3, loss count 0, average rtt 18.03ms

    Subnet NAME PROTOCOL CIDR PRIVATE NAT DEFAULT GATEWAYTYPE USED AVAILABLE join IPv4 100.64.0.0/16 false false false distributed 4 65529 ovn-default IPv4 10.16.0.0/16 false true true distributed 6 65527

    Route Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 192.168.1.1 0.0.0.0 UG 103 0 0 enp189s0f1 10.16.0.0 100.64.0.1 255.255.0.0 UG 0 0 0 ovn0 100.64.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ovn0 128.5.0.0 0.0.0.0 255.255.0.0 U 101 0 0 enp189s0f0 172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 192.168.1.0 0.0.0.0 255.255.255.0 U 103 0 0 enp189s0f1

    opened by abigibonam 10
  • failed to resolve dns kubernetes.default, lookup kubernetes.default on 10.96.0.10:53: dial udp 10.96.0.10:53: i/o timeout

    failed to resolve dns kubernetes.default, lookup kubernetes.default on 10.96.0.10:53: dial udp 10.96.0.10:53: i/o timeout

    System:centos7.7 and centos7.6 Kernel:5.5.7-1.el7.elrepo.x86_64 and 3.10.0-1062.12.1.el7.x86_64 Docker:19.03.6 kubernetes:1.17.3 kube-ovn:1.0.0

    setup k8s master by kubeadm

    $ kubeadm init --config=kubeadm-config.yaml configfile,such as:

    piVersion: kubeadm.k8s.io/v1beta2
    bootstrapTokens:
    - groups:
      - system:bootstrappers:kubeadm:default-node-token
      token: abcdef.0123456789abcdef
      ttl: 24h0m0s
      usages:
      - signing
      - authentication
    kind: InitConfiguration
    localAPIEndpoint:
      advertiseAddress: 192.168.122.8
      bindPort: 6443
    nodeRegistration:
      criSocket: /var/run/dockershim.sock
      name: k8s-master01
      taints:
      - effect: PreferNoSchedule
        key: node-role.kubernetes.io/master
    ---
    apiServer:
      timeoutForControlPlane: 4m0s
    apiVersion: kubeadm.k8s.io/v1beta2
    certificatesDir: /etc/kubernetes/pki
    clusterName: kubernetes
    controllerManager: {}
    dns:
      type: CoreDNS
    etcd:
      local:
        dataDir: /var/lib/etcd
    imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
    kind: ClusterConfiguration
    kubernetesVersion: v1.17.3
    networking:
      dnsDomain: cluster.local
      serviceSubnet: 10.96.0.0/12
    scheduler: {}
    
    ---
    apiVersion: kubeproxy.config.k8s.io/v1alpha1
    kind: KubeProxyConfiguration
    featureGates:
      SupportIPVSProxyMode: true
    mode: ipvs
    

    run nginx pod

    $ kubectl run nginx --image=nginx

    operate ping

    $ kubectl exec -it nginx-6db489d4b7-x898l bash ping www.baidu.com is ok

    edit kubelet config

    $ vi /etc/sysconfig/kubelet KUBELET_EXTRA_ARGS="--fail-swap-on=false --runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice --eviction-hard=memory.available<5%,nodefs.available<10%,nodefs.inodesFree<10% --node-status-update-frequency=10s --eviction-pressure-transition-period=30s --enforce-node-allocatable=pods,kube-reserved --cgroup-driver=cgroupfs --kube-reserved=cpu=1,memory=2Gi,ephemeral-storage=10Gi --kube-reserved-cgroup=/system.slice/kubelet.service --max-open-files=10000"

    $ vi /usr/lib/systemd/system/kubelet.service [Service] ExecStartPre=/bin/mkdir -p /sys/fs/cgroup/cpuset/system.slice/kubelet.service#新增 ExecStartPre=/bin/mkdir -p /sys/fs/cgroup/hugetlb/system.slice/kubelet.service#新增

    restart kubelet

    $ systemctl daemon-reload && systemctl restart kubelet

    operate ping

    $ kubectl exec -it nginx-6db489d4b7-x898l bash ping www.baidu.com is not ok

    kube-ovn-pinger log

    I0307 04:15:25.118370 13026 ping.go:167] start to check apiserver connectivity I0307 04:15:25.119306 13026 ping.go:176] connect to apiserver success in 0.93ms I0307 04:15:25.119326 13026 ping.go:40] start to check node connectivity I0307 04:15:25.922280 13026 ping.go:62] ping node: k8s-master01 192.168.122.201, count: 3, loss count 0, average rtt 1.97ms I0307 04:15:25.922309 13026 ping.go:78] start to check pod connectivity I0307 04:15:26.530532 13026 ping.go:105] ping pod: kube-ovn-pinger-z24rt 10.16.0.4, count: 3, loss count 0, average rtt 0.34ms I0307 04:15:26.530612 13026 ping.go:150] start to check dns connectivity E0307 04:15:36.532114 13026 ping.go:158] failed to resolve dns kubernetes.default, lookup kubernetes.default on 10.96.0.10:53: dial udp 10.96.0.10:53: i/o timeout I0307 04:15:36.532183 13026 ping.go:125] start to check ping external to 114.114.114.114 I0307 04:15:37.244788 13026 ping.go:138] ping external address: 114.114.114.114, total count: 3, loss count 0, average rtt 86.73ms

    this operate use weave is work

    opened by Macolm6 10
  • 【kubeovn1.10.1】同一台宿主机 POD 之间 TCP 不通

    【kubeovn1.10.1】同一台宿主机 POD 之间 TCP 不通

    Expected Behavior

    同一台宿主机 POD 之间 TCP 应该是通的。

    Actual Behavior

    同一台宿主机 POD 之间 TCP 不通

    Steps to Reproduce the Problem

    1. 在 vpc: ovn-cluster 的 subnet:ovn-default 下创建两个 POD,POD yaml如下 ` apiVersion: apps/v1 kind: Deployment metadata: name: deploy spec: selector: matchLabels: app: deploy replicas: 2 template: metadata: labels: app: deploy annotations: ovn.kubernetes.io/default_route: "true" ovn.kubernetes.io/logical_switch: ovn-default spec: nodeSelector: kubernetes.io/hostname: XXX containers:
      • name: centos image: centos:7 command: ["bash","-c","sleep 365d"] imagePullPolicy: Always tolerations:
      • key: key value: value effect: NoSchedule `
    2. 在其中一个 POD(POD-1) 中启动 tcp server,命令为:nc -l -t 12345
    3. 在另一个 POD(POD-2) 中启动 tcp client,命令为:ncat 172.10.0.97 12345
    4. 发现 tcp client 和 tcp server 无法联通。
    5. 在宿主机抓 POD-2 veth 流量如下图 image

    根据抓包分析,tcp 三次握手正常进行,但是当 tcp client 发了一个包后,tcp server 换了一个端口 5511 (本应该是 12345)来回包??

    结果 tcp 链接断掉。


    经过测试:

    • 1.9.2 在 "Rocky Linux 8.6 (Green Obsidian)" 4.18.0-372.9.1.el8.x86_64 不会出现上述问题
    • 1.10.1 在 "Rocky Linux 8.5 (Green Obsidian)" 4.18.0-348.23.1.el8_5.x86_64 不会出现上述问题
    • 1.10.1 在 "Rocky Linux 8.6 (Green Obsidian)" 4.18.0-372.9.1.el8.x86_64 会出现上述问题

    Additional Info

    • Kubernetes version:

      Output of kubectl version:

      Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.6", GitCommit:"ad3338546da947756e8a88aa6822e9c11e7eac22", GitTreeState:"clean", BuildDate:"2022-04-14T08:49:13Z", GoVersion:"go1.17.9", Compiler:"gc", Platform:"linux/amd64"}
      Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.6", GitCommit:"ad3338546da947756e8a88aa6822e9c11e7eac22", GitTreeState:"clean", BuildDate:"2022-04-14T08:43:11Z", GoVersion:"go1.17.9", Compiler:"gc", Platform:"linux/amd64"}
      
    • kube-ovn version:

      v1.10.1,commit 4935fa6adc8a0088b173603e819cec274996ed29
      
    • operation-system/kernel version:

      Output of awk -F '=' '/PRETTY_NAME/ { print $2 }' /etc/os-release: Output of uname -r:

      "Rocky Linux 8.6 (Green Obsidian)"
      4.18.0-372.9.1.el8.x86_64
      
    opened by gugulee 0
  • 【kubeovn1.10版本】使用v1.multus-cni.io/default-network创建macvlan网络的pod,创建不成功

    【kubeovn1.10版本】使用v1.multus-cni.io/default-network创建macvlan网络的pod,创建不成功

    Expected Behavior

    使用v1.multus-cni.io/default-network创建macvlan网络的pod

    Actual Behavior

    pod创建失败,改用例再kube-ovn1.9版本可以创建成功

    Steps to Reproduce the Problem

    1.创建kube-ovn的macvlan网络。 2.使用v1.multus-cni.io/default-network: nsb/netb绑定网络 3.创建pod

    Additional Info

    cat netB2.yaml
    apiVersion: "k8s.cni.cncf.io/v1"
    kind: NetworkAttachmentDefinition
    metadata:
    name: netb2
    namespace: nsb
    spec:
    config: '{
        "cniVersion": "0.3.0",
        "type": "macvlan",
        "master": "ens192",
        "mode": "bridge",
        "ipam": {
          "type": "kube-ovn",
          "server_socket": "/run/openvswitch/kube-ovn-daemon.sock",
          "provider": "netb2.nsb"
        }
      }'
    ---
    apiVersion: kubeovn.io/v1
    kind: Subnet
    metadata:
    name: netb2
    spec:
    protocol: IPv4
    provider: netb2.nsb
    cidrBlock: 192.168.22.0/24
    gateway: 192.168.22.1
    
    
    cat pod42.yaml
    apiVersion: v1
    kind: Pod
    metadata:
    name: pod42
    namespace: nsb
    annotations:
      #k8s.v1.cni.cncf.io/networks: nsb/netb2
      v1.multus-cni.io/default-network: nsb/netb2
    spec:
    containers:
    - name: nginx
      command: ["/bin/sh", "-c", "sleep 2000000000000"]
      image: registry.local:9001/busybox:latest
    
    
    
    • Kubernetes version:

      Output of kubectl version:

      [[email protected] 951-1]# kubectl version
      Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.5", GitCommit:"aea7bbadd2fc0cd689de94a54e5b7b758869d691", GitTreeState:"clean", BuildDate:"2021-09-15T21:10:45Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
      Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.5", GitCommit:"aea7bbadd2fc0cd689de94a54e5b7b758869d691", GitTreeState:"clean", BuildDate:"2021-09-15T21:04:16Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
      
      
    • kube-ovn version:

      kube-ovn1.10
      Image:         kubeovn/kube-ovn:v1.10.1
      
    • operation-system/kernel version:

      Output of awk -F '=' '/PRETTY_NAME/ { print $2 }' /etc/os-release: Output of uname -r:

      [[email protected] 951-1]# awk -F '=' '/PRETTY_NAME/ { print $2 }' /etc/os-release
      "CentOS Linux 7 (Core)"
      [[email protected] 951-1]# uname -r
      5.4.134-1.el7.elrepo.x86_64
      
      
    kubectl -n kube-system logs -f kube-ovn-controller-6c444fc774-kv2w5 
    ....
    
    I0625 14:29:41.230797       6 pod.go:315] handle add pod nsb/pod42
    I0625 14:29:41.232865       6 ipam.go:51] allocate v4 192.168.22.4 v6  mac 00:00:00:83:03:48 for nsb/pod42
    I0625 14:29:41.248912       6 pod.go:321] take 18 ms to handle add pod nsb/pod42
    
    
    [[email protected] 951-1]# kubectl -n nsb describe pod pod42
    Name:         pod42
    Namespace:    nsb
    Priority:     0
    Node:         kk117/111.111.203.117
    Start Time:   Sat, 25 Jun 2022 14:29:41 +0800
    Labels:       <none>
    Annotations:  netb2.nsb.kubernetes.io/allocated: true
                  netb2.nsb.kubernetes.io/cidr: 192.168.22.0/24
                  netb2.nsb.kubernetes.io/gateway: 192.168.22.1
                  netb2.nsb.kubernetes.io/ip_address: 192.168.22.4
                  netb2.nsb.kubernetes.io/logical_switch: netb2
                  netb2.nsb.kubernetes.io/mac_address: 00:00:00:83:03:48
                  netb2.nsb.kubernetes.io/pod_nic_type: veth-pair
                  v1.multus-cni.io/default-network: nsb/netb2
    Status:       Pending
    IP:
    IPs:          <none>
    Containers:
      nginx:
        Container ID:
        Image:         registry.local:9001/busybox:latest
        Image ID:
        Port:          <none>
        Host Port:     <none>
        Command:
          /bin/sh
          -c
          sleep 2000000000000
        State:          Waiting
          Reason:       ContainerCreating
        Ready:          False
        Restart Count:  0
        Environment:    <none>
        Mounts:
          /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zl46q (ro)
    Conditions:
      Type              Status
      Initialized       True
      Ready             False
      ContainersReady   False
      PodScheduled      True
    Volumes:
      kube-api-access-zl46q:
        Type:                    Projected (a volume that contains injected data from multiple sources)
        TokenExpirationSeconds:  3607
        ConfigMapName:           kube-root-ca.crt
        ConfigMapOptional:       <nil>
        DownwardAPI:             true
    QoS Class:                   BestEffort
    Node-Selectors:              <none>
    Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 120s
                                 node.kubernetes.io/unreachable:NoExecute op=Exists for 120s
    Events:
      Type     Reason                  Age                   From               Message
      ----     ------                  ----                  ----               -------
      Normal   Scheduled               38m                   default-scheduler  Successfully assigned nsb/pod42 to kk117
      Warning  FailedCreatePodSandBox  37m                   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "a58cfb79dc83492e8813944cdd014a825f45e8de92b742052f57af566e2cc0fc" network for pod "pod42": networkPlugin cni failed to set up pod "pod42_nsb" network: [nsb/pod42:netb2]: error adding container to network "netb2": request ip return 500 route is not ready for pod nsb/pod42 provider netb2.nsb, please see kube-ovn-controller logs to find errors, failed to clean up sandbox container "a58cfb79dc83492e8813944cdd014a825f45e8de92b742052f57af566e2cc0fc" network for pod "pod42": networkPlugin cni failed to teardown pod "pod42_nsb" network: delegateDel: error invoking ConflistDel - "kube-ovn": conflistDel: error converting the raw bytes into a conflist: error parsing configuration list: no 'plugins' key]
      Warning  FailedCreatePodSandBox  37m                   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "a503f694b6e2511ca3610fb306487a5a2cefc9f53622fc794b67f8739ffe3954" network for pod "pod42": networkPlugin cni failed to set up pod "pod42_nsb" network: [nsb/pod42:netb2]: error adding container to network "netb2": request ip return 500 route is not ready for pod nsb/pod42 provider netb2.nsb, please see kube-ovn-controller logs to find errors, failed to clean up sandbox container "a503f694b6e2511ca3610fb306487a5a2cefc9f53622fc794b67f8739ffe3954" network for pod "pod42": networkPlugin cni failed to teardown pod "pod42_nsb" network: delegateDel: error invoking ConflistDel - "kube-ovn": conflistDel: error converting the raw bytes into a conflist: error parsing configuration list: no 'plugins' key]
      Warning  FailedCreatePodSandBox  37m                   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "596c9f3e74cf1c3fe76cf01a869fe9f8d71dd1e348704a7f685774edb3049525" network for pod "pod42": networkPlugin cni failed to set up pod "pod42_nsb" network: [nsb/pod42:netb2]: error adding container to network "netb2": request ip return 500 route is not ready for pod nsb/pod42 provider netb2.nsb, please see kube-ovn-controller logs to find errors, failed to clean up sandbox container "596c9f3e74cf1c3fe76cf01a869fe9f8d71dd1e348704a7f685774edb3049525" network for pod "pod42": networkPlugin cni failed to teardown pod "pod42_nsb" network: delegateDel: error invoking ConflistDel - "kube-ovn": conflistDel: error converting the raw bytes into a conflist: error parsing configuration list: no 'plugins' key]
    
    bug 
    opened by aslov1 1
  • ci: fix golangci-lint

    ci: fix golangci-lint

    What type of this PR

    • CI
    v3.0.0+ requires explicit setup-go installation step prior to using this action: uses: actions/[email protected]
    The skip-go-installation option has been removed.
    
    ci 
    opened by zhangzujian 0
  • VIP lost after k8s service update

    VIP lost after k8s service update

    Expected Behavior

    service更新功能正常

    Actual Behavior

    service更新后,vip丢失

    Steps to Reproduce the Problem

    1.当前环境

    [[email protected] home]# kubectl get svc
    NAME                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE
    demo                ClusterIP   10.105.42.178   <none>        80/TCP                   5d18h
    
    NAME                ENDPOINTS                                                AGE
    demo                10.16.0.31:80                                            5d18h
    
    [[email protected] home]# kubectl ko nbctl lb-list
    UUID                                    LB                  PROTO      VIP                     IPs
    84200011-4c15-4815-9a34-0e245bf10208    cluster-tcp-load    tcp       10.105.42.178:80        10.16.0.31:80
    

    2.更新service

    [[email protected] home]# kubectl edit svc demo 
    

    3.再次查看lb-list,vip消失

    Additional Info

    • kube-ovn version: 1.10

    代码这里,会对当前的vip进行删除: https://github.com/kubeovn/kube-ovn/blob/16d28f755b22704427c297918c01119955ed6e6d/pkg/controller/service.go#L280

    bug 
    opened by ShaPoHun 1
  • ns multi subnet logic use subnet not speciied to pod to allocate ip

    ns multi subnet logic use subnet not speciied to pod to allocate ip

    Expected Behavior

    1.9 之前 subnet 应该是 not namespaced, vpc, subnet模板中定义属于哪个ns, 1.10 和 1.11 看起来ns支持了多子网,但是双网卡分配ip时,发现会从非指定子网分配ip 且ns未更新annos subnet list (看起来好像是添加subnet未触发update ns annos) 期望 多网卡场景,也仅从pod指定的subnet分配ip

    image

    Actual Behavior

    Steps to Reproduce the Problem

    1. just delete vpc nat gw pod, and see kube-ovn-controller log
    
    [[email protected] vpc]# cat 000-ns.yml
    apiVersion: v1
    kind: Namespace
    metadata:
      name: bb1
    
    [[email protected] vpc]# cat 001-nat-gw-cm.yml 
    kind: ConfigMap
    apiVersion: v1
    data:
      image: kubeovn/vpc-nat-gateway:v1.10.0
      enable-vpc-nat-gw: "true"
    metadata:
      name: ovn-vpc-nat-gw-config
      namespace: kube-system
    [[email protected] vpc]# cat 02-dual-vpc-subnet.yml
    apiVersion: kubeovn.io/v1
    kind: Subnet
    metadata:
      name: bb1-subnet1
    spec:
      cidrBlock: 192.168.0.0/24,fd00:192:168::/120
      default: false
      disableGatewayCheck: false
      disableInterConnection: true
      gatewayNode: ""
      gatewayType: distributed
      natOutgoing: false
      private: false
      protocol: Dual
      provider: ovn
      vpc: bb1
      namespaces:
      - bb1
    [[email protected] vpc]# cat 03-nat-gw.yaml
    kind: VpcNatGateway
    apiVersion: kubeovn.io/v1
    metadata:
      name: gw1
    spec:
      vpc: bb1
      subnet: bb1-subnet1
      lanIp: 192.168.0.254
    
    
    
    
    [[email protected] centos]# kubectl  get ns bb1 -o yaml
    apiVersion: v1
    kind: Namespace
    metadata:
      annotations:
        kubectl.kubernetes.io/last-applied-configuration: |
          {"apiVersion":"v1","kind":"Namespace","metadata":{"annotations":{},"name":"bb1"}}
        ovn.kubernetes.io/cidr: 192.168.0.0/24,fd00:192:168::/120
        ovn.kubernetes.io/exclude_ips: 192.168.0.1,fd00:192:168::1
        ovn.kubernetes.io/logical_switch: bb1-subnet1  # no ovn-default
      creationTimestamp: "2022-06-20T02:11:38Z"
      labels:
        kubernetes.io/metadata.name: bb1
      name: bb1
      resourceVersion: "4582471"
      uid: 27110466-cad0-4a8e-b221-e1ec6a4e6078
    spec:
      finalizers:
      - kubernetes
    status:
      phase: Active
    
    
    [[email protected] centos]# kubectl  get subnet | grep ovn-default
    ovn-default                ovn                                    ovn-cluster   IPv4       10.123.0.0/16                       false     true    true      distributed   87       65446         0        0             ["10.123.0.1"]
    [[email protected] centos]# kubectl  get subnet | grep bb
    bb1-subnet1                ovn                                    bb1           Dual       192.168.0.0/24,fd00:192:168::/120   false     false   false     distributed   2        251           2        251           ["192.168.0.1","fd00:192:168::1"]
    
    
    

    Additional Info

    • Kubernetes version:

      Output of kubectl version:

      (paste your output here)
      
    • kube-ovn version:

    1.10

    
    - operation-system/kernel version:
    
    **Output of `awk -F '=' '/PRETTY_NAME/ { print $2 }' /etc/os-release`:**
    **Output of `uname -r`:**
    

    centos8 5.4.196-1.el8.elrepo.x86_64

    
    <!-- Any other additional information -->
    
    
    
    ![image](https://user-images.githubusercontent.com/7981158/174605672-94983514-ab0e-4f26-a17c-10e7c31c8496.png)
    
    ![image](https://user-images.githubusercontent.com/7981158/174605761-6ee9cde9-a69c-4d00-a254-dfb28e3ada86.png)
    
    
    
    getPodKubeovnNets getNsAvailableSubnets 感觉上, 这两个方法融合一下可能会更好
    
    
    
    
    
    
    ``` yaml
    
    # kubectl  get po -nkube-system            vpc-nat-gw-gw1-54b74748d8-dg4z4    -o yaml
    
    
    apiVersion: v1
    kind: Pod
    metadata:
    annotations:
      k8s.v1.cni.cncf.io/network-status: |-
        [{
            "name": "kube-ovn",
            "ips": [
                "192.168.0.254",
                "fd00:192:168::4"
            ],
            "default": true,
            "dns": {}
        },{
            "name": "kube-system/ovn-vpc-external-network",
            "interface": "net1",
            "ips": [
                "10.122.193.73"
            ],
            "mac": "62:99:92:31:1a:d2",
            "dns": {}
        }]
      k8s.v1.cni.cncf.io/networks: kube-system/ovn-vpc-external-network
      k8s.v1.cni.cncf.io/networks-status: |-
        [{
            "name": "kube-ovn",
            "ips": [
                "192.168.0.254",
                "fd00:192:168::4"
            ],
            "default": true,
            "dns": {}
        },{
            "name": "kube-system/ovn-vpc-external-network",
            "interface": "net1",
            "ips": [
                "10.122.193.73"
            ],
            "mac": "62:99:92:31:1a:d2",
            "dns": {}
        }]
      ovn-vpc-external-network.kube-system.kubernetes.io/allocated: "true"
      ovn-vpc-external-network.kube-system.kubernetes.io/cidr: 10.122.192.0/19
      ovn-vpc-external-network.kube-system.kubernetes.io/gateway: 10.122.223.254
      ovn-vpc-external-network.kube-system.kubernetes.io/ip_address: 10.122.193.73
      ovn-vpc-external-network.kube-system.kubernetes.io/logical_switch: ovn-vpc-external-network
      ovn-vpc-external-network.kube-system.kubernetes.io/mac_address: 00:00:00:BA:94:65
      ovn-vpc-external-network.kube-system.kubernetes.io/pod_nic_type: veth-pair
      ovn.kubernetes.io/allocated: "true"
      ovn.kubernetes.io/cidr: 192.168.0.0/24,fd00:192:168::/120
      ovn.kubernetes.io/gateway: 192.168.0.1,fd00:192:168::1
      ovn.kubernetes.io/ip_address: 192.168.0.254,fd00:192:168::4
      ovn.kubernetes.io/logical_router: bb1
      ovn.kubernetes.io/logical_switch: bb1-subnet1
      ovn.kubernetes.io/mac_address: 00:00:00:70:D0:17
      ovn.kubernetes.io/pod_nic_type: veth-pair
      ovn.kubernetes.io/routed: "true"
      ovn.kubernetes.io/vpc_nat_gw: gw1
      ovn.kubernetes.io/vpc_nat_gw_init: "true"
    creationTimestamp: "2022-06-20T12:42:51Z"
    generateName: vpc-nat-gw-gw1-54b74748d8-
    labels:
      app: vpc-nat-gw-gw1
      ovn.kubernetes.io/vpc-nat-gw: "true"
      pod-template-hash: 54b74748d8
    name: vpc-nat-gw-gw1-54b74748d8-dg4z4
    namespace: kube-system
    ownerReferences:
    - apiVersion: apps/v1
      blockOwnerDeletion: true
      controller: true
      kind: ReplicaSet
      name: vpc-nat-gw-gw1-54b74748d8
      uid: fdf408e1-8cc4-46bd-9547-9cfaf8de5465
    resourceVersion: "5037175"
    uid: ea016627-6542-4a82-a3e8-be40d039cd62
    spec:
    containers:
    - args:
      - -c
      - while true; do sleep 10000; done
      command:
      - bash
      image: kubeovn/vpc-nat-gateway:v1.10.0
      imagePullPolicy: IfNotPresent
      name: vpc-nat-gw
      resources: {}
      securityContext:
        allowPrivilegeEscalation: true
        privileged: true
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: kube-api-access-jgxhg
        readOnly: true
    dnsPolicy: ClusterFirst
    enableServiceLinks: true
    nodeName: hci-compute-10
    preemptionPolicy: PreemptLowerPriority
    priority: 0
    restartPolicy: Always
    schedulerName: default-scheduler
    securityContext: {}
    serviceAccount: default
    serviceAccountName: default
    terminationGracePeriodSeconds: 30
    tolerations:
    - effect: NoExecute
      key: node.kubernetes.io/not-ready
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: node.kubernetes.io/unreachable
      operator: Exists
      tolerationSeconds: 300
    volumes:
    - name: kube-api-access-jgxhg
      projected:
        defaultMode: 420
        sources:
        - serviceAccountToken:
            expirationSeconds: 3607
            path: token
        - configMap:
            items:
            - key: ca.crt
              path: ca.crt
            name: kube-root-ca.crt
        - downwardAPI:
            items:
            - fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
              path: namespace
    status:
    conditions:
    - lastProbeTime: null
      lastTransitionTime: "2022-06-20T12:42:51Z"
      status: "True"
      type: Initialized
    - lastProbeTime: null
      lastTransitionTime: "2022-06-20T12:43:35Z"
      status: "True"
      type: Ready
    - lastProbeTime: null
      lastTransitionTime: "2022-06-20T12:43:35Z"
      status: "True"
      type: ContainersReady
    - lastProbeTime: null
      lastTransitionTime: "2022-06-20T12:42:51Z"
      status: "True"
      type: PodScheduled
    containerStatuses:
    - containerID: containerd://f10668555dc6e9132066af2e7bb5c7c3d5c049fb32710e16683aa875dea65fad
      image: docker.io/kubeovn/vpc-nat-gateway:v1.10.0
      imageID: docker.io/kubeovn/[email protected]:4d30e414ade03c9deb3712b3d3e282be072b0528a3bd12f0b30d316271cbd23a
      lastState: {}
      name: vpc-nat-gw
      ready: true
      restartCount: 0
      started: true
      state:
        running:
          startedAt: "2022-06-20T12:43:35Z"
    hostIP: 10.122.16.110
    phase: Running
    podIP: 192.168.0.254
    podIPs:
    - ip: 192.168.0.254
    - ip: fd00:192:168::4
    qosClass: BestEffort
    startTime: "2022-06-20T12:42:51Z"
    
    
    opened by bobz965 5
Releases(v1.10.1)
  • v1.10.1(Jun 19, 2022)

    v1.10.1 (2022-06-19)

    • 4935fa6a monitor dns in cilium e2e (#1597)
    • 3dc29041 prepare for release 1.10.1
    • e459688e ci: build amd64 images without avx512 (#1584)
    • d7144681 update ovs health check, delete connection to ovn sb db (#1588)
    • cfbe55e0 fix: all cluster pod will be in podadd queue (#1587)
    • 08ba4215 fix pod could not be ready (#1562)
    • c453b7ac fix: delete pod panic when delete vm or statefulset. (#1565)
    • 77044e3d fix: clean CRDs introduced by new vpc-nat-gateway (#1563)
    • e35f90f1 do not gc vm pod lsp when vm still exists (#1558)
    • adabd853 do not delete static routes on controller startup (#1560)
    • 4348e58f replace ovn-nbctl daemon with libovsdb in frequent operations (#1544)
    • 4cacb4b9 fix exec cmd in vpc nat gateway (#1556)
    • 0ed681af CNI: do not return route if nic is not eth0 (#1555)
    • 96f232d4 do not nat packets for incoming traffic when service externalTrafficPolicy is Local
    • bbb8a697 exit kube-ovn-controller on stopped leading (#1536)
    • 4b0bd69e tmp cancel cilium external svc test (#1531)

    Contributors

    • hzma
    • lut777
    • xujunjie-cover
    • zhangzujian
    • 刘睿华
    • 张祖建
    Source code(tar.gz)
    Source code(zip)
  • v1.9.4(Jun 19, 2022)

    v1.9.4 (2022-06-19)

    • c85ab203 ci: disable cilium e2e for release
    • 0a841aa1 prepare for release 1.9.4
    • f99f4e81 update ovs health check, delete connection to ovn sb db (#1588)
    • 82d7dd37 fix: all cluster pod will be in podadd queue (#1587)
    • 3c68cb9b fix pod could not be ready (#1562)
    • f39ff7a8 fix: delete pod panic when delete vm or statefulset. (#1565)
    • 4c60872f fix: keep vm's and statefulset's ips when user specified subnet (#1520)
    • 81781a01 do not gc vm pod lsp when vm still exists (#1558)
    • 4a28c014 fix exec cmd in vpc nat gateway (#1556)
    • 67db2bf3 CNI: do not return route if nic is not eth0 (#1555)
    • d5fce51d exit kube-ovn-controller on stopped leading (#1536)
    • 05a4b4dc remove name for default drop acl in networkpolicy (#1522)
    • 6fcc1975 tmp cancel cilium external svc test (#1531)
    • fe3bb3e5 move dumb-init from base images to kube-ovn image

    Contributors

    • hzma
    • lut777
    • xujunjie-cover
    • 刘睿华
    • 张祖建
    Source code(tar.gz)
    Source code(zip)
  • v1.8.7(Jun 19, 2022)

    v1.8.7 (2022-06-19)

    • 46987551 prepare for release 1.8.7
    • b6796d09 cni handler: do not wait routed annotation for net1 (#1586)
    • f5c3ed3f fix adding static route after LSP deletion (#1571)
    • f7ee860b fix duplicate netns parameter (#1580)
    • 0a3468b1 do not gc vm pod lsp when vm still exists (#1558)
    • d453add3 fix exec cmd in vpc nat gateway (#1556)
    • 8303ace0 CNI: do not return route if nic is not eth0 (#1555)
    • bc758245 exit kube-ovn-controller on stopped leading (#1536)
    • c51b09e8 remove name for default drop acl in networkpolicy (#1522)
    • 9fe8cfcd move dumb-init from base images to kube-ovn image
    • 2a8a45a1 fix defunct ovn-nbctl daemon

    Contributors

    • hzma
    • zhangzujian
    • 张祖建
    Source code(tar.gz)
    Source code(zip)
  • v1.10.0(May 15, 2022)

    New Feature

    • Windows support
    • Reduce ovs-ovn restart downtime
    • Submariner for multi clusters service
    • Iptables eip nats splits
    • support to add multiple subnets for a namespace
    • add custom ACLs for subnet
    • support dpdk hybrid node
    • keep ip for kubevirt pod
    • Support kubevirt vm live migrate for pod static ip
    • feat: support DHCP
    • feat: vpc peering connection
    • add 'virtual' port for vip
    • LSP forwarding external Layer-2 packets
    • update provider network via node annotation
    • feat: add webhook to check subnet deletion
    • add webhook validate the vpc resource whether it can be deleted.

    Performance

    • optimize ovs request in cni
    • optimize node port-group check
    • optimize IPAM initialization
    • manual compile method for ubuntu20.04
    • add repo for tunning packages
    • add kube-ovn-controller switch for EIP and SNAT
    • use router policy for distributed/centralized gateway

    Security

    • update alpine to fix CVE-2022-1271

    Monitoring

    • Add env-check
    • add sb/nb db check bash script
    • update nodeips for restore cmd in ko plugin
    • add restore process for ovn nb db
    • add reset porocess for ovs interface metrics
    • add metric for ovn nb/sb db status

    Bugfix

    • fix defunct ovn-nbctl daemon
    • keep vm's and statefulset's ips when user specified subnet
    • add netem qos when create pod
    • handle the case of error node cidr
    • ovs trace flow always ends with controller action
    • add empty chassis check in ovn db
    • delete ipam record when gc lsp
    • fix wrong vpc-nat-gateway arm image
    • fix pod annotation may override by patch
    • fix: workqueue_depth should show count not rate
    • add delete ovs pods after restore nb db
    • delete monitor noexecute toleration
    • fix routes for packets from Pods to other nodes
    • masquerade packets from Pods to service IP
    • modify init ipam by ip crd only for sts pod
    • fix adding key to delete Pod queue
    • fix IPAM initialization
    • ignore all link local unicast addresses/routes
    • fix error handling for netlink.AddrDel
    • fix provider-networks status
    • recover ips CR on IPAM initialization
    • fix: do not recreate port for terminating pods
    • avoid frequent ipset update
    • add reset for kube-ovn-monitor metrics
    • fix: The underlay physical gateway config by external-gw-addr when use snat&eip
    • fix external egress gateway
    • add missing link scope routes in vpc-nat-gateway
    • modify ipam v6 release ip problem
    • skip ping gateway for pods during live migration
    • don't check conflict for migration pod with only static mac
    • fix usage of ovn commands
    • fix OVS bridge with bond port in mode 6
    • fix underlay subnet in custom VPC
    • configurable kube-ovn cni config filename
    • replace ecmp dp_hash with hash by src_ip
    • set up tunnel correctly in hybrid mode
    • check static route conflict
    • transfer IP/route earlier in OVS startup
    • fix: validate statefulset pod by name

    Mics

    • use inc-engine/recompute instead of deprecated recomput
    • update kind to v0.13.0
    • refactor logical router routes
    • update ovn and ovs
    • add routed check in circulation
    • create ip crd in kube-ovn-controller
    • update cni version to 1.0
    • VIP is decoupled from port security
    • Use go to rerimplement ovn-is-leader.sh
    • Replace command health check with k8s tcpSocket check
    • add gateway check after update subnet
    Source code(tar.gz)
    Source code(zip)
    kube-ovn-win64.zip(71.63 MB)
  • v1.9.3(May 13, 2022)

    Enhancement

    • optimize IPAM initialization
    • reduce ovs-ovn restart downtime
    • optimize node port-group check
    • optimize ovs request in cni

    Bugfix

    • delete ipam record and static route when gc lsp
    • fix: ovs trace flow always ends with controller action
    • fix defunct ovn-nbctl daemon
    Source code(tar.gz)
    Source code(zip)
  • v1.8.6(May 13, 2022)

    Enhancement

    • add kube-ovn-controller switch for EIP and SNAT
    • log: show the reason if get gw node failed
    • append metrics
    • add env-check
    • add delete ovs pods after restore nb db
    • optimize IPAM initialization
    • reduce ovs-ovn restart downtime

    Bugfix

    • recover ips CR on IPAM initialization
    • fix provider-networks status
    • support alloc static ip from any subnet after ns supports multi subnets
    • replace pod name when create ip crd
    • fix error handling for netlink.AddrDel
    • ignore all link local unicast addresses/routes
    • fix IPAM initialization
    • fix adding key to delete Pod queue
    • modify init ipam by ip crd only for sts pod
    • add routed check in circulation
    • delete monitor noexecute toleration
    • delete ipam record and static route when gc lsp
    • fix: ovs trace flow always ends with controller action

    Security

    • update alpine to fix CVE-2022-1271
    • ignore cni cve
    • CVE-2022-27191
    Source code(tar.gz)
    Source code(zip)
  • v1.9.2(Apr 26, 2022)

    Bugfix

    • fix: wrong vpc-nat-gateway arm image
    • add delete ovs pods after restore nb db
    • delete monitor noexecute toleration
    • masquerade packets from Pods to service IP
    • modify init ipam by ip crd only for sts pod
    • modify webhook img to independent image
    • fix adding key to delete Pod queue
    • fix IPAM initialization
    • ignore all link local unicast addresses/routes
    • fix error handling for netlink.AddrDel
    • replace pod name when create ip crd
    • fix provider-networks status
    • recover ips CR on IPAM initialization
    • fix: do not recreate port for terminating pods
    • avoid frequent ipset update
    • fix: The underlay physical gateway config by external-gw-addr when use snat&eip
    • add reset for kube-ovn-monitor metrics
    • check the cidr format whether is correct
    • fix external egress gateway
    • add missing link scope routes in vpc-nat-gateway

    New Feature

    • add env-check
    • add kube-ovn-controller switch for EIP and SNAT
    • add routed check in circulation
    • log: show the reason if get gw node failed
    • support keep-vm-ip and live-migrate at the same time
    • support alloc static ip from any subnet after ns supports multi subnets
    • create ip crd in kube-ovn-controller
    • add condition for triggering the deletion of redundant chassises in sbdb
    Source code(tar.gz)
    Source code(zip)
  • v1.8.4(Mar 29, 2022)

    Bugfix

    • fix provider network range loop
    • increase memory limit of ovn-central
    • add missing link scope routes in vpc-nat-gateway
    • update ip assigned check
    • fix external egress gateway
    • update nodeips for restore cmd in ko plugin
    • append vm deletion check
    • check the cidr format whether is correct
    • add reset for kube-ovn-monitor metrics
    • The underlay physical gateway config by external-gw-addr when use snat&eip
    • avoid frequent ipset update
    • do not recreate port for terminating pods
    • create ip crd in kube-ovn-controller
    Source code(tar.gz)
    Source code(zip)
  • v1.9.1(Mar 9, 2022)

    New Feature

    • Add new arg to configure ns of ExternalGatewayConfig
    • add dnsutils for base image
    • kubectl-ko: support trace Pods being created
    • Add args to configure port ln-ovn-external
    • add back centralized subnet active-standby mode
    • add metric for ovn nb/sb db status
    • update provider network via node annotation
    • support to add multiple subnets for a namespace
    • append add cidr and excludeIps annotation for namespace
    • keep ip for kubevirt pod
    • add webhook for subnet update validation

    Bugfix

    • ignore hostnetwork pod when initipam
    • update check for delete statefulset pod
    • liveMigration with IPv6
    • validate statefulset pod by name
    • add gateway check after update subnet
    • fix clusterrole in ovn-ha.yaml
    • set up tunnel correctly in hybrid mode
    • check static route conflict
    • transfer IP/route earlier in OVS startup
    • only log matched svc with np (#1287)
    • fix underlay subnet in custom VPC
    • resync provider network status periodically
    • fix statefulset Pod deletion
    • Fix usage of ovn commands
    • continue of deletion for del pod failed when can't found vpc or subnet
    • fix OVS bridge with bond port in mode 6
    • replace ecmp dp_hash with hash by src_ip (#1289)
    • fix ips update
    • add check for pod update process
    • skip ping gateway for pods during live migration
    • modify ipam v6 release ip problem
    • fix SNAT/PR on Pod startup
    • update networkpolicy port process
    • append htbqos para in crd yaml
    • update flag parse in webhook
    Source code(tar.gz)
    Source code(zip)
  • v1.8.3(Mar 9, 2022)

    Bugfix

    • update check for delete statefulset pod
    • update networkpolicy port process
    • add back centralized subnet active-standby mode
    • add gateway check after update subnet
    • fix clusterrole in ovn-ha.yaml
    • set up tunnel correctly in hybrid mode
    • check static route conflict
    • transfer IP/route earlier in OVS startup
    • fix only log matched svc with np (#1287)
    • fix statefulset Pod deletion
    • ignore cilint
    • Fix usage of ovn commands
    • fix continue of deletion for del pod failed when can't found vpc or subnet
    • fix OVS bridge with bond port in mode 6
    • fix replace ecmp dp_hash with hash by src_ip (#1289)
    • fix cni deepcopy
    • fix ips update
    • add check for pod update process
    • fix SNAT/PR on Pod startup
    • optimize log for node port-group

    New Feature

    • support to add multiple subnets for a namespace
    • append add cidr and excludeIps annotation for namespace
    • feat optimize log
    • resync provider network status periodically
    • add metric for ovn nb/sb db status
    • feat update provider network via node annotation
    • keep ip for kubevirt pod
    Source code(tar.gz)
    Source code(zip)
  • v1.9.0(Jan 12, 2022)

    New Feature

    • Policy route support in custom VPC
    • VLAN subnet support in custom VPC
    • Load balancer support in custom VPC
    • Add nodeSelector for vpc-nat-gateway pod
    • Add vpc-nat-gateway support for default VPC
    • VIP for pod support
    • Support to set HTB QoS priority
    • Integrate Cilium with Kube-OVN
    • Pod can use multiple nic with the same subnet
    • Support using logical gateway in underlay subnet
    • Add args to configure port ln-ovn-external
    • Other CNI can be used as the default network
    • Add networkpolicy support for attachment cni
    • Add back webhook for subnet and ip validation
    • Sync live migration vm port
    • Update OVN to 21.06
    • Add macvlan CNI binary into image
    • Add kubectl-ko plugin into image

    Performance

    • switch's router port's addresses to "router"
    • do not diagnose external access
    • increase ovn-nb timeout
    • add stt section and update benchmark
    • add fastpath module for 4.x kernel
    • reduce qos query with ovs-vsctl cmd
    • use logical router policy for accessing node
    • do not send multicast packets to conntrack
    • add db compact for nb and sb db
    • do not send traffic to ct if not designate to svc
    • jemalloc and ISA optimization
    • change nbctl args 'wait=sb' to 'no-wait'

    Security

    • security: update base ubuntu image

    Monitoring & Troubleshooting

    • kubectl-ko: support trace Pods being created
    • add dnsutils for base image

    Test

    • add e2e testing for dual stack underlay
    • add ovn-ic e2e
    • add cilium e2e
    • support running ovn-ic e2e on macOS

    Refactor

    • remove ovn ipam option
    • mute ovn0 ping log and add ping details
    • refactor: reuse waitNetworkReady to check ovn0 and slightly improve the installation speed
    • cleanup command flags
    • update klog to v2 which embed log rotation
    • update Go modules
    • delete frequently log
    • add healthcheck cmd to probe live and ready

    Bugfix

    • fix nat-outgoing/policy-routing on pod startup
    • fix variable reference
    • fix e2e testing
    • fix gc lsp statistic for multiple subnet
    • re-check ns annotation to avoid annotations lost
    • append externalIds for pod and node when upgrade
    • fix IPAM for StatefulSet
    • wrong link for iptables
    • fix StatefulSet down scale
    • fix getting LSP UUID by name
    • fix installation scripts
    • init node with wrong ipamkey and lead conflict
    • modify kube-ovn as multus-cni problem
    • In netpol egress rules, except rule should be set to "!=" and should not be "=="
    • replace api for get lsp id by name
    • fix trace command in dual stack underlay networking
    • fix pinger and monitor in underlay networking
    • fix pinger in dual stack cluster
    • fix kubectl-ko diagnose
    • fix cleanup.sh and uninstall.sh
    • fix: check and load ip_tables module
    • add inspection
    • change inspection logic from manually adding lsp to just reading pod queue
    • fix: serialize pod add/delete order
    • fix: delete vpc-nat-gw deployment
    • remove node chassis annotation on cleanup
    • fix: ensure all kube-ovn components deleted before annotate pods
    • fix bug: logical switch ts not ready
    • fix: check allocated annotation in update handler
    • fix LB in dual stack cluster
    • fix: multus-cni subnet allocation
    • fix: trace in custom vpc
    • fix read-only pointer in vlan and provider-network
    • fix ko trace
    • fix: no need to set address for ls to lr port
    • add sg acl check when init
    • add pod in default vpc to node port-group
    • fix LB: skip service without cluster IP
    • fix pinger's compatibility for k8s v1.16
    • deleting all chassises which are not nodes
    • add vendor param for fix list LR
    • fix: add kube-ovn-cni prob timeout
    • update delete operation for statefulset pod
    • fix: add back the leader check
    • when update subnet's except ip,we should filter repeat ip
    • when netpol is added to a workload, the workload's POD can be accessed using service
    • fix: check np switch
    • filter used qos when delete qos
    • add protocol check when subnet is dual-stack
    • pinger: fix getting empty PodIPs
    • delete frequently log
    • fix: do not reuse released ip after subnet updated
    • use multus-cni as default cni to assign ip
    • use different ip crd with provider suffix for pod multus nic
    • move chassis judge to the end of node processing
    • append check for centralized subnet nat process
    • fix installation script
    • fix pod tolerations
    • modify pod's process of update for use multus cni as default cni
    • fix iptables rules and service e2e
    • update check for delete statefulset pod
    • ignore hostnetwork pod when initipam
    Source code(tar.gz)
    Source code(zip)
  • v1.8.2(Jan 7, 2022)

    Bugfix

    • modify kube-ovn as multus-cni problem
    • In netpol egress rules, except rule should be set to "!=" and should not be "=="
    • replace api for get lsp id by name
    • fix trace command in dual stack underlay networking
    • fix pinger and monitor in underlay networking
    • fix pinger in dual stack cluster
    • fix kubectl-ko diagnose
    • fix cleanup.sh and uninstall.sh
    • fix: check and load ip_tables module
    • add inspection
    • change inspection logic from manually adding lsp to just reading pod queue
    • fix: serialize pod add/delete order
    • fix: delete vpc-nat-gw deployment
    • remove node chassis annotation on cleanup
    • fix: ensure all kube-ovn components deleted before annotate pods
    • fix bug: logical switch ts not ready
    • fix: check allocated annotation in update handler
    • fix LB in dual stack cluster
    • fix: multus-cni subnet allocation
    • fix: trace in custom vpc
    • fix read-only pointer in vlan and provider-network
    • fix ko trace
    • fix: no need to set address for ls to lr port
    • add sg acl check when init
    • add pod in default vpc to node port-group
    • fix LB: skip service without cluster IP
    • fix pinger's compatibility for k8s v1.16
    • deleting all chassises which are not nodes
    • add vendor param for fix list LR
    • fix: add kube-ovn-cni prob timeout
    • update delete operation for statefulset pod
    • fix: add back the leader check
    • when update subnet's except ip,we should filter repeat ip
    • when netpol is added to a workload, the workload's POD can be accessed using service
    • fix: check np switch
    • filter used qos when delete qos
    • add protocol check when subnet is dual-stack
    • pinger: fix getting empty PodIPs
    • delete frequently log
    • fix: do not reuse released ip after subnet updated
    • use multus-cni as default cni to assign ip
    • use different ip crd with provider suffix for pod multus nic
    • move chassis judge to the end of node processing
    • append check for centralized subnet nat process
    • fix installation script
    • fix pod tolerations

    Security

    • security: update base ubuntu image

    Test

    • add e2e testing for dual stack underlay

    Performance

    • add db compact for nb and sb db
    • change nbctl args 'wait=sb' to 'no-wait'
    Source code(tar.gz)
    Source code(zip)
  • v1.8.1(Oct 9, 2021)

    Performance

    • switch's router port's addresses to "router"
    • do not diagnose external access
    • increase ovn-nb timeout

    Bugfix

    • fix nat-outgoing/policy-routing on pod startup
    • fix variable reference
    • fix e2e testing
    • fix gc lsp statistic for multiple subnet
    • re-check ns annotation to avoid annotations lost
    • append externalIds for pod and node when upgrade
    • fix IPAM for StatefulSet
    • wrong link for iptables
    • fix StatefulSet down scale
    • fix getting LSP UUID by name
    • fix installation scripts
    • init node with wrong ipamkey and lead conflict

    Refactor

    • remove ovn ipam option
    • mute ovn0 ping log and add ping details
    Source code(tar.gz)
    Source code(zip)
  • v1.7.3(Oct 9, 2021)

    Performance

    • switch's router port's addresses to "router"
    • do not diagnose external access
    • increase ovn-nb timeout

    Bugfix

    • fix nat-outgoing/policy-routing on pod startup
    • fix variable reference
    • fix e2e testing
    • fix gc lsp statistic for multiple subnet
    • re-check ns annotation to avoid annotations lost
    • append externalIds for pod and node when upgrade
    • fix IPAM for StatefulSet
    • wrong link for iptables
    • fix StatefulSet down scale
    • fix getting LSP UUID by name
    • fix installation scripts
    • init node with wrong ipamkey and lead conflict
    • serialize pod add/delete order
    • kubeclient timeout
    • update base image

    Refactor

    • remove ovn ipam option
    • mute ovn0 ping log and add ping details
    Source code(tar.gz)
    Source code(zip)
  • v1.8.0(Sep 8, 2021)

    New Feature

    • Underlay/Vlan network refactoring to adapt more complicate network infrastructure
    • Share same OVN infrastructure with OpenStack and run Pods in OpenStack VPC
    • Support SecurityGroup in VPCs
    • Support Service in VPCs
    • Adapt to vfio-pci driver type device
    • Use annotation to dynamic change tunnel interface
    • Pod level annotation to control traffic mirror
    • Kube-OVN ipam supports custom routes
    • Switch to enable/disable OVN network policy support
    • Switch to enable/disable OVN LB
    • Switch to enable/disable gateway connectivity check

    Performance

    • New fastpath module which can reduce about 40% latency and cpu usage
    • New performance tuning guide to boost the performance and comparison for different network
    • Enable tx offload again as fix for double nat kernel issue

    Monitoring

    • Diagnose command 'kubectl ko' support trace in underlay networking
    • Diagnose command 'kubectl ko' support cluster operations status/kick/backup

    Security

    • fix CVE-2021-33910
    • Add go build security options
    • Fix CVE-2021-3121
    • fix CVE-2021-3538
    • Update base image to ubuntu:21.04

    Mics

    • update kind to v0.11.1
    • fix gofmt lint
    • ci: use stable golang version
    • update ovn to 21.03

    Test

    • update underlay e2e testing
    • fix subnet e2e
    • fix dual stack cluster created by kind
    • Correct vlan e2e testing
    • Remove dpdk ci

    Bugfix

    • fix CRD provider-networks.kubeovn.io
    • fix ipsets, subnets using underlay networking should not be included in ipsets
    • update qos ingress_policing_burst
    • match chassis until timeout
    • delete overlapped var subnet
    • fix: if nftables not exists do no exit
    • delete ecmp route when node is deleted
    • fix cleanup.sh
    • fix image version
    • fix iptables
    • ignore empty strings when counting lbs
    • fix external_ids:pod_netns
    • delete attachment ips
    • update qos process
    • fix: panic when node has nil annotations
    • append pod/exec resource for vpc nat gw
    • fix underlay networking on node reboot
    • fix default bind socket of cni server
    • if the string of ip is empty,program will die
    • fix uninstall.sh
    • ensure provider nic is up
    • fix: bad udp checksum when access nodeport
    • fix IPv6-related issues
    • fix issues in underlay networking
    • avoid Pod IP to be the same with node internal IP
    • fix subnet conflict check for node address
    • fix ipset on pod creation/deletion
    • delete subnet AvailableIPs and UsingIPs para
    • fix: ovn-northd svc flip flop
    • delete residual ovs internal ports
    • initialize ipsets on cni server startup
    • Fix acl overlay issues
    • Fix available ips count of subnet
    • Fix lsp may lost when server pressure is high
    • Cleanup kube-ovn-monitor resource
    • Remove wait ovn sb
    • Remove kube-ovn-controller rollout check
    • Delete process of ip crd delete in cni delete request
    • Delete ecmp route when node is not ready
    • Ignore update pod nic annotation when not nil
    • Clean up gateway chassis list for external gw
    • Node route should filter out 'vpc'
    • Do not delete statefulset pod when update pod
    • Add master check when a node adding to a cluster and config sb/nb address
    • Fix IP/route transfer on node reboot
    • Fix uninstall.sh execution in OVS pods
    • Add node internal ip into ovn-ic advertise blacklist
    • Fix bug for deleting ovn-ic lrp failed
    • Keep subnet's vlan empty if not specified
    • Add field defaultNetworkType in configmap ovn-config
    Source code(tar.gz)
    Source code(zip)
    kube_ovn_fastpath.ko(407.91 KB)
    openvswitch-kmod-2.15.2-1.el7.x86_64.rpm(3.07 MB)
  • v1.7.2(Sep 8, 2021)

    This release mainly fix bugs found in 1.7.1

    Feature

    • update encap ip by node annotation periodic
    • update node labels and provider network's status.readyNodes when provider network is not initialized successfully in a node
    • add ready status for provider network

    Bugfix

    • fix CRD provider-networks.kubeovn.io
    • fix ipsets, subnets using underlay networking should not be included in ipsets
    • update qos ingress_policing_burst
    • match chassis until timeout
    • delete overlapped var subnet
    • fix: if nftables not exists do no exit
    • delete ecmp route when node is deleted
    • fix cleanup.sh
    • fix image version
    • fix iptables
    • ignore empty strings when counting lbs
    • fix external_ids:pod_netns
    • delete attachment ips
    • update qos process
    • fix: panic when node has nil annotations
    • append pod/exec resource for vpc nat gw
    • fix underlay networking on node reboot
    • fix default bind socket of cni server
    • if the string of ip is empty,program will die
    • fix uninstall.sh
    • ensure provider nic is up
    • fix: bad udp checksum when access nodeport
    • fix IPv6-related issues
    • fix issues in underlay networking
    • avoid Pod IP to be the same with node internal IP
    • fix subnet conflict check for node address
    • fix ipset on pod creation/deletion
    • delete subnet AvailableIPs and UsingIPs para
    • fix: ovn-northd svc flip flop
    • delete residual ovs internal ports
    • initialize ipsets on cni server startup

    Test

    • update underlay e2e testing
    • fix subnet e2e
    • fix dual stack cluster created by kind

    Security

    • fix CVE-2021-33910

    Chore

    • update kind to v0.11.1
    • fix gofmt lint
    • ci: use stable golang version
    Source code(tar.gz)
    Source code(zip)
    kube_ovn_fastpath.ko(407.91 KB)
    openvswitch-kmod-2.15.2-1.el7.x86_64.rpm(3.07 MB)
  • v1.7.1(Jul 15, 2021)

    New Feature

    • Underlay/Vlan network refactoring
    • Diagnose command 'kubectl ko' support trace in underlay networking
    • Diagnose command 'kubectl ko' support cluster operations status/kick/backup
    • Support to specify node nic name

    Bugfix

    • Fix acl overlay issues
    • Fix available ips count of subnet
    • Fix lsp may lost when server pressure is high
    • Cleanup kube-ovn-monitor resource
    • Remove wait ovn sb
    • Remove kube-ovn-controller rollout check
    • Delete process of ip crd delete in cni delete request
    • Delete ecmp route when node is not ready
    • Ignore update pod nic annotation when not nil
    • Clean up gateway chassis list for external gw
    • Node route should filter out 'vpc'
    • Do not delete statefulset pod when update pod
    • Add master check when a node adding to a cluster and config sb/nb address
    • Fix IP/route transfer on node reboot
    • Fix uninstall.sh execution in OVS pods
    • Add node internal ip into ovn-ic advertise blacklist
    • Fix bug for deleting ovn-ic lrp failed
    • Keep subnet's vlan empty if not specified
    • Add field defaultNetworkType in configmap ovn-config

    Performance

    • Enable tx offload again as upstream already fix it

    Test

    • Correct vlan e2e testing
    • Remove dpdk ci

    Security

    • Add go build security options
    • Fix CVE-2021-3121
    Source code(tar.gz)
    Source code(zip)
    kube_ovn_fastpath.ko(407.91 KB)
    openvswitch-kmod-2.15.2-1.el7.x86_64.rpm(3.07 MB)
  • v1.7.0(Jun 3, 2021)

    Release for v1.7.0

    New Feature

    • Support configuration for cni-bin-dir and cni-conf-dir
    • Support for vpc nat gateway
    • Support for multus ovn nic
    • Support ecmp static route for centralized gateway
    • Support vxlan tunnel encapsulation
    • Support hybrid mode for geneve and vlan
    • Support external egress gateway
    • Support underlay mode with single nic
    • Support kube-ovn-speaker announce service ip
    • Support kube-ovn-speaker graceful restart
    • Support interconnection between OpenStack and Kubernetes

    Bugfix

    • Restart when init ping failed
    • Make sure northd leader change
    • Wrong split in FindLoadbalancer function
    • Ip6tables check error
    • Reset ovn0 addr
    • Masq traffic to ovn0 from other nodes
    • Add missing ovn-ic-db schema
    • Update ipam cidr when subnet changes
    • Disable offload for genev_sys_6081
    • Configure nic failed when ifname empty
    • Udp checksum offload error
    • Restart ovn-controller to force ovn-ic flows update
    • Update usingips check when update finalizer for subnet
    • Livenessprobe fail if ovn nb/ovn sb not running
    • Release norhtd lock when power off
    • Fix chassis check for node
    • Pod terminating not recycle ip when controller not ready

    Monitoring

    • Split ovn-monitor from ovn-central as an independent deployment
    • Optimization for ovn/ovs status metric
    • Add more command to diagnose results, such as ovs-dpctl, ovs-ofctl, ovs-appctl and so on

    Performance

    • Support use ovs internal-port instead of veth pair to implement pod nic

    Test

    • Add e2e for ofctl/dpctl/appctl
    • Add service e2e
    • Add single node e2e
    • Add e2e tests for external egress gateway

    Mics

    • Update ovn to 20.12 and ovs to 2.15
    • Update Go to 1.16
    Source code(tar.gz)
    Source code(zip)
  • v1.6.3(Jun 3, 2021)

    This release mainly fix bugs found in 1.6.2

    Bugfix

    • fix: do not nat route traffic
    • fix: release ip addresses even if pods not found
    • security: fix crypto CVE
    • fix: add address_set to avoid error message
    • fix: add node to pod allow acl
    • Handler the parse config error before used
    • fix: del might panic if duplicate delete
    • fix: do not re-generate ts port
    • fix: get_leader_ip always return fist node ip
    • fix: do not gc learned routes
    • fix: remove tty error notification
    • fix ovn nb reconnect
    • perf: reclaim heap memory after compaction
    • fix: leader may change during startup, use cluster connection to set options
    • fix SNAT on pod startup
    Source code(tar.gz)
    Source code(zip)
  • v1.6.2(Apr 19, 2021)

    This release mainly fix bugs found in 1.6.1

    Bugfix

    • udp checksum offload error
    • restart ovn-controller to force ovn-ic flows update
    • update usingips check when update finalizer for subnet
    • add node address allocate check when init
    • livenessprobe fail if ovn nb/ovn sb not running
    • ignore ip6tabels check for v4 hostIP
    • release norhtd lock when power off
    • fix chassis check for node
    • configure nic failed when ifname empty
    Source code(tar.gz)
    Source code(zip)
  • v1.6.1(Mar 9, 2021)

    This release mainly fix bugs found in 1.6.0

    Bugfix

    • DualStack error logs
    • IP count error in DualStack mode
    • ip6tables check error
    • Update ipam cidr when subnet changes
    • When address is empty, skip route/nat deletion
    • Waiting pod network ready takes too long
    • Add new iptables clean up commands
    • Pod terminating not recycle ip when controller not ready
    • Restart when init ping failed
    • Make sure northd leader change
    • Check required module before start
    • Underlay gateway flood logs
    • Wrong split in FindLoadbalancer function
    • Reset ovn0 addr
    • Set default db addr same with leader node to fix nb and sb error 'bind: Address already in use'
    • Masq traffic to ovn0 from other nodes

    Misc

    • Update ovn to 20.12 and ovs to 2.15
    Source code(tar.gz)
    Source code(zip)
  • v1.6.0(Jan 4, 2021)

    New Feature

    • Basic support for custom VPC
    • DualStack support
    • Overlay to underlay gateway through BGP
    • Support binding pod to subnet
    • Support distributed eip
    • Support disable interconnection for specific subnet
    • Iface now support regex
    • install.sh supports DPDK resource configuration
    • Remove cluster ip dependency for ovn/ovs components
    • Change base image to ubuntu
    • Update OVN to 20.09

    Monitoring

    • Add OVN/OVS monitor
    • Add ovs client latency metrics
    • Add ping total count metric
    • Add ovs-vsctl show to diagnose results
    • Add kubectl describe no to diagnose results
    • Add ovs-vsctl show to diagnose results
    • Add available IP metrics
    • Add more dashboard

    Mics

    • CI: change to official docker buildx action
    • Perf: remove default acl rules
    • ci: add github code scan
    • Add version info
    • Reduce image size
    • Perf: accelerate ic and ex gw update
    • Refactor iptable logs
    • Tolerate all taints
    • OVN/OVS log rotation
    • Update Go to 1.15
    • Multi arch image

    Bugfix

    • Remove not alive pod in networkpolicy portGroup
    • Delete Pod when marked with deletionTimestamp
    • Use the internal IP when node try to connect to pod
    • Do not advertise node switch cidr when enable ovn-ic
    • Wrong proto str for udp diagnose
    • IPv6 len mismatch
    • Add default SSL var for compatibility
    • Wrong ipv6 network format when update subnet
    • Broken RPM link
    • Default SSL var for compatibility
    • Wrong iptable order
    • Check multicast and loopback subnet
    • CodeQL scan warnings
    • Fix cleanup scripts
    • Check ipv6 requirement before start
    • Check if ovn-central ip exists in NODE_IPS before start
    • Fix the problem of confusion between old and new versions of crd
    • Fix external-address config description
    • Add resources limits to avoid eviction
    • NAT rules can be modified
    • Masquerade other nodes to local pod to avoid NodePort triangle traffic
    • OVN-IC support SSL
    • Pod static ip validation
    • Multiple rule networkpolicy issues
    • Modify service vip parse error
    • CNIServer default encap ip use node ip
    Source code(tar.gz)
    Source code(zip)
  • v1.5.2(Dec 1, 2020)

    New Feature

    • Iface now support regex
    • install.sh supports DPDK resource configuration
    • Masquerade other nodes to local pod to avoid NodePort triangle traffic

    Monitoring

    • Add ping total count metric
    • Add ovs-vsctl show to diagnose results
    • Add kubectl describe no to diagnose results

    Bugfix

    • Fix cleanup scripts
    • Update Go to 1.15
    • Check ipv6 requirement before start
    • Check if ovn-central ip exists in NODE_IPS before start
    • Fix external-address config description
    • Fix the problem of confusion between old and new versions of crd
    • Add resources limits to avoid eviction
    • NAT rules can be modified

    Mics

    • Refactor iptable logs
    • Tolerate all taints
    • OVN/OVS log rotation
    Source code(tar.gz)
    Source code(zip)
  • v1.5.1(Oct 26, 2020)

    This release mainly fix bugs found in v1.5.0 and add function of binding pod to specified subnet

    New Feature

    • Support binding pod to subnet

    Bugfix

    • Remove not alive pod in networkpolicy portGroup
    • Delete Pod when marked with deletionTimestamp
    • Use internal IP when node try to connect to pod
    • Do not advertise node switch cidr when enable ovn-ic
    • Wrong proto str for udp diagnose
    • Wrong ipv6 network format when update subnet
    • Broken RPM link
    • Default SSL var for compatibility
    • Wrong iptable order
    • Check multicast and loopback subnet
    • CodeQL scan warnings

    Mics

    • CI: change to official docker buildx action
    • Perf: remove default acl rules
    • Perf: accelerate ic and ex gw update
    Source code(tar.gz)
    Source code(zip)
  • v1.5.0(Sep 28, 2020)

    From v1.5.0 Kube-OVN takes use of OVN distributed gateway router to implement SNAT and EIP functions. Users now can controller the external IP of a groups of pods. SFC functions is also integrated into OVN to further extend the capability of OVN. In this version, users can also enabled the TLS connection between Kube-OVN components to secure the communication. We also enhance the monitoring and diagnose tools, more metrics and Grafana dashboards are added to better expose the internal stats of the network.

    New Feature

    • Pod level SNAT and EIP support, please check the Guide
    • Integrate SFC function into OVN
    • OVN-Controller graceful stop
    • Mirror config can be updated dynamically
    • Set more metadata to interface external-ids

    Security

    • Support TLS connection between components
    • Change DB file access mode

    Monitoring

    • Add more metrics to pinger dashboard
    • Add more metrics to kube-ovn-cni and a new Grafana dashboard
    • Diagnose show ovn-nb and ovn-sb overview

    Mics

    • Update CI k8s to 1.19
    • Change kube-ovn-cni updateStrategy
    • Move CNI conf when kube-ovn-cni ready

    Bugfix

    • Use NodeName as OVN chassis name
    • Stop OVN-IC if disabled
    • Uninstall scripts will clean up ipv6 iptables and ipset
    • Bridging-mapping may conflict, if enable vlan and external gateway
    • Pinger ipv6 mode fetch portmaping failed
    • Pinger diagnose should reuse cmd args
    Source code(tar.gz)
    Source code(zip)
  • v1.4.0(Sep 1, 2020)

    From 1.4, Kube-OVN can connect multiple cluster into one network. Pods in different cluster can communicate with others by Pod IP directly. This version also add ACL log function that can record when and why a packet is dropped by NetworkPolicy. We also enhance many dependency and improve the performance. Please look the changelog for more detail.

    New Feature

    • Integrate OVN-IC to support multi-cluster networking, Multi-Cluster Networking Steps
    • Enable ACL log to record networkpolicy drop packets
    • Reserve source ip for NodePort service to local pod
    • Support vlan subnet switch to underlay gateway

    Bugfix

    • Add forward accept rules
    • kubectl-ko cannot find nic
    • Prevent vlan/subnet init error logs
    • Subnet ACL might conflict if allSubnets and subnet cidr overlap
    • Missing session lb

    Misc

    • Update ovs to 2.14
    • Update golang to 1.15
    • Suppress logs
    • Add psp rules
    • Remove juju log dependency
    Source code(tar.gz)
    Source code(zip)
    openvswitch-dkms-2.14.0-1.el8.x86_64.rpm(5.84 MB)
  • v1.3.0(Jul 31, 2020)

    From v1.3.0, Kube-OVN support hardware offload which can significantly improve data plane performance and save CPU resource. This version also add gateway qos, pod gateway, session affinity loadbalancer and more features for security and monitoring.

    New Feature

    • Hardware offload to boost performance in Bare-Metal environment
    • Assigning a specific pod as gateway
    • Central gateway QoS
    • Session affinity service
    • Round-robbin IP allocation to relieve IP conflict

    Security

    • Use gosec to audit code security
    • Use trivy to scan and fix image CVEs
    • Update loopback plugin to fix CVEs

    Bugfix

    • Missing package for arm images
    • Node annotation overwrite incorrectly
    • Create/Delete order might lead ip conflict
    • Add MSS rules to resolve MTU issues

    Monitoring

    • kubectl-ko support ovs-tracing
    • Pinger support metrics to resolve external address

    Misc

    • Update OVN to 20.06
    • CRD version upgrade to v1
    • Optimize ARM build
    • Refactor ovs cmd with ovs.Exec
    • OVS-DPDK support config file
    • Add DPDK tools in OVS_DPDK image
    • Reduce image size of OVS-DPDK
    Source code(tar.gz)
    Source code(zip)
  • v1.2.1(Jun 22, 2020)

    This release fix bugs found in v1.2.0

    Bugfix

    • Add back privilege for IPv6
    • Update loopback cni to fix CVE issues
    • Node annotations overwrite incorrectly
    • Create/Delete order might lead to ip conflict
    Source code(tar.gz)
    Source code(zip)
  • v1.2.0(May 30, 2020)

    In this version, Kube-OVN support vlan and dpdk type network interfaces for higher performance requirement. Thanks for Intel and Ruijie Networks guys who contribute these features.

    Previously to expose Pod IP to external network, admins have to manually add static routes. Now admins can try the new BGP features to dynamically announce routes to external network.

    From this version, subnet CIDR can be changed after creation, and routes will be changed if gateway type is modified.

    New Feature

    • Kube-OVN now supports OVS-DPDK, high performance dpdk application can run in pod
    • Kube-OVN now supports vlan underlay network to achieve better network performance
    • Kube-OVN now supports using BGP to announce Pod IP routes to external network
    • Subnet validator will check if subnet CIDR conflicts with svc or node CIDR
    • Subnet CIDR can be changed after creation
    • When subnet gateway changed, routes will aromatically changed

    Monitoring

    • Check if dns and kubernetes svc exist
    • Make grafana dashboard more sensitive to changes

    Misc

    • Patch upstream ovn to reduce lflow count
    • Add support for arm64 platform
    • Add support for kubernetes 1.18
    • Use github action to perform CI tasks
    • Optimize some log information
    • Move image to dockerhub

    Bugfix:

    • OVS local interface table mac_in_use row is lower case, but pod annotation store mac in Upper case
    • fork go-ping to fix ping lost issues
    • Networkpolicy controller will panic if label is nil
    • Some concurrent panic when handle pod and subnet update
    • Some IPv6 break issues
    • Use kubectl version to avoid handshake errors in apiserver
    Source code(tar.gz)
    Source code(zip)
  • v1.1.1(Apr 27, 2020)

    This release fix bugs found in v1.1.0.

    Bugfix

    • Use legacy iptables to replace default iptables in centos:8 image
    • Mount etc/origin/ovn to ovs-ovn
    • Fix bugs in go-ping
    • Fix yaml indent error
    • Fix panic when handles networkpolicy

    Monitoring

    • Make graph more sensitive to changes
    Source code(tar.gz)
    Source code(zip)
K8s-network-config-operator - Kubernetes network config operator to push network config to switches

Kubernetes Network operator Will add more to the readme later :D Operations The

Daniel Hertzberg 6 May 16, 2022
fabric 1.4 bdls protocol on top of the SmartBFT 1.4 project

Hyperledger Fabric Note: This is a read-only mirror of the formal Gerrit repository, where active development is ongoing. Issue tracking is handled in

Ahmed Al Salih 0 Feb 22, 2022
The DGL Operator makes it easy to run Deep Graph Library (DGL) graph neural network training on Kubernetes

DGL Operator The DGL Operator makes it easy to run Deep Graph Library (DGL) graph neural network distributed or non-distributed training on Kubernetes

Qihoo 360 40 Apr 28, 2022
The OCI Service Operator for Kubernetes (OSOK) makes it easy to connect and manage OCI services from a cloud native application running in a Kubernetes environment.

OCI Service Operator for Kubernetes Introduction The OCI Service Operator for Kubernetes (OSOK) makes it easy to create, manage, and connect to Oracle

Oracle 22 Jun 17, 2022
An Easy to use Go framework for Kubernetes based on kubernetes/client-go

k8devel An Easy to use Go framework for Kubernetes based on kubernetes/client-go, see examples dir for a quick start. How to test it ? Download the mo

null 10 Mar 25, 2022
The NiFiKop NiFi Kubernetes operator makes it easy to run Apache NiFi on Kubernetes.

The NiFiKop NiFi Kubernetes operator makes it easy to run Apache NiFi on Kubernetes. Apache NiFI is a free, open-source solution that support powerful and scalable directed graphs of data routing, transformation, and system mediation logic.

konpyutaika 24 Jun 15, 2022
Kubernetes OS Server - Kubernetes Extension API server exposing OS configuration like sysctl via Kubernetes API

KOSS is a Extension API Server which exposes OS properties and functionality using Kubernetes API, so it can be accessed using e.g. kubectl. At the moment this is highly experimental and only managing sysctl is supported. To make things actually usable, you must run KOSS binary as root on the machine you will be managing.

Mateusz Gozdek 3 May 19, 2021
Reconciler - A library to avoid overstuffed Reconcile functions of Kubernetes operators

reconciler A library to avoid overstuffed Reconcile functions of Kubernetes oper

Tiago Angelo 7 May 31, 2022
Package trn introduces a Range type with useful methods to perform complex operations over time ranges

Time Ranges Package trn introduces a Range type with useful methods to perform c

CappuccinoTeam 38 Apr 18, 2022
Planet Scale Robotics - Offload computation-heavy robotic operations to GPU powered world's first cloud-native robotics platform.

robolaunch ?? Planet Scale Robotics - Offload computation-heavy robotic operations to GPU powered world's first cloud-native robotics platform. robola

robolaunch 5 Feb 28, 2022
Go-Mongodb API - A sample REST API ( CRUD operations ) created using Golang

Go-Mongodb_API This is a sample REST API ( CRUD operations ) created using the G

Aman Sharma 1 May 31, 2022
Kubelet-bench - Example Go-based e2e benchmark for various Kubelet operations without spinning up whole K8s cluster

kubelet-bench An example of Go based e2e benchmark for various Kubelet operation

Bartlomiej Plotka 3 Mar 17, 2022
Kilo is a multi-cloud network overlay built on WireGuard and designed for Kubernetes (k8s + wg = kg)

Kilo Kilo is a multi-cloud network overlay built on WireGuard and designed for Kubernetes. Overview Kilo connects nodes in a cluster by providing an e

Lucas Servén Marín 1.4k Jun 25, 2022
Hubble - Network, Service & Security Observability for Kubernetes using eBPF

Network, Service & Security Observability for Kubernetes What is Hubble? Getting Started Features Service Dependency Graph Metrics & Monitoring Flow V

Cilium 2.1k Jun 30, 2022
Shiba - Minimalist Kubernetes network plugin

Shiba(柴) Shiba is a minimalist Kubernetes network plugin, as a replacement for f

Qing Moy 6 Jun 18, 2022
KubeOrbit is an open-source abstraction layer library that turns easy apps testing&debuging on Kubernetes in a new way

KubeOrbit is an open-source abstraction layer library that turns easy apps testing&debuging on Kubernetes in a new way

TeamCode 433 Jun 23, 2022
Litmus helps Kubernetes SREs and developers practice chaos engineering in a Kubernetes native way.

Litmus Cloud-Native Chaos Engineering Read this in other languages. ???? ???? ???? ???? Overview Litmus is a toolset to do cloud-native chaos engineer

Litmus Chaos 3.1k Jun 21, 2022
vcluster - Create fully functional virtual Kubernetes clusters - Each cluster runs inside a Kubernetes namespace and can be started within seconds

Website • Quickstart • Documentation • Blog • Twitter • Slack vcluster - Virtual Clusters For Kubernetes Lightweight & Low-Overhead - Based on k3s, bu

Loft Labs 1.7k Jun 27, 2022