A distributed Layer 2 Direct Server Return (L2DSR) load balancer for Linux using XDP/eBPF

Related tags

Network vc5
Overview

VC5

A distributed Layer 2 Direct Server Return (L2DSR) load balancer for Linux using XDP/eBPF

This is very much a proof of concept at this stage - most everything is incomplete and poorly defined!

Quickstart

It would be recommended to start out using a fresh virtual machine.

First we need a development environment capable of building libbpf and Go binaries. On Ubuntu 20.04 this can be achieved with:

apt-get install git build-essential libelf-dev clang libc6-dev libc6-dev-i386 llvm golang libyaml-perl libjson-perl ethtool bridge-utils

Copy the config.yaml file to vc5.yaml and edit appropriately for your routers/services/backends.

Run make. This will build the binary and transform the YAML config file to an more verbose JSON config.

Run the vc5.sh shell script with arguments of your IP address and network interface name, eg.:

./vc5.sh 10.10.100.200 ens192

If this doesn't bomb out then you should have a load balancer up and running, although by default it will wait for around one minute to learn from other instances via multicast before healthchecking backends and advertising routes. You can add static routing to forward traffic for a VIP to the load balancer, or configure BGP peers in the YAML file to have routes automatically injected. You will see that an virtual ethernet device and a net namespace has been added. These should be removed when the binary exists (use Ctrl-C).

If you don't have a routed environment then you can test with a client machine on the same VLAN. Either add a static route on the client machine pointing to the load balancer, or run BIRD/Quagga on client and add the client's IP address to the BGP section of the YAML config.

Sample bird.conf snippet:

protocol bgp loadbalancers {
     description "loadbalancers";
     local as 65304;
     neighbor range 10.10.100.0/24 as 65304;

     ipv4 {
          export none;
          import filter {
                 if net ~ 192.168.101.0/24 then accept;
                 else reject;
          };
          next hop self;
     };

     passive on;
     direct;
}

If you enable ECMP on your router/client ("merge paths on;" in BIRD's kernel protocol) then you can load balance traffic to multiple load balancers. VC5 uses multicast to share a flow state table so peers will learn about each other's connections and take over in the case of one load balancer going down.

To use native driver mode in XDP a slighly more involved network setup is needed at this stage. Run your physical interface in a bridge (see sample netplan config in bridge.yaml) and add the -b flag to vc5.sh as so:

./vc5.sh -b br0 10.10.100.200 enp130s0f1

Performance

This has mostly been tested using Icecast backend servers with clients pulling a mix of low and high bitrate streams (48kbps - 192kbps).

It seems that a VMWare guest (4 core, 8GB) using the XDP generic driver will comfortably support 100K concurrent clients, 380Mbps/700Kpps through the load balancer and 8Gbps of traffic from the backends directly to the clients.

A server with an Intel Xeon CPU (E52620 @ 2.40GHz) with 6 physical cores and an Intel 10Gbps NIC (ixgbe driver) in native mode will support upwards of 500K clients, 2Gbps/3.5Mpps and 40Gbps traffic back to clients. This was at a load of ~25% on the CPU - clearly it can do significantly more than this, but resources for running more client and backend servers were not available at the time.

TODOs

  • Better check flexibility - eg. 3 out of 5
  • Least conns support
  • Multicast status to help last conns check
  • Take most loaded server out of pool
  • More complete BGP4 implementation
  • BFD implementation
  • VLAN support
  • GRE/DSCP L3 support

Configuration

Route Health Injection

The rhi section contains the AS number to use and a list of peers. Currently, the BGP4 implementation does not listen on port 179. The IP address specified on the command line will be used as the router ID. If the peer has a router ID higher than this IP then according to the RFC the connection should be abandoned, so this may not work in all situations. I'm unsure if using passive option in BIRD helps with this or not; YMMV. This will be addressed soon.

RHI will only advertise a service's IP address if all services on all ports using that IP address pass healthchecks. All healthchecks listed for a port (http/https/tcp) must pass if specified.

Services

A service consist of

  • name - a string, the name of the service
  • desc - a string, a description of the service
  • addr - an VIP address, or list of VIP addresses, for the service
  • port - an integer or list of integers, that the service consist of
  • real - a name, or list of names, that references groups of real IP address in the "reals" section
  • need - number of real servers that need to be available for the service to be declared healthy (default: 1)
  • checks - a list of checks that must all pass for the service to be declared healthy.

Checks

A check consists of

  • name - a string describing the check
  • type - http, https or tcp
  • port - the port that the check will be run against, optional if there is only one port listed in the service
  • path - the path to run http or https checks against, not needed for tcp
Issues
  • go module name

    go module name

    Regarding https://github.com/davidcoles/vc5/blob/master/vc5/go.mod#L1 - I think it would be more idiomatic to put the "FQDN" module name like github.com/davidcoles/vc5 instead of just vc5, and adjust the import paths accordingly. I would also recommend to consider moving go.mod to the repository root (unless you have strong reasons to keep it in subdirectory). The idiomatic module name for example would enable to look up the godoc documentation at the usual place such as https://pkg.go.dev/github.com/davidcoles/vc5

    opened by dmitris 3
Owner
David Coles
David Coles
Ebpfmanager - A golang ebpf libary base on cilium/ebpf and datadog/ebpf

介绍 ebpfmanager参照datadog/ebpf/manager包的思想,基于cilium/ebpf实现的ebpf类库封装。 相比cilium/ebpf

null 122 Jul 31, 2022
Cat Balancer is line based load balancer for net cat nc.

Cat Balancer Cat Balancer is line based load balancer for net cat nc. Usage cb [-p <producers-port>] [-c <consumers-port>] One Producer to One Consum

CGI France 4 Jul 6, 2022
A modern layer 7 load balancer from baidu

BFE BFE is a modern layer 7 load balancer from baidu. Advantages Multiple protocols supported, including HTTP, HTTPS, SPDY, HTTP2, WebSocket, TLS, Fas

null 5.6k Aug 2, 2022
Kiwi-balancer - A balancer is a gateway between the clients and the server

Task description Imagine a standard client-server relationship, only in our case

Jozef Lami 0 Feb 11, 2022
High-performance PHP application server, load-balancer and process manager written in Golang

RoadRunner is an open-source (MIT licensed) high-performance PHP application server, load balancer, and process manager. It supports running as a serv

Spiral Scout 6.6k Aug 2, 2022
High-performance PHP application server, load-balancer and process manager written in Golang

RoadRunner is an open-source (MIT licensed) high-performance PHP application server, load balancer, and process manager. It supports running as a serv

Spiral Scout 6.1k Dec 9, 2021
A load balancer supporting multiple LB strategies written in Go

farely A load balancer supporting multiple LB strategies written in Go. Goal The goal of this project is purley educational, I started it as a brainst

Mehdi Cheracher 11 Aug 7, 2022
gobetween - modern & minimalistic load balancer and reverse-proxy for the ☁️ Cloud era.

gobetween - modern & minimalistic load balancer and reverse-proxy for the ☁️ Cloud era. Current status: Maintenance mode, accepting PRs. Currently in

Yaroslav Pogrebnyak 1.7k Aug 10, 2022
KgLb - L4 Load Balancer

KgLb KgLb is L4 a load balancer built on top of linux ip virtual server (ip_vs). It provides rich functionality such as discovery, health checks for r

Dropbox 129 Mar 10, 2022
Simple Reverse Proxy Load Balancer

lb - a reverse proxy load-balancing server, It implements the Weighted Round Robin Balancing algorithm

Blessing Pariola 3 Mar 23, 2022
Basic Load Balancer

Load Balancer Work flow based on code snippet Trade-offs: 1. Using etcd as a global variable map. 2. Using etcd to store request references rather tha

Nikhil Vasudev 0 Nov 1, 2021
Vippy - A Virtual IP/BGP/IPVS Load-Balancer for Equinix Metal

Vippy - A Virtual IP/BGP/IPVS Load-Balancer for Equinix Metal If I figure out how to make it work.. How it works! The Vippy LB PoC uses BGP/IPVS and E

Daniel Finneran 5 Mar 10, 2022
A Load-balancer made from steel

slb The Steel Load Balancer A load-balancer forged in the fires of Sheffield Getting slb Prebuilt binaries for armv7 and amd64 exist in the releases p

null 74 Apr 18, 2022
Lightweight http response time based load balancer written in Go

HTTP Load Balancer Specifications http servers should always return time taken to proceed request in headers as EXECUTION_TIME in ms this load balance

Gaëtan 0 Feb 22, 2022
A Service Load Balancer for Kubernetes.

PureLB - is a Service Load Balancer for Kubernetes PureLB is a load-balancer orchestrator for Kubernetes clusters. It uses standard Linux networking a

PureLB 26 Jul 29, 2022
Consistelancer - Consistent hashing load balancer for Kubernetes

Setup minikube start kubectl apply -f k8s-env.yaml skaffold dev # test locks ku

null 1 Jan 12, 2022
Simple load-balancer for npchat servers, based on the xor distance metric between node & user id

npchat-helmsman Simple load-balancer for npchat servers, based on the xor distance metric between node & user id. Local Development Clone this reposit

npchat 0 Jan 15, 2022
eBPF library for Go based on Linux libbpf

libbpfgo libbpfgo is a Go library for working with Linux's eBPF. It was created for Tracee, our open source Runtime Security and eBPF tracing tools wr

Aqua Security 326 Aug 3, 2022
eBPF-based EDR for Linux

ebpf-edr A proof-of-concept eBPF-based EDR for Linux Seems to be working fine with the 20 basic rules implemented. Logs the alerts to stdout at the mo

null 15 Aug 3, 2022