An easy-to-use, flexible network simulator library in Go.

Overview

ns-x

Go CodeQL

An easy-to-use, flexible network simulator library for Go.

Feature

  • Programmatically build customizable and scalable network topology from basic nodes.
  • Simulate packet loss, delay, etc. on any nodes, according to any parameters inputs in well-defined models.
  • Collect network and user-defined data in detail, from each and every node.
  • Cross-platform. High precision.

Introduction

Concept

  • Network: a topological graph consist of nodes, reflecting a real-world network for packet to transfer and route through.
  • Node: a physical or logical device in the network deciding what to do when a packet going through. A node usually connect to other nodes.
  • Event: an action to be done at a given time point.
  • Packet: simulated data packets transferring between nodes.
  • Transfer: the behavior of node when a packet going through.

Prerequisites

  • go mod must be supported and enabled.

Usage

Use in three steps: building network, starting network simulation, and collecting data.

1. Building network

The network is built by nodes and edges. Normally an edge connects only two nodes, each on one end. In some special cases, a node may connect to multiple (or none) incoming and/or outgoing nodes.

While nodes are highly customizable, some typical nodes are pre-defined as follows:

  • Broadcast: a node transfers packet from one source to multiple targets.

mermaid diagram 1

Mermaid markup
graph LR;
    In --> Broadcast --> Out1;
    Broadcast --> Out2;
    Broadcast --> Out3;
  • Channel: a node delays, losses or reorders packets passing by.

mermaid diagram 2

Mermaid markup
graph LR;
    In --> Channel -->|Loss & Delay & Reorder| Out;
  • Endpoint: a node where to send and receive packets, usually acting as the endpoint of a chain.

mermaid diagram 3

Mermaid markup
graph LR;
   Nodes... --> Endpoint;
  • Gather: a node gathers packets from multiple sources to a single target.

mermaid diagram 4

Mermaid markup
graph LR;
    In1 --> Gather ==> Out;
    In2 --> Gather;
    In3 --> Gather;
  • Restrict: a node limits pps or bps by dropping packets when its internal buffer overflows.

mermaid diagram 5

Mermaid markup
graph LR;
    In --> Restrict -->|Restricted Speed| Out;
  • Scatter: a node selects which node the incoming packet should be route to according to a given rule.

mermaid diagram 6

Mermaid markup
graph LR;
    In --> Scatter -.-> Out1;
    Scatter -->|Selected Route| Out2;
    Scatter -.-> Out3;

After all necessary nodes created, connect them with code to build the network. To do so, just set the next node correctly for each node until all edges are defined as expected.

In addition, ns-x provides a builder to facilitate the process. Instead of connecting edges, it builds the network by connecting all paths in one line of code.

A Path, aka. a chain, is similar to the path concept in graph theory, representing a route along the edges of a graph.

Methods of the builder:

  • Chain(): saves current chain (path) and start to describe another chain.
  • Node(): appends a given node to current chain.
  • NodeWithName(): same as Node(), with a customizable name to refer to later.
  • NodeByName(): finds (refer to) a node with given name and appends it to current chain.
  • NodeGroup(): given a number of nodes, perform Node() operation on each of them in order.
  • NodeGroupWithName(): same as NodeGroup() with a customizable name.
  • NodeGroupByName(): finds a group with the given name, then perform NodeGroup() operation on it.
  • Build(): is the final trigger of the builder to really build the network. Note that in all nodes used in this builder line, all previously established connections will be overwritten.
2. Starting Network Simulation

Once the network built, start running it so packets can go through nodes.

Guaranteed behaviours of the simulation

  • Order: if any event e at time point t, only generate events at time point not before t, then the handling order of two events at different time point is guaranteed, and the order of events at same time point is undetermined.
  • Accuracy: each event will be handled at the given time point exactly in simulate clock, and the difference between the simulator clock and real clock is as small as possible, usually some microseconds.

See comments in the code for additional node-specific guarantees.

3. Collecting Data

Data could be collected by callback function node.OnTransferCallback(). Also note that time-costing callbacks would slow down the simulation and lead to inaccuracy, so it is highly recommended only collecting data in the callbacks. Further analyses should be done after the simulation.

Example

Following is an example of a network with two entries, one endpoint and two chains.

  • Chain 1: entry1 -> channel1(with 30% packet loss rate) -> restrict (1 pps, 1024 bps, buffer limited to 4096 bytes and 5 packets) -> endpoint
  • Chain 2: entry2 -> channel2(with 10% packet loss rate) -> endpoint
package main

import (
	"github.com/bytedance/ns-x/v2"
	"github.com/bytedance/ns-x/v2/base"
	"github.com/bytedance/ns-x/v2/math"
	"github.com/bytedance/ns-x/v2/node"
	"github.com/bytedance/ns-x/v2/tick"
	"go.uber.org/atomic"
	"math/rand"
	"time"
)

func main() {
	source := rand.NewSource(0)
	random := rand.New(source)
	helper := ns_x.NewBuilder()
	callback := func(packet base.Packet, source, target base.Node, now time.Time) {
		println("emit packet")
	}
	n1 := node.NewEndpointNode()
	t := time.Now()
	network, nodes := helper.
		Chain().
		NodeWithName("entry1", n1).
		Node(node.NewChannelNode(node.WithTransferCallback(callback), node.WithLoss(math.NewRandomLoss(0.1, random)))).
		Node(node.NewRestrictNode(node.WithPPSLimit(1, 20))).
		NodeWithName("endpoint", node.NewEndpointNode()).
		Chain().
		NodeWithName("entry2", node.NewEndpointNode()).
		Node(node.NewChannelNode(node.WithTransferCallback(callback), node.WithLoss(math.NewRandomLoss(0.1, random)))).
		NodeOfName("endpoint").
		Summary().
		Build()
	entry1 := nodes["entry1"].(*node.EndpointNode)
	entry2 := nodes["entry2"].(*node.EndpointNode)
	endpoint := nodes["endpoint"].(*node.EndpointNode)
	count := atomic.NewInt64(0)
	endpoint.Receive(func(packet base.Packet, now time.Time) []base.Event {
		if packet != nil {
			count.Inc()
			println("receive packet at", now.String())
			println("total", count.Load(), "packets received")
		}
		return nil
	})
	total := 20
	events := make([]base.Event, 0, total*2)
	for i := 0; i < 20; i++ {
		events = append(events, entry1.Send(base.RawPacket([]byte{0x01, 0x02}), t))
	}
	for i := 0; i < 20; i++ {
		events = append(events, entry2.Send(base.RawPacket([]byte{0x01, 0x02}), t))
	}
	event := base.NewPeriodicEvent(func(now time.Time) []base.Event {
		for i := 0; i < 10; i++ {
			_ = rand.Int()
		}
		return nil
	}, time.Second, t)
	events = append(events, event)
	network.Run(events, tick.NewStepClock(t, time.Second), 300*time.Second)
	defer network.Wait()
}

Design

Architecture

The simulator is event driven, each event will be handled at the given time point, and generate subsequent events. Behaviors of nodes will be wrapped as events.

Event Loop

The event loop maintains a thread-local event queue, in order to sort the events.

Basic operation is dequeue an event from the queue, handle it and enqueue events generated each time. The event loop will drain the event queue until only events with time point after current time are left.

Once the event queue is drained, event loop will refresh current time from the clock, and redo operations above until no events left in the queue or reach lifetime of the simulation.

Event Queue

The event queue is used to sort events to guarantee the handling order.

As observed, most of the events generated just with a short delay, which form an events cluster. For events cluster, bucket sort is used first to divide events into some buckets; other events are put into another bucket.

For each bucket, a heap sort is used to form the priority queue.

Since buckets are created/destroyed frequently, but total count of buckets at the same time are usually within a bound. All the buckets are stored in a ring queue, to reduce the cost and avoid gc.

Contribution

Future work

  • parallelize main loop (done)
  • implement commonly used protocol stack as a new node type (will be implemented as different packet type)
  • separate send and pass to avoid cumulative error (done)
  • Buffer overflow determination of restrict node should have a more accurate way (done)
  • split event heap when size of heap is large enough (done)
  • implement packets of commonly used protocol

Contributors

contributors

Made with contributors-img.

Releases(v2.4.4)
Owner
Bytedance Inc.
Bytedance Inc.
secure scuttlebutt network simulator

Network Simulator a brief exposition on the network simulator to be built as part of SSB's NGI Pointer grant For the remainder of May—and likely parts

null 21 May 13, 2022
A LoRaWAN nodes' and network simulator that works with a real LoRaWAN environment (such as Chirpstack) and equipped with a web interface for real-time interaction.

LWN Simulator A LoRaWAN nodes' simulator to simulate a LoRaWAN Network. Table of Contents General Info Requirements Installation General Info LWN Simu

ARSLab 26 Jul 16, 2022
Go network programming framework, supports multiplexing, synchronous and asynchronous IO mode, modular design, and provides flexible custom interfaces

Go network programming framework, supports multiplexing, synchronous and asynchronous IO mode, modular design, and provides flexible custom interfaces。The key is the transport layer, application layer protocol has nothing to do

rick.wu 10 Jun 27, 2022
:vulcan_salute: Fast, modern, easy-to-use network scanner

sx is the command-line network scanner designed to follow the UNIX philosophy. The goal of this project is to create the fastest network scanner with

null 974 Aug 10, 2022
A modular is an opinionated, easy-to-use P2P network stack for decentralized applications written in Go.

xlibp2p xlibp2p is an opinionated, easy-to-use P2P network stack for decentralized applications written in Go. xlibp2p is made to be minimal, robust,

XFS Network 63 Jul 10, 2022
Xlibp2p: an opinionated, easy-to-use P2P network stack for decentralized applications written in Go

xlibp2p xlibp2p is an opinionated, easy-to-use P2P network stack for decentraliz

null 2 Feb 21, 2022
A simple low bandwidth simulator written in go

NETSNAIL 0.8 ABOUT Netsnail is a simple network proxy that simulates low bandwidth. RUNNING Usage of netsnail: -d=0: the delay on data transfe

Per Arneng 23 May 19, 2021
FLoC Simulator

FLoC Simulator Command line FLoC simulator to calculate CohortID with using host list and cluster data. Build and Run Demo $ cd $ cd demos/floc_sampl

Shigeki Ohtsu 37 May 5, 2022
connect to microsoft flight simulator 2020 using golang

msfs2020-go simconnect package msfs2020-go/simconnect connects to microsoft flight simulator 2020 using golang. cross-compiles from macos/linux, no ot

lian 357 Aug 1, 2022
AutoTrackIR will automatically re-enabled your Track IR if flight simulator disabled it. (bug introduce in SU7)

AutoTrackIR AutoTrackIR will automatically re-enable your Track IR if flight simulator disabled it. (bug introduce in SU7) How it works ? It just watc

Stéphane Depierrepont 6 May 8, 2022
golife is a cell evolution simulator.

golife WORK IN PROGRESS golife is a cell evolution simulator. It presents autonomous organism with inner properties and behavior (hunter or prey) that

Jean-Thierry BONHOMME 0 Dec 30, 2021
Multithreaded Airline Check-In Simulator

Multithreaded Airline Check-In Simulator This is a task scheduler which includes a single queue and 5 clerks. The queue will be sorted based on priori

null 0 Mar 15, 2022
Use Consul to do service discovery, use gRPC +kafka to do message produce and consume. Use redis to store result.

目录 gRPC/consul/kafka简介 gRPC+kafka的Demo gRPC+kafka整体示意图 限流器 基于redis计数器生成唯一ID kafka生产消费 kafka生产消费示意图 本文kafka生产消费过程 基于pprof的性能分析Demo 使用pprof统计CPU/HEAP数据的

null 43 Jul 9, 2022
Gmqtt is a flexible, high-performance MQTT broker library that fully implements the MQTT protocol V3.1.1 and V5 in golang

中文文档 Gmqtt News: MQTT V5 is now supported. But due to those new features in v5, there area lots of breaking changes. If you have any migration problem

null 712 Aug 15, 2022
girc is a flexible IRC library for Go

girc, a flexible IRC library for Go Status girc is fairly close to marking the 1.0.0 endpoint, which will be tagged as necessary, so you will be able

Liam Stanley 78 Jul 21, 2022
A standalone Web Server developed with the standard http library, suport reverse proxy & flexible configuration

paddy 简介 paddy是一款单进程的独立运行的web server,基于golang的标准库net/http实现。 paddy提供以下功能: 直接配置http响应 目录文件服务器 proxy_pass代理 http反向代理 支持请求和响应插件 部署 编译 $ go build ./main/p

fangyousong 5 May 2, 2022
A golang library about socks5, supports all socks5 commands. That Provides server and client and easy to use. Compatible with socks4 and socks4a.

socks5 This is a Golang implementation of the Socks5 protocol library. To see in this SOCKS Protocol Version 5. This library is also compatible with S

chenhao zhang 40 Jul 4, 2022
Package socket provides a low-level network connection type which integrates with Go's runtime network poller to provide asynchronous I/O and deadline support. MIT Licensed.

socket Package socket provides a low-level network connection type which integrates with Go's runtime network poller to provide asynchronous I/O and d

Matt Layher 44 Aug 3, 2022
Optimize Windows's network/NIC driver settings for NewTek's NDI(Network-Device-Interface).

windows-ndi-optimizer[WIP] Optimize Windows's network/NIC driver settings for NewTek's NDI(Network-Device-Interface). How it works This is batchfile d

Nil Hiiragi 3 Apr 15, 2022