Golang connection multiplexing library

Related tags

Network yamux
Overview

Yamux

Yamux (Yet another Multiplexer) is a multiplexing library for Golang. It relies on an underlying connection to provide reliability and ordering, such as TCP or Unix domain sockets, and provides stream-oriented multiplexing. It is inspired by SPDY but is not interoperable with it.

Yamux features include:

  • Bi-directional streams
    • Streams can be opened by either client or server
    • Useful for NAT traversal
    • Server-side push support
  • Flow control
    • Avoid starvation
    • Back-pressure to prevent overwhelming a receiver
  • Keep Alives
    • Enables persistent connections over a load balancer
  • Efficient
    • Enables thousands of logical streams with low overhead

Documentation

For complete documentation, see the associated Godoc.

Specification

The full specification for Yamux is provided in the spec.md file. It can be used as a guide to implementors of interoperable libraries.

Usage

Using Yamux is remarkably simple:

func client() {
    // Get a TCP connection
    conn, err := net.Dial(...)
    if err != nil {
        panic(err)
    }

    // Setup client side of yamux
    session, err := yamux.Client(conn, nil)
    if err != nil {
        panic(err)
    }

    // Open a new stream
    stream, err := session.Open()
    if err != nil {
        panic(err)
    }

    // Stream implements net.Conn
    stream.Write([]byte("ping"))
}

func server() {
    // Accept a TCP connection
    conn, err := listener.Accept()
    if err != nil {
        panic(err)
    }

    // Setup server side of yamux
    session, err := yamux.Server(conn, nil)
    if err != nil {
        panic(err)
    }

    // Accept a stream
    stream, err := session.Accept()
    if err != nil {
        panic(err)
    }

    // Listen for a message
    buf := make([]byte, 4)
    stream.Read(buf)
}
Issues
  • Fix window update, reduce memory utilization and improve performance

    Fix window update, reduce memory utilization and improve performance

    improve memory utilization in receive buffer, fix flow control

    • flow control was assuming a Read consumed the entire buffer
    • introduced fixed receive buffer on streams to reduce memory utilization when receiving large streams of data
    • use timer pool to reduce allocations
    • use static handler function pointer to avoid closure allocations for every frame

    Obligatory benchmarks

    benchmark                    old ns/op     new ns/op     delta
    BenchmarkSendRecv-2          3942          3655          -7.28%
    BenchmarkSendRecvLarge-2     28408768      23931960      -15.76%
    
    benchmark                    old allocs     new allocs     delta
    BenchmarkSendRecv-2          6              2              -66.67%
    BenchmarkSendRecvLarge-2     4012           1236           -69.19%
    
    benchmark                    old bytes     new bytes     delta
    BenchmarkSendRecv-2          302           83            -72.52%
    BenchmarkSendRecvLarge-2     1538987       57536         -96.26%
    
    opened by stuartcarnie 20
  • optimize ping : if the data is flowing into recvLoop, ping is not required

    optimize ping : if the data is flowing into recvLoop, ping is not required

    consider the following situation:

    1. a link has 200KB/s bandwidth, and the underlying socket buffer is 4MB(large enough)
    2. set yamux.Config MaxStreamWindowSize to sockbuf(large enough)
    3. constantly send data via streams, eg : 2MB/s to make the socket buffer full
    4. then the keepalive cannot respond in 10seconds ( 2MB/200KB)

    I think we don't need to send ping packets when the data is flowing to the receiver.

    opened by xtaci 13
  • Fix failing partial read on last frame of closed session

    Fix failing partial read on last frame of closed session

    If the sending side of a stream closes both their stream and their session, a partial read of the final frame on the receiving side may fail because the receiving side tries to send a window update on a closed session.

    This commit changes Stream.Read() so that it ignores "session shutdown" errors return by Stream.sendWindowUpdate(). This will let Read() calls consume all of the final frame.

    opened by jacobvosmaer 10
  • long hang in stream write

    long hang in stream write

    I finally got around to updating ipfs to use the fixes from #29 and now i'm seeing hangs on the select in stream.go:199. Its reproducible on the live network, but it takes a little while to happen.

    cc @slackpad

    bug 
    opened by whyrusleeping 10
  • Limited of numbers of streams?

    Limited of numbers of streams?

    If I read and understand this code correctly, it seems to me that it imposes a hard limit here.

    So what are the options if you would like to use Yamux in a long running process that could use more than math.MaxUint32-1 streams? Most likely not all at the same time of course, but as the preferred way to manage stream ID's seem to be to increment the ID with 1 you will hit the limit sooner or later.

    So could this logic be updated/changed without breaking existing stuff? And/or are there other ways to (safely) be able to reuse old ID's of streams that aren't used anymore?

    opened by svanharmelen 8
  • OpenStream hanging on synCh

    OpenStream hanging on synCh

    over on the ipfs project we updated the version of yamux we were using to the latest master from 9feabe6 about a month ago. Since then we started noticing random hanging issues, which i got around to debugging today and it looks like the culprit is the synCh channel on the session. Commits that don't contain that logic work just fine, but every commit i've tried with it has hung.

    It looks to be an issue with not replacing tokens in the semaphore in all needed cases, maybe a call to Close isnt actually releasing a stream the way it should?

    Increasing the AcceptBacklog in the config 'fixes' the problem, but that just seems like its just buying me more time until it hangs again.

    opened by whyrusleeping 7
  • How to deal with ErrRecvWindowExceeded

    How to deal with ErrRecvWindowExceeded

    I am working on a port-forwarding service to replace an existing sshd based service, as that one encrypts and decrypts data which is not needed in my case. I started working with yamux, and it looks like I have it basically working. My test scenario so far is putting a forwarded HTTP server under siege, and it fails after a couple of transfers. The number itself varies, usually after 20 to 100 MB yamux returns ErrRecvWindowExceeded. I looked into this, and in that case atomic.LoadUint32(&s.recvWindow) is always 0.

    If I just disable the test below for ErrRecvWindowExceeded, everything works perfect. What could I do wrong here?

    if length > atomic.LoadUint32(&s.recvWindow) {
        s.session.logger.Printf("[ERR] yamux: receive window exceeded")
        return ErrRecvWindowExceeded
    }
    
    opened by radiospiel 6
  • wait for both recv and send routines to complete on close

    wait for both recv and send routines to complete on close

    this change ensures that there is no write operations after close is called. This was not the case before and this led to race conditions in the caller code because there was no way to ensure there would not be any write anymore after the close.

    opened by clems4ever 5
  • don't block writes after session has shut down

    don't block writes after session has shut down

    This should address the issues in #45 . I've applied this to my machine thats been able to semi-reliably reproduce the issue, so we'll see if it crops up again.

    opened by whyrusleeping 5
  • potential race condition in writing while connection drops

    potential race condition in writing while connection drops

    I've been having some issues with yamux writes hanging lately. See goroutine 14008165 in: https://ipfs.io/ipfs/QmQn8YDTHWJzF7GVKUKanBzoxbzd5jQpwsgM2PfFVTKuGR

    I think its also correlated with the internet connection of the machine its running on dropping during a write. Heres what i suspect is happening:

    • A write on a stream goes through, and consumes the entire sendWindow (resulting in s.sendWindow being not zero)
    • Another write comes in, and is delegated to the WAIT code (where we see the write being stuck in the linked stack trace)
    • At this point, the connection drops, resulting in us not receiving the window update
    • Now, since the writer is waiting on either a timeout (which by default is not set) or a notification is received on the recvNotifyCh, we have to hope we receive that notification.

    I'm not sure if the above is exactly whats happening, but i'm quite confident that if we somehow ended up in the write wait loop after the stream has been closed, its possible that the sendNotifyCh signal got missed and we will block forever. To address that possibility, I think that we should close the notify channels when the streams get closed, so that they are always ready to receive on.

    cc @slackpad

    opened by whyrusleeping 5
  • Fixes #12 KeepAlive is not working

    Fixes #12 KeepAlive is not working

    This adds a timeout of ConnectionWriteTimeout for pings. If a keepalive fails, the connection is closed.

    Modify test code to turn off keepalives on tests that want to fail for other reasons (since keepalive will tend to fail them faster than the situation under test).

    opened by rnapier 5
  • stream recvWindow decremented incorretly

    stream recvWindow decremented incorretly

    at stream.readData():

           if _, err := io.Copy(s.recvBuf, conn); err != nil {
    		s.session.logger.Printf("[ERR] yamux: Failed to read stream data: %v", err)
    		s.recvLock.Unlock()
    		return err
    	}
    
    	// Decrement the receive window
    	s.recvWindow -= length <-- this should be number of read bytes.
    	s.recvLock.Unlock()
    

    when there is connection breaks io.Copy() will return partial reads.

    opened by apmattil 4
  • at stream read/write wrap underlying error to returned error

    at stream read/write wrap underlying error to returned error

    I need to recover from errors when connection is broken. So there should be some way to check why the session was closed at client side. e.g

    [ERR] yamux: Failed to read header: read tcp 10.0.1.10:55190->10.0.3.20:443: read: connection timed out

    now read() and write() just return io.EOF

    opened by apmattil 0
  • Add support for closing write ends of streams

    Add support for closing write ends of streams

    When sending payloads of unknown length over a Stream and expecting the server to read it to completion before emitting a response (such as forwarding a byte stream), one difficulty with yamux is communicating that the byte stream's end has been reached.

    One approach is to introduce a higher-level protocol, but when the byte stream is of unknown size this requires essentially reimplementing large parts of yamux's framing protocol.

    A simpler solution is to communicate that EOF has been reached. Yamux provides this capability through (*Stream).Close, but it closes both the read and the write ends, which then prevents the client from reading the response from the server.

    This change introduces a new method, (*Stream).CloseWrite, which only closes the write end of the stream. When encountered on the other end, it sets a flag that the remote's write end has been closed and begins returning EOF from any reads after the receive buffer has been exhausted.

    opened by eandre 1
  • When MaxStreamWindowSize is larger than initialStreamWindow, should lead to short read

    When MaxStreamWindowSize is larger than initialStreamWindow, should lead to short read

    OpenStream call sendWindowUpdate, then sendWindowUpdate will sync window from client to server. incrSendWindow and wirte will happen race. if incrSendWindow first run it is ok, else will lead sendWindow is not the same value to recvWindow, will out of sync.

    // session_test.go
    func TestSendData_Large(t *testing.T) {
        cfg := testConf()
        cfg.MaxStreamWindowSize = 4 * 1024 
        // ...
    }
    // const.go
    const (
        // initialStreamWindow is the initial stream window size
        initialStreamWindow uint32 = 1 * 1024
    )
    
    === RUN   TestSendData_Large
    --- FAIL: TestSendData_Large (5.00s)
            session_test.go:415: short read: 1024
            session_test.go:441: err: stream reset
    panic: timeout [recovered]
            panic: timeout
    
    opened by wllenyj 1
Owner
HashiCorp
Consistent workflows to provision, secure, connect, and run any infrastructure for any application.
HashiCorp
A Stable & Secure Tunnel based on KCP with N:M multiplexing and FEC. Available for ARM, MIPS, 386 and AMD64。KCPプロトコルに基づく安全なトンネル。KCP 프로토콜을 기반으로 하는 보안 터널입니다。

Disclaimer: kcptun maintains a single website — github.com/xtaci/kcptun. Any websites other than github.com/xtaci/kcptun are not endorsed by xtaci. Re

xtaci 13k Aug 3, 2022
network multiplexing and framing protocol for RPC

TChannel Network multiplexing and framing protocol for RPC Read the Docs Languages: Node.js, Python, Go, Java Questions: Open a Github issue Uber's OS

Uber Open Source 1.2k Aug 5, 2022
Go network programming framework, supports multiplexing, synchronous and asynchronous IO mode, modular design, and provides flexible custom interfaces

Go network programming framework, supports multiplexing, synchronous and asynchronous IO mode, modular design, and provides flexible custom interfaces。The key is the transport layer, application layer protocol has nothing to do

rick.wu 10 Jun 27, 2022
🤘 The native golang ssh client to execute your commands over ssh connection. 🚀🚀

Golang SSH Client. Fast and easy golang ssh client module. Goph is a lightweight Go SSH client focusing on simplicity! Installation ❘ Features ❘ Usage

Mohamed El Bahja 1.1k Aug 9, 2022
Send network packets over a TCP or UDP connection.

Packet is the main class representing a single network message. It has a byte code indicating the type of the message and a []byte type payload.

Aero 71 Jul 15, 2022
Package socket provides a low-level network connection type which integrates with Go's runtime network poller to provide asynchronous I/O and deadline support. MIT Licensed.

socket Package socket provides a low-level network connection type which integrates with Go's runtime network poller to provide asynchronous I/O and d

Matt Layher 42 Jul 3, 2022
WebSocket Connection Smuggler

ws-smuggler ws-smuggler is websocket connection smuggling testing tool. It is similar to the this project, but it has been rewritten based on the web

HAHWUL 37 Jun 16, 2022
ConnPool is a thread safe connection pool for net.Conn interface.

ConnPool is a thread safe connection pool for net.Conn interface. It can be used to manage and reuse connections.

Burak Sezer 118 Jul 1, 2022
This package helps establish a websocket connection to the bilibili streaming server.

biliStreamClient This package helps establish a websocket connection to the bilibili streaming server. bilibili直播弹幕的WebSocket协议分析请参考:https://blog.csdn

JINGWEI ZHANG 23 Jun 21, 2022
Uses the Finger user information protocol to open a TCP connection that makes a request to a Finger server

Finger Client This client uses the Finger user information protocol to open a TCP connection that makes a request to a Finger server. Build and Run Ru

Linda Xiao 0 Oct 7, 2021
Make TCP connection storm between server and client for benchmarking network stuff

Make TCP connection storm between server and client for benchmarking network stuff

Masahiro Nagano 2 Nov 14, 2021
A simple proxy to work with tcp connection

Proxy It is simple proxy to work with tcp connection HTTP TCP Getting Started pr

Altynbek Kaliakbarov 0 Dec 16, 2021
Resolved the issue that Windows cannot detect the Internet even if it does have an Internet connection.

win-connect 中文文档 Background This program is built to resolved the issue that Windows cannot detect the Internet even if it does have an Internet conne

null 0 Dec 19, 2021
SSHWaiterUtil - Wait for file to appear over an SSH connection

SSHWaiterUtil Simple util to wait for a remote file to appear, over SSH using pr

George Johnson 0 Jan 11, 2022
Gsshrun - Running commands via ssh on the server/hosting (if ssh support) specified in the connection file

Gsshrun - Running commands via ssh on the server/hosting (if ssh support) specified in the connection file

Məhəmməd 3 Jun 9, 2022
Go-db-connection-api - API REST in Go that connect to SQL DB and manage task of projects

Go Todo REST API Example A RESTful API example for simple application with Go It

Carlos Andres Toro Guerrero 0 Jan 26, 2022
“Dear Port80” is a zero-config TCP proxy server that hides SSH connection behind a HTTP server!

Dear Port80 About The Project: “Dear Port80” is a zero-config TCP proxy server that hides SSH connection behind a HTTP server! +---------------------

Abbas Gheydi 6 Jun 29, 2022
A library for the MIGP (Might I Get Pwned) protocolA library for the MIGP (Might I Get Pwned) protocol

MIGP library This contains a library for the MIGP (Might I Get Pwned) protocol. MIGP can be used to build privacy-preserving compromised credential ch

Cloudflare 21 Dec 16, 2021