wire protocol for multiplexing connections or streams into a single connection, based on a subset of the SSH Connection Protocol

Related tags

Network qmux
Overview

qmux

qmux is a wire protocol for multiplexing connections or streams into a single connection. It is based on the SSH Connection Protocol, which is the simplest, longest running, most widely deployed multiplexing protocol with flow control.

It is meant as a drop-in layer for any stream capable transport (TCP, WebSocket, STDIO, etc) to provide basic multiplexing. This brings any connection API to rough semantic parity with QUIC multiplexing, so it can act as a stopgap or fallback when QUIC is not available. You can then design higher level protocols based on multiplexing semantics that sit on top of QUIC or any other streaming transport with qmux.

Comments
  • Use memory-aligned data structures in qmux protocol

    Use memory-aligned data structures in qmux protocol

    I am currently looking into implementing a TCP-in-TCP connection tunneling protocol very similar to qmux or OpenSSH in Devolutions Gateway + jetsocat, a project written in Rust. The cleaned up protocol specification used in qmux is exactly what I'm looking for, but I've noticed that the message headers haven't been modified to respect memory alignment rules commonly enforced in certain types of processors like ARM. To read a uint32 (4-bytes) then it should be at a memory address that is a multiple of 4 bytes, etc. This is the primary reason why a lot of protocols try hard to align their data structures, and why compilers automatically pad data structures that don't respect those rules. You can still read data structures that are not memory aligned, but it comes with a performance cost.

    I know this would be a breaking change, but would you be interested in making the change anyway? I will likely implement the memory aligned variant on my side and keep everything else as-is. Here is what the current header structure looks like:

      byte      QMUX_MSG_CHANNEL_OPEN
      uint32    sender channel
      uint32    initial window size
      uint32    maximum packet size
    

    And what a memory-aligned variant would look like:

      byte      QMUX_MSG_CHANNEL_OPEN
      byte     reserved[3]
      uint32    sender channel
      uint32    initial window size
      uint32    maximum packet size
    

    This would free 1 byte and a uint16 for other use like flags, etc, if you have such needs. What do you think? Everything else that follows would be nicely memory-aligned with those 3 bytes.

    opened by awakecoding 6
  • golang/session: initial implementation and basic test

    golang/session: initial implementation and basic test

    This is the meat of the implementation. Although I'd like better test coverage and comments, I'm particularly concerned about the use of contexts. I don't believe they are used where given or appropriate where given so I'd like to think about them more.

    Also, I don't often centralize errors into predefined variables, but it might be a good idea to here?

    Double check min-max limits ... there are packet sizes and data packet sizes and I'm not sure they're correctly used?

    After this there is simply a transport subpackage with various transport protocol dialers and such.

    If synchronous review works better, schedule a time with me!

    opened by progrium 5
  • javascript: test channel communication

    javascript: test channel communication

    Debugged some issues with reads blocking.

    This still has a bunch of debugging logs and hacky code, but putting it up for some feedback on my attempts to test this.

    opened by mgood 2
  • Constant vhost errors from server

    Constant vhost errors from server

    These are just from a single client connection: Failed to extract vhost name: read tcp 45.76.255.14:4415->173.44.95.183:50518: i/o timeout Failed to extract vhost name: read tcp 45.76.255.14:4415->173.44.95.183:50524: i/o timeout Failed to extract vhost name: read tcp 45.76.255.14:4415->173.44.95.183:50528: i/o timeout Failed to extract vhost name: read tcp 45.76.255.14:4415->173.44.95.183:50550: i/o timeout Failed to extract vhost name: read tcp 45.76.255.14:4415->173.44.95.183:50552: i/o timeout Failed to extract vhost name: read tcp 45.76.255.14:4415->173.44.95.183:50558: i/o timeout Failed to extract vhost name: read tcp 45.76.255.14:4415->173.44.95.183:50562: i/o timeout

    Client and Server running on two different linux systems

    Server DNS has wildcard subdomains set Server bound to 0.0.0.0 Server port 4415

    Running client on two different PCs simultaneously Client connects to server with port 3415 Running quick test using mini-httpd listening on port 3415 with simple 2-line output html file

    Browsers using generated address to connect to other PC successfully - html displayed

    Plan is to use groktunnel to do crypto wallet transfers (EPIC Cash) to wallet on LAN behind router (using ngrok currently but don't trust them to keep the free access going so we need our own option).

    Load issue with server throwing constant errors even if sending to /dev/nul ?

    Can Server handle thousands of concurrent client connections?

    I tried a wallet transfer from a mining pool to local wallet (these transactions were fine with localtunnel). It's a very small data stream. The server threw a Failed to extract vhost name: EOF and ended the session. Client shows EOF and Exit. So it looks like groktunnel is very http specific and will not allow just the http header then an open tcp tunnel like localtunnel. Will have to use that instead.

    opened by gbkelley 1
  • A few queries regarding using this tool

    A few queries regarding using this tool

    I switched my internet connection recently and they use Carrier Grade NAT.

    1. If I install this on a Linode server, can I install the same code on my local server (Raspberry PI) to enable incoming connections by talking to my Linode server running the same?
    2. Also does this mean, all traffic from outside my home via this grok server (Linode) to my home PI will go through Linode? Or is it just network translation? I am concerned about possible bandwidth usage.
    opened by mathewparet 1
  • javascript: queue generic typing

    javascript: queue generic typing

    Adds a type parameter to queue to help with type checking, and updates the method signatures to include undefined where appropriate.

    Also adds specific types to Array<Function>.

    opened by mgood 1
  • javascript: make codec test match API

    javascript: make codec test match API

    Looks like there were some API changes that weren't integrated into the codec test.

    This looks like it was meant to test what's now Unmarshal and Marshal. Since Unmarshal returns a Message which only has ID, that doesn't match AnyMessage which corresponds to more specific types. The cast here doesn't seem ideal, but should I change the return type or parameter to be compatible?

    opened by mgood 1
  • golang/transport: add tests

    golang/transport: add tests

    Adds tests for the current transports.

    I'm getting a spurious failure for the Pipe-based IO test that I still need to investigate if it's specific to that test implementation, or something else:

    --- FAIL: TestIO (0.00s)
        --- FAIL: TestIO/channel_read (0.00s)
            qmux/golang/transport/transport_test.go:43: io: read/write on closed pipe
    
    opened by mgood 1
  • golang/session: additional tests

    golang/session: additional tests

    Start adding a couple additional tests for session package:

    • context timeout on Open
    • Session.Wait

    Also marks fatal as a test helper so that errors show the caller's line number instead of the line inside fatal.

    opened by mgood 1
  • golang/codec: initial implementation

    golang/codec: initial implementation

    Rather nice and readable. Could use some tests for error cases, but otherwise >90% coverage.

    How important are godoc comments with the spec right there?

    opened by progrium 1
  • typescript: transport for Deno websockets

    typescript: transport for Deno websockets

    Implements a TypeScript transport based on Deno websockets.

    Extracts the buffering of data for pending reads into a ReadBuffer class to reuse.

    Fixes #17

    opened by mgood 0
  • encount error when i try to run my own server instance

    encount error when i try to run my own server instance

    Hi! Thanks for this project! it looks awesome.

    I'm trying to run my own instance on VPS using docker. i can establish connection between client and server. i get back a randomized url but when in try to use it i get this kind of error:

    go_srv_1  | 2022/01/26 17:25:10 groktunnel server [grok.greenponik.com] ready!
    go_srv_1  | 2022/01/26 17:25:11 75npf5e8b1.grok.greenponik.com:9999: start session
    go_srv_1  | 2022/01/26 17:25:11 8m0bcmc5cm.grok.greenponik.com:9999: start session
    go_srv_1  | Failed to extract vhost name: invalid method "\x16\x03\x01\x02\x00\x01\x00\x01\xfc\x03\x03\xf3\xf2V\x9f;dx\xb9y]\t&\x10Q\xcbL\x97\x0f\xd9"
    
    

    On server side: i just change default value for port, host and addr in main.go (args aren't takes when i use them on Dockerfile script) main.go:

            var port = flag.String("p", "9999", "server port to use")
            var host = flag.String("h", "grok.greenponik.com", "server hostname to use")
            var addr = flag.String("b", "54.37.11.133", "ip to bind [server only]")
            flag.Parse()
    

    here my Dockerfile:

    FROM golang:1.16-alpine
    
    WORKDIR /app
    
    COPY ./go.mod .
    COPY ./go.sum .
    RUN go mod download
    
    COPY . /app
    
    RUN go build
    
    CMD ["./groktunnel"]
    

    here my docker-compose.yml:

    version: "3.8"
    
    services:
      go_srv:
        build:
          context: .
          dockerfile: Dockerfile.server
        image: go_srv
        network_mode: host
        restart: always
    

    on client side: main.go:

    	var port = flag.String("p", "9999", "server port to use")
    	var host = flag.String("h", "grok.greenponik.com", "server hostname to use")
    	var addr = flag.String("b", "127.0.0.1", "ip to bind [server only]")
    	flag.Parse()
    

    Dockerfile:

    
    WORKDIR /app
    
    COPY ./go.mod .
    COPY ./go.sum .
    RUN go mod download
    
    COPY . /app
    
    RUN go build
    
    CMD ["./groktunnel", "8000"]
    

    docker-compose.yml:

    version: "3.8"
    
    services:
      tunnel:
        build:
          context: .
          dockerfile: Dockerfile
        image: go_srv
        restart: always
    

    How an i debug this error? is it about the vhost "github.com/inconshreveable/go-vhost" component?

    opened by MkLHX 2
  • typescript: split transport into deno and browser modules

    typescript: split transport into deno and browser modules

    this way transport/index.ts can check for deno and import/re-export the deno module transports, otherwise import/re-export the browser module transports.

    opened by progrium 2
Owner
Jeff Lindsay
Jeff Lindsay
network multiplexing and framing protocol for RPC

TChannel Network multiplexing and framing protocol for RPC Read the Docs Languages: Node.js, Python, Go, Java Questions: Open a Github issue Uber's OS

Uber Open Source 1.2k Nov 4, 2022
A Stable & Secure Tunnel based on KCP with N:M multiplexing and FEC. Available for ARM, MIPS, 386 and AMD64。KCPプロトコルに基づく安全なトンネル。KCP 프로토콜을 기반으로 하는 보안 터널입니다。

Disclaimer: kcptun maintains a single website — github.com/xtaci/kcptun. Any websites other than github.com/xtaci/kcptun are not endorsed by xtaci. Re

xtaci 13.1k Nov 20, 2022
🤘 The native golang ssh client to execute your commands over ssh connection. 🚀🚀

Golang SSH Client. Fast and easy golang ssh client module. Goph is a lightweight Go SSH client focusing on simplicity! Installation ❘ Features ❘ Usage

Mohamed El Bahja 1.2k Nov 27, 2022
Gsshrun - Running commands via ssh on the server/hosting (if ssh support) specified in the connection file

Gsshrun - Running commands via ssh on the server/hosting (if ssh support) specified in the connection file

Məhəmməd 2 Sep 8, 2022
TritonHTTP - A simple web server that implements a subset of the HTTP/1.1 protocol specification

TritonHTTP Spec Summary Here we provide a concise summary of the TritonHTTP spec. You should read the spec doc for more details and clarifications. HT

Boyu Chen 1 Nov 5, 2022
Go network programming framework, supports multiplexing, synchronous and asynchronous IO mode, modular design, and provides flexible custom interfaces

Go network programming framework, supports multiplexing, synchronous and asynchronous IO mode, modular design, and provides flexible custom interfaces。The key is the transport layer, application layer protocol has nothing to do

rick.wu 11 Nov 7, 2022
A little library for turning TCP connections into go channels.

netutils By Tim Henderson ([email protected]) This is a little library that was part of a larger project that I decided to pull out and make public.

Tim Henderson 11 Aug 13, 2020
Connect your devices into a single private WireGuard®-based mesh network.

Wiretrustee A WireGuard®-based mesh network that connects your devices into a single private network. Why using Wiretrustee? Connect multiple devices

null 3.8k Nov 28, 2022
SSHWaiterUtil - Wait for file to appear over an SSH connection

SSHWaiterUtil Simple util to wait for a remote file to appear, over SSH using pr

George Johnson 0 Jan 11, 2022
“Dear Port80” is a zero-config TCP proxy server that hides SSH connection behind a HTTP server!

Dear Port80 About The Project: “Dear Port80” is a zero-config TCP proxy server that hides SSH connection behind a HTTP server! +---------------------

Abbas Gheydi 6 Jun 29, 2022
Eventproc - A proof-of-concept for exploring event based architecture utilizing bi-directional gRPC streams

eventproc A proof-of-concept for exploring event based architecture utilizing bi

null 0 Jan 25, 2022
Extended ssh-agent which supports git commit signing over ssh

ssh-agentx ssh-agentx Rationale Requirements Configuration ssh-agentx Configuration ssh-gpg-signer Linux Windows Signing commits after configuration T

Wim 10 Jun 29, 2022
Golang `net/rpc` over SSH using installed SSH program

Golang net/rpc over SSH using installed SSH program This package implements a helper functions to launch an RPC client and server. It uses the install

null 1 Nov 16, 2022
one simple git ssh server (just for learning git over ssh )

wriet one simple git ssh server use golang write one simple git ssh server how to running starting service docker-compose up -d add authorized_keys i

rong fengliang 2 Mar 5, 2022
Totem - A Go library that can turn a single gRPC stream into bidirectional unary gRPC servers

Totem is a Go library that can turn a single gRPC stream into bidirectional unar

Joe Kralicky 2 Oct 12, 2022
Uses the Finger user information protocol to open a TCP connection that makes a request to a Finger server

Finger Client This client uses the Finger user information protocol to open a TCP connection that makes a request to a Finger server. Build and Run Ru

Linda Xiao 0 Oct 7, 2021
Walrus 🕑 Real-time event streaming platform built on top of gRPC streams

Walrus ?? Real-time event streaming platform built on top of gRPC streams Table of Contents About the project Built With How it works Getting Started

Matheus Mosca 14 Sep 24, 2022
Reflex stream client for redis streams

rredis A reflex stream client for a redis streams using the radix client implementation. It provides an API for inserting data into a stream and for c

null 0 Oct 5, 2021