Like Go channels over the network

Overview

libchan: like Go channels over the network

Circle CI

Libchan is an ultra-lightweight networking library which lets network services communicate in the same way that goroutines communicate using channels:

  • Simple message passing
  • Synchronization for concurrent programming
  • Nesting: channels can send channels

Libchan supports the following transports out of the box:

  • In-memory Go channel
  • Unix socket
  • Raw TCP
  • TLS
  • HTTP2/SPDY
  • Websocket

This provides great flexibility in scaling an application by breaking it down into loosely coupled concurrent services. The same application could be composed of goroutines communicating over in-memory channels; then transition to separate unix processes, each assigned to a processor core, and communicating over high-performance IPC; then to a cluster of machines communicating over authenticated TLS sessions. All along it benefits from the concurrency model which has made Go so popular.

Not all transports have the same semantics. In-memory Go channels guarantee exactly-once delivery; TCP, TLS, and the various HTTP socket families do not guarantee delivery. Messages arrive in order but may be arbitrarily delayed or lost. There are no ordering invariants across channels.

An explicit goal of libchan is simplicity of implementation and clarity of spec. Porting it to any language should be as effortless as humanly possible.

Focused on not reinventing the wheel

Because remote libchan sessions are regular HTTP2 over TLS sessions, they can be used in combination with any standard proxy or authentication middleware. This means libchan, when configured properly, can be safely exposed on the public Internet. It can also be embedded in an existing rest API using an http1 and websocket fallback.

How is it different from RPC or REST?

Modern micro-services are not a great fit for classical RPC or REST protocols because they often rely heavily on events, bi-directional communication, stream multiplexing, and some form of data synchronization. Sometimes these services have a component which requires raw socket access, either for performance (file transfer, event firehose, database access) or simply because they have their own protocol (dns, smtp, sql, ssh, zeromq, etc). These components typically need a separate set of tools because they are outside the scope of the REST and RPC tools. If there is also a websocket or ServerEvents transport, those require yet another layer of tools.

Instead of a clunky patchwork of tools, libchan implements in a single minimalistic library all the primitives needed by modern micro-services:

  • Request/response with arbitrary structured data

  • Asynchronous events flowing in real-time in both directions

  • Requests and responses can flow in any direction, and can be arbitrarily nested, for example to implement a self-registering worker model

  • Any message serialization format can be plugged in: json, msgpack, xml, protobuf.

  • Raw file descriptors can be "attached" to any message, and passed under the hood using the best method available to each transport. The Go channel transport just passes os.File pointers around. The unix socket transport uses fd passing which makes it suitable for high-performance IPC. The tcp transport uses dedicated http2 streams. And as a bonus extension, a built-in tcp gateway can be used to proxy raw network sockets without extra overhead. That means libchan services can be used as smart gateways to a sql database, ssh or file transfer service, with unified auth, discovery and tooling and without performance penalty.

Example usage

Here's an example implementing basic RPC-style request/response. We gloss over error handling to tersely demonstrate the core concepts.

On the client:

var ch libchan.Sender

// Send a message, indicate that we want a return channel to be automatically created
ret1, err := ch.Send(&libchan.Message{Data: []byte("request 1!"), Ret: libchan.RetPipe})

// Send another message on the same channel
ret2, err := ch.Send(&libchan.Message{Data: []byte("request 2!"), Ret: libchan.RetPipe})

// Wait for an answer from the first request.  Set flags to zero
// to indicate we don't want a nested return channel.
msg, err := ret1.Receive(0)

On the server:

var ch libchan.Receiver

// Wait for messages in a loop
// Set the return channel flag to indicate that we
// want to receive nested channels (if any).
// Note: we don't send a nested return channel, but we could.
for {
	msg, err := ch.Receive(libchan.Ret)
	msg.Ret.Send(&libchan.Message{Data: []byte("this is an extremely useful response")});
}

Creators

Solomon Hykes

Additional Implementations

Java

Javascript / Node.js

Copyright and license

Code and documentation copyright 2013-2014 Docker, inc. Code released under the Apache 2.0 license. Docs released under Creative commons.

Comments
  • New channel interface

    New channel interface

    Add a new single channel interface for sending receiving with support for dynamic types. Any type can contain channels or bytestreams as field values and communicate them over a channel. This is a work in progress and up for early review.

    opened by dmcgowan 14
  • Update listener to use stream session

    Update listener to use stream session

    Listener is currently broken when multiple connections are created, using sessions allows proper separation of each connection. A sync condition is added around stream chans to avoid race conditions between setting stream chans and reading them. The original behavior used the global stream chan when the stream was not found, however that led to synchronization issues when sender/receivers sometimes got back different channels.

    opened by dmcgowan 10
  • Streamchan implementation

    Streamchan implementation

    The network implementation is not transport agnostic. Netchan makes use of the stream provider and encoding interface.

    This PR may be split up and dependent on other PRs for new msgpack implementation, protocol updates, and encoding interface. Created this PR now to being reviewing interface.

    Depends on #85

    opened by dmcgowan 8
  • New msgpack implementation

    New msgpack implementation

    I mark this as a work in progress since it is not ready to be merged. The code design may be reviewed since I plan on using this structure.

    Includes:

    • Different msgpack implementation
    • New raw message type
    • New extension implementation

    What this enables:

    • More efficient proxying
    • Zero copy in-memory channels
    • Partial Decoding
    • Sending/receiving non struct/map objects
    opened by dmcgowan 8
  • Suggested order of implementation for the libchan protocol

    Suggested order of implementation for the libchan protocol

    I'm in the process of researching and designing a node.js/javascript implementation of libchan (i'm calling it Graft).

    I'm planning to use node streams as the base of the abstraction, and i'm aiming for a api similar in usage to gulp.

    What I'm wondering is, what you would consider the simplest proof of concept to bootstrap this implementation, so that I can iterate on the various components from there.

    So I guess, what does 'hello world' look like for libchan?

    opened by AdrianRossouw 8
  • Fix references to dotcloud repo.

    Fix references to dotcloud repo.

    Fixes running tests in clean environment.

    go test ./...
    # github.com/docker/libchan
    inmem_test.go:9:2: cannot find package "github.com/dotcloud/docker/pkg/testutils" in any of:
        /Users/pnasrat/Library/homebrew/Cellar/go/1.3/libexec/src/pkg/github.com/dotcloud/docker/pkg/testutils (from $GOROOT)
        /Users/pnasrat/go/src/github.com/dotcloud/docker/pkg/testutils (from $GOPATH)
    FAIL    github.com/docker/libchan [setup failed]
    # github.com/docker/libchan/unix
    inmem_test.go:9:2: cannot find package "github.com/dotcloud/docker/pkg/testutils" in any of:
        /Users/pnasrat/Library/homebrew/Cellar/go/1.3/libexec/src/pkg/github.com/dotcloud/docker/pkg/testutils (from $GOROOT)
        /Users/pnasrat/go/src/github.com/dotcloud/docker/pkg/testutils (from $GOPATH)
    FAIL    github.com/docker/libchan/unix [setup failed]
    # github.com/docker/libchan/utils
    inmem_test.go:9:2: cannot find package "github.com/dotcloud/docker/pkg/testutils" in any of:
        /Users/pnasrat/Library/homebrew/Cellar/go/1.3/libexec/src/pkg/github.com/dotcloud/docker/pkg/testutils (from $GOROOT)
        /Users/pnasrat/go/src/github.com/dotcloud/docker/pkg/testutils (from $GOPATH)
    FAIL    github.com/docker/libchan/utils [setup failed]
    ok      github.com/docker/libchan/data  0.006s
    ok      github.com/docker/libchan/http2 0.045s
    
    opened by pnasrat 7
  • request free ports from the OS

    request free ports from the OS

    Tests were hardcoded to ports that may not be available on the system. Some ports were reused in subsequent tests which caused tests to occasionally fail when the system didn't release these ports.

    opened by drewwells 6
  • Clarify channel semantics

    Clarify channel semantics

    The readme says:

    network services communicate in the same way that goroutines communicate using channels:

    And the Golang spec, regarding asynchronous channels, says:

    elements are received in the order they are sent

    How does libchan enforce the ordering/delivery properties of golang channels when the underlying transport is disrupted--e.g., when re-establishing a TCP link that dropped, how does the channel state machine negotiate redelivery of incomplete in-flight messages? Do libchan channels yield at-least-once delivery? At-most-once? Exactly once? Are there any ordering invariants, or can messages be arbitrarily reordered? If ordering is preserved, what are the space costs on both sides of the network?

    opened by aphyr 6
  • http2 fd too many files

    http2 fd too many files

    hi,

    i'm implementing a server with libchan and http2, they work well, but they fail after some minutes with the error fd too many files open,when debug see the descriptor for libchan.Message.Fd create by the http2.StreamReceiver.Receive method used on the server, they grow and grow.. look, simple test, ulimit -n 80 then running de server the Fd grow:

    • adialer-handler2014/07/09 00:01:14 msg.Fd &{%!s(*os.file=&{9 0})}
    • adialer-handler2014/07/09 00:01:15 msg.Fd &{%!s(*os.file=&{13 0})}
    • adialer-handler2014/07/09 00:01:15 msg.Fd &{%!s(*os.file=&{15 0})}
    • adialer-handler2014/07/09 00:01:17 msg.Fd &{%!s(*os.file=&{23 0})}
    • adialer-handler2014/07/09 00:01:29 msg.Fd &{%!s(*os.file=&{29 0})}
    • ...
    • adialer-handler2014/07/09 00:01:34 msg.Fd &{%!s(*os.file=&{72 0})}

    and boom!, i'm find a solution and it's closing the descriptor (msg.Fd.Close()) when then server finish the response,

    there a better way?

    thanks

    opened by bit4bit 5
  • Use new msgpack implementation

    Use new msgpack implementation

    This change include use of a new msgpack implementation based on a fork of github.com/vmihailenco/msgpack. The changes were to support libchan were extended types, interface encoding, raw messages, and removing of a non-standard time encoding.

    Additional cleanup changes to support this PR include removing of the raw connection encoding in the spdy implementation which was never defined in the libchan protocol.

    Includes:

    • Different msgpack implementation
    • New raw message type
    • New extension implementation

    What this enables:

    • More efficient proxying
    • Zero copy in-memory channel transfer
    • Partial Decoding
    • Sending/receiving non struct/map objects

    fixes #47

    in progress 
    opened by dmcgowan 4
  • [WIP] Bus implementation

    [WIP] Bus implementation

    Implementation of a bus using libchan to pass messages. There is a simple registration and message model to facilitate routing messages to clients connected to the bus. The design and implementation is in progress as this is just a prototype. There is also a higher level plugin interface being prototyped using this but not included in this PR.

    opened by dmcgowan 4
  • 
Fix error formatting based on best practices from Code Review Comments

    Fix error formatting based on best practices from Code Review Comments

    opened by CodeLingoBot 0
  • How to migrate code from v.0.1.0 version ?

    How to migrate code from v.0.1.0 version ?

    I'm trying to do the build of https://github.com/bfosberry/banano/blob/master/server/main.go code and I am getting the following errors:

    • undefined: spdy.NewTransportListener
    • undefined: spdy.NoAuthenticator

    The code is:

    listener, err := net.Listen("tcp", fmt.Sprintf(":%s", port))
    if err != nil {
        log.Fatal(err)
    }
    
    tl, err := spdy.NewTransportListener(listener, spdy.NoAuthenticator)
    if err != nil {
        log.Fatal(err)
    }
    

    This code run with v.0.1.0 of libchain

    I haven't found documentation of changes to be made in code when we want to migrate to the current version of libchain.

    opened by joao-parana 2
  • Encoding interface

    Encoding interface

    Adds encoding package to define encoding interface. Updates spdy package msgpack encoding to using new interface. Moves msgpack encoding into separate package.

    opened by dmcgowan 0
Owner
Docker
Docker provides a simple and powerful developer experience, workflows and collaboration for creating applications.
Docker
More effective network communication, two-way calling, notify and broadcast supported.

ARPC - More Effective Network Communication Contents ARPC - More Effective Network Communication Contents Features Performance Header Layout Installat

lesismal 605 Sep 21, 2022
Parallel Digital Universe - A decentralized identity-based social network

Parallel Digital Universe Golang implementation of PDU. What is PDU? Usage Development Contributing PDU PDU is a decentralized identity-based social n

PDU.PUB 40 Sep 26, 2022
A Golang implementation of the Umee network, a decentralized universal capital facility in the Cosmos ecosystem.

Umee A Golang implementation of the Umee network, a decentralized universal capital facility in the Cosmos ecosystem. Umee is a Universal Capital Faci

null 153 Sep 25, 2022
Network connecter for storage.

database Quick fast connection to database use gorm. Installation $ go get -u github.com/coolstina/connecter Example package main import ( "fmt" "

coolstina 0 Dec 4, 2021
An implementation of a distributed KV store backed by Raft tolerant of node failures and network partitions ๐Ÿšฃ

barge A simple implementation of a consistent, distributed Key:Value store which uses the Raft Concensus Algorithm. This project launches a cluster of

Shehjad Khan 0 Nov 24, 2021
M3U generator for Stirr, optimized for Channels' custom channels.

Stirr for Channels This simple Docker image will generate an M3U playlist and EPG optimized for use in Channels and expose them over HTTP. Channels su

Robbie Trencheny 16 Sep 2, 2022
Guilherme Biff Zarelli 3 Jun 6, 2022
network-node-manager is a kubernetes controller that controls the network configuration of a node to resolve network issues of kubernetes.

Network Node Manager network-node-manager is a kubernetes controller that controls the network configuration of a node to resolve network issues of ku

kakao 98 Sep 22, 2022
K8s-network-config-operator - Kubernetes network config operator to push network config to switches

Kubernetes Network operator Will add more to the readme later :D Operations The

Daniel Hertzberg 6 May 16, 2022
The devs are over here at devzat, chat over SSH!

Devzat Where are the devs at? Devzat! Devzat is chat over SSH Try it out: ssh sshchat.hackclub.com Add this to ~/.ssh/config: Host chat HostName s

Ishan Goel 1.4k Sep 21, 2022
Send network packets over a TCP or UDP connection.

Packet is the main class representing a single network message. It has a byte code indicating the type of the message and a []byte type payload.

Aero 73 Aug 12, 2022
Bulk query SQLite database over the network

SQLiteQueryServer Bulk query SQLite database over the network. Way faster than SQLiteProxy!

Assaf Morami 48 May 20, 2022
Go error library with error portability over the network

cockroachdb/errors: Go errors with network portability This library aims to be used as a drop-in replacement to github.com/pkg/errors and Go's standar

CockroachDB 1.4k Sep 23, 2022
Go (Golang) encrypted deep learning library; Fully homomorphic encryption over neural network graphs

DC DarkLantern A lantern is a portable case that protects light, A dark lantern is one who's light can be hidden at will. DC DarkLantern is a golang i

Raven 1 Dec 2, 2021
๐Ÿ‘ท Library for safely running groups of workers concurrently or consecutively that require input and output through channels

Examples Quickstart Multiple Go Workers Passing Fields Getting Started Pull in the dependency go get github.com/catmullet/go-workers Add the import to

Robert Catmull 144 Sep 22, 2022
A little library for turning TCP connections into go channels.

netutils By Tim Henderson ([email protected]) This is a little library that was part of a larger project that I decided to pull out and make public.

Tim Henderson 11 Aug 13, 2020
Pusher Channels HTTP API library for Go

Pusher Channels HTTP Go Library The Golang library for interacting with the Pusher Channels HTTP API. This package lets you trigger events to your cli

Pusher 185 Sep 27, 2022
dfg - Generates dockerfiles based on various input channels.

dfg - Dockerfile Generator dfg is both a go library and an executable that produces valid Dockerfiles using various input channels. Table of Contents

Ozan KaลŸฤฑkรงฤฑ 134 Sep 2, 2022
Go iter tools (for iterating , mapping, filtering, reducing streams -represented as channels-)

Go iter tools (for iterating , mapping, filtering, reducing streams -represented as channels-)

Hulne 14 Jul 27, 2022