network multiplexing and framing protocol for RPC

Related tags

Network tchannel
Overview

TChannel Build Status

Network multiplexing and framing protocol for RPC

Overview

TChannel is a networking framing protocol used for general RPC, supporting out-of-order responses at extremely high performance where intermediaries can make a forwarding decision quickly. It is easy to implement in multiple languages, especially JavaScript and Python.

Design Goals

  • Easy implementation in multiple languages
  • High performance forwarding path where intermediaries can make forwarding decisions quickly
  • Request/response model with out-of-order responses so that slow requests don't block subsequent faster requests at the head of the line
  • Ability of large requests/responses to be broken into fragments and sent progressively
  • Optional checksums
  • Ability to transport multiple protocols between endpoints (e.g., HTTP+JSON and Thrift)

Components

  • tchannel-protocol TChannel Protocol Documentation
  • tchannel-node TChannel peer library for Node.js
  • tchannel-python TChannel peer library for Python
  • tchannel-go TChannel peer library for Go
  • tchannel-java TChannel peer library for the JVM
  • tcurl TChannel curl program, for making manual one-off requests to TChannel servers
  • tcap TChannel packet capture tool, for eavesdropping and inspecting TChannel traffic

MIT Licensed

Issues
  • Unify and Restructure

    Unify and Restructure

    So far a structure like:

    docs/...
    node/...
    python/...
    go/...
    c++/...
    java/...
    

    Most of what's currently written would be under node/, and we would merge in the .py efforts into python.

    Later we'd add an integration test area.

    We'd convert and land a markdown'd version of the google doc in docs/.

    cc: @Raynos @mranney @sectioneight @kriskowal @jwolski

    opened by jcorbin 15
  • update the thrift meta file for thriftIDL

    update the thrift meta file for thriftIDL

    r @prashantv @Raynos @junchaowu cc @blampe

    @Raynos suggested us to move forward implementing a default handler for thriftIDL ... Node example is here: #1332

    opened by ShanniLi 13
  • docs/protocol: define headers for tchannel language and version

    docs/protocol: define headers for tchannel language and version

    This defines three new init headers:

    • tchannel_language to identify the implementation language
    • tchannel_language_version to identify the version of the implementation language
    • tchannel_version to identify the version of the implementation

    r @breerly @prashantv @mranney @ShanniLi

    opened by jcorbin 12
  • Hyperbahn client should only destroy in HardFail mode

    Hyperbahn client should only destroy in HardFail mode

    When advertisement timeout or other failures happens, hyperbahn client will destroy itself even it's not in HardFail mode. And if advertise is called again, it will trigger AlreadyDestroyed error and return, so hyperbahn client will never be able to get to advertised state again.

    r: @ShanniLi @Raynos @vipulaneja

    opened by anson627 12
  • Drafting Circuit Breaking design.

    Drafting Circuit Breaking design.

    The following PR attempts to:

    • Introduce the problem area and define Circuit Breaking.
    • Choose a simple vocabulary we can use when talking about the problem.
    • Outlines a implementation that prevents cascading failure.
    • Details failure scenarios and what their outcome would be under this design.
    • Highlight challenges we will no doubt face when attempting to implement/rollout.

    Sorry for the typo's and grammatical errors, I didn't get this out till late :sleepy:

    View the rendered markdown here: https://github.com/uber/tchannel/blob/docs_circuit_breaking/docs/circuit-breaking.md

    opened by GoodPete 12
  • Proposal: streaming bit field

    Proposal: streaming bit field

    To improve our timeout implementations I would like to add an "is streaming" bit flag so we can improve the timeout semantics in Hyperbahn.

    • Without streaming bit, the timeout is from the first call request frame until the last call response cont frame
    • With streaming bit, the timeout is from the first call request frame until the first call response frame

    I'll be sending a broader document to define the semantics of Timeouts in more edge cases but getting this PR out to add the new bit field to the protocol as it's slightly orthogonal.

    i.e. we can start sending this bit field before we fix the Timeouts.

    r: @jcorbin @prashantv @mranney

    cc: @junchaowu @blampe @kriskowal

    opened by Raynos 10
  • Fix as raw docs

    Fix as raw docs

    We incorrectly say that arg1 is raw bytes instead of endpoint, leading @willsalz to think there can only be 1 raw endpoint per service and myself to create this https://github.com/uber/tchannel-python/issues/113

    bug 
    opened by GoodPete 9
  • hyperbahn: rename register() to advertise()

    hyperbahn: rename register() to advertise()

    Whilst writing the tchannel guide I realized we used the word register() twice. Once to mean registrying an endpoint with a tchannel server and once to mean registrying a server with hyperbahn.

    Internally hyperbahn calls the "ad" method short for advertise. We've already renamed hyperbahns advertisement protocol to be around the word "advertise".

    This PR changes the hyperbahnClient interface to have an advertise() method. This means that when we document register() it is no longer ambigious.

    I've left the old register() method in place for 100% back-compat.

    r: @shannili @rf @jcorbin

    cc: @prashant @abhinav It would be good to rename python and go. It looks like python already calls this advertise() but it looks like go uses the word register() twice.

    opened by Raynos 9
  • Egress circuit breaker

    Egress circuit breaker

    Introduces a circuit health state machine for each callerName::serviceName::endpointName combination, integrated into the egress node for each service name.

    • [x] tests
    node 
    opened by kriskowal 9
  • v2 protocol errata: max call overhead is insane

    v2 protocol errata: max call overhead is insane

    The max call over head is insane, at 0x1ff25 bytes, aka (128Ki - 219)B:

    • 0x00010 bytes fixed frame header
    • 0x00001 byte flags
    • 0x00004 bytes ttl
    • 0x00019 bytes tracing
    • 0x00100 bytes at most for service name (1 + 0xff)
    • 0x1fe00 bytes at most for transport headers (0xff * (1 + 0xff + 1 + 0xff))
    • 0x00005 bytes for checksum
    • 0x00002 bytes for length of arg1

    We may want to impose some semantic limits on those...

    node python protocol 
    opened by jcorbin 9
  • Allow listening on port 0

    Allow listening on port 0

    Note that this is a backward incompatible change. addPeer depends on the channel to know its own name, and the name is now only knowable after the channel is listening. I have removed support for options.listening and server.listening is now a flag indicating that the server is listening. Tests have been adjusted to listen, and those that required a random, available port, now use port 0 and need not wonder whether the port was actually available.

    opened by kriskowal 9
  • Installation issue with Node 10 - 13

    Installation issue with Node 10 - 13

    I tried to install tchannel node locally on mac using multiple node versions, 10-13. None of them works. I got the same error. I would like to use RingPop and tchannel is a dependency of it. Thanks!

    npm install tchannel

    [email protected] install /Users/congwang/node/node_modules/sse4_crc32 node-gyp rebuild

    CXX(target) Release/obj.target/crc32c_sse42/src/crc32c_sse42.o LIBTOOL-STATIC Release/crc32c_sse42.a libtool: unrecognized option -static' libtool: Trylibtool --help' for more information. make: *** [Release/crc32c_sse42.a] Error 1 gyp ERR! build error gyp ERR! stack Error: make failed with exit code: 2 gyp ERR! stack at ChildProcess.onExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:194:23) gyp ERR! stack at ChildProcess.emit (events.js:219:5) gyp ERR! stack at Process.ChildProcess._handle.onexit (internal/child_process.js:274:12) gyp ERR! System Darwin 18.7.0 gyp ERR! command "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild" gyp ERR! cwd /Users/congwang/node/node_modules/sse4_crc32 gyp ERR! node -v v13.3.0 gyp ERR! node-gyp -v v5.0.5 gyp ERR! not ok npm WARN [email protected] No description npm WARN [email protected] No repository field.

    npm ERR! code ELIFECYCLE npm ERR! errno 1 npm ERR! [email protected] install: node-gyp rebuild npm ERR! Exit status 1 npm ERR! npm ERR! Failed at the [email protected] install script. npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

    npm ERR! A complete log of this run can be found in: npm ERR! /Users/congwang/.npm/_logs/2019-12-11T23_30_29_987Z-debug.log

    opened by congwangeid 1
  • Add java namespace to meta.thrift

    Add java namespace to meta.thrift

    Problem

    The classes generated for this file in Java by thrift compiler are currently put in default package, making them unusable from any other Java package.

    This also means that any other thrift files that include this one, but declare a java namespace will have their thrift compilation successful, error will only be surfaced by Java compiler when those thrift files are used in Java project making chain of updates very costly.

    One example of a workaround is in separate copy of this file in Java tchannel library.

    Backwards compatibility concerns

    I don't expect this diff breaking anything because the generated files could only be referenced from default package and that is not a widely spread practice in Java to begin with. However, repos that commit generated files in the artifacts will have a lot of their code regenerated.

    opened by groz 1
  • Add protocol support for 128-bit (16-byte) trace IDs

    Add protocol support for 128-bit (16-byte) trace IDs

    OpenZipkin has added support for 128-bit trace IDs, which have a number of advantages over 64-bit ones (also enumerated in the linked issue). It would be useful to support 128-bit trace IDs in TChannel, as it would enable more seamless integration with, among others, AWS' X-Amzn-Trace-Id header and GCE's X-Cloud-Trace-Context / Stackdriver Trace. Span IDs remain 64-bit in both Zipkin and Stackdriver, so that field should stay the same. (128-bit span IDs don't make much sense anyway.)

    Due to the fixed nature of TChannel headers, this may end up being something of a pain, possibly involving a new protocol version to allow for negotiation during the init phase.

    opened by springheeledjak 3
  • Include more information in errors

    Include more information in errors

    We often see errors like no handler found for "svc" with method "method". It would make debugging a lot easier if these errors included the host:port (and service name if possible) that the error was generated on.

    Thoughts? @blampe @willhug @abhinav

    opened by prashantv 0
  • Fragmentation: How to collect frame in the same message?

    Fragmentation: How to collect frame in the same message?

    When send large message that is bigger than 64KiB, the message is devide to some fragments. Please tell me how to receiving parse know fragments in the same message? Using header and fragment id?

    opened by nhattanmai 6
Owner
Uber Open Source
Open Source Software at Uber
Uber Open Source
Go network programming framework, supports multiplexing, synchronous and asynchronous IO mode, modular design, and provides flexible custom interfaces

Go network programming framework, supports multiplexing, synchronous and asynchronous IO mode, modular design, and provides flexible custom interfaces。The key is the transport layer, application layer protocol has nothing to do

rick.wu 9 Apr 4, 2022
Antenna RPC is an RPC protocol for distributed computing, it's based on QUIC and Colfer. its currently an WIP.

aRPC - Antenna Remote Procedure Call Antenna remote procedure call (aRPC) is an RPC protocol focused on distributed processing and HPC. aRPC is implem

Raphael de Carvalho Almeida 3 Jun 16, 2021
A Stable & Secure Tunnel based on KCP with N:M multiplexing and FEC. Available for ARM, MIPS, 386 and AMD64。KCPプロトコルに基づく安全なトンネル。KCP 프로토콜을 기반으로 하는 보안 터널입니다。

Disclaimer: kcptun maintains a single website — github.com/xtaci/kcptun. Any websites other than github.com/xtaci/kcptun are not endorsed by xtaci. Re

xtaci 12.9k Jun 21, 2022
Golang connection multiplexing library

Yamux Yamux (Yet another Multiplexer) is a multiplexing library for Golang. It relies on an underlying connection to provide reliability and ordering,

HashiCorp 1.7k Jun 25, 2022
Zero Trust Network Communication Sentinel provides peer-to-peer, multi-protocol, automatic networking, cross-CDN and other features for network communication.

Thank you for your interest in ZASentinel ZASentinel helps organizations improve information security by providing a better and simpler way to protect

ZTALAB 5 May 25, 2022
rpc/v2 support for JSON-RPC 2.0 Specification.

rpc rpc/v2 support for JSON-RPC 2.0 Specification. gorilla/rpc is a foundation for RPC over HTTP services, providing access to the exported methods of

High Performance, Kubernetes Native Object Storage 3 Jul 4, 2021
Go Substrate RPC Client (GSRPC)Go Substrate RPC Client (GSRPC)

Go Substrate RPC Client (GSRPC) Substrate RPC client in Go. It provides APIs and types around Polkadot and any Substrate-based chain RPC calls. This c

Chino Chang 1 Nov 11, 2021
Control your Flipper Zero over Protobuf RPC protocol.

go-flipper Control your Flipper Zero over Protobuf RPC protocol. This library is designed to be transport agnostic, though I've tested it with RPC ove

Flipper Devices 29 May 14, 2022
Aidos Kuneen (v2 network) daemon program that is controlled through the command line and remotely via RPC calls

adk-daemon: aidosd.v2 aidosd (v2) is a deamon which acts as bitcoind for adk. This version has been built specifically for network mesh version 2+ For

Aidos Kuneen 0 Dec 1, 2021
A simple tool to convert socket5 proxy protocol to http proxy protocol

Socket5 to HTTP 这是一个超简单的 Socket5 代理转换成 HTTP 代理的小工具。 如何安装? Golang 用户 # Required Go 1.17+ go install github.com/mritd/[email protected] Docker 用户 docker pull m

mritd 6 Jun 25, 2022
Capture packet request/response pairs for a port and/or IP to aid in Network protocol based Nuclei Templates creation.

network-fingerprint Capture packet request/response pairs for a port and/or IP to aid in Network protocol based Nuclei Templates creation. Resources I

ProjectDiscovery 45 Jun 10, 2022
Fast implementation of the collectd network protocol for go.

go-cdclient A fast, dependency free implementation of the collectd binary network protocol. How fast? This package can form an encrypted metric packet

null 7 Feb 6, 2022
Powerful golang network framework, supporting FFAX Protocol

X.NET framework Install $ go get github.com/RealFax/XNET This is a high-performance network framework, currently only supports tcp and FFAX protocol U

Realfax Messenger 0 Nov 19, 2021
Steam's protocol in Go to allow automation of different actions on the Steam network without running an actual Steam client

Steam's protocol in Go to allow automation of different actions on the Steam network without running an actual Steam client. Includes APIs for friends, chatting, trading, trade offers and TF2 crafting.

Philipp Schröer 303 Jun 19, 2022
Package socket provides a low-level network connection type which integrates with Go's runtime network poller to provide asynchronous I/O and deadline support. MIT Licensed.

socket Package socket provides a low-level network connection type which integrates with Go's runtime network poller to provide asynchronous I/O and d

Matt Layher 41 May 9, 2022
Magma is an open-source software platform that gives network operators an open, flexible and extendable mobile core network solution.

Connecting the Next Billion People Magma is an open-source software platform that gives network operators an open, flexible and extendable mobile core

Magma 1.3k Jul 1, 2022
Optimize Windows's network/NIC driver settings for NewTek's NDI(Network-Device-Interface).

windows-ndi-optimizer[WIP] Optimize Windows's network/NIC driver settings for NewTek's NDI(Network-Device-Interface). How it works This is batchfile d

Nil Hiiragi 3 Apr 15, 2022
A simple network analyzer that capture http network traffic

httpcap A simple network analyzer that captures http network traffic. support Windows/MacOS/Linux/OpenWrt(x64) https only capture clienthello colorful

null 1 Nov 24, 2021
Simple, fast and scalable golang rpc library for high load

gorpc Simple, fast and scalable golang RPC library for high load and microservices. Gorpc provides the following features useful for highly loaded pro

Aliaksandr Valialkin 651 Jun 20, 2022