A native Thrift package for Go

Related tags

Network go-thrift
Overview

Thrift Package for Go

Build Status

API Documentation: http://godoc.org/github.com/samuel/go-thrift

License

3-clause BSD. See LICENSE file.

Overview

Thrift is an IDL that can be used to generate RPC client and server bindings for a variety of languages. This package includes client and server codecs, serialization, and code generation for Go. It tries to be a more natural mapping to the language compared to other implementations. For instance, Go already has the idea of a thrift transport in the ReadWriteCloser interfaces.

Types

Most types map directly to the native Go types, but there are some quirks and limitations.

  • Go supports a more limited set of types for map keys than Thrift

  • To use a set define the field as []type and provide a tag of "set":

      StringSet []string `thrift:"1,set"`
    
  • []byte get encoded/decoded as a string because the Thrift binary type is the same as string on the wire.

RPC

The standard Go net/rpc package is used to provide RPC. Although, one incompatibility is the net/rpc's use of ServiceName.Method for naming RPC methods. To get around this the Thrift ServerCodec prefixes method names with "Thrift".

Transport

There are no specific transport "classes" as there are in most Thrift libraries. Instead, the standard io.ReadWriteCloser is used as the interface. If the value also implements the thrift.Flusher interface then Flush() error is called after protocol.WriteMessageEnd.

Framed transport is supported by wrapping a value implementing io.ReadWriteCloser with thrift.NewFramedReadWriteCloser(value)

One-way requests

Client

One-way request support needs to be enabled on the RPC codec explicitly. The reason they're not allowed by default is because the Go RPC package doesn't actually support one-way requests. To get around this requires a rather janky hack of using channels to track pending requests in the codec and faking responses.

Server

One-way requests aren't yet implemented on the server side.

Parser & Code Generator

The "parser" subdirectory contains a Thrift IDL parser, and "generator" contains a Go code generator. It could be extended to include other languages.

How to use the generator:

$ go install github.com/samuel/go-thrift/generator

$ generator --help
Usage of generator:
  -go.binarystring
        Always use string for binary instead of []byte
  -go.importprefix string
        Prefix for Thrift-generated go package imports
  -go.json.enumnum
        For JSON marshal enums by number instead of name
  -go.pointers
        Make all fields pointers
  -go.signedbytes
        Interpret Thrift byte as Go signed int8 type

$ generator cassandra.thrift $GOPATH/src/

TODO

  • default values
  • oneway requests on the server
Issues
  • Crash Info

    Crash Info

    runtime: g18: leftover defer sp=0x0 pc=0x0
        defer 0xc208140000 sp=0xc20829fce8 pc=0x776a02
        defer 0xc208416040 sp=0x0 pc=0x0
    fatal error: traceback has leftover defers
    
    runtime stack:
    runtime.throw(0xa0a070, 0x1d)
        /home/yanyiwu/local/go/src/runtime/panic.go:492 +0x98
    runtime.gentraceback(0x776dd0, 0xc20829e570, 0x0, 0xc208092120, 0x0, 0x0, 0x7fffffff, 0xa93290, 0x7f19cde98cc0, 0x0, ...)
        /home/yanyiwu/local/go/src/runtime/traceback.go:414 +0x6dc
    runtime.copystack(0xc208092120, 0x4000)
        /home/yanyiwu/local/go/src/runtime/stack1.go:553 +0x187
    runtime.newstack()
        /home/yanyiwu/local/go/src/runtime/stack1.go:725 +0xaa4
    runtime.morestack()
        /home/yanyiwu/local/go/src/runtime/asm_amd64.s:309 +0x7e
    
    goroutine 18 [copystack]:
    github.com/samuel/go-thrift/thrift.(*decoder).readValue(0xc2082808c0, 0xc208280b0c, 0x7c58e0, 0xc208280ba0, 0xd6)
        /home/yanyiwu/golang/src/github.com/samuel/go-thrift/thrift/decoder.go:51 fp=0xc20829e578 sp=0xc20829e570
    github.com/samuel/go-thrift/thrift.(*decoder).readValue(0xc2082808c0, 0xc208280b0c, 0x7c4f20, 0xc208280b90, 0xd6)
        /home/yanyiwu/golang/src/github.com/samuel/go-thrift/thrift/decoder.go:164 +0x1521 fp=0xc20829ea28 sp=0xc20829e578
    github.com/samuel/go-thrift/thrift.(*decoder).readValue(0xc2082808c0, 0xc20846200c, 0x7c5a00, 0xc208462028, 0xd6)
        /home/yanyiwu/golang/src/github.com/samuel/go-thrift/thrift/decoder.go:164 +0x1521 fp=0xc20829eed8 sp=0xc20829ea28
    github.com/samuel/go-thrift/thrift.(*decoder).readValue(0xc2082808c0, 0xc20841620f, 0x7c7620, 0xc208416250, 0xd7)
        /home/yanyiwu/golang/src/github.com/samuel/go-thrift/thrift/decoder.go:212 +0x272d fp=0xc20829f388 sp=0xc20829eed8
    github.com/samuel/go-thrift/thrift.(*decoder).readValue(0xc2082808c0, 0xc20847a00c, 0x7c59a0, 0xc20847a010, 0xd6)
        /home/yanyiwu/golang/src/github.com/samuel/go-thrift/thrift/decoder.go:164 +0x1521 fp=0xc20829f838 sp=0xc20829f388
    github.com/samuel/go-thrift/thrift.(*decoder).readValue(0xc2082808c0, 0xc20847a00c, 0x8bd740, 0xc20847a010, 0xd9)
        /home/yanyiwu/golang/src/github.com/samuel/go-thrift/thrift/decoder.go:164 +0x1521 fp=0xc20829fce8 sp=0xc20829f838
    github.com/samuel/go-thrift/thrift.DecodeStruct(0x7f19cf032dc0, 0xc208047680, 0x7c5760, 0xc20847a010, 0x0, 0x0)
        /home/yanyiwu/golang/src/github.com/samuel/go-thrift/thrift/decoder.go:43 +0x460 fp=0xc20829fda8 sp=0xc20829fce8
    github.com/samuel/go-thrift/thrift.(*clientCodec).ReadResponseBody(0xc2080a21b0, 0x7c5760, 0xc20847a010, 0x0, 0x0)
        /home/yanyiwu/golang/src/github.com/samuel/go-thrift/thrift/client.go:159 +0xa5 fp=0xc20829fe00 sp=0xc20829fda8
    net/rpc.(*Client).input(0xc208094120)
        /home/yanyiwu/local/go/src/net/rpc/client.go:141 +0xa62 fp=0xc20829ffd8 sp=0xc20829fe00
    runtime.goexit()
        /home/yanyiwu/local/go/src/runtime/asm_amd64.s:2403 +0x1 fp=0xc20829ffe0 sp=0xc20829ffd8
    created by net/rpc.NewClientWithCodec
        /home/yanyiwu/local/go/src/net/rpc/client.go:201 +0xdb
    

    Does anyone else run into this bug?

    opened by yanyiwu 5
  • Parser doesn't handle map/list/set consts

    Parser doesn't handle map/list/set consts

    Example:

    const map<string,string> M1 = {"hello": "world", "goodnight": "moon"}
    

    The parser chokes when it gets to the {.

    I would expect this to yield the following Go, I think:

    var M1 = map[string]string{"hello": "world", "goodnight": "moon"}
    

    I'm currently working on fixing this, but any pointers you could give would be great.

    opened by cespare 5
  • Thrift byte type should generate signed int8 slice

    Thrift byte type should generate signed int8 slice

    According to this doc:

    https://diwakergupta.github.io/thrift-missing-guide/#_types

    The Thrift byte type is a signed type. My understanding is that Go's byte is an alias to unsigned uint8.

    Using go-thrift, I get the following transformation:

    Thrift method signature

        AcmResponse buildAcm(1: list<byte> byteList, 2: string dataType,
                           3: map<string, string> propertiesMap)
            throws(1: InvalidInputException ex1, 2: SecurityServiceException ex2),
    

    Interface go-thrift generated

    BuildAcm(byteList []byte, dataType string, propertiesMap map[string]string) (*AcmResponse, error)
    

    The Apache go thrift library generated the following, as a reference:

    BuildAcm(byteList []int8, dataType string, propertiesMap map[string]string) (r *AcmResponse, err error)
    

    I noticed this when interoperating with a Java service via Thrift. In Java, the byte type is a signed 8 bit integer.

    opened by anxiousmodernman 4
  • Is this project still usefull now?

    Is this project still usefull now?

    Since we have "git.apache.org/thrift.git/lib/go/thrift", this project seems like less useful now!

    If it is better to tell us such infos in readme.md

    Thanks.

    opened by zhaoweiguo 4
  • Go RPC style implicit returns for replies.

    Go RPC style implicit returns for replies.

    I'm trying to create an RPC service and expose it via go-thrift so that it supports a Go-RPC style method which takes args and a reply and returns an error. Critically, the modified reply struct should be available at the client. I don't think go-thrift currently allows this, but in a previous discussion, you hinted it could; maybe I'm doing something wrong.

    type Cave struct{}
    
    func (self *Cave) Echo(args *echoer_thrift.EchoArgs, reply *echoer_thrift.EchoReply) (error) {
        reply.Echo = args.Message
        return nil
    }
    

    And the Thrift definition,

    struct EchoArgs {
        1: required string Message
    }
    
    struct EchoReply {
        1: required string Echo
    }
    
    service EchoerThriftface {
        void Echo(
            1: EchoArgs args
            2: EchoReply reply
        )
    }
    

    In a client,

    var args = echoer_thrift.EchoArgs{}
    args.Message = "hello!"
    var reply echoer_thrift.EchoReply
    
    conn, _ := net.Dial("tcp", ":1234")
    thriftClient := thrift.NewClient(thrift.NewFramedReadWriteCloser(conn, 1024), thrift.NewBinaryProtocol(true, true), false)
    client := &echoer_thrift.EchoerThriftfaceClient{thriftClient}
    error := client.Echo(&args, &reply)
    
    fmt.Println(reply.Echo)   // reply is EMPTY !!!
    

    Looking at the code generated by go-thrift, it makes sense to my why this occurs. This is the expected behavior right?

    opened by dghubble 4
  • Scribed is unable to talk to go thrift server

    Scribed is unable to talk to go thrift server

    Prefixing method names with "Thrift." causes Scribed to get confused and unable to talk to the Go Thrift server. I modified server.go so that it removes the prefix from the response and it works now but I don't know enough about Thrift to tell whether this is a good solution.

    opened by micmac 4
  • can't use go-thrift to access hbase

    can't use go-thrift to access hbase

    thrift idl: http://wiki.apache.org/hadoop/Hbase/ThriftApi

    server is writen by java. client use go-thrift to call a remote method will be blocked. the method can't return.

    I found it in Go standard library: // Call invokes the named function, waits for it to complete, and returns its error status. func (client *Client) Call(serviceMethod string, args interface{}, reply interface{}) error { call := <-client.Go(serviceMethod, args, reply, make(chan *Call, 1)).Done return call.Error }

    client is blocked here. the length of the "Done" channel is 0, so will block here. But I don't know why the "Done" is empty.

    Hope resolve it... thx~

    opened by skoo87 4
  • Support 'union' structs

    Support 'union' structs

    Currently when I try to parse a Thrift file using union, I'll get the following error messages:

    2015/08/17 10:37:41 Could not parse .thrift file: <reader>:164:1 (3915): rule SyntaxError: parser: syntax error
    

    By some search, below is a real world example Thrift file using union quite a lot. Would be nice to also add this to test files. https://github.com/apache/aurora/blob/master/api/src/main/thrift/org/apache/aurora/gen/api.thrift

    opened by ximyu 3
  • Generate typedef'ed types.

    Generate typedef'ed types.

    Previously, the generated code only referred to a typedef's underlying resolved type. It's useful (and probably expected) to formally declare and use any custom types in the Go-level interface.

    This is similar to #20 (although greatly simplified) and closes #19.

    opened by jparise 3
  • Compatible with the official generated codes?

    Compatible with the official generated codes?

    I have writen some service using the code generated by apache official tool,then use go-thrift in the client,but the client failed to call the service.Is it compatible with codes generated by the official tool ?

    opened by zhaojkun 3
  • Poor compatibility with C++

    Poor compatibility with C++

    I tried to make a client with go-thrift and communicate with C++ server, however after the client connects to the server, it just hangs there and never return. Though go-thrift is cleaner, I have to switch to thrift4go now. Hopefully go-thrift will add more compatibility tests and become better in the future.

    opened by h12w 3
  • CodeLingo Setup

    CodeLingo Setup

    opened by Daanikus 0
  • Syntax error on list when `[` is in new line

    Syntax error on list when `[` is in new line

    Example:

    struct Foo {
      1: optional list<string> Bar = 
      [
        "hello",
        "world"
      ]
    }
    

    Returns:

    $ generator example.thrift $GOPATH/src
    example.thrift:1:1 (0): rule SyntaxError: parser: syntax error
    

    This only happens if you're defining a list within a struct, and doesn't happen with list const. Example:

    const list<string> Bar = 
    [
      "hello",
      "world"
    ]
    

    Works as expected.

    opened by jameshong92 0
  • Syntax error on map const when missing a comma

    Syntax error on map const when missing a comma

    Example:

    const map<string, string> Foo = {
      "hello": "it's me"
      "hi": "how are you",
    }
    

    Returns:

    $ generator example.thrift $GOPATH/src
    example.thrift:1:1 (0): rule SyntaxError: parser: syntax error
    

    It works as expected if I add a trailing comma:

    const map<string, string> Foo = {
      "hello": "it's me",
      "hi": "how are you",
    }
    
    opened by jameshong92 0
  • package name conflict

    package name conflict

    Go packages can't be nested but you can nest namespaces in Thrift. If you have two Thrift files with the same "leaf" namespace name then when both namespaces are used together the generated Go code won't compile. The error is redeclared as imported package name ...

    Eg for this thrift:

      include "Person.Address.thrift"
      include "Computer.Address.thrift"
    
      struct Details
      {
    	1 : optional Person.Address.Primary addr,
    	2 : optional Computer.Address.Primary ip,
      }
    

    the Go code looks something like this:

      import (
    	"Person/Address"
    	"Computer/Address"     // Address redeclared as imported package name
    	"git.apache.org/thrift.git/lib/go/thrift"
      )
      .....
      type Details struct {
    	Addr             *Address.Primary
    	Ip               *Address.Primary
      }
    

    This is not a big problem but could be avoided by importing the package under a non-conflicting name like this:

      import (
    	PersonAddress   "Person/Address"
    	ComputerAddress "Computer/Address"
    	"git.apache.org/thrift.git/lib/go/thrift"
      )
      .....
      type Details struct {
    	Addr             *PersonAddress.Primary
    	Ip               *ComputerAddress.Primary
      }
    
    opened by AndrewWPhillips 0
  • Add support for implicit tag if struct field not explicitly marked required/optional

    Add support for implicit tag if struct field not explicitly marked required/optional

    • Normalizes as Required for field name vs Optional
    • Adds tag 'implicit' when not declared.
    • Don't error when decoding if a required field is implicit.

    Reference: https://thrift.apache.org/docs/idl#field-requiredness

    optional

    • Write: Optional fields are only written when they are set
    • Read: Optional fields may, or may not be part of the input stream.
    • Default values: are written when the isset flag is set

    Most language implementations use the recommended practice of so-called "isset" flags to indicate whether a particular optional field is set or not. Only fields with this flag set are written, and conversely the flag is only set when a field value has been read from the input stream.

    opened by lucasmoten 0
Owner
Samuel Stauffer
Samuel Stauffer
Paster 服务端核心模块,使用字节跳动开源的微服务 RPC 框架 KiteX ,以 Apache Thrift 作为通信协议

paster_core Paster 服务端核心模块,使用字节跳动开源的微服务 RPC 框架 KiteX ,以 Apache Thrift 作为通信协议。 Todo: 实现 KiteX 服务注册扩展接口,使用 Consul 服务注册 新增 frame 层,通过 PreProcessor, PostP

AmeiDance 10 Dec 22, 2021
Paster 服务端核心模块,使用字节跳动开源的微服务 RPC 框架 KiteX 通过 Thrift 协议与上游门面模块 paster_facade 通信

paster_core Paster 服务端核心模块,使用字节跳动开源的微服务 RPC 框架 KiteX 通过 Thrift 协议与上游门面模块 paster_facade 通信。 Todo: 实现 KiteX 服务注册扩展接口,使用 Consul 服务注册 新增 frame 层 Processor

AmeiDance 8 Aug 7, 2021
Package rsync contains a native Go rsync implementation.

gokrazy rsync Package rsync contains a native Go rsync implementation. ⚠ Beware: very fresh. Might eat your data. You have been warned! ⚠ The only com

null 282 Aug 1, 2022
A cloud native distributed streaming network telemetry.

Panoptes Streaming Panoptes Streaming is a cloud native distributed streaming network telemetry. It can be installed as a single binary or clustered n

Yahoo 36 Jun 28, 2022
A simple TUN/TAP library written in native Go.

water water is a native Go library for TUN/TAP interfaces. water is designed to be simple and efficient. It wraps almost only syscalls and uses only G

Song Gao 1.5k Jul 27, 2022
Golang client for NATS, the cloud native messaging system.

NATS - Go Client A Go client for the NATS messaging system. Installation # Go client go get github.com/nats-io/nats.go/ # Server go get github.com/na

NATS - The Cloud Native Messaging System 4.1k Aug 6, 2022
The Cloud Native Application Proxy

Traefik (pronounced traffic) is a modern HTTP reverse proxy and load balancer that makes deploying microservices easy. Traefik integrates with your ex

Traefik Labs 39k Jul 29, 2022
🤘 The native golang ssh client to execute your commands over ssh connection. 🚀🚀

Golang SSH Client. Fast and easy golang ssh client module. Goph is a lightweight Go SSH client focusing on simplicity! Installation ❘ Features ❘ Usage

Mohamed El Bahja 1.1k Jul 27, 2022
grobotstxt is a native Go port of Google's robots.txt parser and matcher library.

grobotstxt grobotstxt is a native Go port of Google's robots.txt parser and matcher C++ library. Direct function-for-function conversion/port Preserve

Jim Smart 86 Apr 7, 2022
MOSN is a cloud native proxy for edge or service mesh. https://mosn.io

中文 MOSN is a network proxy written in Golang. It can be used as a cloud-native network data plane, providing services with the following proxy functio

MOSN 3.8k Aug 4, 2022
Cloud Native Tunnel

inlets is a Cloud Native Tunnel written in Go Expose your local endpoints to the Internet or within a remote network, without touching firewalls. Foll

inlets 8.4k Jan 4, 2022
Native ZooKeeper client for Go. This project is no longer maintained. Please use https://github.com/go-zookeeper/zk instead.

Native Go Zookeeper Client Library License 3-clause BSD. See LICENSE file. This Repository is No Longer Maintained Please use https://github.com/go-zo

Samuel Stauffer 1.6k Aug 2, 2022
go implementation of fissions web-native file system

wnfs-go go language implementation of the fission web-native file system, using the typescript implementation as a reference. Development Status: Work

Qri.io 13 Mar 28, 2022
Native macOS networking for QEMU using vmnet.framework and socket networking.

qemu-vmnet Native macOS networking for QEMU using vmnet.framework and socket networking. Getting started TODO -netdev socket,id=net0,udp=:1234,localad

Alessio Dionisi 24 Jul 21, 2022
zMemif is a native golang based library for memif to interworking with dpdk.

zMemif zMemif is a native golang based library for memif to interworking with dpdk. it can simply provide 20Mpps recv and 10Mpps xmit capability. The

Mie~~~ 31 Jul 25, 2022
Fetch-npm-package - A small utility that can be used to fetch a given version of a NPM package

Use fetch-npm-package <package> <version> <output-dir> E.g. fetch-npm-package is

Bjørn Erik Pedersen 1 May 21, 2022
Package arp implements the ARP protocol, as described in RFC 826. MIT Licensed.

arp Package arp implements the ARP protocol, as described in RFC 826. MIT Licensed. Portions of this code are taken from the Go standard library. The

Matt Layher 289 Jul 27, 2022
Package dhcp6 implements a DHCPv6 server, as described in RFC 3315. MIT Licensed.

dhcp6 Package dhcp6 implements a DHCPv6 server, as described in IETF RFC 3315. MIT Licensed. At this time, the API is not stable, and may change over

Matt Layher 74 May 8, 2022
A Go package for sending and receiving ethernet frames. Currently supporting Linux, Freebsd, and OS X.

ether ether is a go package for sending and receiving ethernet frames. Currently supported platform: BPF based OS X FreeBSD AF_PACKET based Linux Docu

Song Gao 77 Jun 20, 2022