Easily generate gRPC services in Go ⚡️

Overview

logo

Actions Status

Lile is a application generator (think create-react-app, rails new or django startproject) for gRPC services in Go and a set of tools/libraries.

The primary focus of Lile is to remove the boilerplate when creating new services by creating a basic structure, test examples, Dockerfile, Makefile etc.

Lile comes with basic pre setup with pluggable options for things like...

Installation

Installing Lile is easy, using go get you can install the cmd line app to generate new services and the required libraries. First you'll need Google's Protocol Buffers installed.

$ brew install protobuf
$ go get -u github.com/lileio/lile/...

Guide

Creating a Service

Lile comes with a 'generator' to quickly generate new Lile services.

Lile follows Go's conventions around $GOPATH (see How to Write Go) and is smart enough to parse your new service's name to create the service in the right place.

If your Github username was tessthedog and you wanted to create a new service for posting to Slack you might use the following command.

lile new --name tessthedog/slack

Follow the command line instructions and this will create a new project folder for you with everything you need to continue.

Service Definitions

Lile creates gRPC and therefore uses protocol buffers as the language for describing the service methods, the requests and responses.

I highly recommend reading the Google API Design docs for good advice around general naming of RPC methods and messages and how they might translate to REST/JSON, via the gRPC gateway

An example of a service definition can be found in the Lile example project account_service

service AccountService {
  rpc List (ListAccountsRequest) returns (ListAccountsResponse) {}
  rpc GetById (GetByIdRequest) returns (Account) {}
  rpc GetByEmail (GetByEmailRequest) returns (Account) {}
  rpc AuthenticateByEmail (AuthenticateByEmailRequest) returns (Account) {}
  rpc GeneratePasswordToken (GeneratePasswordTokenRequest) returns (GeneratePasswordTokenResponse) {}
  rpc ResetPassword (ResetPasswordRequest) returns (Account) {}
  rpc ConfirmAccount (ConfirmAccountRequest) returns (Account) {}
  rpc Create (CreateAccountRequest) returns (Account) {}
  rpc Update (UpdateAccountRequest) returns (Account) {}
  rpc Delete (DeleteAccountRequest) returns (google.protobuf.Empty) {}
}

Generating RPC Methods

By default Lile will create a example RPC method and a simple message for request and response.

syntax = "proto3";
option go_package = "github.com/tessthedog/slack";
package slack;

message Request {
  string id = 1;
}

message Response {
  string id = 1;
}

service Slack {
  rpc Read (Request) returns (Response) {}
}

Let's modify this to be a real service and add our own method.

We're going to create an Announce method that will announce a message to a Slack room.

We're assuming that the Slack team and authentication is already handled by the services configuration, so a user of our service only needs to provide a room and their message. The service is going to send the special Empty response, since we only need to know if an error occurred and don't need to know anything else.

Our proto file now looks like this...

syntax = "proto3";
option go_package = "github.com/tessthedog/slack";
import "google/protobuf/empty.proto";
package slack;

message AnnounceRequest {
  string channel = 1;
  string msg = 2;
}

service Slack {
  rpc Announce (AnnounceRequest) returns (google.protobuf.Empty) {}
}

We now run the protoc tool with our file and the Lile method generator plugin.

protoc -I . slack.proto --lile-server_out=. --go_out=plugins=grpc:$GOPATH/src

Handily, Lile provides a Makefile with each project that has a proto build step already configured. So we can just run that.

make proto

We can see that Lile will create two files for us in the server directory.

$ make proto
protoc -I . slack.proto --lile-server_out=. --go_out=plugins=grpc:$GOPATH/src
2017/07/12 15:44:01 [Creating] server/announce.go
2017/07/12 15:44:01 [Creating test] server/announce_test.go

Let's take a look at the announce.go file that's created for us.

package server

import (
    "errors"

    "github.com/golang/protobuf/ptypes/empty"
    "github.com/tessthedog/slack"
    context "golang.org/x/net/context"
)

func (s SlackServer) Announce(ctx context.Context, r *slack.AnnounceRequest) (*empty.Empty, error) {
  return nil, errors.New("not yet implemented")
}

We can now fill in this generated method with the correct implementation. But let's start with a test!

Running & Writing Tests

When you generate an RPC method with Lile a counterpart test file is also created. For example, given our announce.go file, Lile will create announce_test.go in the same directory.

This should look something like the following..

package server

import (
	"testing"

	"github.com/tessthedog/slack"
	"github.com/stretchr/testify/assert"
	context "golang.org/x/net/context"
)

func TestAnnounce(t *testing.T) {
	ctx := context.Background()
	req := &slack.AnnounceRequest{}

	res, err := cli.Announce(ctx, req)
	assert.Nil(t, err)
	assert.NotNil(t, res)
}

You can now run the tests using the Makefile and running make test...

$ make test
=== RUN   TestAnnounce
--- FAIL: TestAnnounce (0.00s)
        Error Trace:    announce_test.go:16
        Error:          Expected nil, but got: &status.statusError{Code:2, Message:"not yet implemented", Details:[]*any.Any(nil)}
        Error Trace:    announce_test.go:17
        Error:          Expected value not to be nil.
FAIL
coverage: 100.0% of statements
FAIL    github.com/tessthedog/slack/server  0.011s
make: *** [test] Error 2

Our test failed because we haven't implemented our method, at the moment we're returning an error of "unimplemented" in our method.

Let's implement the Announce method in announce.go, here's an example using nlopes' slack library.

package server

import (
	"os"
	"google.golang.org/grpc/codes"
	"google.golang.org/grpc/status"

	"github.com/golang/protobuf/ptypes/empty"
	"github.com/tessthedog/slack"
	sl "github.com/nlopes/slack"
	context "golang.org/x/net/context"
)

var api = sl.New(os.Getenv("SLACK_TOKEN"))

func (s SlackServer) Announce(ctx context.Context, r *slack.AnnounceRequest) (*empty.Empty, error) {
	_, _, err := api.PostMessage(r.Channel, r.Msg, sl.PostMessageParameters{})
	if err != nil {
		return nil, status.Errorf(codes.Internal, err.Error())
	}

	return &empty.Empty{}, nil
}

Let's fill out our testing request and then run our tests again...

package server

import (
	"testing"

	"github.com/tessthedog/slack"
	"github.com/stretchr/testify/assert"
	context "golang.org/x/net/context"
)

func TestAnnounce(t *testing.T) {
	ctx := context.Background()
	req := &slack.AnnounceRequest{
		Channel: "@alex",
		Msg:     "hellooo",
	}

	res, err := cli.Announce(ctx, req)
	assert.Nil(t, err)
	assert.NotNil(t, res)
}

Now if I run the tests with my Slack token as an ENV variable, I should see a passing test!

$ [email protected]: SLACK_TOKEN=zbxkkausdkasugdk make test
go test -v ./... -cover
?       github.com/tessthedog/slack [no test files]
=== RUN   TestAnnounce
--- PASS: TestAnnounce (0.32s)
PASS
coverage: 75.0% of statements
ok      github.com/tessthedog/slack/server  0.331s  coverage: 75.0% of statements
?       github.com/tessthedog/slack/slack   [no test files]
?       github.com/tessthedog/slack/slack/cmd       [no test files]
?       github.com/tessthedog/slack/subscribers     [no test files]

Using the Generated cmds

Lile generates a cmd line application based on cobra when you generate your service. You can extend the app with your own cmds or use the built-in cmds to run the service.

Running the cmd line app without any arguments will print the generated help.

For example go run orders/main.go

up

Running up will run both the RPC server and the pubsub subscribers.

go run orders/main.go up

Adding your own cmds

To add your own cmd, you can use the built in generator from cobra which powers Lile's cmds

$ cd orders
$ cobra add import

You can now edit the file generated to create your cmd, cobra will automatically add the cmd's name to the help.

Issues
  • logger write escape sequence on Windows command prompt

    logger write escape sequence on Windows command prompt

    logrus in default output escape sequence like this:

    See https://github.com/sirupsen/logrus#formatters

    opened by mattn 9
  • service discovery initial code

    service discovery initial code

    in a second step, I'll make an interface out of the current registry implementation.

    opened by nadilas 8
  • Service Discovery integration

    Service Discovery integration

    As discussed a consul integration would be nice. I am currently looking at a shutdown hook, to deregister the service. I can't seem to find, where do you stop the service?

    opened by nadilas 5
  • lile new throws [ERROR]: 1:13: expected ';', found '-'

    lile new throws [ERROR]: 1:13: expected ';', found '-'

    I tried to generate a new service, but got this error:

    $ protoc --version
    libprotoc 3.5.1
    
    $ lile new arne-cl/grpc-test
    Creating project in /home/arne/go/src/github.com/arne-cl/grpc-test
    Is this OK? [y]es/[n]o
    y
    [ERROR]: 1:13: expected ';', found '-'
    

    Interestingly, if I use your example repo name, it works:

    $ lile new lileio/slack
    Creating project in /home/arne/go/src/github.com/lileio/slack
    Is this OK? [y]es/[n]o
    y
    .
    ├── server
    │   ├── server.go
    │   └── server_test.go
    ├── subscribers
    │   └── subscribers.go
    ├── slack
    │   ├── cmd
    │   │   ├── root.go
    │   │   └── up.go
    │   └── main.go
    ├── slack.proto
    ├── client.go
    ├── Makefile
    ├── Dockerfile
    └── .gitignore
    
    opened by arne-cl 5
  • Lile bootstrap does not work as in README

    Lile bootstrap does not work as in README

    In README:

    Getting Started
    To generate a new service, run lile new with a short folder path.
    
    $ lile new lileio/users
    

    In real life:

    ➜  ~ lile new piotrkochan/lile-test
    Error: required flag(s) "name" not set
    

    Trying with --name arg as in example (--name string the module name i.e (github.com/username/project):

    ➜  ~ lile new piotrkochan/lile-test --name "github.com/piotrkochan/lile-test"
    Creating project in lile-test
    Is this OK? [y]es/[n]o
    Y
    [...some code output...]
    Could not format Go file {client.go lile-test/client.go client.tmpl}
    [ERROR]: 1:13: expected ';', found '-'
    panic: 1:13: expected ';', found '-'
    

    Other try:

    ➜  ~ lile new piotrkochan/lile-test --name "github.com/piotrkochan"
    Creating project in piotrkochan
    Is this OK? [y]es/[n]o
    Y
    .
    ├── server
    │   ├── server.go
    │   └── server_test.go
    ├── subscribers
    │   └── subscribers.go
    ├── piotrkochan
    │   ├── cmd
    │   │   ├── root.go
    │   │   └── up.go
    │   └── main.go
    ├── piotrkochan.proto
    ├── client.go
    ├── Makefile
    ├── Dockerfile
    ├── go.mod
    └── .gitignore
    
    l➜  ~ pwd
    <my home>
    l➜  ~ ls -l | grep lile
    drwxr-xr-x   4 piotr  staff   128 Nov 20 20:55 lile-test
    

    Generated code is not placed in $GOPATH as described but in the directory I'm currently in (is smart enough to parse your new service's name to create the service in the right place. seems a little bit wrong :))

    ➜  ~ go version
    go version go1.12.5 darwin/amd64
    
    opened by piotrkochan 4
  • support windows

    support windows

    opened by mattn 4
  • Allow to use gogo/protobuf (importing files with no go_package)

    Allow to use gogo/protobuf (importing files with no go_package)

    When I try to use gogo along with lile (protoc -I../vendor -I ../models -I . ecommerce.proto --lile-server_out=. --gogo_out=plugins=grpc:$$GOPATH/src)

    I get

    2017/09/29 18:35:42 No go_package option defined for import github.com/gogo/protobuf/gogoproto/gogo.proto
    
    opened by facundomedica 4
  • fix to use GOPATH

    fix to use GOPATH

    Recently, many people doesn't set $GOPATH.

    opened by mattn 4
  • Fix index out of range

    Fix index out of range

    I get this error on the subscriber side when a message is received.

    panic: runtime error: index out of range
    
    goroutine 132 [running]:
    github.com/lileio/lile/pubsub.Client.On.func2(0x1927b60, 0xc4203ba8a0, 0xc4203f2570, 0xf, 0xc4203ba720, 0xc4203f23d0, 0xa, 0xa, 0xc4203f2680, 0xc4203f2690, ...)
            /Users/baopham/Projects/Go/src/github.com/lileio/lile/pubsub/subscribe.go:81 +0x2b8
    github.com/lileio/lile/pubsub/google.(*GoogleCloud).subscribe.func1.1(0x1fa5910, 0xc4203ba8a0, 0xc420246850)
            /Users/baopham/Projects/Go/src/github.com/lileio/lile/pubsub/google/google.go:160 +0x575
    cloud.google.com/go/pubsub.(*Subscription).receive.func2(0xc4203d8f30, 0xc420409000, 0xc420246850, 0xc420388660, 0x1c90fc0, 0xc42034af00)
            /Users/baopham/Projects/Go/src/cloud.google.com/go/pubsub/subscription.go:392 +0xa2
    created by cloud.google.com/go/pubsub.(*Subscription).receive
            /Users/baopham/Projects/Go/src/cloud.google.com/go/pubsub/subscription.go:393 +0x311
    exit status 2
    

    My subscriber is standard following README example https://github.com/lileio/lile#subscribing-to-events

    opened by baopham 3
  • GO111MODULE=on go get -u github.com/lileio/lile/... fails

    GO111MODULE=on go get -u github.com/lileio/lile/... fails

    Problem: Running GO111MODULE=on go get -u github.com/lileio/lile/... fails with:

    go get: github.com/openzipkin/[email protected] updating to
    	github.com/openzipkin/[email protected]: parsing go.mod:
    	module declares its path as: github.com/openzipkin-contrib/zipkin-go-opentracing
    	        but was required as: github.com/openzipkin/zipkin-go-opentracing
    

    Solutions:

    Short Term Fix: I haven't found one, adding the following to go.mod

    replace (
    	github.com/openzipkin/zipkin-go-opentracing v0.3.4 => github.com/openzipkin-contrib/zipkin-go-opentracing v0.3.4
    	github.com/openzipkin/zipkin-go-opentracing v0.4.3 => github.com/openzipkin-contrib/zipkin-go-opentracing v0.4.3
    	github.com/openzipkin/zipkin-go-opentracing v0.4.5 => github.com/openzipkin-contrib/zipkin-go-opentracing v0.4.5
    )
    

    leads to the following new error

    go: github.com/openzipkin-contrib/[email protected] used for two different module paths (github.com/openzipkin-contrib/zipkin-go-opentracing and github.com/openzipkin/zipkin-go-opentracing)
    

    Longer Term Fix:

    • Change dependency name to github.com/openzipkin-contrib/zipkin-go-opentracing and bump version if needed (GO111MODULE=on go get github.com/openzipkin-contrib/[email protected]). Happy to submit a PR if this feels like the right approach.
    opened by jason-riddle 0
  • Adding Client-side code generators and Ask Advise for dependent servers

    Adding Client-side code generators and Ask Advise for dependent servers

    Let's take a look at your slack example and say we have two server:

    1. ServerOne implements Announce() via gRPC interface (say to announce notif to slack)
    2. ServerTwo needs a server to announce something to slack. It needs to call ServerOne

    Well then, I build ServerOne with lileio/lile and get an autogenerated code in Go. Neat and pretty.

    How about ServerTwo? Hmmm its a bit tedious because:

    1. To commnicate with ServerOne, developer need to read and understand the proto file of ServerOne. Good thing if ServerOne developers kindly have an API docs.
    2. Then come changes in ServerOne proto file which change the comminication contract between ServerOne and ServerTwo. Then these changes also affect the dependent microservice (ServerTwo)

    Ideas

    There are two ideas for Go developer to resolve this:

    1. Versioning of API: ServerOne do two version of API: the old schema and new schema. Allowing ServerTwo to slowly migrate.
    2. Autogenerated Client-side: ServerOne can give its protofile to ServerTwo and we can build a code generator for client side.

    I want to highlight on the (2) Ideas here and whether it should be supported, what are the Pros and Cons.

    Pros

    • Easy migration for dependent services. Using protofile as single source of truth

    Cons

    • Generated code in Golang doesn't always adhere to how Go code is written. Some developers think its OK because its an autogenerated code. Hence, if we want this feature we have to make it autgenerated but clean
    • Same problem as above, but this time autogenerated code like that of thrift can yield freaking 50,000 lines of code just for 12 endpoints. This pollute the codebase, git commits, and slows down IDE and code intellisense (VSCode, Vim, GoLand, you name it).
    opened by iqDF 1
  • First impl of protocopy!

    First impl of protocopy!

    opened by arbarlow 0
  • Add TODOs Badge to README

    Add TODOs Badge to README

    Hi there! I wanted to propose adding the following badge to the README to indicate how many TODO comments are in this codebase:

    TODOs

    The badge links to tickgit.com which is a free service that indexes and displays TODO comments in public github repos. It can help surface latent work and be a way for contributors to find areas of code to improve, that might not be otherwise documented.

    The markdown is:

    [![TODOs](https://badgen.net/https/api.tickgit.com/badgen/github.comgithub.com/lileio/lile)](https://www.tickgit.com/browse?repo=github.com/lileio/lile)
    

    Thanks for considering, feel free to close this issue if it's not appropriate or you prefer not to!

    opened by patrickdevivo 0
Owner
Lile
gRPC services and service generators! ⚡️
Lile
A suite of gRPC debugging tools. Like Fiddler/Charles but for gRPC.

grpc-tools A suite of tools for gRPC debugging and development. Like Fiddler/Charles but for gRPC! The main tool is grpc-dump which transparently inte

Bradley Kemp 938 Jul 23, 2021
Use Consul to do service discovery, use gRPC +kafka to do message produce and consume. Use redis to store result.

目录 gRPC/consul/kafka简介 gRPC+kafka的Demo gRPC+kafka整体示意图 限流器 基于redis计数器生成唯一ID kafka生产消费 kafka生产消费示意图 本文kafka生产消费过程 基于pprof的性能分析Demo 使用pprof统计CPU/HEAP数据的

null 38 Jan 31, 2021
The Go language implementation of gRPC. HTTP/2 based RPC

gRPC-Go The Go implementation of gRPC: A high performance, open source, general RPC framework that puts mobile and HTTP/2 first. For more information

grpc 14.1k Jul 23, 2021
grpc-http1: A gRPC via HTTP/1 Enabling Library for Go

grpc-http1: A gRPC via HTTP/1 Enabling Library for Go This library enables using all the functionality of a gRPC server even if it is exposed behind a

StackRox 53 Jun 3, 2021
gRPC over WebRTC

gRPC over WebRTC Just a proof of concept, please be kind. How to Start all the things Client, create-react-app + grpc-web signaling + webrtc extension

Jean-Sebastien Mouret 227 Jul 10, 2021
protoc-gen-grpc-gateway-ts is a Typescript client generator for the grpc-gateway project. It generates idiomatic Typescript clients that connect the web frontend and golang backend fronted by grpc-gateway.

protoc-gen-grpc-gateway-ts protoc-gen-grpc-gateway-ts is a Typescript client generator for the grpc-gateway project. It generates idiomatic Typescript

gRPC Ecosystem 25 Jul 21, 2021
In memory Key/Value store in go using gRPC.

In memory cache, using gRPC Contents About Running Server Local Docker Kubernetes Example Helm Terraform API Add Get GetByPrefix GetAllItems DeleteKey

Kautilya Tripathi 136 Jul 23, 2021
A Protocol Buffers compiler that generates optimized marshaling & unmarshaling Go code for ProtoBuf APIv2

vtprotobuf, the Vitess Protocol Buffers compiler This repository provides the protoc-gen-go-vtproto plug-in for protoc, which is used by Vitess to gen

PlanetScale 262 Jul 23, 2021
Fast time-series data storage server accessible over gRPC

tstorage-server Persistent fast time-series data storage server accessible over gRPC. tstorage-server is lightweight local on-disk storage engine serv

Bartlomiej Mika 3 Jul 17, 2021
gRPC/REST proxy for Kafka

Kafka-Pixy (gRPC/REST Proxy for Kafka) Kafka-Pixy is a dual API (gRPC and REST) proxy for Kafka with automatic consumer group control. It is designed

Mailgun Team 646 Jul 16, 2021
Create a gRPC Server from Database

xo-grpc Create a gRPC Server from the generated code by the xo project. Requirements Go 1.16 or superior protoc xo, protoc-gen-go and protoc-gen-go-gr

Walter Wanderley 17 Jun 23, 2021
Easily generate gRPC services in Go ⚡️

Lile is a application generator (think create-react-app, rails new or django startproject) for gRPC services in Go and a set of tools/libraries. The p

Lile 1.3k Jul 18, 2021
Create a gRPC server from code generated by sqlc

sqlc-grpc Create a gRPC Server from the generated code by the awesome sqlc project. Dependencies Go 1.16 or superior protoc sqlc, protoc-gen-go and pr

Walter Wanderley 35 Jul 8, 2021
gproxy is a tiny service/library for creating lets-encrypt/acme secured gRPC and http reverse proxies

gproxy is a reverse proxy service AND library for creating flexible, expression-based, lets-encrypt/acme secured gRPC/http reverse proxies GProxy as a

null 13 Feb 23, 2021