Erda's infrastructure framework

Overview

Erda Infra

License codecov

Translations: English | 简体中文

Erda Infra is a lightweight microservices framework implements by golang, which offers many useful modules and tools to help you quickly build a module-driven application.

Many Go projects are built using Erda Infra including:

Features

  • modular design to drive the implementation of the application system, and each module is pluggable.
  • each module is configurable,supports setting defaults、reading from files (YAML, HCL,、JSON、TOML、.env file)、environment、flags.
  • manage the lifecycle of the module, includes initialization, startup, and shutdown.
  • manage dependencies between modules.
  • support Dependency Injection between modules.
  • offers many commonly modules that can be used directly.
  • support define APIs and models in protobuf file to expose both gRPC and HTTP APIs.
  • offers tools to help you quickly build a module.
  • etc.

Concept

  • Service, represents a function.
  • Provider, service provider, equivalent to module, provide some services. It can also depend on other services, witch provide by other provider.
  • ProviderDefine, describe a provider, includes provider's name, constructor function of provider, services, etc. Register by it servicehub.RegisterProvider function.
  • Hub, is a container for all providers, and manage the life cycle of all loaded providers.

A configuration is used to determine whether all registered Providers are loaded, and the Hub initializes, starts, and closes the loaded Providers.

Define Provider

Define a provider by implementing the servicehub.ProviderDefine interface, and register it through the servicehub.RegisterProvider function.

But, it is simpler to describe a provider through servicehub.Spec and register it through the servicehub.Register function.

Examples

Quick Start

# create service interface
➜ ROOT_PATH=$(pwd)${ROOT_PATH}/tools/protoc.sh protocol "examples/protocol/*.proto"
➜ 
➜ # create module 
➜ mkdir -p examples/server/helloworld
➜ cd examples/server/helloworld
➜ ${ROOT_PATH}/tools/protoc.sh init "${ROOT_PATH}/examples/protocol/*.proto"
➜ 
➜ # implement the service interface in examples/server/helloworld directory
➜ 
➜ cd ${ROOT_PATH}/examples/server
➜ 
➜ # create main.go, like examples/server/main.go# create server.yaml, like examples/server/server.yaml
➜ 
➜ go run main.go

example

Hello World ( Server | Client )

Useful Providers

Many available providers have been packaged in this project, it can be found in the providers/ directory.

Under each module, there is an examples directory, which contains examples of the use of the module.

  • elasticsearch, provide elasticsearch client APIs, and it easy to write batch data.
  • etcd, provide etcd client APIs.
  • etcd-mutex, distributed lock implemented by etcd.
  • grpcserver, start a grpc server.
  • grpcclient, provide gRPC client, and manage client's connection.
  • health, provide health check API, and can register some health check function into this provider.
  • httpserver, provide an HTTP server, support any form of handle function, interceptor, parameter binding, parameter verification, etc.
  • i18n, provide internationalization support, manage i18n files, support templates.
  • kafka, provide kafka sdk, and easy to produce and consume messages.
  • kubernetes, provide kubernetes client APIs.
  • mysql, provide mysql client APIs.
  • pprof, expose pprof HTTP APIs By httpserver.
  • redis, provide redis client APIs.
  • zk-master-election, provide interface about master-slave election, it is implemented by zookeeper.
  • zookeeper, provide zookeeper client.
  • cassandra, provide cassandra APIs, and easy to write batch data
  • serviceregister, use it to register services to expose gRPC and HTTP APIs.

Tools

protoc-gen-go-* tools depends on protobuf compiler,see protobuf to install protoc。

You can also use the following tools through a Docker container.

docker run --rm -ti -v $(pwd):/go \
    registry.cn-hangzhou.aliyuncs.com/dice/erda-tools:1.0 protoc.sh usage
  • protoc-gen-go-grpc, according to *.proto file, provide gRPC server and client support
  • protoc-gen-go-http, according to the *.proto file, provide HTTP server support for the defined service.
  • protoc-gen-go-form, according to the *.proto file, provide HTTP form codec support for the defined message.
  • protoc-gen-go-client, according to the *.proto file, generate clients and provider for defined service.
  • protoc-gen-go-register, provide some functions to help a provider register services.
  • protoc-gen-go-provider, according to the *.proto file, generate a provider template that implements the service interface to facilitate rapid module development.
  • protoc.sh, wrap the protoc-gen-go-* series of tools for the development.

License

Erda Infra is under the Apache 2.0 license. See the LICENSE file for details.

Comments
  • feat: Component protocol default protocol yml support i18n

    feat: Component protocol default protocol yml support i18n

    What this PR does / why we need it:

    Component protocol default protocol yml support i18n

    Which issue(s) this PR fixes:

    Fixes #

    Specified Reivewers:

    /assign @sfwn

    approved 
    opened by shuofan 4
  • provider mysqlxorm support session for transparent transaction and init-ping

    provider mysqlxorm support session for transparent transaction and init-ping

    What type of this PR

    /kind feature

    What this PR does / why we need it:

    • support ping when init to check database advanced
    • migrated feature: support session for transparent transaction delivery, see example for usage, which is already used in pipeline
    approved 
    opened by sfwn 4
  • support mac M1

    support mac M1

    What type of this PR

    Add one of the following kinds: /kind bug

    What this PR does / why we need it:

    under mac M1, runtime.GOARCH value is arm64. because currently there is no real protobuf release for darwin-arm64 and the darwin-x64 binary can run on mac M1 with the rosetta compatibility, so here just use the same release.

    Specified Reivewers:

    /assgin @liuhaoyang @recallsong

    lgtm 
    opened by snakorse 4
  • feat: Get component from default protocol if not exist

    feat: Get component from default protocol if not exist

    What this PR does / why we need it:

    Get component from default protocol if not exist

    Which issue(s) this PR fixes:

    Fixes #

    Specified Reivewers:

    /assign @sfwn

    approved feature 
    opened by shuofan 3
  • feat: cp: optionally register component as provider

    feat: cp: optionally register component as provider

    What this PR does / why we need it:

    Sometimes, a component is a provider first, then become a component. For this situation, we provide a more convenient method to use.

    Previous: image

    Now: image

    Specified Reivewers:

    /assign @Effet

    feature 
    opened by sfwn 3
  • message list, need client data when click

    message list, need client data when click

    What type of this PR

    Add one of the following kinds: /kind bug message list, need client data when click

    What this PR does / why we need it:

    Which issue(s) this PR fixes:

    Fixes #

    Specified Reivewers:

    /assign

    approved 
    opened by harverywxu 3
  • feat(component-protocol): add component cardlist

    feat(component-protocol): add component cardlist

    What type of this PR

    Add one of the following kinds: /kind feature

    What this PR does / why we need it:

    Which issue(s) this PR fixes:

    Fixes #

    Specified Reivewers:

    /assign @sfwn

    approved 
    opened by bugaolengdeyuxiaoer 3
  • add debug log

    add debug log

    What type of this PR

    Add one of the following kinds:

    /kind bug

    What this PR does / why we need it:

    add debug logs

    Specified Reivewers:

    /assign @johnlanni

    approved 
    opened by CraigMChen 3
  • add error encoder

    add error encoder

    What type of this PR

    Add one of the following kinds:

    /kind bug

    What this PR does / why we need it:

    add encoder to handle cp error

    Which issue(s) this PR fixes:

    Fixes #

    Specified Reivewers:

    /assign @sfwn

    approved 
    opened by CraigMChen 3
  • add X-Need-User-Info header in response

    add X-Need-User-Info header in response

    What type of this PR

    Add one of the following kinds:

    /kind feature

    What this PR does / why we need it:

    Add "X-Need-User-Indo" header to tell openapi to inject user info in response. Otherwise, openapi will skip injecting user info.

    Specified Reivewers:

    /assign @johnlanni

    approved 
    opened by CraigMChen 3
  • upgrade gocql to fix reconnection

    upgrade gocql to fix reconnection

    What type of this PR

    Add one of the following kinds: /kind bug

    What this PR does / why we need it:

    In order to avoid 'gocql: no hosts available in the pool ' beacuse of reconnection issue (https://github.com/gocql/gocql/issues/915)

    Upgrade to fix it (https://github.com/gocql/gocql/pull/1555)

    Which issue(s) this PR fixes:

    Fixes #

    Specified Reivewers:

    /assgin @your-reviewer

    approved 
    opened by erenming 3
  • feat: support custom http code

    feat: support custom http code

    What this PR does / why we need it:

    feat: support custom http code

    you can define method that reference github.com/erda-project/erda-infra/proto-go/http.HttpInfo in your proto message:

    import "infra/http/infra-http.proto";
    
    service MyService {
      rpc HelloWorld(infra.http.Void) returns(HelloWorldResp) {
        option(google.api.http) = {
          get: "/api/hello-world",
        };
      }
    }
    
    message HelloWorldResp {
      string msg = 1;
      infra.http.HttpInfo httpInfo = 2 [json_name = "-"];
    }
    

    you can return custom HTTP Status Code in your handler now:

    func (p *MyService) HelloWorld(ctx context.Context, void *http1.Void) (*pb.HelloWorldResp, error) {
    	return &pb.HelloWorldResp{
    		Msg: time.Now().Format(time.RFC3339),
    		HttpInfo: &http1.HttpInfo{
    			Status: http.StatusAccepted,
    			Headers: []*http1.HttpHeader{
    				{
    					Key:    "x-custom-key-a",
    					Values: []string{"v1", "v2"},
    				}, {
    					Key:    "x-custom-key-b",
    					Values: []string{"something", "anything"},
    				},
    			},
    		},
    	}, nil
    }
    

    the cURL result:

    # curl the-host/api/hello-world -v
    > GET /api/hello-world HTTP/1.1
    > User-Agent: curl/7.29.0
    > Accept: */*
    > 
    < HTTP/1.1 202 Accepted
    < Vary: Origin
    < X-Custom-Key-A: v1
    < X-Custom-Key-A: v2
    < X-Custom-Key-B: something
    < X-Custom-Key-B: anything
    < Date: Tue, 19 Jul 2022 17:05:46 GMT
    < Content-Length: 196
    < Content-Type: text/plain; charset=utf-8
    < 
    {"success":true,"data":{"msg":"2022-07-20T01:05:46+08:00"}}
    

    Which issue(s) this PR fixes:

    Fixes #

    Specified Reviewers:

    /assign @your-reviewer

    feature 
    opened by dspo 1
  • can't download package

    can't download package

    opened by mlboy 2
Owner
Erda
Erda Project
Erda
EasyAgent is an infrastructure component, applied to manage the life-cycle of services on the remote host.

Easyagent English | 中文 介绍 easyagent是在袋鼠云内部广泛使用的基础架构组件,最佳应用场景包括ELK体系beats等数据采集器的管控和配置管理、数栈体系自动化部署等 基本原理 easyagent主要有sidecar和server两个组件,sidecar部署在主机端,si

袋鼠云 24 Nov 13, 2022
Run commands on remote hosts, inspecting key indicators to manage infrastructure

inspector This is a very basic ssh helper tool to manage a smaller (few 100s up to a few 1000s) fleet of servers. The main point of inspector is to pr

null 17 Mar 3, 2022
Server and relay side infrastructure for RDA

BitterJohn Server and relay side infrastructure for RDA. Usage install sudo ./BitterJohn install -g systemctl enable --now BitterJohn upgrade sudo ./B

null 5 Nov 6, 2022
An easy-to-use platform for creating microservices without complex infrastructure solutions.

RPCPlatform An easy-to-use platform for creating microservices without complex infrastructure solutions. Only etcd required. Out of the box you get a

Oleg Trifonov 0 Jan 4, 2022
🚀 gnet is a high-performance, lightweight, non-blocking, event-driven networking framework written in pure Go./ gnet 是一个高性能、轻量级、非阻塞的事件驱动 Go 网络框架。

English | ???? 中文 ?? Introduction gnet is an event-driven networking framework that is fast and lightweight. It makes direct epoll and kqueue syscalls

Andy Pan 7.1k Nov 21, 2022
An experimental go FTP server framework

graval Go FTP server framework. By providing a simple driver class that responds to a handful of methods you can have a complete FTP server. Some samp

Koofr 28 Sep 27, 2022
NFF-Go -Network Function Framework for GO (former YANFF)

Network Function Framework for Go (former YANFF) Wonderful news : we are now supporting AF_XDP and supporting(almost) getting packets directly from Li

Intel Go Team 1.3k Nov 25, 2022
A TCP Server Framework with graceful shutdown, custom protocol.

xtcp A TCP Server Framework with graceful shutdown,custom protocol. Usage Define your protocol format: Before create server and client, you need defin

xfx 138 Nov 22, 2022
hc is a lightweight framework to develop HomeKit accessories in Go.

hc hc is a lightweight framework to develop HomeKit accessories in Go. It abstracts the HomeKit Accessory Protocol (HAP) and makes it easy to work wit

Matthias 1.7k Nov 20, 2022
⚡️Lightweight framework for microservices & web services in golang

Quickstart Zepto is a lightweight framework for the development of microservices & web services in golang. As an opinionated framework, zepto proposes

null 117 Jun 19, 2022
A modern, fast and scalable websocket framework with elegant API written in Go

About neffos Neffos is a cross-platform real-time framework with expressive, elegant API written in Go. Neffos takes the pain out of development by ea

Gerasimos (Makis) Maropoulos 475 Nov 13, 2022
GO2P is a P2P framework, designed with flexibility and simplicity in mind

go2p golang p2p framework By v-braun - viktor-braun.de. Description GO2P is a P2P framework, designed with flexibility and simplicity in mind. You can

Viktor Braun 89 Nov 9, 2022
A Xray backend framework that can easily support many panels. 一个基于Xray的后端框架,支持V2ay,Trojan,Shadowsocks协议,极易扩展,支持多面板对接

XRayR A Xray backend framework that can easily support many panels. 一个基于Xray的后端框架,支持V2ay,Trojan,Shadowsocks协议,极易扩展,支持多面板对接。 如果您喜欢本项目,可以右上角点个star+watch

null 433 Nov 22, 2022
High-precision indoor positioning framework, version 3.

The Framework for Internal Navigation and Discovery (FIND) is like indoor GPS for your house or business, using only a simple smartphone or laptop. Th

Zack 4.4k Nov 22, 2022
Subfinder is a subdomain discovery tool that discovers valid subdomains for websites. Designed as a passive framework to be useful for bug bounties and safe for penetration testing.

Fast passive subdomain enumeration tool. Features • Install • Usage • API Setup • License • Join Discord Subfinder is a subdomain discovery tool that

ProjectDiscovery 6.5k Nov 21, 2022
Extensible network application framework inspired by netty

GO-NETTY 中文介绍 Introduction go-netty is heavily inspired by netty Feature Extensible transport support, default support TCP, UDP, QUIC, KCP, Websocket

The Go-Netty Project 336 Oct 30, 2022
A simple RPC framework with protobuf service definitions

Twirp is a framework for service-to-service communication emphasizing simplicity and minimalism. It generates routing and serialization from API defin

Twitch 6.2k Nov 25, 2022