Orion - a small lightweight framework written around grpc/protobuf with the aim to shorten time to build microservices at Carousell.

Overview

Orion Build Status Go Report Card codecov GoDoc

Orion is a small lightweight framework written around grpc/protobuf with the aim to shorten time to build microservices at Carousell.

It is derived from 'Framework' a small microservices framework written and used inside https://carousell.com, It comes with a number of sensible defaults such as zipkin tracing, hystrix, live reload of configuration, etc.

Getting Started

Follow the guide at https://github.com/carousell/Orion/blob/master/orion/README.md

Setup Instructions

Orion is written in golang, please follow instructions on https://golang.org/doc/install to install, or you can also run

brew install golang

or

sudo dnf install golang

Make sure you are on go 1.9 or later add the following lines to your ~/.profile

export GOPATH="$HOME/code/go"
export GOBIN="$GOPATH/bin"
export PATH="$GOBIN:$PATH"
export PATH="$HOME/.gotools:$PATH"

source your ~/.profile

source ~/.profile

then create the code dir

mkdir -p $GOPATH

we use govendor to vendor package in Orion, install it by running

go get -u github.com/kardianos/govendor

another helpful tool to check for unupdated packages is Go-Package-Store, install it by running

go get -u github.com/shurcooL/Go-Package-Store/cmd/Go-Package-Store

now clone this repo

mkdir -p $GOPATH/src/github.com/carousell/
git clone [email protected]:carousell/Orion.git $GOPATH/src/github.com/carousell/Orion

You need the following tools to better develop for go

go get -u golang.org/x/lint/golint

now you can build the package by using make build

gRPC

for gRPC, you need to follow the following steps

get gRPC codebase

go get -u google.golang.org/grpc

install protobuf

brew install protobuf

install the protoc plugin for go

go get -u github.com/golang/protobuf/{proto,protoc-gen-go}

install the protoc plugin for orion

go get -u github.com/carousell/Orion/protoc-gen-orion

Project Status

Orion is in use at production at Carousell and powers multiple (100+) services serving thousands of requests per second, we ensure all updates are backward compatible unless it involves a major bug or security issue.

License

This code is available under the following https://github.com/carousell/Orion/blob/master/LICENSE

Comments
  • Config to disable default interceptors

    Config to disable default interceptors

    This PR is allow service to disable Orion's default interceptors by setting config

    [orion]
    DisableDefaultInterceptors=true
    

    This could be useful to disable interceptors such as ResponseTimeLoggingInterceptor which logs massively for each endpoint calls.

    opened by achichen 7
  • Fix Potential Race on Logger

    Fix Potential Race on Logger

    Issue Description When multiple parallel routine try to log like log.GetLogger().Log(...), there is a potential race condition in logger.GetLogger.

    ==================
    WARNING: DATA RACE
    Write at 0x0000014994b0 by goroutine 9:
      github.com/carousell/Orion/utils/log.GetLogger.func1()
          /.../Orion/utils/log/log.go:65 +0xaa
      sync.(*Once).Do()
          /usr/local/go/src/sync/once.go:44 +0xde
      github.com/carousell/Orion/utils/log.GetLogger()
          /.../Orion/utils/log/log.go:64 +0xaf
      github.com/carousell/Orion.main.func1()
          /.../Orion/racecheck.go:22 +0x187
    
    Previous read at 0x0000014994b0 by goroutine 8:
      github.com/carousell/Orion/utils/log.GetLogger()
          /.../Orion/utils/log/log.go:63 +0x3e
      github.com/carousell/Orion.main.func1()
          /.../Orion/racecheck.go:22 +0x187
    
    Goroutine 9 (running) created at:
      github.com/carousell/Orion.main()
          /.../Orion/racecheck.go:18 +0xb5
      github.com/carousell/Orion.Test_main.func1()
          /.../Orion/racecheck_test.go:18 +0x2f
      testing.tRunner()
          /usr/local/go/src/testing/testing.go:865 +0x163
    
    Goroutine 8 (running) created at:
      github.com/carousell/Orion.main()
          /.../Orion/racecheck.go:18 +0xb5
      github.com/carousell/Orion.Test_main.func1()
          /.../Orion/racecheck_test.go:18 +0x2f
      testing.tRunner()
          /usr/local/go/src/testing/testing.go:865 +0x163
    ==================
    

    Test Code

    go test -v -race -run main

    racecheck_test.go a empty test just for go test race check

    package main
    
    import "testing"
    
    func Test_main(t *testing.T) {
    
            // TODO: Init instances and mocks
    
            testCases := []struct {
                    name string
            }{
                    {
                            name: "hi",
                    },
            }
            for _, tt := range testCases {
                    t.Run(tt.name, func(t *testing.T) {
                            main()
                    })
            }
    }
    

    racecheck.go simulate parallel GetLogger() case

    package main
    
    import (
            "context"
            "fmt"
            "sync"
    
            "github.com/carousell/Orion/utils/errors"
            "github.com/carousell/Orion/utils/log"
            "github.com/carousell/Orion/utils/log/loggers"
    )
    
    func main() {
            var wg sync.WaitGroup
    
            for i := 0; i < 2; i++ {
                    wg.Add(1)
                    go func(wg *sync.WaitGroup, routineIndex int) {
                            defer wg.Done()
                            errWithStack := errors.WrapWithSkip(fmt.Errorf("error %d", routineIndex), "", 1)
                            log.GetLogger().Log(context.Background(), loggers.ErrorLevel, 1, "err", errWithStack, "stack", errWithStack.StackFrame())
                    }(&wg, i)
            }
            wg.Wait()
    }
    
    opened by stevenCarousell 7
  • Support tool installation after Go 1.17

    Support tool installation after Go 1.17

    Situation:

    Starting in Go 1.17, installing executables with go get is deprecated. go install may be used instead. And go get is no longer supported outside a module in Go 1.18. (Source)

    Which means Orion is no longer available in Go 1.18. We will stuck with older version of Go in the future.

    Install protoc-gen-orion with different Go version.

    ||Go 1.16|Go 1.17|Go 1.18| |-|-|-|-| |go get|Passed|Warned|Failed| |go install|Failed|Failed|Failed|

    go get -u github.com/carousell/Orion/protoc-gen-orion go install github.com/carousell/Orion/[email protected]

    Target:

    Allow protoc-gen-orion to be installed by go install.

    Action:

    Remove replace directives from go.mod.

    Result:

    We can install protoc-gen-go with go install command in Go 1.16, 1.17, and 1.18.

    Install protoc-gen-orion with different Go version.

    ||Go 1.16|Go 1.17|Go 1.18| |-|-|-|-| |go get|Passed|Warned|Failed| |go install|Passed|Passed|Passed|

    go get -u github.com/carousell/Orion/[email protected] go install github.com/carousell/Orion/[email protected] afd0407a = this commit

    opened by cs-lexliu 6
  • init incoming context as default for HTTP server

    init incoming context as default for HTTP server

    We've onboarded forward metadata via incoming and outgoing context in https://github.com/carousell/Orion/pull/149, but in HTTP server we haven't inited an incoming context inside, which means we can use the metadata in the incoming context that will be transformed to outgoing when we call to other services.

    Also, in our current interface of encoders: https://github.com/carousell/Orion/blob/master/orion/handlers/types.go#L40-L44, it doesn't support context assigning back, which will cause new context with incoming context lost. We will have to init an incoming context in preparation stage. (Btw, for gRPC server, the context always has incoming context regardless)

    opened by hothero 4
  • Add GRPCRecovery Interceptor

    Add GRPCRecovery Interceptor

    • main purpose is to prevent silent crash because of server panic, should recover and send the error out (through notifier/log).
    • add GRPCRecoveryInterceptor to DefaultInterceptors()
    • in GRPCRecoveryInterceptor, it will use grpc_recovery middleware with defined handler return an Orion/utils/errors ErrorExt. The error will skip 4 stack frames and show the function triggered the panic.

    Concept Flow not include the change for http middlewares https://drive.google.com/file/d/13AkM-68szUG_ITRaWJUpnJGcQF4Xqb3J/view?usp=sharing

    Panic Log with no skip

    {"@timestamp":"2019-07-01T16:19:33.3244498Z","caller":"interceptors/interceptors.go:137","err":"runtime error: index out of range","grpcMethod":"/{service}/{rpcName}","level":"error",
    "stack":[
    {"file":"/go/src/github.com/.../vendor/github.com/carousell/Orion/interceptors/interceptors.go","line":153,"function":"GRPCRecoveryInterceptor.func1.1"},
    {"file":"/usr/local/go/src/runtime/asm_amd64.s","line":522,"function":"call32"},
    {"file":"/usr/local/go/src/runtime/panic.go","line":513,"function":"gopanic"},
    {"file":"/usr/local/go/src/runtime/panic.go","line":44,"function":"panicindex"},
    {"file":"/go/src/github.com/.../service.go","line":442,"function":"{panic func}"},
    ...
    

    so skip 4 stack frames to remove non-related information.

    Example on Sentry (Internal) https://sentry.carou.sl/carousell/home/issues/244262/?query=is:unresolved

    opened by stevenCarousell 4
  • Allow *.orion.pb.go files to access the exported-service-desc from *.pb.go

    Allow *.orion.pb.go files to access the exported-service-desc from *.pb.go

    Allow *.orion.pb.go files to access the exported-service-desc from *.pb.go

    Add two execution parameters.

    • standalone-mode: Allow you to place *.orion.pb.go and *.pb.go into different folder.
    • exported-service-desc: Allow you to use the higher version (v1.20.0) of protoc-gen-go.

    Read go_package option from protobufs files. In standalone mode, the external import path is getting from the go_package option.

    https://github.com/carousell/Orion/issues/186

    opened by cs-lexliu 3
  • reorder defaultclientinterceptors

    reorder defaultclientinterceptors

    A request that failed could be many reasons including network-level issues, connection re-establishment, etc.. Hystrix as implementing circuit breaker pattern considers how to avoid intermittent network hiccups as well. Also, we don't want to have circuit open because of too many retries that were probably intended.

    In other words, we consider Hystrix Command as a whole for a request to other services regardless of whether retry or not.

    However, it may be a breaking change to some services that have been using grpc_retry options. We will have to notice those services after we merge this PR.

    opened by hothero 3
  • Add version config to kafka utils

    Add version config to kafka utils

    Problem

    In a recent exercise to onboard trace-id in logging, we upgraded Orion to the latest commit. However, due to complex dependencies between several internal modules, we are forced to upgrade sarama to v1.30.0 (the dependent version by carousell/messaging).

    With this upgrade, it breaks out publisher to old v0.11.2 kafka cluster. Connecting to kafka client also failed with error error creating async producer: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

    Root Cause

    The root cause is since sarama v1.27.1, the default kafka version in config is changed from v0.8.2.0 to v1.0.0.0.

    If you are connecting to old kafka cluster using v1.27.1+, you have to explicitly specify the Version in sarama config https://github.com/Shopify/sarama/blob/v1.27.1/config.go#L425-L431.

    PR Change Summary

    However, Orion's NewProducer() doesn't expose an option for callers to set version config.

    This PR introduces a new version option allowing callers to specify kafka version when connecting to old version cluster.

    Usage

    import (
        "github.com/Shopify/sarama"
        "github.com/carousell/Orion/utils/kafka"
    )
    
    producer, err := kafka.NewProducer(
        []string{"broker1", "broker2", "broker3"}
        kafka.WithVersion(sarama.V0_11_0_2),
        // other options go here...
    )
    
    opened by achichen 2
  • Adding sonar.properties and .golangci.yml template files

    Adding sonar.properties and .golangci.yml template files

    • Adding sonarqube properties files as a part of service creation
    • Adding .golangci.yml as a part of service creation
    • Adding phase to add golangci dependency
    opened by priyankshah217 2
  • Fix travis ci error

    Fix travis ci error

    We have an error when travis building code on tip env with go1.19. Run go mod tidy for download missing packages on the env. Add more go version for building the code.

    opened by cs-lexliu 2
  • Forward headers package

    Forward headers package

    The main idea is to have the capability to whitelist outgoing headers and not break our DefaultClientInterceptor.

    • Create a forwardheaders package responsible for server and client interceptors, allowlist management through context and some util functions
    • Extend orion.config to have forwardheaders as the allowlist
    • Extend orion.config to have NoDefaultInterceptors (we have the capability but seems no one set it yet)
    • Extend DefaultServerInterceptors to set allowlist for forwardheaders
    • Transform HTTP headers into gRPC headers (metadata)
    opened by hothero 2
  • Allow *.orion.pb.go files to be generated into a different package from the generated protobufs

    Allow *.orion.pb.go files to be generated into a different package from the generated protobufs

    Currently the generated *.orion.pb.go file needs to be placed as a sibling to the generated protobufs, because the orion file references an unexported variable:

    https://github.com/carousell/Orion/blob/ec021a79aefef548f78fd349884cb3f50b8db9ed/protoc-gen-orion/orion.go#L262

    In later versions of protoc-gen-go-grpc, this variable is now exported. https://github.com/carousell/Orion/issues/171 is a related issue to allow protoc-gen-orion to use the exported variable.

    The variable being exported now also means that it should be possible to place the *.orion.pb.go file in a separate package. In Carousell we're considering having a Go module for all generated protobufs, and another module for all generated orion *.orion.pb.go files.

    Ideally the module for generated protobufs will only contain two dependencies:

    github.com/golang/protobuf
    google.golang.org/grpc
    

    Having the orion files as a sibling will require github.com/carousell/Orion to be added as another dependency. The orion dependency is too heavy for us to include. Not every service that consumes our generated protobufs will need Orion.

    Thus ideally the orion file will live in a separate package. Example file structure:

    gen/
      go/
        serviceA/serviceA.pb.go
        serviceB/serviceB.pb.go
        go.mod
      orion/
        serviceA/serviceB.orion.pb.go
        go.mod
    

    I have some ideas of how to extend protoc-gen-orion to support this. I'll add more comments to the issue later.

    opened by benjaminheng 6
  • [RE-5863] Resolve potential race condition in log fields

    [RE-5863] Resolve potential race condition in log fields

    Description

    In current version of logFields, we discovered that there is a potential risk of having racing condition when multiple goroutine try to access logFields under an identical context (ctx). As you may see from the report below, when running parallelism test on the current implementation, a race condition is emerged even with a very little sample rate. Therefore, in this PR, we are turning the old logFields into protected logFields by applying read/write lock. Now when logFields being accessed, it will first check if some other instance is also accessing it, if yes, it will wait until the access is freed up so no simultaneous write will happen. In effects, FromContext doesn't seems to have it's performance decreased, while AddToLogContext increased about 10% of it's process time, and it's all being said with race condition is no longer a concern.

    Review Guideline

    FromContext Applied with read lock as it's a public method, now we are return a copy of the map instead of giving access directly to logFields

    fromContext Same implementation as before, but now it's only for private access

    AddToLogContext Now write access is locked when writing new data to logFields

    How to Test

    Parallelism Test Previously

    Reader 2 read from logfields map[k1:v1 k2:v2]
    Reader 3 read from logfields map[k1:v1 k2:v2]
    Reader 1 read from logfields map[k1:v1 k2:v2]
    ==================
    WARNING: DATA RACE
    Write at 0x00c00009f380 by goroutine 14:
      runtime.mapassign_faststr()
          /Users/chiahewen/.goenv/versions/1.16.8/src/runtime/map_faststr.go:202 +0x0
      github.com/carousell/Orion/utils/log/loggers.LogFields.Add()
          /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/fields.go:19 +0x11d
      github.com/carousell/Orion/utils/log/loggers.AddToLogContext()
          /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/fields.go:38 +0xcd
      github.com/carousell/Orion/utils/log/loggers/test_test.writeWorker()
          /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/test/parallelism_test.go:29 +0x1ce
      github.com/carousell/Orion/utils/log/loggers/test_test.TestParallelWrite.func1()
          /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/test/parallelism_test.go:61 +0xa4
    
    Previous write at 0x00c00009f380 by goroutine 16:
      runtime.mapassign_faststr()
          /Users/chiahewen/.goenv/versions/1.16.8/src/runtime/map_faststr.go:202 +0x0
      github.com/carousell/Orion/utils/log/loggers.LogFields.Add()
          /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/fields.go:19 +0x11d
      github.com/carousell/Orion/utils/log/loggers.AddToLogContext()
          /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/fields.go:38 +0xcd
      github.com/carousell/Orion/utils/log/loggers/test_test.writeWorker()
          /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/test/parallelism_test.go:29 +0x1ce
      github.com/carousell/Orion/utils/log/loggers/test_test.TestParallelWrite.func1()
          /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/test/parallelism_test.go:61 +0xa4
    
    Goroutine 14 (running) created at:
      github.com/carousell/Orion/utils/log/loggers/test_test.TestParallelWrite()
          /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/test/parallelism_test.go:59 +0x204
      testing.tRunner()
          /Users/chiahewen/.goenv/versions/1.16.8/src/testing/testing.go:1193 +0x202
    
    Goroutine 16 (running) created at:
      github.com/carousell/Orion/utils/log/loggers/test_test.TestParallelWrite()
          /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/test/parallelism_test.go:59 +0x204
      testing.tRunner()
          /Users/chiahewen/.goenv/versions/1.16.8/src/testing/testing.go:1193 +0x202
    ==================
    Writer 3 wrote key3:val8081
    Writer 2 wrote key2:val1847
    Writer 1 wrote key1:val7887
    lf map[key1:val7887 key2:val1847 key3:val8081 test-key:test-value]
    --- FAIL: TestParallelWrite (0.25s)
        testing.go:1092: race detected during execution of test
    ==================
    WARNING: DATA RACE
    Write at 0x00c000196090 by goroutine 23:
      runtime.mapassign_faststr()
          /Users/chiahewen/.goenv/versions/1.16.8/src/runtime/map_faststr.go:202 +0x0
      github.com/carousell/Orion/utils/log/loggers.LogFields.Add()
          /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/fields.go:19 +0x11d
      github.com/carousell/Orion/utils/log/loggers.AddToLogContext()
          /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/fields.go:38 +0xcd
      github.com/carousell/Orion/utils/log/loggers/test_test.writeWorker()
          /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/test/parallelism_test.go:29 +0x1ce
      github.com/carousell/Orion/utils/log/loggers/test_test.TestParallelReadAndWrite.func2()
          /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/test/parallelism_test.go:93 +0xa4
    
    Previous write at 0x00c000196090 by goroutine 22:
      runtime.mapassign_faststr()
          /Users/chiahewen/.goenv/versions/1.16.8/src/runtime/map_faststr.go:202 +0x0
      github.com/carousell/Orion/utils/log/loggers.LogFields.Add()
          /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/fields.go:19 +0x11d
      github.com/carousell/Orion/utils/log/loggers.AddToLogContext()
          /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/fields.go:38 +0xcd
      github.com/carousell/Orion/utils/log/loggers/test_test.writeWorker()
          /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/test/parallelism_test.go:29 +0x1ce
      github.com/carousell/Orion/utils/log/loggers/test_test.TestParallelReadAndWrite.func2()
          /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/test/parallelism_test.go:93 +0xa4
    
    Goroutine 23 (running) created at:
      github.com/carousell/Orion/utils/log/loggers/test_test.TestParallelReadAndWrite()
          /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/test/parallelism_test.go:91 +0x2c5
      testing.tRunner()
          /Users/chiahewen/.goenv/versions/1.16.8/src/testing/testing.go:1193 +0x202
    
    Goroutine 22 (running) created at:
      github.com/carousell/Orion/utils/log/loggers/test_test.TestParallelReadAndWrite()
          /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/test/parallelism_test.go:91 +0x2c5
      testing.tRunner()
          /Users/chiahewen/.goenv/versions/1.16.8/src/testing/testing.go:1193 +0x202
    ==================
    Writer 3 wrote key3:val1318
    Reader 3 read from logfields map[key1:val4059 key2:val2081 key3:val1318 test-key:test-value]
    ==================
    WARNING: DATA RACE
    Read at 0x00c00019c098 by goroutine 20:
      reflect.typedmemmove()
          /Users/chiahewen/.goenv/versions/1.16.8/src/runtime/mbarrier.go:177 +0x0
      reflect.copyVal()
          /Users/chiahewen/.goenv/versions/1.16.8/src/reflect/value.go:1310 +0x7d
      reflect.(*MapIter).Value()
          /Users/chiahewen/.goenv/versions/1.16.8/src/reflect/value.go:1264 +0x186
      internal/fmtsort.Sort()
          /Users/chiahewen/.goenv/versions/1.16.8/src/internal/fmtsort/sort.go:65 +0x2d9
      fmt.(*pp).printValue()
          /Users/chiahewen/.goenv/versions/1.16.8/src/fmt/print.go:769 +0x14cf
      fmt.(*pp).printArg()
          /Users/chiahewen/.goenv/versions/1.16.8/src/fmt/print.go:712 +0x284
      fmt.(*pp).doPrintf()
          /Users/chiahewen/.goenv/versions/1.16.8/src/fmt/print.go:1026 +0x330
      fmt.Fprintf()
          /Users/chiahewen/.goenv/versions/1.16.8/src/fmt/print.go:204 +0x84
      fmt.Printf()
          /Users/chiahewen/.goenv/versions/1.16.8/src/fmt/print.go:213 +0x111
      github.com/carousell/Orion/utils/log/loggers/test_test.readWorker()
          /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/test/parallelism_test.go:23 +0x67
      github.com/carousell/Orion/utils/log/loggers/test_test.TestParallelReadAndWrite.func1()
          /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/test/parallelism_test.go:85 +0x99
    
    Previous write at 0x00c00019c098 by goroutine 22:
      github.com/carousell/Orion/utils/log/loggers.LogFields.Add()
          /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/fields.go:19 +0x132
      github.com/carousell/Orion/utils/log/loggers.AddToLogContext()
          /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/fields.go:38 +0xcd
      github.com/carousell/Orion/utils/log/loggers/test_test.writeWorker()
          /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/test/parallelism_test.go:29 +0x1ce
      github.com/carousell/Orion/utils/log/loggers/test_test.TestParallelReadAndWrite.func2()
          /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/test/parallelism_test.go:93 +0xa4
    
    Goroutine 20 (running) created at:
      github.com/carousell/Orion/utils/log/loggers/test_test.TestParallelReadAndWrite()
          /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/test/parallelism_test.go:83 +0x20e
      testing.tRunner()
          /Users/chiahewen/.goenv/versions/1.16.8/src/testing/testing.go:1193 +0x202
    
    Goroutine 22 (running) created at:
      github.com/carousell/Orion/utils/log/loggers/test_test.TestParallelReadAndWrite()
          /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/test/parallelism_test.go:91 +0x2c5
      testing.tRunner()
          /Users/chiahewen/.goenv/versions/1.16.8/src/testing/testing.go:1193 +0x202
    ==================
    Reader 2 read from logfields map[key1:val4059 key2:val2081 key3:val1318 test-key:test-value]
    Reader 1 read from logfields map[key1:val4059 key2:val2081 key3:val1318 test-key:test-value]
    Writer 1 wrote key1:val4059
    Writer 2 wrote key2:val2081
    map[key1:val4059 key2:val2081 key3:val1318 test-key:test-value]
    --- FAIL: TestParallelReadAndWrite (0.25s)
        testing.go:1092: race detected during execution of test
    FAIL
    exit status 1
    FAIL    github.com/carousell/Orion/utils/log/loggers/test       1.136s
    

    Now

    Reader 9 read from logfields map[k1:v1 k2:v2]
    Reader 2 read from logfields map[k1:v1 k2:v2]
    ...
    Writer 47 wrote key47:val4078
    Writer 48 wrote key48:val6159
    lf map[key1:val1847 key10:val8511 key11:val3300 key12:val694 key13:val1211 key14:val8162 key15:val3237 key16:val495 key17:val9947 key18:val4728 key19:val3274 key2:val7887 key20:val5089 key21:val1445 key22:val9106 key23:val1528 key24:val5466 key25:val6258 key26:val8047 key27:val8287 key28:val2790 key29:val2888 key3:val1318 key30:val3015 key31:val5541 key32:val408 key33:val7387 key34:val6831 key35:val5429 key36:val1737 key37:val631 key38:val1485 key39:val5194 key4:val8081 key40:val5356 key41:val6413 key42:val3090 key43:val5026 key44:val4147 key45:val563 key46:val4324 key47:val4078 key48:val6159 key49:val2433 key5:val2081 key50:val1353 key6:val4059 key7:val2540 key8:val4425 key9:val456 test-key:test-value]
    Reader 29 read from logfields map[test-key:test-value]
    Reader 18 read from logfields map[test-key:test-value]
    ...
    map[key1:val1957 key10:val9355 key11:val2451 key12:val4538 key13:val8266 key14:val156 key15:val2605 key16:val5561 key17:val7202 key18:val9828 key19:val4783 key2:val3721 key20:val4376 key21:val5746 key22:val1563 key23:val9002 key24:val9718 key25:val5447 key26:val5094 key27:val1577 key28:val7463 key29:val7996 key3:val3000 key30:val6420 key31:val8623 key32:val953 key33:val1137 key34:val3133 key35:val9241 key36:val59 key37:val3033 key38:val8643 key39:val3891 key4:val7189 key40:val2002 key41:val8878 key42:val9336 key43:val2546 key44:val9107 key45:val7940 key46:val6503 key47:val552 key48:val9843 key49:val2205 key5:val2199 key50:val1598 key6:val8705 key7:val9703 key8:val2888 key9:val8510 test-key:test-value]
    PASS
    ok      github.com/carousell/Orion/utils/log/loggers/test       1.200s
    

    Benchmark stat Previously

    cpu: Intel(R) Core(TM) i7-1068NG7 CPU @ 2.30GHz
    BenchmarkFromContext
    BenchmarkFromContext-8                  41110058                25.81 ns/op
    BenchmarkFromAddToLogContext
    BenchmarkFromAddToLogContext-8           2947244               409.4 ns/op
    

    Now

    cpu: Intel(R) Core(TM) i7-1068NG7 CPU @ 2.30GHz
    BenchmarkFromContext
    BenchmarkFromContext-8                  39115351                25.86 ns/op
    BenchmarkFromAddToLogContext
    BenchmarkFromAddToLogContext-8           2701092               442.9 ns/op
    

    Checklist

    • [ ] Test on local
    • [ ] Test on stage
    • [ ] run go mod tidy and go mod vendor
    • [ ] change module's version back to master

    Screenshots (Optional)

    Related PRs (Optional)

    Impacted Endpoints (Optional)

    opened by jalacardio 1
  • Encoder and Decoder concept seems opposite

    Encoder and Decoder concept seems opposite

    Orion defines func prototypes Encoder and Decoder for handling http request/response as https://github.com/carousell/Orion/blob/master/orion/handlers/types.go#L40-L44:

    //Encoder is the function type needed for request encoders
    type Encoder func(req *http.Request, reqObject interface{}) error
    
    //Decoder is the function type needed for response decoders
    type Decoder func(ctx context.Context, w http.ResponseWriter, encodeError, endpointError error, respObject interface{})
    

    Encoder extracts information from incoming http request and convert to a request object. Decoder takes a response object to write it into http response.

    Essentially, Encoders does unmarshalling and deserializing action, while Decoders does marshalling and serializing. The name and what exactly it does seems opposite.

    Even in Orion's code, we can find this confusion: https://github.com/carousell/Orion/blob/master/orion/handlers/http/http.go#L176-L188. In the beginning of the section, the comment says "decoder func", but it calls encoders later.

    Should them switch names?

    opened by achichen 2
  • Unable to register service to Orion Server

    Unable to register service to Orion Server

    Context

    The grpc.ServiceDesc has been exported from protoc-gen-go-grpc v1.1.0. The protoc-gen-orion generates the file that contains an unresolved reference to the unexported name of service desc.

    Process

    Use the Orion example.

    Execute protoc -I. --go-grpc_out=. --go_out=. --orion_out=. simple.proto.

    Environment

    go version         1.17.6
    protoc-gen-go-grpc v1.2.0
    protoc-gen-orion   latest
    protoc-gen-go      v1.27.1
    

    Expected result

    The simple service should be registered in Orion Serve, and simple_proto/simple.proto.orion.pb.go should not contain any error.

    Current result

    In simple_proto/simple.proto.orion.pb.go, we got a following error.

    Unresolved reference '_SimpleService_serviceDesc'
    

    Possible Fix

    Follow the protoc-gen-go-grpc naming role in https://github.com/grpc/grpc-go/pull/4035. And modify the generating code. https://github.com/carousell/Orion/blob/ec021a79aefef548f78fd349884cb3f50b8db9ed/protoc-gen-orion/orion.go#L262

    opened by cs-lexliu 0
  • Bump go.elastic.co/apm from 1.4.0 to 1.11.0

    Bump go.elastic.co/apm from 1.4.0 to 1.11.0

    Bumps go.elastic.co/apm from 1.4.0 to 1.11.0.

    Release notes

    Sourced from go.elastic.co/apm's releases.

    v1.11.0

    v1.10.0

    • module/apmsql: add tracingDriver.Unwrap method to get underlying driver #849
    • module/apmgopgv10: add support for github.com/go-pg/pg/v10 #857 (thanks @​macnibblet!)
    • Enable central configuration of "sanitize_field_names" #856
    • module/apmgrpc: set span destination context #861

    v1.9.0

    • module/apmgoredisv8: introduce new package to support go-redis v8 (#780)
    • module/apmhttp: introduce httptrace client option (#788)
    • module/apmsql: add support for database/sql/driver.Validator (#791)
    • Record sample rate on transactions and spans, propagate through tracestate (#804)
    • module/apmredigo: change redigo dependency to v1.8.2 (#807)
    • Deprecate IGNORE_URLS, replace with TRANSACTION_IGNORE_URLS (#811)
    • Tracer.Close now waits for the transport goroutine to end before returning (#816)
    • Relax Kubernetes pod UID discovery rules (#819)
    • Add transaction and span outcome (#820)
    • Add cloud metadata, configurable with ELASTIC_APM_CLOUD_PROVIDER (#823)
    • Round ELASTIC_APM_SAMPLING_RATE with 4 digits precision (#828)
    • module/apmhttp: implement io.ReaderFrom in wrapped http.ResponseWriter (#830)
    • Fixed Transaction.Discard so that it sets TransactionData to nil (#836)
    • module/apmsql/pgxv4: add support for pgx driver (#831 -- thanks @​Deepak13245!)
    • module/apmgormv2: add support for gorm.io (GORM v2) (#825 -- thanks @​Deepak13245!)

    v1.8.0

    • Add "recording" config option, to dynamically disable event recording (#737)
    • Enable central configuration of "stack_frames_min_duration" and "stack_trace_limit" (#742)
    • Implement "CloseIdleConnections" on the Elasticsearch RoundTripper (#750 - thanks @​pebrc!)
    • Fix apmot nil pointer dereference in Tracer.Inject (#763 - thanks @​randomswdev!)

    v1.7.2

    • Update cucumber/godog to 0.8.1 (#733)

    v1.7.1

    • Fix segfault on 32-bit architectures (#728)

    v1.7.0

    • Add span.context.destination.* #664
    • transport: fix Content-Type for pprof data #679
    • Add "tracestate" propagation #690
    • Add support for API Key auth #698
    • module/apmsql: report rows affected #700

    v1.6.0

    ... (truncated)

    Changelog

    Sourced from go.elastic.co/apm's changelog.

    ==== 1.11.0 - 2021/02/01

    https://github.com/elastic/apm-agent-go/releases/tag/v1.11.0[View release]

    • Make TRANSACTION_IGNORE_URLS dynamically configurable: {pull}872#872

    [[release-notes-1.10.0]] ==== 1.10.0 - 2021/01/20

    https://github.com/elastic/apm-agent-go/releases/tag/v1.10.0[View release]

    • module/apmsql: add tracingDriver.Unwrap method to get underlying driver {pull}#849[#(849)]
    • module/apmgopgv10: add support for github.com/go-pg/pg/v10 {pull}857[(#857)]
    • Enable central configuration of "sanitize_field_names" {pull}856[(#856)]
    • module/apmgrpc: set span destination context {pull}861[(#861)]

    [[release-notes-1.9.0]] ==== 1.9.0 - 2020/11/02

    https://github.com/elastic/apm-agent-go/releases/tag/v1.9.0[View release]

    • module/apmgoredisv8: introduce new package to support go-redis v8 {pull}780[#(780)]
    • module/apmhttp: introduce httptrace client option {pull}788[#(788)]
    • module/apmsql: add support for database/sql/driver.Validator {pull}791[#(791)]
    • Record sample rate on transactions and spans, propagate through tracestate {pull}804[#(804)]
    • module/apmredigo: change redigo dependency to v1.8.2 {pull}807[#(807)]
    • Deprecate IGNORE_URLS, replace with TRANSACTION_IGNORE_URLS {pull}811[(#811)]
    • Tracer.Close now waits for the transport goroutine to end before returning {pull}816[#(816)]
    • Relax Kubernetes pod UID discovery rules {pull}819[#(819)]
    • Add transaction and span outcome {pull}820[#(820)]
    • Add cloud metadata, configurable with ELASTIC_APM_CLOUD_PROVIDER {pull}823[#(823)]
    • Round ELASTIC_APM_SAMPLING_RATE with 4 digits precision {pull}828[#(828)]
    • module/apmhttp: implement io.ReaderFrom in wrapped http.ResponseWriter {pull}830[#(830)]
    • Fixed Transaction.Discard so that it sets TransactionData to nil {pull}836[#(836)]
    • module/apmsql/pgxv4: add support for pgx driver {pull}831[#(831)]
    • module/apmgormv2: add support for gorm.io (GORM v2) {pull}825[#(825)]

    [[release-notes-1.8.0]] ==== 1.8.0 - 2020/05/06

    https://github.com/elastic/apm-agent-go/releases/tag/v1.8.0[View release]

    • Add "recording" config option, to dynamically disable event recording {pull}737[(#737)]
    • Enable central configuration of "stack_frames_min_duration" and "stack_trace_limit" {pull}742[(#742)]
    • Implement "CloseIdleConnections" on the Elasticsearch RoundTripper {pull}750[(#750)]
    • Fix apmot nil pointer dereference in Tracer.Inject {pull}763[(#763)]

    [[release-notes-1.7.2]] ==== 1.7.2 - 2020/03/19

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 2
Owner
Carousell
Carousell is a community marketplace that makes selling as simple as taking a photo, buying as simple as chatting.
Carousell
Ephemeral One Time/Build-Time gRPC TLS PKI system.

PkiSauce Ephemeral Build Time TLS PKI saucing for your intra services GRPC (or not) communications. Description A simple attempt to avoid deploying co

Ermites.IO 6 Jul 4, 2022
Go-grpc-template - A small template for quickly bootstrapping a, developer platform independent gRPC golang application

go-grpc-template A small template for quickly bootstrapping a developer platform

null 1 Jan 20, 2022
protoc-gen-grpc-gateway-ts is a Typescript client generator for the grpc-gateway project. It generates idiomatic Typescript clients that connect the web frontend and golang backend fronted by grpc-gateway.

protoc-gen-grpc-gateway-ts protoc-gen-grpc-gateway-ts is a Typescript client generator for the grpc-gateway project. It generates idiomatic Typescript

gRPC Ecosystem 82 Sep 28, 2022
Our aim is to expand the capabilities of blockchain and make a secure way for transferring NFT between RMRK and MOVR blockchain.

remov Inspiration Our aim is to expand the capabilities of blockchain and make a secure way for transferring NFT between RMRK and MOVR blockchain. The

RMRK Team 3 Jul 25, 2022
The aim of this project is to publish and archive newsletters to a target email address.

Publish Newsletter Curated by a Group of People Even though the name says Group of People, it can be just you. The aim of this project is to publish a

코딩냄비 12 Jan 30, 2022
⚡️Lightweight framework for microservices & web services in golang

Quickstart Zepto is a lightweight framework for the development of microservices & web services in golang. As an opinionated framework, zepto proposes

null 117 Jun 19, 2022
Using Wireshark to decrypt TLS gRPC Client-Server protobuf messages

Using Wireshark to decrypt TLS gRPC Client-Server protobuf messages Sample client server in golang that demonstrates how to decode protobuf messages f

null 4 Sep 8, 2022
A simple RPC framework with protobuf service definitions

Twirp is a framework for service-to-service communication emphasizing simplicity and minimalism. It generates routing and serialization from API defin

Twitch 6.1k Sep 27, 2022
Lightweight http response time based load balancer written in Go

HTTP Load Balancer Specifications http servers should always return time taken to proceed request in headers as EXECUTION_TIME in ms this load balance

Gaëtan 0 Feb 22, 2022
A suite of gRPC debugging tools. Like Fiddler/Charles but for gRPC.

grpc-tools A suite of tools for gRPC debugging and development. Like Fiddler/Charles but for gRPC! The main tool is grpc-dump which transparently inte

Bradley Kemp 1.1k Sep 22, 2022
grpc-http1: A gRPC via HTTP/1 Enabling Library for Go

grpc-http1: A gRPC via HTTP/1 Enabling Library for Go This library enables using all the functionality of a gRPC server even if it is exposed behind a

StackRox 85 Aug 18, 2022
Server and client implementation of the grpc go libraries to perform unary, client streaming, server streaming and full duplex RPCs from gRPC go introduction

Description This is an implementation of a gRPC client and server that provides route guidance from gRPC Basics: Go tutorial. It demonstrates how to u

Joram Wambugu 0 Nov 24, 2021
Go based grpc - grpc gateway micro service example

go-grpc-gateway-server This repository provides an example for go based microservice. Go micro services developed based on gRPC protobuf's and also us

Suresh Yekasiri 0 Dec 8, 2021
Simple grpc web and grpc transcoding with Envoy

gRPC Web and gRPC Transcoding with Envoy This is a simple stand-alone set of con

null 0 Dec 25, 2021
Go-grpc - This is grpc server for golang.

go-grpc This is grpc server for golang. protocのインストール brew install protoc Golang用のプラグインのインストール go install google.golang.org/protobuf/cmd/protoc-gen-go

jotaro yuza 1 Jan 2, 2022
GRPC - Creating a gRPC service from scratch

#Go gRPC services course Creating a gRPC service from scratch Command line colle

Rafael Diaz Miles 1 Jan 2, 2022
Totem - A Go library that can turn a single gRPC stream into bidirectional unary gRPC servers

Totem is a Go library that can turn a single gRPC stream into bidirectional unar

Joe Kralicky 2 Jan 10, 2022
Grpc-gateway-map-null - gRPC Gateway test using nullable values in map

Demonstrate gRPC gateway behavior with nullable values in maps Using grpc-gatewa

null 1 Jan 6, 2022
Todo-app-grpc - Go/GRPC codebase containing RealWorld examples (CRUD, auth, advanced patterns, etc)

Go/GRPC codebase containing RealWorld examples (CRUD, auth, advanced patterns, e

Sammi Aldhi Yanto 4 Aug 12, 2022