Opinionated boilerplate Golang HTTP server with CORS, OPA, Prometheus, rate-limiter for API and static website.

Overview

Teal.Finance/Garcon

logo Opinionated boilerplate HTTP server with CORS, OPA, Prometheus, rate-limiter… for API and static website.

Origin

This library was originally developed as part of the project Rainbow during hackathons, based on older Teal.Finance products, and then moved to its own repository.

Features

Teal.Finance/Garcon supports:

  • Metrics server exporting data to Prometheus or other monitoring services ;
  • File server intended for static web files ;
  • HTTP/REST server for API endpoints (compatible any Go-standard HTTP handlers) ;
  • Chained middlewares (fork of github.com/justinas/alice)
  • Auto-completed error response in JSON format ;
  • Middleware: authentication rules based on Datalog/Rego files using Open Policy Agent ;
  • Middleware: rate limiter to prevent flooding by incoming requests ;
  • Middleware: logging of incoming requests ;
  • Middleware: Cross-Origin Resource Sharing (CORS).

License

LGPL-3.0-or-later: GNU Lesser General Public License v3.0 or later (tl;drLegal, Choosealicense.com). See the LICENSE file.

Except the two example files under CC0-1.0 (Creative Commons Zero v1.0 Universal) and the file chain.go (fork) under the MIT License.

Examples

See also a complete real example in the repo github.com/teal-finance/rainbow.

High-level

The following code uses the high-level function Garcon.RunServer().

package main

import (
    "log"

    "github.com/teal-finance/garcon"
)

func main() {
    s := garcon.Garcon{
        Version:        "MyApp-1.2.0",
        Resp:           "https://my.dns.co/doc",
        AllowedOrigins: []string{"https://my.dns.co"},
        OPAFilenames:   []string{"example-auth.rego"},
    }

    h := myHandler()

    // main port 8080, export port 9093, rate limiter 10 20, dev mode 
    log.Fatal(s.RunServer(h, 8080, 9093, 10, 20, true))
}

Run the high-level example:

$ go build ./examples/high-level && ./high-level
2021/10/19 23:41:50 Prometheus export http://localhost:9093
2021/10/19 23:41:50 CORS: Set origin: https://my.dns.co
2021/10/19 23:41:50 Middleware CORS: {AllowedOrigins:[] AllowOriginFunc:0x556b48e30960 AllowOriginRequestFunc:<nil> AllowedMethods:[GET] AllowedHeaders:[Origin Accept Content-Type Authorization Cookie] ExposedHeaders:[] MaxAge:60 AllowCredentials:true OptionsPassthrough:false Debug:true}
2021/10/19 23:41:50 OPA: load "example-auth.rego"
2021/10/19 23:41:50 Middleware OPA: map[example-auth.rego:package auth

default allow = false
tokens := {"Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJsb2dnZWRJbkFzIjoiYWRtaW4iLCJpYXQiOjE0MjI3Nzk2Mzh9.gzSraSYS8EXBxLN_oWnFSRgCzcmJmMjLiuyu5CSpyHI"} { true }
allow = true { __local0__ = input.token; data.auth.tokens[__local0__] }]
2021/10/19 23:41:50 Middleware response HTTP header: Set Server MyApp-1.2.0
2021/10/19 23:41:50 Middleware RateLimiter: burst=100 rate=5/s
2021/10/19 23:41:50 Middleware logger: log requested URLs and remote addresses
2021/10/19 23:41:50 Server listening on http://localhost:8080

Test the API endpoint with default curl HTTP headers:

$ curl -D - http://localhost:8080/api/v1/items
HTTP/1.1 401 Unauthorized
Content-Type: application/json
Server: MyApp-1.2.0
Vary: Origin
X-Content-Type-Options: nosniff
Date: Tue, 19 Oct 2021 21:42:29 GMT
Content-Length: 77

{"error":"Unauthorized","doc":"https://my.dns.co/doc","path":"/api/v1/items"}

The corresponding garcon logs in debug mode:

2021/10/19 23:42:29 in  GET /api/v1/items [::1]:54796
[cors] 2021/10/19 23:42:29 Handler: Actual request
[cors] 2021/10/19 23:42:29   Actual request no headers added: missing origin
2021/10/19 23:42:29 OPA unauthorize [::1]:54796 /api/v1/items
2021/10/19 23:42:29 out GET /api/v1/items 337.751µs

Test the API endpoint with valid Authorization header:

$ curl -D - -H 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJsb2dnZWRJbkFzIjoiYWRtaW4iLCJpYXQiOjE0MjI3Nzk2Mzh9.gzSraSYS8EXBxLN_oWnFSRgCzcmJmMjLiuyu5CSpyHI' http://localhost:8080/api/v1/items
HTTP/1.1 200 OK
Content-Type: application/json
Server: MyApp-1.2.0
Vary: Origin
Date: Tue, 19 Oct 2021 21:43:55 GMT
Content-Length: 25

["item1","item2","item3"]

The corresponding garcon logs:

2021/10/19 23:43:55 in  GET /api/v1/items [::1]:54798
[cors] 2021/10/19 23:43:55 Handler: Actual request
[cors] 2021/10/19 23:43:55   Actual request no headers added: missing origin
2021/10/19 23:43:55 out GET /api/v1/items 235.621µs

Test the API endpoint with valid Authorization and Origin headers:

$ curl -D - -H 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJsb2dnZWRJbkFzIjoiYWRtaW4iLCJpYXQiOjE0MjI3Nzk2Mzh9.gzSraSYS8EXBxLN_oWnFSRgCzcmJmMjLiuyu5CSpyHI' -H 'Origin: https://my.dns.co' http://localhost:8080/api/v1/items
HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: https://my.dns.co
Content-Type: application/json
Server: MyApp-1.2.0
Vary: Origin
Date: Tue, 19 Oct 2021 21:45:00 GMT
Content-Length: 25

["item1","item2","item3"]

The corresponding garcon logs:

2021/10/19 23:45:00 in  GET /api/v1/items [::1]:54800
[cors] 2021/10/19 23:45:00 Handler: Actual request
[cors] 2021/10/19 23:45:00   Actual response added headers: map[Access-Control-Allow-Credentials:[true] Access-Control-Allow-Origin:[https://my.dns.co] Server:[MyApp-1.2.0] Vary:[Origin]]
2021/10/19 23:45:00 out GET /api/v1/items 341.432µs

Low-level

See the low-level example.

The following code can be replaced by the high-level function Garcon.RunServer() presented in the previous chapter. The following code is intended to show that the Teal.Finance/Garcon can be customized to meet specific requirements.

package main

import (
    "log"
    "net"
    "net/http"
    "time"

    "github.com/teal-finance/garcon"
    "github.com/teal-finance/garcon/chain"
    "github.com/teal-finance/garcon/cors"
    "github.com/teal-finance/garcon/export"
    "github.com/teal-finance/garcon/limiter"
    "github.com/teal-finance/garcon/opa"
    "github.com/teal-finance/garcon/reserr"
)

func main() {
    middlewares, connState := setMiddlewares()

    h := myHandler()
    h = middlewares.Then(h)

    runServer(h, connState)
}

func setMiddlewares() (middlewares chain.Chain, connState func(net.Conn, http.ConnState)) {
    // Uniformize error responses with API doc
    resErr := reserr.New("https://my.dns.co/doc")

    // Start a metrics server in background if export port > 0.
    // The metrics server is for use with Prometheus or another compatible monitoring tool.
    metrics := export.Metrics{}
    middlewares, connState = metrics.StartServer(9093, true)

    // Limit the input request rate per IP
    reqLimiter := limiter.New(10, 20, true, resErr)
    middlewares = middlewares.Append()

    // Endpoint authentication rules (Open Policy Agent)
    policy, err := opa.New(resErr, []string{"example-auth.rego"})
    if err != nil {
        log.Fatal(err)
    }

    // CORS
    allowedOrigins := []string{"https://my.dns.co"}

    middlewares = middlewares.Append(
        garcon.LogRequests,
        reqLimiter.Limit,
        garcon.ServerHeader("MyServerName-1.2.0"),
        policy.Auth,
        cors.Handle(allowedOrigins, true),
    )

    return middlewares, connState
}

// runServer runs in foreground the main server.
func runServer(h http.Handler, connState func(net.Conn, http.ConnState)) {
    server := http.Server{
        Addr:              ":8080",
        Handler:           h,
        TLSConfig:         nil,
        ReadTimeout:       1 * time.Second,
        ReadHeaderTimeout: 1 * time.Second,
        WriteTimeout:      1 * time.Second,
        IdleTimeout:       1 * time.Second,
        MaxHeaderBytes:    222,
        TLSNextProto:      nil,
        ConnState:         connState,
        ErrorLog:          log.Default(),
        BaseContext:       nil,
        ConnContext:       nil,
    }

    log.Print("Server listening on http://localhost", server.Addr)

    log.Fatal(server.ListenAndServe())
}
Comments
  • Fix security issues reported by CodeQL from GitHub

    Fix security issues reported by CodeQL from GitHub

    Issues fixed by this PR:

    • Log injection
    • Path traversal attacks (was already fixed, but improve code quality)
    • Logging sensitive information in clear-text
    opened by 0uep 0
  • Bump github.com/open-policy-agent/opa from 0.36.1 to 0.37.0

    Bump github.com/open-policy-agent/opa from 0.36.1 to 0.37.0

    Bumps github.com/open-policy-agent/opa from 0.36.1 to 0.37.0.

    Release notes

    Sourced from github.com/open-policy-agent/opa's releases.

    v0.37.0

    This release contains a number of fixes and enhancements.

    This is the first release that includes a binary and a docker image for linux/arm64, opa_linux_arm64_static and openpolicyagent/opa:0.37.0-static. Thanks to @​ngraef for contributing the build changes necessary.

    Strict Mode

    There have been numerous possible checks in the compiler that fall into this category:

    1. They would help avoid common mistakes; but
    2. Introducing them would potentially break some uncommon, but legitimate use.

    We've thus far refrained from introducing them. Now, a new "strict mode" allows you to opt-in to these checks, and we encourage you to do so!

    With OPA 1.0, they will become the new default behaviour.

    For more details, see the docs on Compiler Strict Mode.

    Delta Bundles

    Delta bundles provide a more efficient way to make data changes by containing patches to data instead of snapshots. Using them together with HTTP Long Polling, you can propagate small changes to bundles without waiting for polling delays.

    See the documentation for more details.

    Tooling and Runtime

    • Bundles bug fix: Roundtrip manifest before hashing to allow changing the manifest and still using signature verification of bundles (#4233), reported by @​CristianJena

    • The test runner now also supports custom builtins, when invoked through the Golang interface (authored by @​MIA-Deltat1995)

    • The compile package and the opa build command support a new output format: "plan". It represents a query plan, steps needed to take to evaluate a query (with policies). The plan format is a JSON encoding of the intermediate representation (IR) used for compiling queries and policies into Wasm.

      When calling opa build -t plan ..., the plan can be found in plan.json at the top- level directory of the resulting bundle.tar.gz. See the documentation for details..

    ... (truncated)

    Changelog

    Sourced from github.com/open-policy-agent/opa's changelog.

    0.37.0

    This release contains a number of fixes and enhancements.

    This is the first release that includes a binary and a docker image for linux/arm64, opa_linux_arm64_static and openpolicyagent/opa:0.37.0-static. Thanks to @​ngraef for contributing the build changes necessary.

    Strict Mode

    There have been numerous possible checks in the compiler that fall into this category:

    1. They would help avoid common mistakes; but
    2. Introducing them would potentially break some uncommon, but legitimate use.

    We've thus far refrained from introducing them. Now, a new "strict mode" allows you to opt-in to these checks, and we encourage you to do so!

    With OPA 1.0, they will become the new default behaviour.

    For more details, see the docs on Compiler Strict Mode.

    Delta Bundles

    Delta bundles provide a more efficient way to make data changes by containing patches to data instead of snapshots. Using them together with HTTP Long Polling, you can propagate small changes to bundles without waiting for polling delays.

    See the documentation for more details.

    Tooling and Runtime

    • Bundles bug fix: Roundtrip manifest before hashing to allow changing the manifest and still using signature verification of bundles (#4233), reported by @​CristianJena

    • The test runner now also supports custom builtins, when invoked through the Golang interface (authored by @​MIA-Deltat1995)

    • The compile package and the opa build command support a new output format: "plan". It represents a query plan, steps needed to take to evaluate a query (with policies). The plan format is a JSON encoding of the intermediate representation (IR) used for compiling queries and policies into Wasm.

      When calling opa build -t plan ..., the plan can be found in plan.json at the top- level directory of the resulting bundle.tar.gz. See the documentation for details..

    ... (truncated)

    Commits
    • ad4f4f1 Prepare 0.37.0 release (#4309)
    • a18f53d compile: adds metadata field to .manifest (#4306)
    • 01ca4a6 build(deps): bump ansi-regex in /docs/website/scripts/live-blocks (#4308)
    • 42a559c build(deps): bump github.com/prometheus/client_golang (#4307)
    • 503a520 ast: Deprecating any() and all() built-in functions (#4271)
    • 59810d0 ast: Making input and data reserved keywords in strict-mode (#4301)
    • dd02a7f Add support for delta bundles
    • cb867a1 ast/compile: 'every' rewriting steps (#4231)
    • d12fb7c docs: Update generated CLI docs
    • 4449d96 docs: add explanation of the IR
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
  • Affero notice and source code

    Affero notice and source code

    When a Garcon-based web application is released under the GNU AGPL, the web application should offer its source to those users in some way. For example, the interface of the web application could display a “Source” link that leads users to an archive of the code. The GNU AGPL is flexible enough to choose a method that's suitable for the specific program. See section 13 of the AGPL for details.

    See https://github.com/zipizap/EmbeddedSource

    opened by 0uep 0
  • Monitor web traffic

    Monitor web traffic

    Privacy-friendly web analytics in Garcon

    Pirsch

    https://github.com/pirsch-analytics/pirsch

    On server-side, Pirsch concatenates IP, User-Agent, date, then salt, hash and store it using ClickHouse. No cookies. The stored data is analyzed to extract web traffic info: unique visitors, sessions, bounces, views, average session duration, languages, operating system and browser, referrers, countries…

    The metrics may be sent to Prom or Loki and Grafana…

    GoatCounter

    https://github.com/arp242/goatcounter

    Active community-driven project. :heart:

    ZerØ Analytics

    https://github.com/ncarlier/za

    Sends metrics to Loki to Grafana.

    Fathom Lite

    https://github.com/usefathom/fathom

    The project is unmaintained but there are forks with random fixes. The community needs to rally around a fork.

    https://github.com/usefathom/fathom/network

    Repo | Stars | Forks | Ahead | Behind | Last Push -- | -- | -- | -- | -- | -- https://github.com/JokerQyou/fathom | 2 | 0 | 41 | 0 | 2020-06-03 https://github.com/samuelmeuli/fathom | 2 | 1 | 22 | 0 | 2020-05-31 https://github.com/bjkippax/fathom-lite-gdprcompliant | 0 | 0 | 3 | 0 | 2022-02-03 https://github.com/status-im/fathom | 0 | 0 | 2 | 0 | 2021-07-21 https://github.com/mauriciodulce/fathom | 0 | 0 | 1 | 0 | 2020-05-23 https://github.com/flopp/fathom | 0 | 0 | 1 | 0 | 2020-08-28 https://github.com/javascript-indonesias/fathom | 0 | 0 | 16 | 0 | 2020-02-15 https://github.com/aarroyoc/fathom | 0 | 0 | 1 | 0 | 2020-11-22 https://github.com/emarsden/fathom | 0 | 0 | 1 | 0 | 2020-02-16 https://github.com/KerstinMaur/fathom | 0 | 0 | 1 | 0 | 2021-10-03 https://github.com/kasuboski/fathom | 0 | 0 | 6 | 0 | 2020-08-18 https://github.com/featherbear/fathom | 0 | 0 | 4 | 0 | 2021-06-12 https://github.com/MikeLindenau/fathom | 0 | 0 | 1 | 0 | 2020-05-06 https://github.com/dworznik/fathom | 0 | 0 | 2 | 0 | 2021-04-11

    https://github.com/teal-finance/garcon/issues/9

    Plausible Analytics (Elixir)

    Self-hosted with ClickHouse https://github.com/plausible/analytics

    Shynet (Python)

    https://github.com/milesmcc/shynet

    Offen

    https://github.com/offen/offen

    Matomo and OWA

    Matomo and OWA require JavaScript tracking on the browser side. I do not like it.

    Prometheus

    The Caddy doc provides some Prometheus queries. Garcon should use the same metrics naming as Caddy.

    The Prometheus client provides monitoring functions: pkg.go.dev

    An example: GitHub

    Garcon should use them!

    opened by 0uep 0
  • Document privacy compliance

    Document privacy compliance

    Privacy laws:

    • GDPR
    • Schrems II
    • PECR
    • ePrivacy (cookie law)
    • COPPA
    • CCPA

    Be inspired by:

    • https://usefathom.com/blog/huge-growth
    • https://usefathom.com/compliance/pecr-compliant-website-analytics (see also the links at the bottom)
    opened by 0uep 0
  • Set the URL path prefix (BASE_URL)

    Set the URL path prefix (BASE_URL)

    When the back-end is served by a reverse-proxy, the reverse-proxy may strip the URL path prefix. This path prefix may be used to set the Cookie path in lieu of "/" and for the default Response Error (doc URL).

    opened by 0uep 0
Owner
Teal.Finance
Data for All to Verify or to Trade
Teal.Finance
⚡Simple cors middleware package for minima

This is package is wrapper based on rs/cors package made for minima. Geting Started Install the package using go get github.com/gominima/cors and call

Minima 5 Mar 7, 2022
Boilerplate API template includes all the common packages and setup used for API development in this Company

Boilerplate API Boilerplate API template includes all the common packages and setup used for API development in this Company. Development Copy .env.ex

null 11 Feb 19, 2022
A Golang restful API boilerplate based on Echo framework v4

A Golang restful API boilerplate based on Echo framework v4. Includes tools for module generation, db migration, authorization, authentication and more.

Dzung Tran 7 Aug 17, 2022
Opinionated Go starter with gin for REST API, logrus for logging, viper for config with added graceful shutdown

go-gin-starter An opinionated starter for Go Backend projects using: gin-gonic/gin as the REST framework logrus for logging viper for configs Docker f

Udaya Prakash 67 Aug 8, 2022
REST API boilerplate built with go and clean architecture - Echo Framework

GO Boilerplate Prerequisite Install go-migrate for running migration https://github.com/golang-migrate/migrate App requires 2 database (postgreSQL an

Syahid Firdaus 9 Sep 22, 2022
Headless CMS with automatic JSON API. Featuring auto-HTTPS from Let's Encrypt, HTTP/2 Server Push, and flexible server framework written in Go.

Ponzu Watch the video introduction Ponzu is a powerful and efficient open-source HTTP server framework and CMS. It provides automatic, free, and secur

Ponzu 5.5k Sep 30, 2022
An opinionated productive web framework that helps scaling business easier.

appy An opinionated productive web framework that helps scaling business easier, i.e. focus on monolith first, only move to microservices with GRPC la

appist 127 Sep 12, 2022
Download an entire website with Go

Go Download Web A simple command-line application to download an entire online website, including CSS, JSS, and other assets. Coded with Go. Project s

Antonio Sánchez 15 Aug 19, 2022
Go Fiber Boilerplate with Gorm ORM

Go Fiber Boilerplate with Gorm ORM This boilerplate app is using Go version 1.17 because I think for now this is the most updated release. Installatio

Muhammad Iqbal 2 Jul 30, 2022
Generate boilerplate + endpoints for Fiber REST APIs.

gomakeme Generate boilerplate + endpoints for Fiber REST APIs. Never spend 6 minutes doing something by hand when you can spend 1 week to automate it

null 4 May 9, 2022
⚡ Rux is an simple and fast web framework. support middleware, compatible http.Handler interface. 简单且快速的 Go web 框架,支持中间件,兼容 http.Handler 接口

Rux Simple and fast web framework for build golang HTTP applications. NOTICE: v1.3.x is not fully compatible with v1.2.x version Fast route match, sup

Gookit 81 Jul 20, 2022
A minimal framework to build web apps; with handler chaining, middleware support; and most of all standard library compliant HTTP handlers(i.e. http.HandlerFunc).

WebGo v4.1.3 WebGo is a minimalistic framework for Go to build web applications (server side) with zero 3rd party dependencies. Unlike full-fledged fr

Kamaleshwar 258 Sep 2, 2022
Muxie is a modern, fast and light HTTP multiplexer for Go. Fully compatible with the http.Handler interface. Written for everyone.

Muxie ?? ?? ?? ?? ?? ?? Fast trie implementation designed from scratch specifically for HTTP A small and light router for creating sturdy backend Go a

Gerasimos (Makis) Maropoulos 277 Sep 24, 2022
Prometheus middleware for wish

promwish Package promwish provides a simple wish middleware exposing some Prometheus metrics. Example Usage You can add promwish as a middleware to yo

Charm 28 Sep 1, 2022
Tigo is an HTTP web framework written in Go (Golang).It features a Tornado-like API with better performance. Tigo是一款用Go语言开发的web应用框架,API特性类似于Tornado并且拥有比Tornado更好的性能。

Tigo(For English Documentation Click Here) 一个使用Go语言开发的web框架。 相关工具及插件 tiger tiger是一个专门为Tigo框架量身定做的脚手架工具,可以使用tiger新建Tigo项目或者执行其他操作。

Karl 1.4k Sep 25, 2022
HTTP Rest API in Golang

Database configuration (for MacOS) Useful article - click to read Install postgres Run cd /Library/PostgreSQL/13/bin Run sudo -u postgres ./createdb r

null 0 Dec 14, 2021
CRUD API server of Clean Architecture with Go(Echo), Gorm, MySQL, Docker and Swagger

CRUD API Server of Clean Architecture Go(echo) gorm mysql docker swagger build docker-compose up -d --build API Postman and Fiddler is recommended to

null 34 May 14, 2022
henrylee2cn 1.6k Sep 16, 2022
弹弹play 资源搜索节点 API 实现,基于 Cloudflare Workers 或 Golang 的两种实现。 || API implementations for "dandanplay" resource search service.

dandanplay-resource-service API implementations for "dandanplay" resource search service. 弹弹play 资源搜索节点的 API 实现。 提供基于 Cloudflare Workers 和 Golang 的两种实

Lussac 52 Sep 14, 2022