Opinionated boilerplate Golang HTTP server with CORS, OPA, Prometheus, rate-limiter for API and static website.

Overview

Teal.Finance/Server

logo Opinionated boilerplate HTTP server with CORS, OPA, Prometheus, rate-limiter… for API and static website.

Origin

This library was originally developed as part of the project Rainbow during hackathons, based on older Teal.Finance products, and then moved to its own repository.

Features

Teal.Finance/Server supports:

  • Metrics server exporting data to Prometheus or other monitoring services ;
  • File server intended for static web files ;
  • HTTP/REST server for API endpoints (compatible any Go-standard HTTP handlers) ;
  • Chained middlewares (fork of github.com/justinas/alice)
  • Auto-completed error response in JSON format ;
  • Middleware: authentication rules based on Datalog/Rego files using Open Policy Agent ;
  • Middleware: rate limiter to prevent flooding by incoming requests ;
  • Middleware: logging of incoming requests ;
  • Middleware: Cross-Origin Resource Sharing (CORS).

License

LGPL-3.0-or-later: GNU Lesser General Public License v3.0 or later (tl;drLegal, Choosealicense.com). See the LICENSE file.

Except the two example files under CC0-1.0 (Creative Commons Zero v1.0 Universal) and the file chain.go (fork) under the MIT License.

High-level usage

See the high-level example.

The following Go code uses the high-level function Server.RunServer().

package main

import (
    "log"

    "github.com/teal-finance/server"
)

func main() {
    s := server.Server{
        Version:        "MyApp-1.2.3",
        Resp:           "https://my-dns.com/doc",
        AllowedOrigins: []string{"http://my-dns.com"},
        OPAFilenames:   []string{"example-auth.rego"},
    }

    h := myHandler()

    // main port 8080, export port 9093, rate limiter 10 20, debug mode 
    log.Fatal(s.RunServer(h, 8080, 9093, 10, 20, true))
}

Low-level example

See the low-level example.

See also a complete real example in the repo github.com/teal-finance/rainbow.

The following Go code can be replaced by the high-level function Server.RunServer() presented in the previous chapter. The following Go code is intended to show that the Teal.Finance/Server can be customized to meet specific requirements.

0. // The metrics server is for use with Prometheus or another compatible monitoring tool. metrics := export.Metrics{} middlewares, connState = metrics.StartServer(9093, true) // Limit the input request rate per IP reqLimiter := limiter.New(10, 20, true, resErr) middlewares = middlewares.Append() // Endpoint authentication rules (Open Policy Agent) policy, err := opa.New(resErr, []string{"example-auth.rego"}) if err != nil { log.Fatal(err) } // CORS allowedOrigins := []string{"http://my-dns.com"} middlewares = middlewares.Append( server.LogRequests, reqLimiter.Limit, server.Header("MyServerName-1.2.3"), policy.Auth, cors.HandleCORS(allowedOrigins), ) return middlewares, connState } // runServer runs in foreground the main server. func runServer(h http.Handler, connState func(net.Conn, http.ConnState)) { server := http.Server{ Addr: ":8080", Handler: h, TLSConfig: nil, ReadTimeout: 1 * time.Second, ReadHeaderTimeout: 1 * time.Second, WriteTimeout: 1 * time.Second, IdleTimeout: 1 * time.Second, MaxHeaderBytes: 222, TLSNextProto: nil, ConnState: connState, ErrorLog: log.Default(), BaseContext: nil, ConnContext: nil, } log.Print("Server listening on http://localhost", server.Addr) log.Fatal(server.ListenAndServe()) } ">
package main

import (
    "log"
    "net"
    "net/http"
    "time"

    "github.com/teal-finance/server"
    "github.com/teal-finance/server/chain"
    "github.com/teal-finance/server/cors"
    "github.com/teal-finance/server/export"
    "github.com/teal-finance/server/limiter"
    "github.com/teal-finance/server/opa"
    "github.com/teal-finance/server/reserr"
)

func main() {
    middlewares, connState := setMiddlewares()

    h := myHandler()
    h = middlewares.Then(h)

    runServer(h, connState)
}

func setMiddlewares() (middlewares chain.Chain, connState func(net.Conn, http.ConnState)) {
    // Uniformize error responses with API doc
    resErr := reserr.New("https://my-dns.com/doc")

    // Start a metrics server in background if export port > 0.
    // The metrics server is for use with Prometheus or another compatible monitoring tool.
    metrics := export.Metrics{}
    middlewares, connState = metrics.StartServer(9093, true)

    // Limit the input request rate per IP
    reqLimiter := limiter.New(10, 20, true, resErr)
    middlewares = middlewares.Append()

    // Endpoint authentication rules (Open Policy Agent)
    policy, err := opa.New(resErr, []string{"example-auth.rego"})
    if err != nil {
        log.Fatal(err)
    }

    // CORS
    allowedOrigins := []string{"http://my-dns.com"}

    middlewares = middlewares.Append(
        server.LogRequests,
        reqLimiter.Limit,
        server.Header("MyServerName-1.2.3"),
        policy.Auth,
        cors.HandleCORS(allowedOrigins),
    )

    return middlewares, connState
}

// runServer runs in foreground the main server.
func runServer(h http.Handler, connState func(net.Conn, http.ConnState)) {
    server := http.Server{
        Addr:              ":8080",
        Handler:           h,
        TLSConfig:         nil,
        ReadTimeout:       1 * time.Second,
        ReadHeaderTimeout: 1 * time.Second,
        WriteTimeout:      1 * time.Second,
        IdleTimeout:       1 * time.Second,
        MaxHeaderBytes:    222,
        TLSNextProto:      nil,
        ConnState:         connState,
        ErrorLog:          log.Default(),
        BaseContext:       nil,
        ConnContext:       nil,
    }

    log.Print("Server listening on http://localhost", server.Addr)

    log.Fatal(server.ListenAndServe())
}
Comments
  • Fix security issues reported by CodeQL from GitHub

    Fix security issues reported by CodeQL from GitHub

    Issues fixed by this PR:

    • Log injection
    • Path traversal attacks (was already fixed, but improve code quality)
    • Logging sensitive information in clear-text
    opened by 0uep 0
  • Bump github.com/open-policy-agent/opa from 0.36.1 to 0.37.0

    Bump github.com/open-policy-agent/opa from 0.36.1 to 0.37.0

    Bumps github.com/open-policy-agent/opa from 0.36.1 to 0.37.0.

    Release notes

    Sourced from github.com/open-policy-agent/opa's releases.

    v0.37.0

    This release contains a number of fixes and enhancements.

    This is the first release that includes a binary and a docker image for linux/arm64, opa_linux_arm64_static and openpolicyagent/opa:0.37.0-static. Thanks to @​ngraef for contributing the build changes necessary.

    Strict Mode

    There have been numerous possible checks in the compiler that fall into this category:

    1. They would help avoid common mistakes; but
    2. Introducing them would potentially break some uncommon, but legitimate use.

    We've thus far refrained from introducing them. Now, a new "strict mode" allows you to opt-in to these checks, and we encourage you to do so!

    With OPA 1.0, they will become the new default behaviour.

    For more details, see the docs on Compiler Strict Mode.

    Delta Bundles

    Delta bundles provide a more efficient way to make data changes by containing patches to data instead of snapshots. Using them together with HTTP Long Polling, you can propagate small changes to bundles without waiting for polling delays.

    See the documentation for more details.

    Tooling and Runtime

    • Bundles bug fix: Roundtrip manifest before hashing to allow changing the manifest and still using signature verification of bundles (#4233), reported by @​CristianJena

    • The test runner now also supports custom builtins, when invoked through the Golang interface (authored by @​MIA-Deltat1995)

    • The compile package and the opa build command support a new output format: "plan". It represents a query plan, steps needed to take to evaluate a query (with policies). The plan format is a JSON encoding of the intermediate representation (IR) used for compiling queries and policies into Wasm.

      When calling opa build -t plan ..., the plan can be found in plan.json at the top- level directory of the resulting bundle.tar.gz. See the documentation for details..

    ... (truncated)

    Changelog

    Sourced from github.com/open-policy-agent/opa's changelog.

    0.37.0

    This release contains a number of fixes and enhancements.

    This is the first release that includes a binary and a docker image for linux/arm64, opa_linux_arm64_static and openpolicyagent/opa:0.37.0-static. Thanks to @​ngraef for contributing the build changes necessary.

    Strict Mode

    There have been numerous possible checks in the compiler that fall into this category:

    1. They would help avoid common mistakes; but
    2. Introducing them would potentially break some uncommon, but legitimate use.

    We've thus far refrained from introducing them. Now, a new "strict mode" allows you to opt-in to these checks, and we encourage you to do so!

    With OPA 1.0, they will become the new default behaviour.

    For more details, see the docs on Compiler Strict Mode.

    Delta Bundles

    Delta bundles provide a more efficient way to make data changes by containing patches to data instead of snapshots. Using them together with HTTP Long Polling, you can propagate small changes to bundles without waiting for polling delays.

    See the documentation for more details.

    Tooling and Runtime

    • Bundles bug fix: Roundtrip manifest before hashing to allow changing the manifest and still using signature verification of bundles (#4233), reported by @​CristianJena

    • The test runner now also supports custom builtins, when invoked through the Golang interface (authored by @​MIA-Deltat1995)

    • The compile package and the opa build command support a new output format: "plan". It represents a query plan, steps needed to take to evaluate a query (with policies). The plan format is a JSON encoding of the intermediate representation (IR) used for compiling queries and policies into Wasm.

      When calling opa build -t plan ..., the plan can be found in plan.json at the top- level directory of the resulting bundle.tar.gz. See the documentation for details..

    ... (truncated)

    Commits
    • ad4f4f1 Prepare 0.37.0 release (#4309)
    • a18f53d compile: adds metadata field to .manifest (#4306)
    • 01ca4a6 build(deps): bump ansi-regex in /docs/website/scripts/live-blocks (#4308)
    • 42a559c build(deps): bump github.com/prometheus/client_golang (#4307)
    • 503a520 ast: Deprecating any() and all() built-in functions (#4271)
    • 59810d0 ast: Making input and data reserved keywords in strict-mode (#4301)
    • dd02a7f Add support for delta bundles
    • cb867a1 ast/compile: 'every' rewriting steps (#4231)
    • d12fb7c docs: Update generated CLI docs
    • 4449d96 docs: add explanation of the IR
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
  • Affero notice and source code

    Affero notice and source code

    When a Garcon-based web application is released under the GNU AGPL, the web application should offer its source to those users in some way. For example, the interface of the web application could display a “Source” link that leads users to an archive of the code. The GNU AGPL is flexible enough to choose a method that's suitable for the specific program. See section 13 of the AGPL for details.

    See https://github.com/zipizap/EmbeddedSource

    opened by 0uep 0
  • Monitor web traffic

    Monitor web traffic

    Privacy-friendly web analytics in Garcon

    Pirsch

    https://github.com/pirsch-analytics/pirsch

    On server-side, Pirsch concatenates IP, User-Agent, date, then salt, hash and store it using ClickHouse. No cookies. The stored data is analyzed to extract web traffic info: unique visitors, sessions, bounces, views, average session duration, languages, operating system and browser, referrers, countries…

    The metrics may be sent to Prom or Loki and Grafana…

    GoatCounter

    https://github.com/arp242/goatcounter

    Active community-driven project. :heart:

    ZerØ Analytics

    https://github.com/ncarlier/za

    Sends metrics to Loki to Grafana.

    Fathom Lite

    https://github.com/usefathom/fathom

    The project is unmaintained but there are forks with random fixes. The community needs to rally around a fork.

    https://github.com/usefathom/fathom/network

    Repo | Stars | Forks | Ahead | Behind | Last Push -- | -- | -- | -- | -- | -- https://github.com/JokerQyou/fathom | 2 | 0 | 41 | 0 | 2020-06-03 https://github.com/samuelmeuli/fathom | 2 | 1 | 22 | 0 | 2020-05-31 https://github.com/bjkippax/fathom-lite-gdprcompliant | 0 | 0 | 3 | 0 | 2022-02-03 https://github.com/status-im/fathom | 0 | 0 | 2 | 0 | 2021-07-21 https://github.com/mauriciodulce/fathom | 0 | 0 | 1 | 0 | 2020-05-23 https://github.com/flopp/fathom | 0 | 0 | 1 | 0 | 2020-08-28 https://github.com/javascript-indonesias/fathom | 0 | 0 | 16 | 0 | 2020-02-15 https://github.com/aarroyoc/fathom | 0 | 0 | 1 | 0 | 2020-11-22 https://github.com/emarsden/fathom | 0 | 0 | 1 | 0 | 2020-02-16 https://github.com/KerstinMaur/fathom | 0 | 0 | 1 | 0 | 2021-10-03 https://github.com/kasuboski/fathom | 0 | 0 | 6 | 0 | 2020-08-18 https://github.com/featherbear/fathom | 0 | 0 | 4 | 0 | 2021-06-12 https://github.com/MikeLindenau/fathom | 0 | 0 | 1 | 0 | 2020-05-06 https://github.com/dworznik/fathom | 0 | 0 | 2 | 0 | 2021-04-11

    https://github.com/teal-finance/garcon/issues/9

    Plausible Analytics (Elixir)

    Self-hosted with ClickHouse https://github.com/plausible/analytics

    Shynet (Python)

    https://github.com/milesmcc/shynet

    Offen

    https://github.com/offen/offen

    Matomo and OWA

    Matomo and OWA require JavaScript tracking on the browser side. I do not like it.

    Prometheus

    The Caddy doc provides some Prometheus queries. Garcon should use the same metrics naming as Caddy.

    The Prometheus client provides monitoring functions: pkg.go.dev

    An example: GitHub

    Garcon should use them!

    opened by 0uep 0
  • Document privacy compliance

    Document privacy compliance

    Privacy laws:

    • GDPR
    • Schrems II
    • PECR
    • ePrivacy (cookie law)
    • COPPA
    • CCPA

    Be inspired by:

    • https://usefathom.com/blog/huge-growth
    • https://usefathom.com/compliance/pecr-compliant-website-analytics (see also the links at the bottom)
    opened by 0uep 0
  • Set the URL path prefix (BASE_URL)

    Set the URL path prefix (BASE_URL)

    When the back-end is served by a reverse-proxy, the reverse-proxy may strip the URL path prefix. This path prefix may be used to set the Cookie path in lieu of "/" and for the default Response Error (doc URL).

    opened by 0uep 0
Owner
teal.finance
teal.finance
🪐 ✨ GO server for lorem.space website ✨ 🪐

Lorem.space server The backend of https://lorem.space lives here. Features Scan the directory & categorize the image resources (only JPEG for now) Pro

null 34 Aug 29, 2022
a simple http server as replacement of python -m http.server

ser a simple http server as replacement of python -m http.server

Changkun Ou 4 Feb 16, 2022
Static Content Web Server

Static Content Web Server The main purpose of the project is to develop static server that can be used with modern javascript frameworks (React, vue.j

Rinat 1 Dec 17, 2021
Gin Server Demo, with Features: Swagger UI, Prometheus, Shell scripts

Gin Server Gin Server Features Swagger UI Docker Prometheus Shell generator Production Cypher Service And we can get the authorization key SECURITY_CA

song 0 Nov 1, 2021
Fishserver is designed to quickly add HTTP handlers to HTTP servers. It supports registration of various HTTP

Fishserver is designed to quickly add HTTP handlers to HTTP servers. It supports registration of various HTTP. Handler interface types such as Gin Engine, Go's built-in HTTP. HandlerFunc, or http.ServeMux. The HTTP server can be configured quickly with options and can be used for test cases.

coolstina 0 Nov 1, 2021
OpenAPI specs for your Go server, generated at server runtime. No CLI, no code generation, and no HTTP

Overview "oas" is short for "OpenAPI Spec". Go package for generating OpenAPI docs at runtime. Non-features: No code generation. No CLI. No magic comm

Nelo Mitranim 0 Dec 3, 2021
A simple http-web server logging incoming requests to stdout with simple http-interface.

http-cli-echo-logger A simple http-web server logging incoming requests to stdout with simple http-interface. Run locally go run ./cmd/main.go Default

Andrii Bosonchenko 5 Jul 18, 2022
Formrecevr is a simple and lightweight from receiver backend primarily designed for (but not limited to) static websites.

Formrecevr Formrecevr (pronunced "Form receiver") is a simple and lightweight from receiver backend primarily designed for (but not limited to) static

Dorian Zedler 2 Apr 17, 2022
Fileserver: A simple static fileserver

Fileserver A simple static fileserver. Serves requests to local files in a directory of your choosing so that you can preview files in your browser. B

Liam Egan 1 May 26, 2022
Turn Nginx logs into Prometheus metrics

Nginx log parser and Prometheus exporter This service parses incoming syslog messages from Nginx sent over UDP and converts them into Prometheus metri

Igor Novgorodov 32 Aug 10, 2022
✨ A lightweight HTTP server based on GO, will try to detect your OS and architecture and return as SHELL script. ✨

✨ A lightweight HTTP server based on GO, will try to detect your OS and architecture and return as SHELL script. ✨

Salvydas Lukosius 6 May 5, 2022
Simple HTTP server written in golang

Simple HTTP server written in golang Simple webserver in golang, to demonstrate basic functionalities like e.g. sending back some request header info,

null 1 Aug 31, 2022
Go-simplehttp - Simple HTTP server written in golang

Simple HTTP server written in golang Simple webserver in golang, to demonstrate

null 0 Jan 1, 2022
:tophat: Small self-contained pure-Go web server with Lua, Markdown, HTTP/2, QUIC, Redis and PostgreSQL support

Web server with built-in support for QUIC, HTTP/2, Lua, Markdown, Pongo2, HyperApp, Amber, Sass(SCSS), GCSS, JSX, BoltDB (built-in, stores the databas

Alexander F. Rødseth 2.1k Sep 23, 2022
A feature flag solution, with only a YAML file in the backend (S3, GitHub, HTTP, local file ...), no server to install, just add a file in a central system and refer to it. 🎛️

??️ go-feature-flag A feature flag solution, with YAML file in the backend (S3, GitHub, HTTP, local file ...). No server to install, just add a file i

Thomas Poignant 515 Sep 24, 2022
HTTP server receives events and publishes them to STAN

Publishes events to Nats Streaming(STAN) synchornously and asynchronously. Cache events's publish-state using Redis and Store events using MongoDB.

Mehdi 16 Sep 9, 2022
Go web server - A web server that can accept a GET request and serve a response.

go_web_server A web server that can accept a GET request and serve a response. Go is a great language for creating simple yet efficient web servers an

Gilbert Ngeno 0 Jan 3, 2022
A simple HTTP Server to share files over WiFi via Qr Code

go-fileserver A simple HTTP server to share files over WiFi via QRCode Installation You can download compressed version from

Pradeep Khileri 68 Sep 12, 2022