An Advanced HTTP Reverse Proxy with Dynamic Sharding Strategies

Overview

Weaver - A modern HTTP Proxy with Advanced features

Build Status Go Report Card Coverage Status GitHub Release

Description

Weaver is a Layer-7 Load Balancer with Dynamic Sharding Strategies. It is a modern HTTP reverse proxy with advanced features.

Features:

  • Sharding request based on headers/path/body fields
  • Emits Metrics on requests per route per backend
  • Dynamic configuring of different routes (No restarts!)
  • Is Fast
  • Supports multiple algorithms for sharding requests (consistent hashing, modulo, s2 etc)
  • Packaged as a single self contained binary
  • Logs on failures (Observability)

Installation

Build from source

  • Clone the repo:
git clone [email protected]:gojektech/weaver.git
  • Build to create weaver binary
make build

Binaries for various architectures

Download the binary for a release from: here

Architecture

Weaver uses etcd as a control plane to match the incoming requests against a particular route config and shard the traffic to different backends based on some sharding strategy.

Weaver can be configured for different routes matching different paths with various sharding strategies through a simple route config named ACL.

The various sharding strategies supported by weaver are:

  • Consistent hashing (hashring)
  • Simple lookup based
  • Modulo
  • Prefix lookup
  • S2 based

Deploying to Kubernetes

Currently we support deploying to kubernetes officially. You can check the doc here

Examples

We have examples defined to deploy it to kubernetes and using acls. Please checkout out examples

Configuration

Defining ACL's

Details on configuring weaver can be found here

Please note

As the famous saying goes, All Load balancers are proxies, but not every proxy is a load balancer, weaver currently does not support load balancing.

Contributing

If you'd like to contribute to the project, refer to the contributing documentation

License

Copyright 2018, GO-JEK Tech (http://gojek.tech)

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Issues
  • go sum update

    go sum update

    Build failing due to. Updated by regenerating go.sum

    go: verifying github.com/coreos/[email protected]+incompatible: checksum mismatch
            downloaded: h1:v3H7yHgF+94suF7Xg6V7Haq6Anac3X6WosuKGTTJCGM=
            go.sum:     h1:WyMdmwfw02n2i3gTkzy+HBR2dsh1axjxn808iAe9PLI=
    
    opened by irfn 3
  • Sometimes tests are stuck for longer time.

    Sometimes tests are stuck for longer time.

    Sometimes when running tests, they are stuck for longer time, couldn't get them to complete.

    ╭─TheInsider ~/Documents/repos/gojektech/weaver ‹ruby-2.5.0› ‹weaver-cli*›
    ╰─➤  mk test
    cp weaver.conf.yaml.sample weaver.conf.yaml
    GO111MODULE=on go test ./...
    ok  	github.com/gojektech/weaver	(cached)
    ?   	github.com/gojektech/weaver/cmd/weaver	[no test files]
    ok  	github.com/gojektech/weaver/config	0.021s
    ^C{"level":"info","msg":"bootstrapping router using etcd on /weaver/acls/","time":"2019-03-19T07:53:25+05:30"}
    {"level":"info","msg":"creating router namespace on etcd using /weaver/acls/","time":"2019-03-19T07:53:25+05:30"}
    FAIL	github.com/gojektech/weaver/etcd	14.251s
    make: *** [Makefile:40: test] Error 1
    

    Here is the message, that I got when trying to reproduce it.

    opened by gwthm-in 2
  • weaver as go library

    weaver as go library

    Hello, thanks for sharing this amazing project. Any option to release it as go library so we can integrate it a a more large application? I'm speaking something as https://github.com/devopsfaith/krakend

    please let me know Thanks

    opened by bsctl 1
  • Helm charts to deploy to kubenetes

    Helm charts to deploy to kubenetes

    Pull request to close #1. It is still draft pull request.

    Currently: ETCD is optional, can deploy or ignore. configurations are mentioned in the chart itself via env variable. Currently these are not configured in chart, but hardcoded.

    TODO: - Have to extract env variables - Have to configure statsd, newrelic (work required on weaver) - All env keys should not be mandatory (work required on weaver) - Sentry should be optional (work required on weaver) - Helm chart to deploy statsd (in progress)

    opened by gwthm-in 1
  • Full Documentation

    Full Documentation

    Required Topics include:

    • [x] Usage Guidelines
    • [x] Development Guidelines
    • [x] Contribution Guidelines
    • [x] Weaver Design Overview
    • [x] ACL Basics, Formats and Management
    • [x] Routing and Sharding (Working & available strategies)
    help wanted documentation 
    opened by spy16 1
  • Correct the spelling of Kubernets into Kubernetes

    Correct the spelling of Kubernets into Kubernetes

    I noticed that many parts of the markdown files contain misspelled "Kubernets". This pull request fixes all those occurrences in the markdown files to "Kubernetes".

    opened by mfaarabi 0
  • #6 Can run tests in docker

    #6 Can run tests in docker

    Fix #6 Next, remove env variables from docker-compose and use hostname directly to simplify. This required little more clean up.

    @rShetty if it looks good, please merge.

    opened by gwthm-in 0
  • update Dockerfile, remove mkdir command, WORKDIR  will create if it doen't exist

    update Dockerfile, remove mkdir command, WORKDIR will create if it doen't exist

    as documented here:

    If the WORKDIR doesn’t exist, it will be created even if it’s not used in any subsequent Dockerfile instruction.

    so there is no need RUN mkdir /weaver

    opened by brownchow 0
  • deployment error

    deployment error

    When I am trying to deploy the weaver I am getting the error :

    version for aks I am using 1.19.0 helm version : version.BuildInfo{Version:"v3.3.4", GitCommit:"a61ce5633af99708171414353ed49547cf05013d", GitTreeState:"clean", GoVersion:"go1.14.9"}

    When I run the command I get the following error , Please help me.

    [email protected]:~/weaver$ helm upgrade --debug --install proxy-cluster ./deployment/weaver -f ./deployment/weaver/values-env.yaml history.go:53: [debug] getting history for release proxy-cluster upgrade.go:121: [debug] preparing upgrade for proxy-cluster Error: UPGRADE FAILED: error validating "": error validating data: [ValidationError(Deployment): unknown field "name" in io.k8s.api.apps.v1.Deployment, ValidationError(Deployment.spec): missing required field "selector" in io.k8s.api.apps.v1.DeploymentSpec] helm.go:94: [debug] error validating "": error validating data: [ValidationError(Deployment): unknown field "name" in io.k8s.api.apps.v1.Deployment, ValidationError(Deployment.spec): missing required field "selector" in io.k8s.api.apps.v1.DeploymentSpec] helm.sh/helm/v3/pkg/kube.scrubValidationError /home/circleci/helm.sh/helm/pkg/kube/client.go:566 helm.sh/helm/v3/pkg/kube.(*Client).Build /home/circleci/helm.sh/helm/pkg/kube/client.go:159 helm.sh/helm/v3/pkg/action.validateManifest /home/circleci/helm.sh/helm/pkg/action/upgrade.go:451 helm.sh/helm/v3/pkg/action.(*Upgrade).prepareUpgrade /home/circleci/helm.sh/helm/pkg/action/upgrade.go:233 helm.sh/helm/v3/pkg/action.(*Upgrade).Run /home/circleci/helm.sh/helm/pkg/action/upgrade.go:122 main.newUpgradeCmd.func2 /home/circleci/helm.sh/helm/cmd/helm/upgrade.go:154 github.com/spf13/cobra.(*Command).execute /go/pkg/mod/github.com/spf13/[email protected]/command.go:842 github.com/spf13/cobra.(*Command).ExecuteC /go/pkg/mod/github.com/spf13/[email protected]/command.go:950 github.com/spf13/cobra.(*Command).Execute /go/pkg/mod/github.com/spf13/[email protected]/command.go:887 main.main /home/circleci/helm.sh/helm/cmd/helm/helm.go:93 runtime.main /usr/local/go/src/runtime/proc.go:203 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1373 UPGRADE FAILED main.newUpgradeCmd.func2 /home/circleci/helm.sh/helm/cmd/helm/upgrade.go:156 github.com/spf13/cobra.(*Command).execute /go/pkg/mod/github.com/spf13/[email protected]/command.go:842 github.com/spf13/cobra.(*Command).ExecuteC

    opened by surendra-devops 0
  • Adding ACL List, Show Cli

    Adding ACL List, Show Cli

    Weaver CLI is ready for review. Commands will register them selves with CLI. In main we need to import the commands that we want them to be with the binary.

    Let's say we want to separate out acls commands, we can create weaverctl in cmd/weaverctl and import only acl commands. Binary produced from weaverctl will have only acls commands and will not have acl commands.

    View layers currently renders output to json, this can manipulated easily using a flag and rending to outputs like yaml, table and etc. This will be a enhancement. For now will stick to json output.

    opened by gwthm-in 1
  • Command Line tool to manage ACLs

    Command Line tool to manage ACLs

    Description: Build an admin tool to add ACL's to the control plane (etcd)

    This tool will be a command line utility inside Weaver (weaverctl) which helps management (addition/deletion/updation) of all the ACL's

    enhancement help wanted 
    opened by rShetty 0
Releases(v0.0.1)
Owner
Gojek
SuperApp from Southeast Asia
Gojek
mt-multiserver-proxy is a reverse proxy designed for linking multiple Minetest servers together

mt-multiserver-proxy mt-multiserver-proxy is a reverse proxy designed for linking multiple Minetest servers together. It is the successor to multiserv

null 11 Jul 31, 2022
Tcp-proxy - A dead simple reverse proxy server.

tcp-proxy A proxy that forwords from a host to another. Building go build -ldflags="-X 'main.Version=$(git describe --tags $(git rev-list --tags --max

Injamul Mohammad Mollah 0 Jan 2, 2022
The Dual-Stack Dynamic DNS client, the world's first dynamic DNS client built for IPv6.

dsddns DsDDNS is the Dual-Stack Dynamic DNS client. A dynamic DNS client keeps your DNS records in sync with the IP addresses associated with your hom

Ryan Young 12 Aug 10, 2022
Open Source HTTP Reverse Proxy Cache and Time Series Dashboard Accelerator

Trickster is an HTTP reverse proxy/cache for http applications and a dashboard query accelerator for time series databases. Learn more below, and chec

null 1.7k Aug 7, 2022
A standalone Web Server developed with the standard http library, suport reverse proxy & flexible configuration

paddy 简介 paddy是一款单进程的独立运行的web server,基于golang的标准库net/http实现。 paddy提供以下功能: 直接配置http响应 目录文件服务器 proxy_pass代理 http反向代理 支持请求和响应插件 部署 编译 $ go build ./main/p

fangyousong 5 May 2, 2022
Goproxy - HTTP/HTTPS Forward and Reverse Proxy

Go HTTP(s) Forward/Reverse Proxy This is intended to provide the proxy for the goproxy frontend. It is currently a work in progress, and is not very s

David Christenson 0 Jan 4, 2022
Websockify-go - A reverse proxy that support tcp, http, https, and the most important, noVNC, which makes it a websockify

websockify-go | mproxy a reverse proxy that support tcp, http, https, and the mo

null 2 Mar 19, 2022
Http-logging-proxy - A HTTP Logging Proxy For Golang

http-logging-proxy HTTP Logging Proxy Description This project builds a simple r

null 3 Aug 1, 2022
A load balancer supporting multiple LB strategies written in Go

farely A load balancer supporting multiple LB strategies written in Go. Goal The goal of this project is purley educational, I started it as a brainst

Mehdi Cheracher 11 Aug 7, 2022
Go HTTP tunnel is a reverse tunnel based on HTTP/2.

Go HTTP tunnel is a reverse tunnel based on HTTP/2. It enables you to share your localhost when you don't have a public IP.

Michal Jan Matczuk 2.9k Aug 12, 2022
Reverse cwmp proxy

cwmp-proxy Integration of the proxy will provide you the ability to place CPEs and ACS servers in different networks. What about if the devices are pl

Ivan Stefanov 14 Feb 21, 2020
A fast reverse proxy to help you expose a local server behind a NAT or firewall to the internet.

frp README | 中文文档 What is frp? frp is a fast reverse proxy to help you expose a local server behind a NAT or firewall to the Internet. As of now, it s

null 58.8k Aug 9, 2022
Dead simple reverse proxy for all your containerized needss

Whats this ? Pawxi is yet another reverse proxy designed with simplicity in mind. Born out of a certain users frustration at the complexity of setting

null 14 May 26, 2022
Simple edge server / reverse proxy

reproxy Reproxy is simple edge HTTP(s) sever / reverse proxy supporting various providers (docker, static, file). One or more providers supply informa

Umputun 1k Aug 11, 2022
gobetween - modern & minimalistic load balancer and reverse-proxy for the ☁️ Cloud era.

gobetween - modern & minimalistic load balancer and reverse-proxy for the ☁️ Cloud era. Current status: Maintenance mode, accepting PRs. Currently in

Yaroslav Pogrebnyak 1.7k Aug 10, 2022
Reverse Proxy for geth node

geth-proxy Reverse Proxy for geth node gcr.io/moonrhythm-containers/geth-proxy Features Health check base on last synced block timestamp Merge websock

Moon Rhythm 9 Jul 26, 2022
4chain is a simple、fast reverse proxy to help you expose a local server behind a NAT or firewall to the Internet.

4Chain What is 4chain? 4chain is a simple、fast reverse proxy to help you expose a local server behind a NAT or firewall to the Internet. Using the ssh

null 6 Jan 4, 2022
TProx is a fast reverse proxy path traversal detector and directory bruteforcer.

TProx is a fast reverse proxy path traversal detector and directory bruteforcer Install • Usage • Examples • Join Discord Install Options From Source

Krypt0mux 25 Aug 8, 2022
Example of how to write reverse proxy in Go that runs on Cloud Run with Tailscale

Cloudrun Tailscale Reverse Proxy Setup Create a ephemeral key in Tailscale Set TAILSCALE_AUTHKEY in your Cloud Run environment variables Set TARGET_UR

ThreeComma.io 8 Jul 17, 2022