:zap: Go web framework benchmark

Overview

go-web-framework-benchmark

This benchmark suite aims to compare the performance of Go web frameworks. It is inspired by Go HTTP Router Benchmark but this benchmark suite is different with that. Go HTTP Router Benchmark suit aims to compare the performance of routers but this Benchmark suit aims to compare whole HTTP request processing.

Last Test Updated: 2020-05

test environment

  • CPU: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz (2 cpus, 10 physical cores, 40 logical cores)
  • Memory: 32G
  • Go: go1.12.7 linux/amd64
  • OS: CentOS Linux release 7.3.1611 (Core)

Tested web frameworks (in alphabetical order)

Only test those webframeworks which are stable

Motivation

When I investigated performance of Go web frameworks, I found Go HTTP Router Benchmark, created by Julien Schmidt. He also developed a high performance http router: httprouter. I had thought I got the performance result until I created a piece of codes to mock the real business logics:

api.Get("/rest/hello", func(c *XXXXX.Context) {
	sleepTime := strconv.Atoi(os.Args[1]) //10ms
	if sleepTime > 0 {
		time.Sleep(time.Duration(sleepTime) * time.Millisecond)
	}

	c.Text("Hello world")
})

When I use the above codes to test those web frameworks, the token time of route selection is not so important in the whole http request processing, although performance of route selection of web frameworks are very different.

So I create this project to compare performance of web frameworks including connection, route selection, handler processing. It mocks business logics and can set a special processing time.

The you can get some interesting results if you use it to test.

Implementation

When you test a web framework, this test suit will starts a simple http server implemented by this web framework. It is a real http server and only contains GET url: "/hello".

When this server processes this url, it will sleep n milliseconds in this handler. It mocks the business logics such as:

  • read data from sockets
  • write data to disk
  • access databases
  • access cache servers
  • invoke other microservices
  • ……

It contains a test.sh that can do those tests automatically.

It uses wrk to test.

Basic Test

The first test case is to mock 0 ms, 10 ms, 100 ms, 500 ms processing time in handlers.

Benchmark (Round 3) the concurrency clients are 5000.

Latency (Round 3) Latency is the time of real processing time by web servers. The smaller is the better.

Allocs (Round 3) Allocs is the heap allocations by web servers when test is running. The unit is MB. The smaller is the better.

If we enable http pipelining, test result as below:

benchmark pipelining (Round 2)

Concurrency Test

In 30 ms processing time, the test result for 100, 1000, 5000 clients is:

concurrency (Round 3)

Latency (Round 3)

Latency (Round 3)

If we enable http pipelining, test result as below:

concurrency pipelining(Round 2)

cpu-bound case Test

cpu-bound (5000 concurrency)

Usage

You should install this package first if you want to run this test.

go get github.com/smallnest/go-web-framework-benchmark

It takes a while to install a large number of dependencies that need to be downloaded. Once that command completes, you can run:

cd $GOPATH/src/github.com/smallnest/go-web-framework-benchmark
go build -o  gowebbenchmark *.go
./test.sh

It will generate test results in processtime.csv and concurrency.csv. You can modify test.sh to execute your customized test cases.

  • If you also want to generate latency data and allocation data, you can run the script:
./test-latency.sh
  • If you don't want use keepalive, you can run:
./test-latency-nonkeepalive.sh
  • If you want to test http pipelining, you can run:
./test-pipelining.sh
  • If you want to test some of web frameworks, you can modify the test script and only keep your selected web frameworks:
……
web_frameworks=( "default" "ace" "beego" "bone" "denco" "echov1" "echov2standard" "echov2fasthttp" "fasthttp-raw" "fasthttprouter" "fasthttp-routing" "gin" "gocraftWeb" "goji" "gojiv2" "gojsonrest" "gorestful" "gorilla" "httprouter" "httptreemux" "lars" "lion" "macaron" "martini" "pat" "r2router" "tango" "tiger" "traffic" "violetear" "vulcan")
……
  • If you want to test all cases, you can run:
./test-all.sh

Plot

you can run the shell script plot.sh in testresults directory and it can generate all images in its parent directory.

Add new web framework

Welcome to add new Go web frameworks. You can follow the below steps and send me a pull request.

  1. add your web framework link in README
  2. add a hello implementation in server.go
  3. add your webframework in libs.sh

Please add your web framework alphabetically.

Issues
  • What happens ?

    What happens ?

    Before the last iris was first with fasthttp-raw, before 18 days, as this commit also says: https://github.com/smallnest/go-web-framework-benchmark/commit/0874ce64aa6d78235c9452bfed0b8fb6fb4f4547.

    Also I didn't change any performance-releated with SetBody & router, I just tested with a version we had before a month and the benchmark results are the same as the latest iris version.

    Can you tell me when you upgraded iris? what version did you had before (or a date) you did several tests within this period, this is the first time iris is down.

    And if you didn't changed anything(iris-releated) from the previous week, could you please re-run the tests?

    Edit2:

    I just ran the benchmarks from docker with docker pull smallnest/go-web-framework-benchmark docker run -v /opt/data:/data smallnest/go-web-framework-benchmark .

    Iris is still side-by-side with fasthttp. Some other frameworks I saw on this repo's README that seems to be top of iris, in real, they have lowest results...

    For now, only the basic tests (0ms 5000 clients) are finished:

    fasthttp and others

    iris

    I ran it from a 'clean/fresh' docker*

    but on README's Basic tests are:

    README page

    How is that even possible?

    Edit 3: 10ms 5k clients finished also:

    fasthttp and others 10ms

    iris 10ms

    I will leave the machine open until the graphical results also... I really need to sleep now

    But you got the idea.. something happened on your last machine's benchmark testing

    Thanks for everything!!

    opened by ghost 10
  • Add pipeline to tests

    Add pipeline to tests

    init = function(args)
       request_uri = args[1]
       depth = tonumber(args[2]) or 1
    
       local r = {}
       for i=1,depth do
         r[i] = wrk.format(nil, request_uri)
       end
       req = table.concat(r)
    end
    
    request = function()
       return req
    end
    
    
    $ wrk .... -s pipeline.lua 
    

    Taken from here

    enhancement 
    opened by ghost 10
  • docker image not creating images with labels

    docker image not creating images with labels

    Hi, first at all, many thanks for sharing this project, has helped me to improve and detect some issues within my code and therefore learn more about how to profile/test in general.

    Currently, I am giving a try to the docker image but is not creating the images by just running this:

    docker run  -v /opt/data:/data smallnest/go-web-framework-benchmark
    

    What I am doing to try to generate them, is to execute the plot.sh something like:

    docker run -v /tmp/data:/data -t -i --entrypoint /bin/bash smallnest/go-web-framework-benchmark
    

    then within the container, I run:

    docker-test.sh
    

    When finished I run plot.sh within the testresults this creates the images but without labels, an example of the output:

    benchmark_alloc

    Any idea of how to fix this? and if is not too much asking in the meantime could the benchmark results be updated?

    thanks in advance.

    opened by nbari 8
  • http: Accept error: accept tcp [::]:8080: accept4: too many open files

    http: Accept error: accept tcp [::]:8080: accept4: too many open files

    Hello,

    I made a fresh setup today and started to get these errors. Any idea ?

    Host machine is Ubuntu 20.04 with go 1.15.

    2020/08/26 14:31:24 - atreugo - Temporary error when accepting new connections: accept tcp4 0.0.0.0:8080: accept4: too many open files 2020/08/26 14:31:25 - atreugo - Temporary error when accepting new connections: accept tcp4 0.0.0.0:8080: accept4: too many open files 2020/08/26 14:31:26 - atreugo - Temporary error when accepting new connections: accept tcp4 0.0.0.0:8080: accept4: too many open files throughput: 118162.25 requests/second ./test.sh: line 18: 3216 Killed ./$server_bin_name $2 $3 finsihed testing atreugo

    testing web framework: beego • Initialization package=gramework version=1.7.0-rc3 • node info cputicks=2.53G package=gramework ram=911.18M used / 7.51G total swap=0.00 used / 2.00G total • load average is good fifteen=0.600 five=1.670 one=4.920 package=gramework • node uptime package=gramework uptime= 0:01 2020/08/26 14:31:30 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 5ms 2020/08/26 14:31:30 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 10ms 2020/08/26 14:31:30 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 20ms 2020/08/26 14:31:30 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 40ms 2020/08/26 14:31:30 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 80ms 2020/08/26 14:31:30 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 160ms 2020/08/26 14:31:30 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 320ms 2020/08/26 14:31:30 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 640ms 2020/08/26 14:31:31 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 1s 2020/08/26 14:31:32 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 1s

    Best Regards,

    opened by Karif 4
  • Processing time and Latency

    Processing time and Latency

    According to the Benchmark of different processing time (Latency), In all frameworks except fasthttp, for 0ms processing time, we have more latencies compared to 10ms! It's not clear to me. Maybe because of sleeps in the processing code. What do you think?

    Thanks for your useful benchmarks.

    opened by mehdishojaei 4
  • fixed gear’s bench

    fixed gear’s bench

    非常感谢你的评测,我发现 Gear 的测试逻辑不对,所以导致很多 bench 图表一枝独秀。

    Gear 的 Router 也是一个的 Gear 中间件, 不会默认挂载,这里修复了一下。

    目前的 Gear 版本 handle 了 context canceled error, 回头修复。

    另外,按照目前的测试逻辑——简单的回写一个 hello,基于原生 http server 的框架区别不会很大,无非就是多了一个初始化逻辑。我本地对比跑了一下 gin 和 gear,gear 稍低一点。因为 Gear 内置的初始化逻辑确实多了一些,包括检测 Compress 组件、recover 异常、初始化 cookie 模块、初始化 context 等

    opened by zensh 3
  • 本地测试结果与公布数据差距很大,

    本地测试结果与公布数据差距很大,

    测试环境 go1.7 最新 beego ->default路径无返回值 硬件: 40 Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz Mem: 32765096 测试结果: [[email protected] wrk]# ./wrk -t10 -c5000 -d60s http://127.0.0.1:8080/ Running 1m test @ http://127.0.0.1:8080/ 10 threads and 5000 connections Thread Stats Avg Stdev Max +/- Stdev Latency 402.30ms 565.62ms 2.00s 81.05% Req/Sec 0.86k 639.03 13.29k 78.62% 476512 requests in 1.00m, 70.44MB read Socket errors: connect 0, read 0, write 0, timeout 40477 Requests/sec: 7932.59 Transfer/sec: 1.17MB

    Requests/sec 远远低于您第一项测试中0ms 100ms 等测试项的公布值, 不知道问题出在哪里.

    opened by ghost 3
  • missing dependencies

    missing dependencies

    Forked this repository and tried running. Following is the output

    ../go-web-framework-benchmark$ go build -o gowebfmbench
    # github.com/cloudfoundry/gosigar
    ../../../../pkg/mod/github.com/cloudfoundry/[email protected]/concrete_sigar.go:20:11: cpuUsage.Get undefined (type Cpu has no field or method Get)
    ../../../../pkg/mod/github.com/cloudfoundry/[email protected]/concrete_sigar.go:30:13: cpuUsage.Get undefined (type Cpu has no field or method Get)
    ../../../../pkg/mod/github.com/cloudfoundry/[email protected]/concrete_sigar.go:49:10: l.Get undefined (type LoadAverage has no field or method Get)
    ../../../../pkg/mod/github.com/cloudfoundry/[email protected]/concrete_sigar.go:55:10: m.Get undefined (type Mem has no field or method Get)
    ../../../../pkg/mod/github.com/cloudfoundry/[email protected]/concrete_sigar.go:61:10: s.Get undefined (type Swap has no field or method Get)
    ../../../../pkg/mod/github.com/cloudfoundry/[email protected]/sigar_shared.go:12:20: procTime.Get undefined (type *ProcTime has no field or method Get)
    

    Go env

    $ go env
    GO111MODULE=""
    GOARCH="amd64"
    GOBIN=""
    GOCACHE="/Users/kamaleshwar/Library/Caches/go-build"
    GOENV="/Users/kamaleshwar/Library/Application Support/go/env"
    GOEXE=""
    GOFLAGS=""
    GOHOSTARCH="amd64"
    GOHOSTOS="darwin"
    GOINSECURE=""
    GOOS="darwin"
    GOPATH="/Users/kamaleshwar/go"
    GOPROXY="https://proxy.golang.org,direct"
    GOROOT="/usr/local/go"
    GOSUMDB="sum.golang.org"
    GOTMPDIR=""
    GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
    GCCGO="gccgo"
    AR="ar"
    CC="clang"
    CXX="clang++"
    CGO_ENABLED="0"
    GOMOD=""
    CGO_CFLAGS="-g -O2"
    CGO_CPPFLAGS=""
    CGO_CXXFLAGS="-g -O2"
    CGO_FFLAGS="-g -O2"
    CGO_LDFLAGS="-g -O2"
    PKG_CONFIG="pkg-config"
    GOGCCFLAGS="-fPIC -m64 -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/lv/0wsxrb6123s8slgb2lp_6_3m0000gp/T/go-build461564637=/tmp/go-build -gno-record-gcc-switches -fno-common"
    
    opened by bnkamalesh 2
  • Remove iris

    Remove iris

    Given how the awesome-go PR went (it ended up with a ban of the iris from the list from just a removal) I considered creating one for this place as well. I'm asking to have iris removed based on the widespread evidence that this is a project toxic for the Go community and the fact that it is using this benchmark suite only to promote itself.

    I've also took the liberty to bump up the docker container to 1.7.1 which is the latest version of Go (1.7.2 might be released soon this week) as a small token of saying thank you for this project.

    Thank you for your consideration.

    opened by dlsniper 2
Owner
smallnest
Author of 《Scala Collections Cookbook》
smallnest
Go benchmark harness.

autobench autobench is a framework to compare the performance of Go 1.2 and Go 1.3. usage autobench downloads and builds the latest Go 1.1 and Go tip

Dave Cheney 92 Nov 12, 2021
Key/Value database benchmark

USAGE: kvbench OPTIONS OVERVIEW kvbench is a simple benchmarking tool to evaluate the read performance of a key/value store while writes are being app

James A. Robinson 24 Nov 15, 2021
A rapid http(s) benchmark tool written in Go

gonetx/httpit httpit is a rapid http(s) benchmark tool which on top of fasthttp. Also thanks to cobra and bubbletea. Installation Get binaries from re

null 139 May 20, 2022
Cache benchmark for Golang

Cache comparison benchmark for Go This benchmark compares cache algorithms using scrambled zipfian distribution (a few occur very often while many oth

Vladimir Mihailenco 39 May 5, 2022
Go http server benchmark

go-http-server-benchmark The more connections, nbio cost the less memory, and performance the better than other frameworks. We can serve for 1000k or

lesismal 1 Oct 27, 2021
Golang json encoders\decoders benchmark

Json encoder and decoder benchmark Test Encode for: Build-in encoder https://github.com/mailru/easyjson https://github.com/json-iterator/go Test Decod

Vladislav S 0 Dec 3, 2021
Go Script Lang Benchmark

Go Script Lang Benchmark git clone github.com/akkuman/go_script_lang_benchmark c

null 2 Feb 10, 2022
:zap: Go web framework benchmark

go-web-framework-benchmark This benchmark suite aims to compare the performance of Go web frameworks. It is inspired by Go HTTP Router Benchmark but t

smallnest 1.7k May 18, 2022
Redis-benchmark - Simple get, mget and pipelined get benchmark.

redis-benchmark Simple get, mget and pipelined get benchmark. Usage git clone https://github.com/Ali-A-A/redis-benchmark.git cd ./redis-benchmark dock

Ali Abbasi Alaei 0 Dec 31, 2021
Bxd redis benchmark - Redis benchmark tool for golang

使用 redis benchmark 工具, 测试 10 20 50 100 200 1k 5k 字节 value 大小,redis get set 性能。 r

bingxindan 0 Jan 22, 2022
Go-driver-benchmark - Driver benchmark with golang

We use ClickkHouse for time-series databases, and the driver's performance is ve

null 3 Jan 27, 2022
Benchmark - Benchmark of logr implementations

Benchmark of logr implementations Implementations a function (can bridge to non-

null 2 Mar 23, 2022
Go HTTP request router and web framework benchmark

Go HTTP Router Benchmark This benchmark suite aims to compare the performance of HTTP request routers for Go by implementing the routing structure of

Julien Schmidt 1.6k May 17, 2022
Go HTTP request router and web framework benchmark

Go HTTP Router Benchmark This benchmark suite aims to compare the performance of HTTP request routers for Go by implementing the routing structure of

Peter Kieltyka 23 Apr 21, 2022
:zap: boilerplate template manager that generates files or directories from template repositories

Boilr Are you doing the same steps over and over again every time you start a new programming project? Boilr is here to help you create projects from

Tamer Tas 1.5k May 19, 2022
:zap: Transfer files over wifi from your computer to your mobile device by scanning a QR code without leaving the terminal.

$ qrcp Transfer files over Wi-Fi from your computer to a mobile device by scanning a QR code without leaving the terminal. You can support development

Claudio d'Angelis 8.6k May 16, 2022
Analyzer: zapvet is static analysis tool for zap

zapvet zapvet is static analysis tool for zap. fieldtype: fieldtype finds confliction type of field Install You can get zapvet by go install command (

GoStaticAnalysis 4 Aug 12, 2021
微服务架构-micro-基于go-zero zrpc etcd 单独集成orm-gorm 日志-uber/zap

micro目录结构 pkg ├── api 业务接口逻辑层 所有的业务逻辑存放目录。 │ ├── model 数据模型 数据管理层,仅用于操作管理数据,如数据库操作 ├── assets

jiwei 8 Apr 26, 2022
A Handy FieldBuilder Template for uber-go/zap

zap-fieldbuilder-template A Handy Generic Logging FieldBuilder Template around uber-go/zap. You can check this tiny blog for further information ?? Wh

Yavuz Selim Yesilyurt 7 Dec 16, 2021
Package zaperations provides a Google Cloud operations suite (formerly Stackdriver) compatible config for the uber-go/zap logger.

Package zaperations provides a Google Cloud Operations (formerly Stackdriver) compatible config for the excellent uber-go/zap logger. Example This exa

Wayne 0 Nov 6, 2021
Wrap zap for easy using.

go-project-pkg/log Wrap zap for easy using. Installation $ go get -u github.com/go-project-pkg/log Usage Use default logger: import "github.com/go-pro

Go Project PKG 1 Nov 8, 2021
Hook for sending events zap logger to telegram.

zaptelegram Hook for sending events to telegram for zap logger. Install: go get -u github.com/strpc/zaptelegram Basic usage: package main import ( "

null 4 Mar 29, 2022
logger wraps uber/zap and trace with opentelemetry

logger 特性 支持 uber/zap 日志 支持 log rolling,使用 lumberjace 支持日志追踪 支持debug、info、warn、e

xiaolei 3 Mar 6, 2022
Go starter project with Gin, Viper, postgres , redis, zap, prometheus metrics etc setup

Go REST Service Starter/Boilerplate Easily extendible REST API Service boilerplate aiming to follow idiomatic go and best practice. Any feedback and p

Kshitij Singh 0 Dec 27, 2021
Alternative logging through zap.

wzap Alternative logging through zap Usage Start using it Download and install it: go get github.com/wyy-go/wzap Import it in your code: import "githu

null 0 Jan 8, 2022
Rzap - Log rotate for uber-zap

rzap Log rotate for uber-zap How to install go get github.com/winking324/rzap Ho

ShanHui 1 Jan 26, 2022
Logger - A thin wrapper of uber-go/zap logger for personal project

a thin wraper of uber-go/zap logger for personal project 0. thanks uber-go/zap B

tsingson 0 May 2, 2022
Log to cloud object storage for golang. Zap extension.

cos-logger Log to cloud object storage for golang implemented as io.Writer. Use it as a plugin/extension to uber-go/zap logger Configure logger and ad

Prashant Sharma 0 Jan 19, 2022
A logrus.Hook that logs with a zap.Logger

zaprus Ever had a 3rd-party dependency requiring a logrus, but you're using zap? zaprus provides a logrus.Hook that makes a logrus.(Entry|Logger) repl

Blox Staking 1 Feb 27, 2022