:zap: Go web framework benchmark

Overview

go-web-framework-benchmark

This benchmark suite aims to compare the performance of Go web frameworks. It is inspired by Go HTTP Router Benchmark but this benchmark suite is different with that. Go HTTP Router Benchmark suit aims to compare the performance of routers but this Benchmark suit aims to compare whole HTTP request processing.

Last Test Updated: 2020-05

test environment

  • CPU: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz (2 cpus, 10 physical cores, 40 logical cores)
  • Memory: 32G
  • Go: go1.12.7 linux/amd64
  • OS: CentOS Linux release 7.3.1611 (Core)

Tested web frameworks (in alphabetical order)

Only test those webframeworks which are stable

Motivation

When I investigated performance of Go web frameworks, I found Go HTTP Router Benchmark, created by Julien Schmidt. He also developed a high performance http router: httprouter. I had thought I got the performance result until I created a piece of codes to mock the real business logics:

api.Get("/rest/hello", func(c *XXXXX.Context) {
	sleepTime := strconv.Atoi(os.Args[1]) //10ms
	if sleepTime > 0 {
		time.Sleep(time.Duration(sleepTime) * time.Millisecond)
	}

	c.Text("Hello world")
})

When I use the above codes to test those web frameworks, the token time of route selection is not so important in the whole http request processing, although performance of route selection of web frameworks are very different.

So I create this project to compare performance of web frameworks including connection, route selection, handler processing. It mocks business logics and can set a special processing time.

The you can get some interesting results if you use it to test.

Implementation

When you test a web framework, this test suit will starts a simple http server implemented by this web framework. It is a real http server and only contains GET url: "/hello".

When this server processes this url, it will sleep n milliseconds in this handler. It mocks the business logics such as:

  • read data from sockets
  • write data to disk
  • access databases
  • access cache servers
  • invoke other microservices
  • ……

It contains a test.sh that can do those tests automatically.

It uses wrk to test.

Basic Test

The first test case is to mock 0 ms, 10 ms, 100 ms, 500 ms processing time in handlers.

Benchmark (Round 3) the concurrency clients are 5000.

Latency (Round 3) Latency is the time of real processing time by web servers. The smaller is the better.

Allocs (Round 3) Allocs is the heap allocations by web servers when test is running. The unit is MB. The smaller is the better.

If we enable http pipelining, test result as below:

benchmark pipelining (Round 2)

Concurrency Test

In 30 ms processing time, the test result for 100, 1000, 5000 clients is:

concurrency (Round 3)

Latency (Round 3)

Latency (Round 3)

If we enable http pipelining, test result as below:

concurrency pipelining(Round 2)

cpu-bound case Test

cpu-bound (5000 concurrency)

Usage

You should install this package first if you want to run this test.

go get github.com/smallnest/go-web-framework-benchmark

It takes a while to install a large number of dependencies that need to be downloaded. Once that command completes, you can run:

cd $GOPATH/src/github.com/smallnest/go-web-framework-benchmark
go build -o  gowebbenchmark *.go
./test.sh

It will generate test results in processtime.csv and concurrency.csv. You can modify test.sh to execute your customized test cases.

  • If you also want to generate latency data and allocation data, you can run the script:
./test-latency.sh
  • If you don't want use keepalive, you can run:
./test-latency-nonkeepalive.sh
  • If you want to test http pipelining, you can run:
./test-pipelining.sh
  • If you want to test some of web frameworks, you can modify the test script and only keep your selected web frameworks:
……
web_frameworks=( "default" "ace" "beego" "bone" "denco" "echov1" "echov2standard" "echov2fasthttp" "fasthttp-raw" "fasthttprouter" "fasthttp-routing" "gin" "gocraftWeb" "goji" "gojiv2" "gojsonrest" "gorestful" "gorilla" "httprouter" "httptreemux" "lars" "lion" "macaron" "martini" "pat" "r2router" "tango" "tiger" "traffic" "violetear" "vulcan")
……
  • If you want to test all cases, you can run:
./test-all.sh

Plot

you can run the shell script plot.sh in testresults directory and it can generate all images in its parent directory.

Add new web framework

Welcome to add new Go web frameworks. You can follow the below steps and send me a pull request.

  1. add your web framework link in README
  2. add a hello implementation in server.go
  3. add your webframework in libs.sh

Please add your web framework alphabetically.

Issues
  • What happens ?

    What happens ?

    Before the last iris was first with fasthttp-raw, before 18 days, as this commit also says: https://github.com/smallnest/go-web-framework-benchmark/commit/0874ce64aa6d78235c9452bfed0b8fb6fb4f4547.

    Also I didn't change any performance-releated with SetBody & router, I just tested with a version we had before a month and the benchmark results are the same as the latest iris version.

    Can you tell me when you upgraded iris? what version did you had before (or a date) you did several tests within this period, this is the first time iris is down.

    And if you didn't changed anything(iris-releated) from the previous week, could you please re-run the tests?

    Edit2:

    I just ran the benchmarks from docker with docker pull smallnest/go-web-framework-benchmark docker run -v /opt/data:/data smallnest/go-web-framework-benchmark .

    Iris is still side-by-side with fasthttp. Some other frameworks I saw on this repo's README that seems to be top of iris, in real, they have lowest results...

    For now, only the basic tests (0ms 5000 clients) are finished:

    fasthttp and others

    iris

    I ran it from a 'clean/fresh' docker*

    but on README's Basic tests are:

    README page

    How is that even possible?

    Edit 3: 10ms 5k clients finished also:

    fasthttp and others 10ms

    iris 10ms

    I will leave the machine open until the graphical results also... I really need to sleep now

    But you got the idea.. something happened on your last machine's benchmark testing

    Thanks for everything!!

    opened by ghost 10
  • Add pipeline to tests

    Add pipeline to tests

    init = function(args)
       request_uri = args[1]
       depth = tonumber(args[2]) or 1
    
       local r = {}
       for i=1,depth do
         r[i] = wrk.format(nil, request_uri)
       end
       req = table.concat(r)
    end
    
    request = function()
       return req
    end
    
    
    $ wrk .... -s pipeline.lua 
    

    Taken from here

    enhancement 
    opened by ghost 10
  • docker image not creating images with labels

    docker image not creating images with labels

    Hi, first at all, many thanks for sharing this project, has helped me to improve and detect some issues within my code and therefore learn more about how to profile/test in general.

    Currently, I am giving a try to the docker image but is not creating the images by just running this:

    docker run  -v /opt/data:/data smallnest/go-web-framework-benchmark
    

    What I am doing to try to generate them, is to execute the plot.sh something like:

    docker run -v /tmp/data:/data -t -i --entrypoint /bin/bash smallnest/go-web-framework-benchmark
    

    then within the container, I run:

    docker-test.sh
    

    When finished I run plot.sh within the testresults this creates the images but without labels, an example of the output:

    benchmark_alloc

    Any idea of how to fix this? and if is not too much asking in the meantime could the benchmark results be updated?

    thanks in advance.

    opened by nbari 8
  • http: Accept error: accept tcp [::]:8080: accept4: too many open files

    http: Accept error: accept tcp [::]:8080: accept4: too many open files

    Hello,

    I made a fresh setup today and started to get these errors. Any idea ?

    Host machine is Ubuntu 20.04 with go 1.15.

    2020/08/26 14:31:24 - atreugo - Temporary error when accepting new connections: accept tcp4 0.0.0.0:8080: accept4: too many open files 2020/08/26 14:31:25 - atreugo - Temporary error when accepting new connections: accept tcp4 0.0.0.0:8080: accept4: too many open files 2020/08/26 14:31:26 - atreugo - Temporary error when accepting new connections: accept tcp4 0.0.0.0:8080: accept4: too many open files throughput: 118162.25 requests/second ./test.sh: line 18: 3216 Killed ./$server_bin_name $2 $3 finsihed testing atreugo

    testing web framework: beego • Initialization package=gramework version=1.7.0-rc3 • node info cputicks=2.53G package=gramework ram=911.18M used / 7.51G total swap=0.00 used / 2.00G total • load average is good fifteen=0.600 five=1.670 one=4.920 package=gramework • node uptime package=gramework uptime= 0:01 2020/08/26 14:31:30 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 5ms 2020/08/26 14:31:30 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 10ms 2020/08/26 14:31:30 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 20ms 2020/08/26 14:31:30 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 40ms 2020/08/26 14:31:30 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 80ms 2020/08/26 14:31:30 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 160ms 2020/08/26 14:31:30 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 320ms 2020/08/26 14:31:30 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 640ms 2020/08/26 14:31:31 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 1s 2020/08/26 14:31:32 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 1s

    Best Regards,

    opened by Karif 4
  • Processing time and Latency

    Processing time and Latency

    According to the Benchmark of different processing time (Latency), In all frameworks except fasthttp, for 0ms processing time, we have more latencies compared to 10ms! It's not clear to me. Maybe because of sleeps in the processing code. What do you think?

    Thanks for your useful benchmarks.

    opened by mehdishojaei 4
  • fixed gear’s bench

    fixed gear’s bench

    非常感谢你的评测,我发现 Gear 的测试逻辑不对,所以导致很多 bench 图表一枝独秀。

    Gear 的 Router 也是一个的 Gear 中间件, 不会默认挂载,这里修复了一下。

    目前的 Gear 版本 handle 了 context canceled error, 回头修复。

    另外,按照目前的测试逻辑——简单的回写一个 hello,基于原生 http server 的框架区别不会很大,无非就是多了一个初始化逻辑。我本地对比跑了一下 gin 和 gear,gear 稍低一点。因为 Gear 内置的初始化逻辑确实多了一些,包括检测 Compress 组件、recover 异常、初始化 cookie 模块、初始化 context 等

    opened by zensh 3
  • 本地测试结果与公布数据差距很大,

    本地测试结果与公布数据差距很大,

    测试环境 go1.7 最新 beego ->default路径无返回值 硬件: 40 Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz Mem: 32765096 测试结果: [[email protected] wrk]# ./wrk -t10 -c5000 -d60s http://127.0.0.1:8080/ Running 1m test @ http://127.0.0.1:8080/ 10 threads and 5000 connections Thread Stats Avg Stdev Max +/- Stdev Latency 402.30ms 565.62ms 2.00s 81.05% Req/Sec 0.86k 639.03 13.29k 78.62% 476512 requests in 1.00m, 70.44MB read Socket errors: connect 0, read 0, write 0, timeout 40477 Requests/sec: 7932.59 Transfer/sec: 1.17MB

    Requests/sec 远远低于您第一项测试中0ms 100ms 等测试项的公布值, 不知道问题出在哪里.

    opened by ghost 3
  • missing dependencies

    missing dependencies

    Forked this repository and tried running. Following is the output

    ../go-web-framework-benchmark$ go build -o gowebfmbench
    # github.com/cloudfoundry/gosigar
    ../../../../pkg/mod/github.com/cloudfoundry/[email protected]/concrete_sigar.go:20:11: cpuUsage.Get undefined (type Cpu has no field or method Get)
    ../../../../pkg/mod/github.com/cloudfoundry/[email protected]/concrete_sigar.go:30:13: cpuUsage.Get undefined (type Cpu has no field or method Get)
    ../../../../pkg/mod/github.com/cloudfoundry/[email protected]/concrete_sigar.go:49:10: l.Get undefined (type LoadAverage has no field or method Get)
    ../../../../pkg/mod/github.com/cloudfoundry/[email protected]/concrete_sigar.go:55:10: m.Get undefined (type Mem has no field or method Get)
    ../../../../pkg/mod/github.com/cloudfoundry/[email protected]/concrete_sigar.go:61:10: s.Get undefined (type Swap has no field or method Get)
    ../../../../pkg/mod/github.com/cloudfoundry/[email protected]/sigar_shared.go:12:20: procTime.Get undefined (type *ProcTime has no field or method Get)
    

    Go env

    $ go env
    GO111MODULE=""
    GOARCH="amd64"
    GOBIN=""
    GOCACHE="/Users/kamaleshwar/Library/Caches/go-build"
    GOENV="/Users/kamaleshwar/Library/Application Support/go/env"
    GOEXE=""
    GOFLAGS=""
    GOHOSTARCH="amd64"
    GOHOSTOS="darwin"
    GOINSECURE=""
    GOOS="darwin"
    GOPATH="/Users/kamaleshwar/go"
    GOPROXY="https://proxy.golang.org,direct"
    GOROOT="/usr/local/go"
    GOSUMDB="sum.golang.org"
    GOTMPDIR=""
    GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
    GCCGO="gccgo"
    AR="ar"
    CC="clang"
    CXX="clang++"
    CGO_ENABLED="0"
    GOMOD=""
    CGO_CFLAGS="-g -O2"
    CGO_CPPFLAGS=""
    CGO_CXXFLAGS="-g -O2"
    CGO_FFLAGS="-g -O2"
    CGO_LDFLAGS="-g -O2"
    PKG_CONFIG="pkg-config"
    GOGCCFLAGS="-fPIC -m64 -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/lv/0wsxrb6123s8slgb2lp_6_3m0000gp/T/go-build461564637=/tmp/go-build -gno-record-gcc-switches -fno-common"
    
    opened by bnkamalesh 2
  • Remove iris

    Remove iris

    Given how the awesome-go PR went (it ended up with a ban of the iris from the list from just a removal) I considered creating one for this place as well. I'm asking to have iris removed based on the widespread evidence that this is a project toxic for the Go community and the fact that it is using this benchmark suite only to promote itself.

    I've also took the liberty to bump up the docker container to 1.7.1 which is the latest version of Go (1.7.2 might be released soon this week) as a small token of saying thank you for this project.

    Thank you for your consideration.

    opened by dlsniper 2
Owner
smallnest
Author of 《Scala Collections Cookbook》
smallnest
Golanger Web Framework is a lightweight framework for writing web applications in Go.

/* Copyright 2013 Golanger.com. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except

golanger 299 Mar 3, 2022
The jin is a simplified version of the gin web framework that can help you quickly understand the core principles of a web framework.

jin About The jin is a simplified version of the gin web framework that can help you quickly understand the core principles of a web framework. If thi

null 8 Dec 27, 2021
⚡ Rux is an simple and fast web framework. support middleware, compatible http.Handler interface. 简单且快速的 Go web 框架,支持中间件,兼容 http.Handler 接口

Rux Simple and fast web framework for build golang HTTP applications. NOTICE: v1.3.x is not fully compatible with v1.2.x version Fast route match, sup

Gookit 77 Apr 16, 2022
henrylee2cn 1.6k May 17, 2022
Roche is a Code Generator and Web Framework, makes web development super concise with Go, CleanArch

It is still under development, so please do not use it. We plan to release v.1.0.0 in the summer. roche is a web framework optimized for microservice

Riita 14 Apr 24, 2022
A powerful go web framework for highly scalable and resource efficient web application

webfr A powerful go web framework for highly scalable and resource efficient web application Installation: go get -u github.com/krishpranav/webfr Exa

Krisna Pranav 13 Nov 28, 2021
A powerful go web framework for highly scalable and resource efficient web application

A powerful go web framework for highly scalable and resource efficient web application

null 20 Feb 2, 2022
A web app built using Go Buffalo web framework

Welcome to Buffalo Thank you for choosing Buffalo for your web development needs. Database Setup It looks like you chose to set up your application us

Mike Okoth 0 Feb 7, 2022
laravel for golang,goal,fullstack framework,api framework

laravel for golang,goal,fullstack framework,api framework

桥边红药 17 Feb 24, 2022
Gin is a HTTP web framework written in Go (Golang).

Gin is a HTTP web framework written in Go (Golang). It features a Martini-like API with much better performance -- up to 40 times faster. If you need smashing performance, get yourself some Gin.

Gin-Gonic 58.9k May 15, 2022
An ideally refined web framework for Go.

Air An ideally refined web framework for Go. High-performance? Fastest? Almost all web frameworks are using these words to tell people that they are t

Aofei Sheng 414 Apr 12, 2022
An opinionated productive web framework that helps scaling business easier.

appy An opinionated productive web framework that helps scaling business easier, i.e. focus on monolith first, only move to microservices with GRPC la

appist 121 Apr 20, 2022
BANjO is a simple web framework written in Go (golang)

BANjO banjo it's a simple web framework for building simple web applications Install $ go get github.com/nsheremet/banjo Example Usage Simple Web App

Nazarii Sheremet 19 Jan 24, 2022
beego is an open-source, high-performance web framework for the Go programming language.

Beego Beego is used for rapid development of enterprise application in Go, including RESTful APIs, web apps and backend services. It is inspired by To

astaxie 456 May 9, 2022
High performance, minimalist Go web framework

Supported Go versions As of version 4.0.0, Echo is available as a Go module. Therefore a Go version capable of understanding /vN suffixed imports is r

LabStack LLC 22.4k May 12, 2022
⚡️ Express inspired web framework written in Go

Fiber is an Express inspired web framework built on top of Fasthttp, the fastest HTTP engine for Go. Designed to ease things up for fast development w

Fiber 20k May 17, 2022
Go web framework with a natural feel

Fireball Overview Fireball is a package for Go web applications. The primary goal of this package is to make routing, response writing, and error hand

Zack Patrick 57 Mar 26, 2022
Gearbox :gear: is a web framework written in Go with a focus on high performance

gearbox ⚙️ is a web framework for building micro services written in Go with a focus on high performance. It's built on fasthttp which is up to 10x fa

Gearbox 634 May 13, 2022
Goa is a web framework based on middleware, like koa.js.

Goa Goa is under construction, if you are familiar with koa or go and interested in this project, please join us. What is goa? goa = go + koa Just lik

null 47 Apr 26, 2022