:zap: Go web framework benchmark

Overview

go-web-framework-benchmark

This benchmark suite aims to compare the performance of Go web frameworks. It is inspired by Go HTTP Router Benchmark but this benchmark suite is different with that. Go HTTP Router Benchmark suit aims to compare the performance of routers but this Benchmark suit aims to compare whole HTTP request processing.

Last Test Updated: 2020-05

test environment

  • CPU: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz (2 cpus, 10 physical cores, 40 logical cores)
  • Memory: 32G
  • Go: go1.12.7 linux/amd64
  • OS: CentOS Linux release 7.3.1611 (Core)

Tested web frameworks (in alphabetical order)

Only test those webframeworks which are stable

Motivation

When I investigated performance of Go web frameworks, I found Go HTTP Router Benchmark, created by Julien Schmidt. He also developed a high performance http router: httprouter. I had thought I got the performance result until I created a piece of codes to mock the real business logics:

api.Get("/rest/hello", func(c *XXXXX.Context) {
	sleepTime := strconv.Atoi(os.Args[1]) //10ms
	if sleepTime > 0 {
		time.Sleep(time.Duration(sleepTime) * time.Millisecond)
	}

	c.Text("Hello world")
})

When I use the above codes to test those web frameworks, the token time of route selection is not so important in the whole http request processing, although performance of route selection of web frameworks are very different.

So I create this project to compare performance of web frameworks including connection, route selection, handler processing. It mocks business logics and can set a special processing time.

The you can get some interesting results if you use it to test.

Implementation

When you test a web framework, this test suit will starts a simple http server implemented by this web framework. It is a real http server and only contains GET url: "/hello".

When this server processes this url, it will sleep n milliseconds in this handler. It mocks the business logics such as:

  • read data from sockets
  • write data to disk
  • access databases
  • access cache servers
  • invoke other microservices
  • ……

It contains a test.sh that can do those tests automatically.

It uses wrk to test.

Basic Test

The first test case is to mock 0 ms, 10 ms, 100 ms, 500 ms processing time in handlers.

Benchmark (Round 3) the concurrency clients are 5000.

Latency (Round 3) Latency is the time of real processing time by web servers. The smaller is the better.

Allocs (Round 3) Allocs is the heap allocations by web servers when test is running. The unit is MB. The smaller is the better.

If we enable http pipelining, test result as below:

benchmark pipelining (Round 2)

Concurrency Test

In 30 ms processing time, the test result for 100, 1000, 5000 clients is:

concurrency (Round 3)

Latency (Round 3)

Latency (Round 3)

If we enable http pipelining, test result as below:

concurrency pipelining(Round 2)

cpu-bound case Test

cpu-bound (5000 concurrency)

Usage

You should install this package first if you want to run this test.

go get github.com/smallnest/go-web-framework-benchmark

It takes a while to install a large number of dependencies that need to be downloaded. Once that command completes, you can run:

cd $GOPATH/src/github.com/smallnest/go-web-framework-benchmark
go build -o  gowebbenchmark *.go
./test.sh

It will generate test results in processtime.csv and concurrency.csv. You can modify test.sh to execute your customized test cases.

  • If you also want to generate latency data and allocation data, you can run the script:
./test-latency.sh
  • If you don't want use keepalive, you can run:
./test-latency-nonkeepalive.sh
  • If you want to test http pipelining, you can run:
./test-pipelining.sh
  • If you want to test some of web frameworks, you can modify the test script and only keep your selected web frameworks:
……
web_frameworks=( "default" "ace" "beego" "bone" "denco" "echov1" "echov2standard" "echov2fasthttp" "fasthttp-raw" "fasthttprouter" "fasthttp-routing" "gin" "gocraftWeb" "goji" "gojiv2" "gojsonrest" "gorestful" "gorilla" "httprouter" "httptreemux" "lars" "lion" "macaron" "martini" "pat" "r2router" "tango" "tiger" "traffic" "violetear" "vulcan")
……
  • If you want to test all cases, you can run:
./test-all.sh

Plot

you can run the shell script plot.sh in testresults directory and it can generate all images in its parent directory.

Add new web framework

Welcome to add new Go web frameworks. You can follow the below steps and send me a pull request.

  1. add your web framework link in README
  2. add a hello implementation in server.go
  3. add your webframework in libs.sh

Please add your web framework alphabetically.

Issues
  • Update results?

    Update results?

    Maybe, it's time to update benchmark results with go 1.8?

    opened by kirillDanshin 11
  • Add pipeline to tests

    Add pipeline to tests

    init = function(args)
       request_uri = args[1]
       depth = tonumber(args[2]) or 1
    
       local r = {}
       for i=1,depth do
         r[i] = wrk.format(nil, request_uri)
       end
       req = table.concat(r)
    end
    
    request = function()
       return req
    end
    
    
    $ wrk .... -s pipeline.lua 
    

    Taken from here

    enhancement 
    opened by ghost 10
  • What happens ?

    What happens ?

    Before the last iris was first with fasthttp-raw, before 18 days, as this commit also says: https://github.com/smallnest/go-web-framework-benchmark/commit/0874ce64aa6d78235c9452bfed0b8fb6fb4f4547.

    Also I didn't change any performance-releated with SetBody & router, I just tested with a version we had before a month and the benchmark results are the same as the latest iris version.

    Can you tell me when you upgraded iris? what version did you had before (or a date) you did several tests within this period, this is the first time iris is down.

    And if you didn't changed anything(iris-releated) from the previous week, could you please re-run the tests?

    Edit2:

    I just ran the benchmarks from docker with docker pull smallnest/go-web-framework-benchmark docker run -v /opt/data:/data smallnest/go-web-framework-benchmark .

    Iris is still side-by-side with fasthttp. Some other frameworks I saw on this repo's README that seems to be top of iris, in real, they have lowest results...

    For now, only the basic tests (0ms 5000 clients) are finished:

    fasthttp and others

    iris

    I ran it from a 'clean/fresh' docker*

    but on README's Basic tests are:

    README page

    How is that even possible?

    Edit 3: 10ms 5k clients finished also:

    fasthttp and others 10ms

    iris 10ms

    I will leave the machine open until the graphical results also... I really need to sleep now

    But you got the idea.. something happened on your last machine's benchmark testing

    Thanks for everything!!

    opened by ghost 10
  • New Benchmark?

    New Benchmark?

    Hello can you please provide a new benschmark? I can't get in to run on my Alpine Linux Server... because no wrk on ARM available...

    Thank you!

    opened by Dexus 10
  • docker image not creating images with labels

    docker image not creating images with labels

    Hi, first at all, many thanks for sharing this project, has helped me to improve and detect some issues within my code and therefore learn more about how to profile/test in general.

    Currently, I am giving a try to the docker image but is not creating the images by just running this:

    docker run  -v /opt/data:/data smallnest/go-web-framework-benchmark
    

    What I am doing to try to generate them, is to execute the plot.sh something like:

    docker run -v /tmp/data:/data -t -i --entrypoint /bin/bash smallnest/go-web-framework-benchmark
    

    then within the container, I run:

    docker-test.sh
    

    When finished I run plot.sh within the testresults this creates the images but without labels, an example of the output:

    benchmark_alloc

    Any idea of how to fix this? and if is not too much asking in the meantime could the benchmark results be updated?

    thanks in advance.

    opened by nbari 8
  • add latency and memory usage data in test result

    add latency and memory usage data in test result

    Latency and memory usage are also important measures to compare go web frameworks.

    opened by smallnest 5
  • What about echo v4?

    What about echo v4?

    What about echo v4?

    opened by frederikhors 4
  • Processing time and Latency

    Processing time and Latency

    According to the Benchmark of different processing time (Latency), In all frameworks except fasthttp, for 0ms processing time, we have more latencies compared to 10ms! It's not clear to me. Maybe because of sleeps in the processing code. What do you think?

    Thanks for your useful benchmarks.

    opened by mehdishojaei 4
  • Add SIRIS

    Add SIRIS

    opened by Dexus 4
  • fixed gear’s bench

    fixed gear’s bench

    非常感谢你的评测,我发现 Gear 的测试逻辑不对,所以导致很多 bench 图表一枝独秀。

    Gear 的 Router 也是一个的 Gear 中间件, 不会默认挂载,这里修复了一下。

    目前的 Gear 版本 handle 了 context canceled error, 回头修复。

    另外,按照目前的测试逻辑——简单的回写一个 hello,基于原生 http server 的框架区别不会很大,无非就是多了一个初始化逻辑。我本地对比跑了一下 gin 和 gear,gear 稍低一点。因为 Gear 内置的初始化逻辑确实多了一些,包括检测 Compress 组件、recover 异常、初始化 cookie 模块、初始化 context 等

    opened by zensh 3
  • can you add the `goframe` to the benchmark

    can you add the `goframe` to the benchmark

    https://github.com/gogf/gf this framework is more and mor populor

    opened by ayamzh 0
  • add emir

    add emir

    opened by emirmuminoglu 1
  • add iris to the benchmark

    add iris to the benchmark

    Iris is a fast, simple yet fully featured and very efficient web framework for Go.

    opened by tuhao1020 1
  • http: Accept error: accept tcp [::]:8080: accept4: too many open files

    http: Accept error: accept tcp [::]:8080: accept4: too many open files

    Hello,

    I made a fresh setup today and started to get these errors. Any idea ?

    Host machine is Ubuntu 20.04 with go 1.15.

    2020/08/26 14:31:24 - atreugo - Temporary error when accepting new connections: accept tcp4 0.0.0.0:8080: accept4: too many open files 2020/08/26 14:31:25 - atreugo - Temporary error when accepting new connections: accept tcp4 0.0.0.0:8080: accept4: too many open files 2020/08/26 14:31:26 - atreugo - Temporary error when accepting new connections: accept tcp4 0.0.0.0:8080: accept4: too many open files throughput: 118162.25 requests/second ./test.sh: line 18: 3216 Killed ./$server_bin_name $2 $3 finsihed testing atreugo

    testing web framework: beego • Initialization package=gramework version=1.7.0-rc3 • node info cputicks=2.53G package=gramework ram=911.18M used / 7.51G total swap=0.00 used / 2.00G total • load average is good fifteen=0.600 five=1.670 one=4.920 package=gramework • node uptime package=gramework uptime= 0:01 2020/08/26 14:31:30 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 5ms 2020/08/26 14:31:30 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 10ms 2020/08/26 14:31:30 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 20ms 2020/08/26 14:31:30 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 40ms 2020/08/26 14:31:30 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 80ms 2020/08/26 14:31:30 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 160ms 2020/08/26 14:31:30 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 320ms 2020/08/26 14:31:30 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 640ms 2020/08/26 14:31:31 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 1s 2020/08/26 14:31:32 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 1s

    Best Regards,

    opened by Karif 3
  • What about echo v4?

    What about echo v4?

    First image on Readme doesn't show number for Echo v4. Why?

    image

    opened by frederikhors 1
  • Wrong link in README.md for libs.version

    Wrong link in README.md for libs.version

    Wrong link in README.md:

    version of all web frameworks is at: libs.version.

    opened by frederikhors 0
Owner
smallnest
Author of 《Scala Collections Cookbook》
smallnest
Go benchmark harness.

autobench autobench is a framework to compare the performance of Go 1.2 and Go 1.3. usage autobench downloads and builds the latest Go 1.1 and Go tip

Dave Cheney 92 Nov 12, 2021
Key/Value database benchmark

USAGE: kvbench OPTIONS OVERVIEW kvbench is a simple benchmarking tool to evaluate the read performance of a key/value store while writes are being app

James A. Robinson 24 Nov 15, 2021
A rapid http(s) benchmark tool written in Go

gonetx/httpit httpit is a rapid http(s) benchmark tool which on top of fasthttp. Also thanks to cobra and bubbletea. Installation Get binaries from re

null 96 Oct 16, 2021
Cache benchmark for Golang

Cache comparison benchmark for Go This benchmark compares cache algorithms using scrambled zipfian distribution (a few occur very often while many oth

Vladimir Mihailenco 29 Nov 17, 2021
Go http server benchmark

go-http-server-benchmark The more connections, nbio cost the less memory, and performance the better than other frameworks. We can serve for 1000k or

lesismal 1 Oct 27, 2021
Golang json encoders\decoders benchmark

Json encoder and decoder benchmark Test Encode for: Build-in encoder https://github.com/mailru/easyjson https://github.com/json-iterator/go Test Decod

Vladislav S 0 Dec 3, 2021
:zap: Go web framework benchmark

go-web-framework-benchmark This benchmark suite aims to compare the performance of Go web frameworks. It is inspired by Go HTTP Router Benchmark but t

smallnest 1.6k Dec 3, 2021
Go HTTP request router and web framework benchmark

Go HTTP Router Benchmark This benchmark suite aims to compare the performance of HTTP request routers for Go by implementing the routing structure of

Julien Schmidt 1.5k Nov 23, 2021
:zap: boilerplate template manager that generates files or directories from template repositories

Boilr Are you doing the same steps over and over again every time you start a new programming project? Boilr is here to help you create projects from

Tamer Tas 1.4k Nov 26, 2021
:zap: Transfer files over wifi from your computer to your mobile device by scanning a QR code without leaving the terminal.

$ qrcp Transfer files over Wi-Fi from your computer to a mobile device by scanning a QR code without leaving the terminal. You can support development

Claudio d'Angelis 8.2k Dec 3, 2021
Analyzer: zapvet is static analysis tool for zap

zapvet zapvet is static analysis tool for zap. fieldtype: fieldtype finds confliction type of field Install You can get zapvet by go install command (

GoStaticAnalysis 4 Aug 12, 2021
微服务架构-micro-基于go-zero zrpc etcd 单独集成orm-gorm 日志-uber/zap

micro目录结构 pkg ├── api 业务接口逻辑层 所有的业务逻辑存放目录。 │ ├── model 数据模型 数据管理层,仅用于操作管理数据,如数据库操作 ├── assets

jiwei 6 Nov 15, 2021
A Handy FieldBuilder Template for uber-go/zap

zap-fieldbuilder-template A Handy Generic Logging FieldBuilder Template around uber-go/zap. You can check this tiny blog for further information ?? Wh

Yavuz Selim Yesilyurt 7 Nov 7, 2021
Package zaperations provides a Google Cloud operations suite (formerly Stackdriver) compatible config for the uber-go/zap logger.

Package zaperations provides a Google Cloud Operations (formerly Stackdriver) compatible config for the excellent uber-go/zap logger. Example This exa

Wayne 0 Nov 6, 2021
Wrap zap for easy using.

go-project-pkg/log Wrap zap for easy using. Installation $ go get -u github.com/go-project-pkg/log Usage Use default logger: import "github.com/go-pro

Go Project PKG 1 Nov 8, 2021
Hook for sending events zap logger to telegram.

zaptelegram Hook for sending events to telegram for zap logger. Install: go get -u github.com/strpc/zaptelegram Basic usage: package main import ( "

null 2 Nov 18, 2021
Flamingo Framework and Core Library. Flamingo is a go based framework for pluggable web projects. It is used to build scalable and maintainable (web)applications.

Flamingo Framework Flamingo is a web framework based on Go. It is designed to build pluggable and maintainable web projects. It is production ready, f

Flamingo 241 Nov 26, 2021
Golanger Web Framework is a lightweight framework for writing web applications in Go.

/* Copyright 2013 Golanger.com. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except

golanger 302 Nov 17, 2021
Flamingo Framework and Core Library. Flamingo is a go based framework for pluggable web projects. It is used to build scalable and maintainable (web)applications.

Flamingo Framework Flamingo is a web framework based on Go. It is designed to build pluggable and maintainable web projects. It is production ready, f

Flamingo 242 Nov 29, 2021
Golanger Web Framework is a lightweight framework for writing web applications in Go.

/* Copyright 2013 Golanger.com. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except

golanger 302 Nov 17, 2021
The jin is a simplified version of the gin web framework that can help you quickly understand the core principles of a web framework.

jin About The jin is a simplified version of the gin web framework that can help you quickly understand the core principles of a web framework. If thi

null 7 Oct 22, 2021
7 days golang programs from scratch (web framework Gee, distributed cache GeeCache, object relational mapping ORM framework GeeORM, rpc framework GeeRPC etc) 7天用Go动手写/从零实现系列

7 days golang programs from scratch README 中文版本 7天用Go从零实现系列 7天能写什么呢?类似 gin 的 web 框架?类似 groupcache 的分布式缓存?或者一个简单的 Python 解释器?希望这个仓库能给你答案

Dai Jie 9.1k Dec 4, 2021
Go benchmark harness.

autobench autobench is a framework to compare the performance of Go 1.2 and Go 1.3. usage autobench downloads and builds the latest Go 1.1 and Go tip

Dave Cheney 92 Nov 12, 2021
Key/Value database benchmark

USAGE: kvbench OPTIONS OVERVIEW kvbench is a simple benchmarking tool to evaluate the read performance of a key/value store while writes are being app

James A. Robinson 24 Nov 15, 2021
A rapid http(s) benchmark tool written in Go

gonetx/httpit httpit is a rapid http(s) benchmark tool which on top of fasthttp. Also thanks to cobra and bubbletea. Installation Get binaries from re

null 96 Oct 16, 2021
Cache benchmark for Golang

Cache comparison benchmark for Go This benchmark compares cache algorithms using scrambled zipfian distribution (a few occur very often while many oth

Vladimir Mihailenco 29 Nov 17, 2021
A CLI to turn Go's benchmark output into pictures

benchdraw benchdraw allows you to make easy to read picture plots from data in Go's benchmark format, implemented in pure Go. Benchdraw does not try t

Jack Lindamood 340 Nov 22, 2021
Continuous Benchmark for Go Project

Abstract cob compares benchmarks between the latest commit (HEAD) and the previous commit (HEAD{@1}). The program will fail if the change in score is

Teppei Fukuda 343 Nov 7, 2021
Redis Sorted Sets Benchmark

redis-zbench-go Redis Sorted Sets Benchmark Overview This repo contains code to trigger load ( ZADD ) or query (ZRANGEBYLEX key min max) benchmarks, w

filipe oliveira 3 May 18, 2021