Gzip Middleware for Go

Overview

English | 中文

Gzip Middleware for Go

GoDoc Build status codecov Lint status Go Report Card

An out-of-the-box, also customizable gzip middleware for Gin and net/http.

Golang Gzip Middleware

Examples

Use DefaultHandler() to create a ready-to-go gzip middleware.

Gin

import github.com/nanmu42/gzip

func main() {
	g := gin.Default()

    // use default settings
	g.Use(gzip.DefaultHandler().Gin)

	g.GET("/", func(c *gin.Context) {
		c.JSON(http.StatusOK, map[string]interface{}{
			"code": 0,
			"msg":  "hello",
			"data": fmt.Sprintf("l%sng!", strings.Repeat("o", 1000)),
		})
	})

	log.Println(g.Run(fmt.Sprintf(":%d", 3000)))
}

net/http

import github.com/nanmu42/gzip

func main() {
	mux := http.NewServeMux()
	mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		writeString(w, fmt.Sprintf("This content is compressed: l%sng!", strings.Repeat("o", 1000)))
	})

    // wrap http.Handler using default settings
	log.Println(http.ListenAndServe(fmt.Sprintf(":%d", 3001), gzip.DefaultHandler().WrapHandler(mux)))
}

func writeString(w http.ResponseWriter, payload string) {
	w.Header().Set("Content-Type", "text/plain; charset=utf8")
	_, _ = io.WriteString(w, payload+"\n")
}

Customize Handler

Handler can be customized using NewHandler and Config:

import github.com/nanmu42/gzip

handler := gzip.NewHandler(gzip.Config{
    // gzip compression level to use
	CompressionLevel: 6, 
    // minimum content length to trigger gzip, the unit is in byte.
	MinContentLength: 1024,
    // RequestFilter decide whether or not to compress response judging by request.
    // Filters are applied in the sequence here.
	RequestFilter: []RequestFilter{
	    NewCommonRequestFilter(),
	    DefaultExtensionFilter(),
	},
    // ResponseHeaderFilter decide whether or not to compress response
    // judging by response header
	ResponseHeaderFilter: []ResponseHeaderFilter{
		NewSkipCompressedFilter(),
		DefaultContentTypeFilter(),
	},
})

RequestFilter and ResponseHeaderFilter are interfaces. You may define one that specially suits your need.

Performance

  • When response payload is small, the handler is smart enough to skip compression automatically, which takes neglectable overhead.
  • At the time when the payload is big enough, gzip kicks in and there is a reasonable price.
$ go test -benchmem -bench .
goos: linux
goarch: amd64
pkg: github.com/nanmu42/gzip
BenchmarkSoleGin_SmallPayload-4                          4104684               276 ns/op              64 B/op          2 allocs/op
BenchmarkGinWithDefaultHandler_SmallPayload-4            1683307               707 ns/op              96 B/op          3 allocs/op
BenchmarkSoleGin_BigPayload-4                            4198786               274 ns/op              64 B/op          2 allocs/op
BenchmarkGinWithDefaultHandler_BigPayload-4                44780             27636 ns/op             190 B/op          5 allocs/op
PASS
ok      github.com/nanmu42/gzip 6.373s

Note: due to an awkward man-mistake, benchmark of and before v1.0.0 are not accurate.

Limitation

  • You should always provide a Content-Type in http response's header, though handler guesses by http.DetectContentType()as makeshift;
  • When Content-Length is not available, handler may buffer your writes to decide if its big enough to do a meaningful compression. A high MinContentLength may bring memory overhead, although the handler tries to be smart by reusing buffers and testing if len(data) of the first http.ResponseWriter.Write(data []byte) calling suffices or not.

Status: Stable

All APIs are finalized, and no breaking changes will be made in the 1.x series of releases.

Acknowledgement

During the development of this work, the author took following works/materials as reference:

This package uses klauspost's compress package to handle gzip compression.

Logo generated at Gopherize.me.

License

MIT License
Copyright (c) 2019 LI Zhennan

Caddy is licensed under the Apache License
Copyright 2015 Light Code Labs, LLC
Comments
  • Compression doesn't work for large responces

    Compression doesn't work for large responces

    Hello. My api gives different response sizes. Where the response is small (13kb) compression works fine and returns 1.78kb. But if the size is large (2.75mb), then the response is returned uncompressed, although the Content-Encoding:gzip header is added. I also see that for large responses, the Transfer-Encoding:chunked header is returned, but no Content-Length is returned. What should be done to compress all responses? Compressed: image

    Not compressed: image

    opened by dmitrydvm 3
  • 赞一个!

    赞一个!

    必须要赞一个!Gin,Echo的中间件感觉各种考虑不完善,其他语言的web框架基本上标配的功能,Go语言里居然找不到一个完全没问题的。

    Gin的gzip长期没人维护,新功能只能手动拉代码到本地,搞个排除文件扩展名和路径的配置,配也不是,不配也不是... Echo的gzip就是默认所有文档都开启压缩,自己想配个Skipper,结果搞了半天获取不到header里的值(可能是我太菜...)

    总之这个就很好,真正的开箱即用,菜鸟福音,请求/响应的文档类型都有识别,哪些文档类型该不该压缩,多大的文件需要压缩,这本来不就是一个gzip库该去识别的吗...

    希望作者继续维护下去~

    用的有个疑问: 1、gzip.NewHandler(gzip.Config{})似乎必须要传入所有参数,仅传入CompressionLevel会报

    invalid MinContentLength: 0
    

    2、使用MinContentLength来判断文档大小,是否可以改为按照文档size来识别?比如配置文档大于4kb才开启gzip

    opened by hyingreborn 2
  • Switch compression package (3x faster)

    Switch compression package (3x faster)

    Use github.com/klauspost/compress package for compression.

    Benchmarks did not actually compress since the header was carried over between calls.

    Before/after are both with the benchmark fix in place.

    λ benchcmp before.txt after.txt
    benchmark                                          old ns/op     new ns/op     delta
    BenchmarkSoleGin_SmallPayload-12                   279           284           +1.79%
    BenchmarkGinWithDefaultHandler_SmallPayload-12     653           662           +1.38%
    BenchmarkSoleGin_BigPayload-12                     271           284           +4.80%
    BenchmarkGinWithDefaultHandler_BigPayload-12       83649         25616         -69.38%
    

    I believe the last benchmark is the only one actually to be compressing, correct?

    I didn't go out of my way to clean up the benchmark since the difference is rather obvious.

    The compression ratio is likely a bit less, but among many other changes the default settings have been tweaked to give the most reasonable speed/compression tradeoff.

    opened by klauspost 1
  • Do not change response code after 204

    Do not change response code after 204

    There's a bug in the writerwrapper. If you call WriteHeader with either 204 or 304, shouldCompress would be set to false and then all subsequent calls to Write will result in directly calling the OriginWriter.Write but since the status code is not written to OriginWriter, it will set it to 200. Another side effect of this is that Write will not only set the status code to 200 but it will also write the content of data to Body which would not normally happen if the gzip middleware is not used. Instead, if you set the status code to 204 or 304, any calls to Write will fail with an error. The correct unit test for this would be:

    func Test_writeWrapper_does_not_change_status_code_after_204(t *testing.T) {
    	wrapper, recorder := newWrapper()
    
    	wrapper.WriteHeader(http.StatusNoContent)
    	n, err := wrapper.Write([]byte("something"))
    
    	assert.Equal(t, 0, n)
    	assert.Error(t, err)
    	assert.Equal(t, http.StatusNoContent, recorder.Code)
    	assert.Equal(t, []byte{}, recorder.Body.Bytes())
    }
    

    The problem is that recorder actually behaves differently from server.response and won't check for 204, 304 (the actual call that server.response makes is bodyAllowedForStatus(w.status)) so recorder will both manage to record the 204 status code and the body (which won't happen in reality). Yet, I didn't find a way to instantiate the actual server.response and test against it. So, in the unit tests, the body will be written but in actual use cases, an error will be thrown as expected.

    opened by hrist0stoichev 0
  • Add support for serving precompressed static files from ServeFile

    Add support for serving precompressed static files from ServeFile

    Thanks for providing this useful middleware first of all. I'm wondering if the Static files from http.ServeFile method is also supported. Especially,

    It would also be 'nice to have' if the http.ServeFile method would check for the presence of xxx.gz and serve it if present and the accept header includes gzip. This is a trick that nginx does and it greatly accelerates the serving of static css, js etc files because they can be compressed just once during the build process. There's the added benefit too that they are smaller files to read, as well as needing fewer bytes on the wire

    thanks

    enhancement help wanted 
    opened by suntong 3
Releases(v1.2.0)
  • v1.2.0(Sep 18, 2021)

    v1.2.0 is a bug fix release:

    • fix: Do not change response code after 204 and 304 (#5)

    Thanks to @hstoychev-cb for the wonderful contribution.

    Source code(tar.gz)
    Source code(zip)
  • v1.1.0(Nov 29, 2020)

    • refactor: switch gzip implmentation from official gzip to github.com/klauspost/compress(#3, thanks to @klauspost)
    • vendor: bump dependency version
    • doc: acknoledgement and performance correction
    • refactor: simplify writerWrapper.enoughContentLength()
    Source code(tar.gz)
    Source code(zip)
  • v1.0.0(Apr 26, 2020)

Owner
LI Zhennan
To build beautiful things beautifully.
LI Zhennan
⚡ Rux is an simple and fast web framework. support middleware, compatible http.Handler interface. 简单且快速的 Go web 框架,支持中间件,兼容 http.Handler 接口

Rux Simple and fast web framework for build golang HTTP applications. NOTICE: v1.3.x is not fully compatible with v1.2.x version Fast route match, sup

Gookit 84 Dec 8, 2022
A minimal framework to build web apps; with handler chaining, middleware support; and most of all standard library compliant HTTP handlers(i.e. http.HandlerFunc).

WebGo v4.1.3 WebGo is a minimalistic framework for Go to build web applications (server side) with zero 3rd party dependencies. Unlike full-fledged fr

Kamaleshwar 266 Jan 1, 2023
Bootstrapper and middleware for Echo framework in golang.

rk-echo Interceptor & bootstrapper designed for echo framework. Currently, supports bellow functionalities. Name Description Start with YAML Start ser

PointGoal 5 Jul 9, 2022
Prometheus middleware for wish

promwish Package promwish provides a simple wish middleware exposing some Prometheus metrics. Example Usage You can add promwish as a middleware to yo

Charm 28 Nov 26, 2022
X-Response-Time middleware for fiber/v2

fiber-responsetime X-Response-Time middleware for fiber/v2 go get github.com/husanu/fiber-responsetime/v2 package main import ( "time" "github.com

Andrei Husanu 0 Dec 17, 2021
Auth0 Middleware for go labstack/echo

go-echo-auth0-middleware Auth0 Middleware for go labstack/echo Example package main import ( "net/http" "github.com/auth0/go-jwt-middleware/v2/val

Satish Babariya 6 Nov 24, 2022
Gin middleware/handler to enable CORS support.

wcors Gin middleware/handler to enable CORS support. Usage Start using it Download and install it: go get github.com/wyy-go/wcors Import it in your co

null 0 Jan 8, 2022
Episodio V: Rise of the middleware

APITrials0.5 Episodio V: Rise of the middleware Captain's log: Up to date this file contains some simple errors to be corrected. They come from severa

Rafael Diaz Miles 0 Jan 10, 2022
EchoMiddleware - Echo Middleware with golang

EchoMiddleware middleware for echo server usage import ( "github.com/universe-3

universe30 0 Jan 4, 2022
⚡Simple cors middleware package for minima

This is package is wrapper based on rs/cors package made for minima. Geting Started Install the package using go get github.com/gominima/cors and call

Minima 5 Mar 7, 2022
Go parallel gzip (de)compression

pgzip Go parallel gzip compression/decompression. This is a fully gzip compatible drop in replacement for "compress/gzip". This will split compression

Klaus Post 997 Dec 29, 2022
Split text files into gzip files with x lines

hakgzsplit split lines of text into multiple gzip files

Luke Stephens (hakluke) 6 Jun 21, 2022
a little app to gzip+base64 encode and decode

GO=GZIP64 A little golang console utility that reads a file and either: 1) Encodes it - gzip compress followed by base64 encode writes

Steve White 1 Oct 16, 2021
Parallel implementation of Gzip for modern multi-core machines written in Go

gzip Parallel implementation of gzip for modern multi-core machines written in Go Usage: gzip [OPTION]... [FILE] Compress or uncompress FILE (by defau

Pedro Albanese 0 Nov 16, 2021
Ripgrep but for gzip-compressed files over http

Juicer It's ripgrep but for Gzip-compressed files over HTTP! This tool was primarily designed to scan thru the Common Crawl dataset for URLs without s

Boom 2 Feb 21, 2022
Gin-errorhandling - Gin Error Handling Middleware is a middleware for the popular Gin framework

Gin Error Handling Middleware Gin Error Handling Middleware is a middleware for

Joseph Woodward 9 Sep 19, 2022
This package provides json web token (jwt) middleware for goLang http servers

jwt-auth jwt auth middleware in goLang. If you're interested in using sessions, checkout my sessions library! README Contents: Quickstart Performance

Adam Hanna 224 Dec 5, 2022
:closed_lock_with_key: Middleware for keeping track of users, login states and permissions

Permissions2 Middleware for keeping track of users, login states and permissions. Online API Documentation godoc.org Features and limitations Uses sec

Alexander F. Rødseth 470 Dec 31, 2022
A dead simple, highly performant, highly customizable sessions middleware for go http servers.

If you're interested in jwt's, see my jwt library! Sessions A dead simple, highly performant, highly customizable sessions service for go http servers

Adam Hanna 70 Dec 19, 2022
Golang telegram bot API wrapper, session-based router and middleware

go-tgbot Pure Golang telegram bot API wrapper generated from swagger definition, session-based routing and middlewares. Usage benefits No need to lear

Oleg Lebedev 118 Nov 16, 2022