Rolling writer is an IO util for auto rolling write in go.

Overview

RollingWriter Build Status Go Report Card GoDoc codecov Awesome

RollingWriter is an auto rotate io.Writer implementation. It can works well with logger.

Awesome Go popular log helper

New Version v2.0 is comming out! Much more Powerfull and Efficient. Try it by follow the demo

RollingWriter contains 2 separate patrs:

  • Manager: decide when to rotate the file with policy. RlingPolicy give out the rolling policy

    • WithoutRolling: no rolling will happen
    • TimeRolling: rolling by time
    • VolumeRolling: rolling by file size
  • IOWriter: impement the io.Writer and do the io write

    • Writer: not parallel safe writer
    • LockedWriter: parallel safe garented by lock
    • AsyncWtiter: parallel safe async writer
    • BufferWriter: merge serval write into one file.Write()

Features

  • Auto rotate with multi rotate policies
  • Implement go io.Writer, provide parallel safe writer
  • Max remain rolling files with auto cleanup
  • Easy for user to implement your manager

Benchmark

goos: darwin
goarch: amd64
pkg: github.com/arthurkiller/rollingWriter
BenchmarkWrite-4                          300000              5952 ns/op               0 B/op          0 allocs/op
BenchmarkParallelWrite-4                  200000              7846 ns/op               0 B/op          0 allocs/op
BenchmarkAsynWrite-4                      200000              7917 ns/op           16324 B/op          1 allocs/op
BenchmarkParallelAsynWrite-4              200000              8632 ns/op           12513 B/op          1 allocs/op
BenchmarkLockedWrite-4                    200000              5829 ns/op               0 B/op          0 allocs/op
BenchmarkParallelLockedWrite-4            200000              7796 ns/op               0 B/op          0 allocs/op
BenchmarkBufferWrite-4                    200000              6943 ns/op            1984 B/op          4 allocs/op
BenchmarkParallelBufferWrite-4           1000000              1026 ns/op            7129 B/op          1 allocs/op
PASS
ok      github.com/arthurkiller/rollingWriter   14.867s

Quick Start

	writer, err := rollingwriter.NewWriterFromConfig(&config)
	if err != nil {
		panic(err)
	}

	writer.Write([]byte("hello, world"))

Want more? View demo for more details.

Any suggestion or new feature inneed, please put up an issue

Comments
  • 请问master代码是否可用?

    请问master代码是否可用?

    跑了一个小例子:

    config := rollingwriter.Config{
    	LogPath:           "./log",
    	TimeTagFormat:     "060102150405",
    	FileName:          "test",
    	MaxRemain:         5,
    	RollingPolicy:     rollingwriter.VolumeRolling,
    	RollingVolumeSize: "2K",
    	Asynchronous:      true,
    }
    writer, err := rollingwriter.NewWriterFromConfig(&config)
    if err != nil {
    	// 应该处理错误
    	panic(err)
    }
    wg := sync.WaitGroup{}
    wg.Add(1)
    go func() {
    	for {
    		bf := []byte(fmt.Sprintf("rollingwriter test write gorounter time %v\n", time.Now()))
    		writer.Write(bf)
    	}
    }()
    wg.Wait()
    

    运行了一段时间,输出如下,也没有按照2k切分: default 里面的内容也是不可读的

    opened by spiderpoison 18
  • 配合 zap 使用,切分时报错

    配合 zap 使用,切分时报错

    在配合 zap 使用时,当要切分时,出现以下错误 write error: rename log/logs.log log/logs.log.201905111642: The process cannot access the file because it is being used by another process.

    从提示上看,当要切分时,log 文件被占用了,不能重命名

    我构建 zap logger 代码如下: ` // 基本上与 demo 例子相同的配置进行构建的 RollingWriter 实例 rw, _ := newRollingWriter(&config)

    // 基于 RollingWriter 构建 *zap.Logger
    ws := zapcore.AddSync(rw)
    core := zapcore.NewCore(
    	zapcore.NewJSONEncoder(zap.NewProductionEncoderConfig()),
    	ws,
    	zap.InfoLevel,
    )
    
    logger = zap.New(core)`
    

    ps. rollingWriter 代码获取的是 master 分支

    help wanted wontfix 
    opened by liudaoping 16
  • 更新写入非追加写入!!!

    更新写入非追加写入!!!

    go env:1.13.1 go.mod:

    require (
    	github.com/arthurkiller/rollingwriter v1.1.1
    	github.com/gin-contrib/logger v0.0.1
    	github.com/gin-gonic/gin v1.4.0
    	github.com/gobuffalo/envy v1.7.1
    	github.com/rs/zerolog v1.9.1
    )
    
    replace github.com/ugorji/go v1.1.4 => github.com/ugorji/go/codec v0.0.0-20190204201341-e444a5086c43
    
    config := &rollingwriter.Config{
    		LogPath:       "./storage/logs",        //日志路径
    		TimeTagFormat: "20060102", //时间格式串
    		FileName:      "test",         //日志文件名
    		MaxRemain:     5,              //配置日志最大存留数
    
    		// 目前有2中滚动策略: 按照时间滚动按照大小滚动
    		// - 时间滚动: 配置策略如同 crontable, 例如,每天0:0切分, 则配置 0 0 0 * * *
    		// - 大小滚动: 配置单个日志文件(未压缩)的滚动大小门限, 入1G, 500M
    		RollingPolicy:      rollingwriter.TimeRolling, //配置滚动策略 norolling timerolling volumerolling
    		RollingTimePattern: "* * * * * *",             //配置时间滚动策略
    		RollingVolumeSize:  "2k",                      //配置截断文件下限大小
    
    		// writer 支持4种不同的 mode:
    		// 1. none 2. lock
    		// 3. async 4. buffer
    		// - 无保护的 writer: 不提供并发安全保障
    		// - lock 保护的 writer: 提供由 mutex 保护的并发安全保障
    		// - 异步 writer: 异步 write, 并发安全. 异步开启后忽略 Lock 选项
    		WriterMode: "lock",
    		// BufferWriterThershould in B
    		BufferWriterThershould: 8 * 1024 * 1024,
    		// Compress will compress log file with gzip
    		Compress: false,
    	}
    
    
    	writer ,_:= rollingwriter.NewWriterFromConfig(config)
    
            logger = zerolog.New(writer).With().Timestamp().Logger()
    
           router.GET("/ping", func(c *gin.Context) {
    		logger.Info().Msg("test")
    		c.String(200, "pong "+fmt.Sprint(time.Now().Unix()))
    	})
    
    

    不断刷新 /ping

    {"level":"info","time":"2019-10-30T19:29:14+08:00","message":"test"} //仅有此
    

    test.log 里只有会一条数据,日志内容的时间戳会不断的更新,判断你写入文件的时候使用的是覆盖

    help wanted 
    opened by hookover 9
  • data race in manager

    data race in manager

    1. env

    ▶ go version
    go version go1.12.4 darwin/amd64
    ---
    GoLand 2019.1.1
    Build #GO-191.6707.68, built on April 17, 2019
    JRE: 1.8.0_202-release-1483-b44 x86_64
    JVM: OpenJDK 64-Bit Server VM by JetBrains s.r.o
    macOS 10.14.4
    

    2. 复现步骤

    go run -race demo/writer.go
    

    3. 结果

    WARNING: DATA RACE
    Read at 0x00c00009e0a8 by goroutine 25:
      github.com/arthurkiller/rollingWriter.(*manager).GenLogFileName()
          /Users/chaomai/Documents/workspace/github/rollingWriter/manager.go:140 +0x88
      github.com/arthurkiller/rollingWriter.NewManager.func1()
          /Users/chaomai/Documents/workspace/github/rollingWriter/manager.go:43 +0x53
      github.com/robfig/cron.FuncJob.Run()
          /Users/chaomai/Documents/workspace/go/pkg/mod/github.com/robfig/[email protected]/cron.go:92 +0x34
      github.com/robfig/cron.(*Cron).runWithRecovery()
          /Users/chaomai/Documents/workspace/go/pkg/mod/github.com/robfig/[email protected]/cron.go:165 +0x68
    
    Previous write at 0x00c00009e0a8 by goroutine 22:
      github.com/arthurkiller/rollingWriter.(*manager).GenLogFileName()
          /Users/chaomai/Documents/workspace/github/rollingWriter/manager.go:145 +0x243
      github.com/arthurkiller/rollingWriter.NewManager.func1()
          /Users/chaomai/Documents/workspace/github/rollingWriter/manager.go:43 +0x53
      github.com/robfig/cron.FuncJob.Run()
          /Users/chaomai/Documents/workspace/go/pkg/mod/github.com/robfig/[email protected]/cron.go:92 +0x34
      github.com/robfig/cron.(*Cron).runWithRecovery()
          /Users/chaomai/Documents/workspace/go/pkg/mod/github.com/robfig/[email protected]/cron.go:165 +0x68
    
    Goroutine 25 (running) created at:
      github.com/robfig/cron.(*Cron).run()
          /Users/chaomai/Documents/workspace/go/pkg/mod/github.com/robfig/[email protected]/cron.go:199 +0xa65
    
    Goroutine 22 (finished) created at:
      github.com/robfig/cron.(*Cron).run()
          /Users/chaomai/Documents/workspace/go/pkg/mod/github.com/robfig/[email protected]/cron.go:199 +0xa65
    
    

    4. 分析

    初步看了,(*manager).GenLogFileName()最终是由github.com/robfig/cron/cron.go执行的,可见cron中并无无任何同步机制。

    极端情况下,如果(*manager).GenLogFileName()的执行时间长于调度间隔,那同一时刻就会有两个goroutine在执行(*manager).GenLogFileName()m.startAt就可能会有问题了。

    5. 解决

    type manager struct {
    	// ...
    	lock          sync.Mutex
    }
    
    // ...
    
    // GenLogFileName generate the new log file name, filename should be absolute path
    func (m *manager) GenLogFileName(c *Config) (filename string) {
    	m.lock.Lock()
    	// ...
    	m.lock.Lock()
    	// reset the start time to now
    	m.startAt = time.Now()
    	return
    }
    
    bug question 
    opened by chaomai 8
  • Fix operations with open files

    Fix operations with open files

    Found a bug when using it on windows: renaming fails because the file is open. I modified the code to only do renaming and removing operations with closed files to avoid these issues on windows. Also, there are some auto-fixes (just 3 space deletes and moving one defer) by the IDE that are nice IMO :)

    Closes: #40

    Signed-off-by: Guillem Bonet [email protected]

    opened by Guillembonet 7
  • fix data race in manager and writer, update writer_test v2

    fix data race in manager and writer, update writer_test v2

    修复了上次提到的几个问题。

    但是遇到了一个新的问题,开启compress的情况下,会随机出现压缩后文件为空的情况。暂时没有找到是什么原因导致的,还请帮忙看一下。

    我移除return os.Remove(cmpname + ".tmp") // remove *.log.tmp file后测试,确认了原文件是正常的。

    如果不传递oldfile指针,在CompressFile中打开原文件,这样不会出现压缩后文件为空的情况。

    opened by chaomai 6
  • Dead lock detected?

    Dead lock detected?

    Thanks for the great package! We use it with zerolog, and it works well, however we met a bug recently, large amount of goroutines hang at writing log, all with the following stack trace:

    (dlv) bt
     0  0x0000000000439405 in runtime.gopark
        at go/src/runtime/proc.go:337
     1  0x000000000044a385 in runtime.goparkunlock
        at go/src/runtime/proc.go:342
     2  0x000000000044a385 in runtime.semacquire1
        at go/src/runtime/sema.go:144
     3  0x0000000000468967 in sync.runtime_SemacquireMutex
        at go/src/runtime/sema.go:71
     4  0x00000000004721e5 in sync.(*Mutex).lockSlow
        at go/src/sync/mutex.go:138
     5  0x0000000000b4ee0b in sync.(*Mutex).Lock
        at go/src/sync/mutex.go:81
     6  0x0000000000b4ee0b in github.com/arthurkiller/rollingwriter.(*LockedWriter).Write
        at go-mod-cache/github.com/arthurkiller/[email protected]/writer.go:254
     7  0x0000000000695703 in github.com/rs/zerolog.levelWriterAdapter.WriteLevel
        at go-mod-cache/github.com/rs/[email protected]/writer.go:20
     8  0x0000000000695703 in github.com/rs/zerolog.(*levelWriterAdapter).WriteLevel
        at <autogenerated>:1
     9  0x0000000000689625 in github.com/rs/zerolog.(*Event).write
        at go-mod-cache/github.com/rs/[email protected]/event.go:76
    10  0x0000000000689b1b in github.com/rs/zerolog.(*Event).msg
        at go-mod-cache/github.com/rs/[email protected]/event.go:140
    11  0x0000000000b39f70 in github.com/rs/zerolog.(*Event).Msg
        at go-mod-cache/github.com/rs/[email protected]/event.go:106
    ...
    

    I'm not familiar with the code, however, after a quick search, I found the following suspicious function:

    func (w *LockedWriter) Write(b []byte) (n int, err error) {
    	w.Lock()
    	select {
    	case filename := <-w.fire:
    		if err := w.Reopen(filename); err != nil {
    			return 0, err                      // <-- here
    		}
    	default:
    	}
    	n, err = w.file.Write(b)
    	w.Unlock()
    	return
    }
    

    I don't know if there is a chance that the function returns at the comment line, leaves the lock acquired without release. Is this the root cause of the bug above? Any help is appreciated, thanks.

    opened by sillykelvin 5
  • goroutne leak

    goroutne leak

    图片

    当 RollingPolicy 为 VolumeRolling 时你启动了一个 goroutine, 然后我没有人关闭它, 调 writer 的 Close() 时不能关闭这个 goroutine

    而且这个 goroutine 中的 timer 就算是 goroutine 退出了,也没有有停止这个 Ticker

    opened by mei-rune 4
  • 当长时间没有写日志操作,就会导致Writer中fire阻塞,后续写日志的时候,容易归档到错误的文件里面

    当长时间没有写日志操作,就会导致Writer中fire阻塞,后续写日志的时候,容易归档到错误的文件里面

    在基于TimeRolling和VolumeRolling 切割的时候,容易阻塞

    switch c.RollingPolicy {
    	default:
    		fallthrough
    	case WithoutRolling:
    		return m, nil
    	case TimeRolling:
    		if err := m.cr.AddFunc(c.RollingTimePattern, func() {
    			m.fire <- m.GenLogFileName(c) // 长时间没有写日志,会阻塞
    		}); err != nil {
    			return nil, err
    		}
    		m.cr.Start()
    	case VolumeRolling:
    		m.ParseVolume(c)
    		m.wg.Add(1)
    		go func() {
    			timer := time.Tick(time.Duration(Precision) * time.Second)
    			filepath := LogFilePath(c)
    			var file *os.File
    			var err error
    			m.wg.Done()
    
    			for {
    				select {
    				case <-m.context:
    					return
    				case <-timer:
    					if file, err = os.Open(filepath); err != nil {
    						continue
    					}
    					if info, err := file.Stat(); err == nil && info.Size() > m.thresholdSize {
    						m.fire <- m.GenLogFileName(c) // 长时间没有写日志,会阻塞
    					}
    					file.Close()
    				}
    			}
    		}()
    		m.wg.Wait()
    	}
    	return m, nil
    

    针对VolumeRolling而言,一旦阻塞,会导致日志归档到错误的文件中。我觉得可以通过如下方式来修复一下:

    func (w *LockedWriter) Write(b []byte) (n int, err error) {
    	w.Lock()
    DoWrite:// 解决fire阻塞带来的文件错位问题
    	select {
    	case filename := <-w.fire:
    		if err := w.Reopen(filename); err != nil {
    			return 0, err
    		}
    		goto DoWrite
    	default:
    	}
    	n, err = w.file.Write(b)
    	w.Unlock()
    	return
    }
    
    bug investigate 
    opened by dequanLi 3
  • Buffer writer concurrency unsafe

    Buffer writer concurrency unsafe

    https://github.com/arthurkiller/rollingwriter/blob/master/writer.go#L365

    func (w *BufferWriter) Write(b []byte) (int, error) {
    	// ...
    	buf := append(*w.buf, b...)
    	atomic.StorePointer((*unsafe.Pointer)(unsafe.Pointer(&w.buf)), (unsafe.Pointer)(&buf))
    

    Hi,我使用到了贵包rollingwriter的buffer writer,测试时候发现会有丢日志的情况(高并发场景)。初步分析发现应该是这里的buffer append 非并发安全导致的。

    比如buffer里已有 line1,并发两个Write,分别写入 line2、line3,竞争激烈的情况下,StorePointer可能会出现 [line1, line3] 覆盖 [line1, line2],从而导致line2丢失。

    测试时加上了给这行代码加了个mutex or trylock,发现可以解决丢日志的问题。这个问题也希望跟 @arthurkiller 作者大大确认下有无理解上的偏差。

    From

    buf := append(*w.buf, b...)
    atomic.StorePointer((*unsafe.Pointer)(unsafe.Pointer(&w.buf)), (unsafe.Pointer)(&buf))
    

    To

    w.lockBuf.Lock()
    *(w.buf) = append(*w.buf, b...)
    w.lockBuf.Unlock()
    

    另,spinlock简单实现

    
    type Locker struct {
    	lock uintptr
    }
    
    func (l *Locker) Lock() {
    	for !atomic.CompareAndSwapUintptr(&l.lock, 0, 1) {
    		runtime.Gosched()
    	}
    }
    
    func (l *Locker) Unlock() {
    	atomic.StoreUintptr(&l.lock, 0)
    }
    
    opened by koveango 3
  • Error when renaming file

    Error when renaming file

    Hello,

    I use the package on windows with go version 1.16. If I execute the following code, I get an error:

    config := rollingwriter.NewDefaultConfig()
    config.RollingPolicy = rollingwriter.VolumeRolling
    config.RollingVolumeSize = "10M"
    w, _ := rollingwriter.NewWriterFromConfig(&config)
    
    for i := 0; i < 5000000; i++ {
      _, err := w.Write([]byte(fmt.Sprintf("This is a test %d", i)))
      if err != nil {
        fmt.Println(err)
        break
      }
    }
    

    The printed error is: "rename log/log.log log/log.log.202104121537: The process cannot access the file because it is being used by another process"

    If I look deeper into the returned error, the inner errors is golang.org/x/sys/windows.ERROR_SHARING_VIOLATION (32).

    Is this a known error and is there a workaround?

    Thank you and best regards, Niklas

    opened by nfoecking 2
  • Why we need _asyncBufferPool for async writer?

    Why we need _asyncBufferPool for async writer?

    We already have []byte allocated, since the signature of Write([]byte) method, why use _asyncBufferPool? It add additional copy which seems unnecessary?

    opened by mm2175 2
  • FileName use template string

    FileName use template string

    For FileName, consider using template strings, such as foo-${date}.log.gz, which makes the file suffix reasonable. It's strange to force a TimeTag spliced to the end of FileName. Last, thanks for your components, which helped me a lot~

    opened by frank-yf 2
Releases(v1.1.2.1)
Owner
Arthur Lee
Arthur Lee
Lumberjack is a Go package for writing logs to rolling files.

Lumberjack is a Go package for writing logs to rolling files.

Go微服务 4 Feb 24, 2022
Time based rotating file writer

cronowriter This is a simple file writer that it writes message to the specified format path. The file path is constructed based on current date and t

Yuta UEKUSA 50 Sep 26, 2022
Rotating File Writer

Rotating File writer An io.writer & io.Closer compliant file writer that will always write to the path that you give it, even if somebody deletes/rena

Michael Pearson 34 Jul 12, 2021
Write log entries, get X-Ray traces.

logtoxray Write to logs, get X-Ray traces. No distributed tracing instrumenation library required. ?? ?? ?? THIS PROJECT IS A WORK-IN-PROGRESS PROTOTY

JBD 27 Apr 24, 2022
The AlfheimDB's high performance write-ahead log.

The AlfheimDB's high performance write-ahead log.

chunming.dong 5 Jul 18, 2022
Git-auto-push - Auto commit and push to github repositories

Auto commit and push to github repositories. How to use git clone https://github

Z.Q.K 4 Oct 30, 2022
Write controller-runtime based k8s controllers that read/write to git, not k8s

Git Backed Controller The basic idea is to write a k8s controller that runs against git and not k8s apiserver. So the controller is reading and writin

Darren Shepherd 50 Dec 10, 2021
lumberjack is a log rolling package for Go

lumberjack Lumberjack is a Go package for writing logs to rolling files. Package lumberjack provides a rolling logger. Note that this is v2.0 of lumbe

Nate Finch 3.7k Dec 2, 2022
The utility that created for easily database and their tables rolling up

The utility that created for easily database and their tables rolling up

null 0 Nov 6, 2021
Lumberjack is a Go package for writing logs to rolling files.

Lumberjack is a Go package for writing logs to rolling files.

Go微服务 4 Feb 24, 2022
A Rolling Window Demo

模仿Hystrix实现一个滑动窗口计数器 需求 参考 Hystrix 实现一个滑动窗口计数器 实现思路 实习滑动窗口的核心代码位置是 /pkg/hystrix 定义一个放置请求结果的对象Bucket(/pkg/hystrix/bucket.go) 定义一个滑动窗口对象RollingWindown(/

Yuki Chen 1 Jun 8, 2022
Calculate slope and r2 in your dataset for a defined rolling window

Hit The Slopes Calculate slope and r2 in your dataset for a defined rolling window. This is useful if you want to see the evolution of trends in your

Tom Larsen 0 Feb 9, 2022
FSManager - Tree view Simple util to displays the directory structure of a path or of the disk in a drive graphically.

FSManager - Tree view Simple util to displays the directory structure of a path or of the disk in a drive graphically. If you don't specify a drive or

Kostiantyn Denysov 1 Oct 9, 2021
redis-util business-friendly encapsulation of redis operations, such as the common cache set get operation

redis-util 方便业务使用的redis操作封装,比如常见的缓存set get操作, 一行代码搞定,不像开源库需要写好多行 使用方法

connor 1 Oct 22, 2021
Magic util that "bridges" Wireguard with OpenVPN without a TUN/TAP interface

wg-ovpn Magic util that "bridges" Wireguard with OpenVPN without a TUN/TAP interface Warning: really ugly and unstable code! Building Obtain latest so

Patrycja 7 Sep 27, 2022
A great util to format you git commit message!

A great util to format you git commit message!

常乐村喵蕉君 0 Dec 2, 2021
Simple CLI util for running OCR on images through PERO OCR API

pero_ocr Simple CLI util for running OCR on images through PERO OCR API Usage: Usage of batch_pero_ocr: -c string cancel request with given

Moravian Library in Brno 2 Dec 1, 2021
A command line util created to simulate an alien invasion described in the task

Alien Invasion Simulation A command line util created to simulate an alien invasion described in the task. Requirements Go 1.17 Graphviz (optional) Us

Artur Mruk 0 Nov 24, 2021
A comprehensive, efficient, and reusable util function library of go.

Lancet Lancet is a comprehensive, efficient, and reusable util function library of go. Inspired by the java apache common package and lodash.js. Engli

Beyond 1.9k Dec 2, 2022
Go cmd util that prints cmd-line args with their index

echoidx This is an exercise of the book The Go Programming Language, by Alan A.

Santiago Rodriguez 0 Dec 18, 2021