Port of perl5 File::RotateLogs to Go

Overview

file-rotatelogs

Provide an io.Writer that periodically rotates log files from within the application. Port of File::RotateLogs from Perl to Go.

Build Status

GoDoc

SYNOPSIS

import (
  "log"
  "net/http"

  apachelog "github.com/lestrrat-go/apache-logformat"
  rotatelogs "github.com/lestrrat-go/file-rotatelogs"
)

func main() {
  mux := http.NewServeMux()
  mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { ... })

  logf, err := rotatelogs.New(
    "/path/to/access_log.%Y%m%d%H%M",
    rotatelogs.WithLinkName("/path/to/access_log"),
    rotatelogs.WithMaxAge(24 * time.Hour),
    rotatelogs.WithRotationTime(time.Hour),
  )
  if err != nil {
    log.Printf("failed to create rotatelogs: %s", err)
    return
  }

  // Now you must write to logf. apache-logformat library can create
  // a http.Handler that only writes the approriate logs for the request
  // to the given handle
  http.ListenAndServe(":8080", apachelog.CombinedLog.Wrap(mux, logf))
}

DESCRIPTION

When you integrate this to into your app, it automatically write to logs that are rotated from within the app: No more disk-full alerts because you forgot to setup logrotate!

To install, simply issue a go get:

go get github.com/lestrrat-go/file-rotatelogs

It's normally expected that this library is used with some other logging service, such as the built-in log library, or loggers such as github.com/lestrrat-go/apache-logformat.

import(
  "log"
  "github.com/lestrrat-go/file-rotatelogs"
)

func main() {
  rl, _ := rotatelogs.New("/path/to/access_log.%Y%m%d%H%M")

  log.SetOutput(rl)

  /* elsewhere ... */
  log.Printf("Hello, World!")
}

OPTIONS

Pattern (Required)

The pattern used to generate actual log file names. You should use patterns using the strftime (3) format. For example:

  rotatelogs.New("/var/log/myapp/log.%Y%m%d")

Clock (default: rotatelogs.Local)

You may specify an object that implements the roatatelogs.Clock interface. When this option is supplied, it's used to determine the current time to base all of the calculations on. For example, if you want to base your calculations in UTC, you may specify rotatelogs.UTC

  rotatelogs.New(
    "/var/log/myapp/log.%Y%m%d",
    rotatelogs.WithClock(rotatelogs.UTC),
  )

Location

This is an alternative to the WithClock option. Instead of providing an explicit clock, you can provide a location for you times. We will create a Clock object that produces times in your specified location, and configure the rotatelog to respect it.

LinkName (default: "")

Path where a symlink for the actual log file is placed. This allows you to always check at the same location for log files even if the logs were rotated

  rotatelogs.New(
    "/var/log/myapp/log.%Y%m%d",
    rotatelogs.WithLinkName("/var/log/myapp/current"),
  )
  // Else where
  $ tail -f /var/log/myapp/current

Links that share the same parent directory with the main log path will get a special treatment: namely, linked paths will be RELATIVE to the main log file.

Main log file name Link name Linked path
/path/to/log.%Y%m%d /path/to/log log.YYYYMMDD
/path/to/log.%Y%m%d /path/to/nested/log ../log.YYYYMMDD
/path/to/log.%Y%m%d /foo/bar/baz/log /path/to/log.YYYYMMDD

If not provided, no link will be written.

RotationTime (default: 86400 sec)

Interval between file rotation. By default logs are rotated every 86400 seconds. Note: Remember to use time.Duration values.

  // Rotate every hour
  rotatelogs.New(
    "/var/log/myapp/log.%Y%m%d",
    rotatelogs.WithRotationTime(time.Hour),
  )

MaxAge (default: 7 days)

Time to wait until old logs are purged. By default no logs are purged, which certainly isn't what you want. Note: Remember to use time.Duration values.

  // Purge logs older than 1 hour
  rotatelogs.New(
    "/var/log/myapp/log.%Y%m%d",
    rotatelogs.WithMaxAge(time.Hour),
  )

RotationCount (default: -1)

The number of files should be kept. By default, this option is disabled.

Note: MaxAge should be disabled by specifing WithMaxAge(-1) explicitly.

  // Purge logs except latest 7 files
  rotatelogs.New(
    "/var/log/myapp/log.%Y%m%d",
    rotatelogs.WithMaxAge(-1),
    rotatelogs.WithRotationCount(7),
  )

Handler (default: nil)

Sets the event handler to receive event notifications from the RotateLogs object. Currently only supported event type is FiledRotated

  rotatelogs.New(
    "/var/log/myapp/log.%Y%m%d",
    rotatelogs.WithHandler(rotatelogs.HandlerFunc(func(e rotatelogs.Event) {
      if e.Type() != rotatelogs.FileRotatedEventType {
        return
      }

      // Do what you want with the data. This is just an idea:
      storeLogFileToRemoteStorage(e.(*rotatelogs.FileRotatedEvent).PreviousFile())
    })),
  )

ForceNewFile

Ensure a new file is created every time New() is called. If the base file name already exists, an implicit rotation is performed.

  rotatelogs.New(
    "/var/log/myapp/log.%Y%m%d",
    rotatelogs.ForceNewFile(),
  )

Rotating files forcefully

If you want to rotate files forcefully before the actual rotation time has reached, you may use the Rotate() method. This method forcefully rotates the logs, but if the generated file name clashes, then a numeric suffix is added so that the new file will forcefully appear on disk.

For example, suppose you had a pattern of '%Y.log' with a rotation time of 86400 so that it only gets rotated every year, but for whatever reason you wanted to rotate the logs now, you could install a signal handler to trigger this rotation:

rl := rotatelogs.New(...)

signal.Notify(ch, syscall.SIGHUP)

go func(ch chan os.Signal) {
  <-ch
  rl.Rotate()
}()

And you will get a log file name in like 2018.log.1, 2018.log.2, etc.

Issues
  • set link to relative path file

    set link to relative path file

    logf := rotatelogs.New(
    		"/var/log/xxx/runtime.%Y%m%d",
    		rotatelogs.WithLinkName("/var/log/xxx/runtime"), 
    		rotatelogs.WithRotationTime(24*time.Hour),
    		rotatelogs.WithMaxAge(24*time.Hour*7),
    	)
    

    if my go app run in docker,with volume mount like this:

    volumes:
     - hostPath:
          path: /var/lib/k8s/log/xxx
          type: File
        name: log
    volumeMounts:
    - mountPath: /var/log/xxx
      name: log
    

    and when i go to the container's hypervisor, going to the path /var/lib/k8s/log/xxx, link file runtime cannot open(error: No such file or directory.) since the link file linked /var/log/xxx/runtime.%Y%m%d which is in the container path not the hypervisor path.

    So we need make the link file link to relative path. And I see the k8s log link file also link to relative path.

    kube-apiserver.INFO -> kube-apiserver.MASTER1.root.log.INFO.20200814-101251.2053
    
    
    opened by shiyan2016 11
  • Support rotationCount option

    Support rotationCount option

    It is useful if count of rotation option is also supported. To keep compatibility, maxAge is still required option. rotationCount overrides maxAge option if it is specified.

    opened by kenhys 5
  • Fix handling of MaxAge/RotationCount

    Fix handling of MaxAge/RotationCount

    fixes #16

    • New method, "Rotate" is introduced.
    • MaxAge default is now "time.Duration(0)", which is the same as disabled
    • RotationCount default is now "0", which is the same as disabled
    • RotationCount is now uint, instead of int
    • options are now name/value pairs, instead of doing silly method calls to Configure
    opened by lestrrat 4
  • Create directory if it does not already exist

    Create directory if it does not already exist

    Case:

            logf, err := rotatelogs.New(
                    "./foo/bar.log.%Y%m%d%H",
                    rotatelogs.WithLinkName("./foo/bar.log"),
                    rotatelogs.WithMaxAge(24*time.Hour),
                    rotatelogs.WithRotationTime(time.Hour),
            )
    

    and when the ./foo directory isn't existed, error occur!

    open ./foo/bar.log.2018101616: no such file or directory
    

    We must ensure that the dir is existed by doing so can we use os.OpenFile to create or append the filename

    opened by caibirdme 2
  • Increase the presence of a detection file when writing a log

    Increase the presence of a detection file when writing a log

    After the program is started, the log files are deleted manually, but no files are written. So, Increase the presence of a detection file when writing a log

    opened by w2m 2
  • After a forced generational rotate, ensure new writes go to the new file.

    After a forced generational rotate, ensure new writes go to the new file.

    Keep track of the base filename for log files, so when a forced generational rotate is performed, the existing log file becomes a new version and subsequent writes go to the new file.

    opened by bobmcn 1
  • fix: set default maxAge by 7 days

    fix: set default maxAge by 7 days

    In the current version, default maxAge as 0 will failed when rotate:

    Soure:

    rl, _ := rotatelogs.NewRotateLogs("/path/to/access_log.%Y%m%d%H%M")
    

    Output:

    failed to rotate: maxAge not set, not rotating
    

    Breaking line:

    https://github.com/lestrrat/go-file-rotatelogs/blob/master/rotatelogs.go#L237

    Thie PR will fix this by add a default value in New method.

    opened by liubin 1
  • tebeka/strftime from Github instead of Bitbucket

    tebeka/strftime from Github instead of Bitbucket

    @lestrrat

    I find that Miki Tebeka had already imported his projects into Github

    and the strftime for golang is here: https://github.com/tebeka/strftime

    it would be easier for people to install rotatelogs without install the HG

    opened by qindj 1
  • README: Fix the code example to use the v2 API.

    README: Fix the code example to use the v2 API.

    What is this patch?

    For now, the code example in README.md does not work because it relies on the old API (which has been deprecated since v2.0.0).

    This small patch fixes this issue by migrating the code to use the newer interface. With this patch applied, the code example will work fine.

    opened by fujimotos 0
  • Use lestrrat/go-strftime

    Use lestrrat/go-strftime

    BACKWARDS INCOMPATIBLE CHANGE! New() now returns an object AND an error.

    Otherwise, the file generation is 3~4x faster, and you no longer have to install hg to install this library. Your welcome :D

    opened by lestrrat 0
Releases(v2.4.0)
  • v2.4.0(Sep 8, 2020)

    v2.4.0 - 8 Sep 2020
      * Add WithRotationSize option to specify log ration when
        a certain file size is reached. (NOTE: this does not guarantee
        that the file size is exactly the size specified, but that it
        *exceeds* the specified size)
    
    Source code(tar.gz)
    Source code(zip)
  • v2.2.0(Jun 7, 2018)

    • Added RotationCount
    • When files are rotated and the name of the new file clashes, we will attempt to create a unique name
    • Semantics of WithLocation was wrong, because time.Truncate worked its magic in UTC time.
    • Added Handler, and FileRotatedEvent
    Source code(tar.gz)
    Source code(zip)
Owner
@lestrrat 's Go projects
null
Parse awesome-go README file and generate a new README file with repo info.

Awesome Go Extra All data are from awesome-go and GitHub API. Audio and Music Libraries for manipulating audio. Name Description Star Open Issues Crea

Wendell Sun 21 Jun 22, 2022
Time based rotating file writer

cronowriter This is a simple file writer that it writes message to the specified format path. The file path is constructed based on current date and t

Yuta UEKUSA 47 Feb 8, 2022
A simple logging module for go, with a rotating file feature and console logging.

A simple logging module for go, with a rotating file feature and console logging. Installation go get github.com/jbrodriguez/mlog Usage Sample usage W

Juan B. Rodriguez 25 Jun 11, 2022
Rotating File Writer

Rotating File writer An io.writer & io.Closer compliant file writer that will always write to the path that you give it, even if somebody deletes/rena

Michael Pearson 34 Jul 12, 2021
A hack example to servce file as a webserver with a Google Cloud Functions

Overview This project is a hack example to servce file as a webserver with a Google Cloud Functions (normally single purpose, only one entry point). I

guillaume blaquiere 0 Nov 26, 2021
Goimportcycle - a tool to visualize Go imports resolved to the file level

Go Import Cycle goimportcycle is a tool to visualize Go imports resolved to the

Sam L 28 Apr 24, 2022
Filez - A tiny package showing you File info

filez A tiny package showing you File info Install go get -v github.com/Cne3Rd/f

Mukaila Samsondeen 1 Feb 4, 2022
The full power of the Go Compiler directly in your browser, including a virtual file system implementation. Deployable as a static website.

Static Go Playground Features Full Go Compiler running on the browser. Supports using custom build tags. Incremental builds (build cache). Supports mu

null 25 Jun 16, 2022
A little websocket TCP proxy to let browsers talk to a fixed port on arbitrary hosts. Built for Gemini (gemini://, port 1965)

Kepler A little websocket TCP proxy built to let Amfora talk to Gemini servers when running in a browser. Usage $ git clone https://github.com/awfulco

mooff 1 May 27, 2022
Port-proxy - Temporary expose port for remote connections

Port proxy util Temporary expose port for remote connections. E.g. database/wind

Vadym S 1 Jan 27, 2022
This is a simple file storage server. User can upload file, delete file and list file on the server.

Simple File Storage Server This is a simple file storage server. User can upload file, delete file and list file on the server. If you want to build a

BH_Lin 0 Jan 19, 2022
Lima launches Linux virtual machines on macOS, with automatic file sharing, port forwarding, and containerd.

Lima: Linux-on-Mac ("macOS subsystem for Linux", "containerd for Mac")

Akihiro Suda 8.4k Jun 26, 2022
A feature flag solution, with only a YAML file in the backend (S3, GitHub, HTTP, local file ...), no server to install, just add a file in a central system and refer to it. 🎛️

??️ go-feature-flag A feature flag solution, with YAML file in the backend (S3, GitHub, HTTP, local file ...). No server to install, just add a file i

Thomas Poignant 447 Jun 29, 2022
A Go port of Ruby's dotenv library (Loads environment variables from `.env`.)

GoDotEnv A Go (golang) port of the Ruby dotenv project (which loads env vars from a .env file) From the original Library: Storing configuration in the

John Barton 5k Jun 27, 2022
Port of LZ4 lossless compression algorithm to Go

go-lz4 go-lz4 is port of LZ4 lossless compression algorithm to Go. The original C code is located at: https://github.com/Cyan4973/lz4 Status Usage go

Бранимир Караџић 209 Jun 14, 2022
Port of Google's Keyczar cryptography library to Go

Important note: Keyczar is deprecated. The Keyczar developers recommend Tink. This is a port of Google's Keyczar library to Go. Copyright (c) 2011 Dam

Damian Gryski 111 Apr 16, 2022
Port of webcolors library from Python to Go

go-webcolors A library for working with color names and color value formats defined by the HTML and CSS specifications for use in documents on the Web

Jyotiska Khasnabish 26 Jan 23, 2022
Port of D. J. Bernstein's primegen prime number generator to Go

primegen primegen is a Go package that generates prime numbers in order using the Sieve of Atkin instead of the traditional Sieve of Eratosthenes. It

John Barham 26 Jul 20, 2021
Golang port of Petrovich - an inflector for Russian anthroponyms.

Petrovich is the library which inflects Russian names to given grammatical case. This is the Go port of https://github.com/petrovich. Installation go

Ivan Ivanov 39 May 24, 2022
A Go port of the Rapid Automatic Keyword Extraction algorithm (RAKE)

A Go implementation of the Rapid Automatic Keyword Extraction (RAKE) algorithm as described in: Rose, S., Engel, D., Cramer, N., & Cowley, W. (2010).

Abdullah Joseph 93 Apr 16, 2022
Go port of Coda Hale's Metrics library

go-metrics Go port of Coda Hale's Metrics library: https://github.com/dropwizard/metrics. Documentation: http://godoc.org/github.com/rcrowley/go-metri

Richard Crowley 3.2k Jun 21, 2022
Go binding to libserialport for serial port functionality.

Go Serial Package serial provides a binding to libserialport for serial port functionality. Serial ports are commonly used with embedded systems, such

Jacob Michael Lee 51 Jun 15, 2022
Port of the lemon parser generator to the Go programming language

From the golang-nuts mailing list (with few modifications): --== intro ==-- Hi. I just want to announce a simple port of the lemon parser generator

null 54 Feb 17, 2022
An experimental port of TinyRb to Google go, both as a means of learning go and exploring alternate approaches to implementing Ruby. Work is currently focused on the GoLightly VM.

tinyrb¶ ↑ A tiny subset of Ruby with a Lua'esc VM. Everything in TinyRb should run in the big Ruby. (except bugs and things that don't comply to the p

Eleanor McHugh 72 Feb 27, 2022
Interact with Chromium-based browsers' debug port to view open tabs, installed extensions, and cookies

WhiteChocolateMacademiaNut Description Interacts with Chromium-based browsers' debug port to view open tabs, installed extensions, and cookies. Tested

Justin Bui 85 Jun 14, 2022
A go port of numpy-financial functions and more.

go-financial This package is a go native port of the numpy-financial package with some additional helper functions. The functions in this package are

Razorpay 265 Jun 15, 2022
grobotstxt is a native Go port of Google's robots.txt parser and matcher library.

grobotstxt grobotstxt is a native Go port of Google's robots.txt parser and matcher C++ library. Direct function-for-function conversion/port Preserve

Jim Smart 86 Apr 7, 2022
Router socks. One port socks for all the others.

Router socks The next step after compromising a machine is to enumerate the network behind. Many tools exist to expose a socks port on the attacker's

null 60 Apr 5, 2022
A port of Rebecca Murphey's js-assessment for Go

go-assessment Your Job Is To Make The Tests Pass! What is this? This is a tool for assessing or practicing beginner level programming in Golang. It is

david howard 19 Dec 16, 2021