Go package for reading from continously updated files (tail -f)

Overview

Build Status Build status

Go package for tail-ing files

A Go package striving to emulate the features of the BSD tail program.

t, err := tail.TailFile("/var/log/nginx.log", tail.Config{Follow: true})
for line := range t.Lines {
    fmt.Println(line.Text)
}

See API documentation.

Log rotation

Tail comes with full support for truncation/move detection as it is designed to work with log rotation tools.

Installing

go get github.com/hpcloud/tail/...

Windows support

This package needs assistance for full Windows support.

Issues
  • Fix fsnotify repository relocation

    Fix fsnotify repository relocation

    The https://github.com/go-fsnotify/fsnotify redirect has been disabled and replaced with an empty repo, the new path for gopkg.in is: http://gopkg.in/fsnotify/fsnotify.v1 as of yesterday.

    • Modify fsnotify import paths to: gopkg.in/fsnotify/fsnotify.v1
    • Copy vendor/fsnotify.v1 to the new import path
    opened by promiseofcake 17
  • Using last stable api (v0) fetches master

    Using last stable api (v0) fetches master

    A go get gopkg.in/ActiveState/tail.v0 (to use the old stable API) will also fetch the master repo, which is due to the import of the fullly-qualified "github.com/ActiveState/tail/..." URL in several .go files.

    Replacing them with relative URLs does not work at all:

    > make fulltest
    [...]
    can't load package: /go/src/github.com/ActiveState/tail/tail.go:15:2: local import "./ratelimiter" in non-local package
    

    Consequently it becomes impossible to go get on the old API unless the user manually clones the repo under $GOPATH/src/github.com/ActiveState/tail

    (this shouldn't affect the Stackato folks who use git commit hash to clone the repo)

    opened by srid 7
  • Reopen deleted files properly, and fix BlockUntilExists for relative paths

    Reopen deleted files properly, and fix BlockUntilExists for relative paths

    This fixes two bugs:

    1. Issue #57 - the inotify watcher wouldn't notice when a file was deleted because the inotify event actually comes on the directory and it only watches the file itself.
    2. Watching a file in the current directory with a relative path that doesn't exist yet wouldn't work because the inotify events are for "./foo" not "foo". I found https://github.com/hpcloud/tail/pull/48 afterwards which also fixes this bug.

    Also many of the tests in tail_test.go weren't actually working properly because the test function wasn't waiting for the goroutine to exit. One of the tests was actually exercising the first bug, but wasn't failing for this reason.

    opened by davidsansome 6
  • Single shared Watcher used to avoid inotify limit

    Single shared Watcher used to avoid inotify limit

    When we were using this tail library, we found that the system limit on inotify watches would become an issue when we we try to tail many files at once (if many Tail objects are open simultaneously). To circumvent this, we changed the InotifyTracker struct to contain a Watcher that is shared by all Tail objects that are created.

    opened by andyouyang22 6
  • Half-a-lines

    Half-a-lines

    I've got a file which is growing fast (a kernel packet log). This is line based, and lines are usually around 900 bytes long.

    Sometimes range t.Lines gives a partial line (not ending in "\n"), even though I have not set MaxLineSize. This line is then 512 bytes long.

    It seems this is caused by this if: https://github.com/ActiveState/tail/blob/master/tail.go#L217 although I would expect the next 'if' to be the one we want in this case (wait for more data to complete the line).

    Linux 3.13.7, go 1.3 and this is used to open the file:

          t, err := tail.TailFile(file, tail.Config{
              Follow: true,
              ReOpen: true,
              Location: &tail.SeekInfo{
                  Offset: 0,
                  Whence: os.SEEK_END,
              },
          })
    
    opened by alicebob 6
  • fix block until exists function

    fix block until exists function

    When we run the flow program, then execute echo 1 > not_exist in shell, the result is not we want to.

    package main
    
    import (
        "fmt"
    
        "github.com/hpcloud/tail"
    )
    
    func main() {
        t, err := tail.TailFile("not_exists", tail.Config{
            Follow:    true,
            MustExist: false,
        })
    
        if err != nil {
            fmt.Printf("err=%+v\n", err)
            return
        }
    
        for {
            select {
            case line := <-t.Lines:
                fmt.Printf("line=%s\n", line)
            }
        }
    }
    
    opened by miraclesu 5
  • Single inotify instance with multiple watchers

    Single inotify instance with multiple watchers

    An fd-per-watch quickly consumes more file descriptors than are allowed. Would you accept a patch that uses a single inotify instance and multiple watchers? This appears to be the approach recommended by the Linux kernel docs.

    There's a stumbling block that would require modifying the API. The new approach would eliminate the global inotify tracker variable and create new file watchers using a supplied instance of a tracker.

    Please let me know what you think!

    P.S. This is for Papertrail's remote_syslog2, to resolve this issue.

    opened by joelreymont 5
  • Add logging options to config

    Add logging options to config

    The library generates logging output when files are closed or recreated. I didn't want to display this to the user of my program so I added extra options to modify the way tail logs.

    There's Config.Logger, allowing you to set your own Logger (using logfile, or other prefix or flags). And there's Config.DisableLogging, which disables logging all together.. This doesn't change anything for existing users, as Config.Logger is set to the log package's default logger (Log.std) when it's nil.

    I hope this code will be useful for anyone else out there :smile:

    enhancement 
    opened by GeertJohan 5
  • Add support for arbitrary Location

    Add support for arbitrary Location

    This adds support for arbitrary Location values, both positive and negative. -n=0 and -n=-1 behave the same as before (tail from end and tail from start). To keep the code simple, we introduced a little "off by one" (so you actually use N+1 or -(N+1) to seek to the N-th byte from the end or start).

    To be closer to the original unix tail, it would be nicer to seek lines, not bytes. But for our application, it's not necessary and a lot slower.

    cc @camilo @snormore

    opened by fw42 5
  • Failed test under macOS

    Failed test under macOS

    running Go 1.8 under macOS Sierra 10.12.3 (16D32), HEAD at faf842bde7ed83bbc3c65a2c454fae39bc29a95f

    ~/.../hpcloud/tail $ go test
    2017/02/23 23:27:19 Waiting for /no/such/file to appear...
    2017/02/23 23:27:19 Waiting for .test/waits-for-file-to-exist/test.txt to appear...
    2017/02/23 23:27:19 Waiting for test.txt to appear...
    2017/02/23 23:27:20 Waiting for _no_such_file to appear...
    2017/02/23 23:27:20 Seeked .test/location-end/test.txt - &{Offset:0 Whence:2}
    2017/02/23 23:27:20 Seeked .test/location-middle/test.txt - &{Offset:-6 Whence:2}
    2017/02/23 23:27:21 Re-opening moved/deleted file .test/reopen-inotify/test.txt ...
    2017/02/23 23:27:21 Waiting for .test/reopen-inotify/test.txt to appear...
    2017/02/23 23:27:22 Re-opening moved/deleted file .test/reopen-polling/test.txt ...
    2017/02/23 23:27:22 Waiting for .test/reopen-polling/test.txt to appear...
    2017/02/23 23:27:22 Successfully reopened .test/reopen-polling/test.txt
    2017/02/23 23:27:22 Re-opening moved/deleted file .test/reopen-polling/test.txt ...
    2017/02/23 23:27:22 Waiting for .test/reopen-polling/test.txt to appear...
    2017/02/23 23:27:23 Successfully reopened .test/reopen-polling/test.txt
    2017/02/23 23:27:23 Re-opening moved/deleted file .test/reopen-polling/test.txt ...
    2017/02/23 23:27:23 Waiting for .test/reopen-polling/test.txt to appear...
    2017/02/23 23:27:23 Stopping tail as file no longer exists: .test/reseek-inotify/test.txt
    --- FAIL: TestReSeekInotify (0.20s)
    	tail_test.go:499: tail ended early; expecting more: [h311o w0r1d endofworld]
    2017/02/23 23:27:23 Re-opening truncated file .test/reseek-polling/test.txt ...
    2017/02/23 23:27:23 Successfully reopened truncated .test/reseek-polling/test.txt
    2017/02/23 23:27:24 Leaky bucket full (.test/rate-limiting/test.txt); entering 1s cooloff period.
    2017/02/23 23:27:24 Stopping tail as file no longer exists: .test/reseek-polling/test.txt
    2017/02/23 23:27:25 Leaky bucket full (.test/rate-limiting/test.txt); entering 1s cooloff period.
    2017/02/23 23:27:25 Seeked .test/tell-position/test.txt - &{Offset:0 Whence:0}
    2017/02/23 23:27:25 Seeked .test/tell-position/test.txt - &{Offset:12 Whence:0}
    2017/02/23 23:27:25 Waiting for .test/block-until-file-exists/test.txt to appear...
    FAIL
    exit status 1
    FAIL	github.com/hpcloud/tail	5.565s
    
    opened by pyed 4
  • Issue on Stop Tailing

    Issue on Stop Tailing

    Hello,

    I'm using this func err := t.Stop()

    to stop tailing activity on a file, however this not stops the tailing. Is this the correct aproach to stop tailing ? If not, how can I achieve my goal ? Thanks in advance

    opened by marioffmarques 4
  • Can solve the problem of file scrolling in Linux?

    Can solve the problem of file scrolling in Linux?

    like the file'/root/.bash_history'

    when my '/root/.bash_history' start roll

    [email protected]:~# wc -l /root/.bash_history 
    
    2000 /root/.bash_history
    
    [email protected]:~# echo $HISTSIZE
    
    1000
    

    this util work abnormal ,but "tail -f /root/.bash_history " can still work

    this is my config

    t, err := tail.TailFile(p, tail.Config{
    	Location: &tail.SeekInfo{
    		Offset: 0,
    		Whence: io.SeekEnd,
    	},
    	ReOpen:      true,
    	Follow:      true,
    	Poll:        true,
    	MustExist:   false,
    	Pipe:        false,
    	MaxLineSize: 10000
    })
    
    opened by hz-kelpie 0
  • fsnotify doesn't work with Windows, polling does

    fsnotify doesn't work with Windows, polling does

    The simple example in the README doesn't work with Windows 10 Pro x64 (Version 10.0.19042 Build 19042). Adding Poll: true, insted, works:

    t, err := tail.TailFile(fpath, tail.Config{Follow: true, Poll: true})
    for line := range t.Lines {
    	fmt.Println(line.Text)
    }
    

    I think you could mention it in the README

    opened by tommyblue 1
  • how to close tails goroutine

    how to close tails goroutine

    goroutine 889 2020/11/17 09:13:00 Seeked /opt/6938e171-7ffd-49f8-8e62-c2daa1c8d63c/1.1.4/log.txt - &{Offset:0 Whence:2} 2020/11/17 09:13:00 Seeked /opt/acf8de17-6e26-4dda-b185-67e4070f3bf8/1.1.3/log.txt - &{Offset:0 Whence:2} goroutine 894 goroutine 894

    i use it to watch an log ,but it aways add goroutine numbers ,how to work out it ?

    opened by Linuxlovers 1
Owner
HPE Helion
HPE Helion
Lumberjack is a Go package for writing logs to rolling files.

Lumberjack is a Go package for writing logs to rolling files.

Go微服务 4 Feb 24, 2022
Peimports - based on golang's debug/pe this package gives quick access to the ordered imports of pe files with ordinal support

This code is almost entirely derived from the Go standard library's debug/pe package. It didn't provide access to ordinal based entries in the IAT and

Mike Wiacek 0 Jan 5, 2022
A simple daemon which will watch files on your filesystem, mirror them to MFS, automatically update related pins, and update related IPNS keys.

ipfs-sync is a simple daemon which will watch files on your filesystem, mirror them to MFS, automatically update related pins, and update related IPNS keys, so you can always access your directories from the same address. You can use it to sync your documents, photos, videos, or even a website!

null 72 Jun 14, 2022
A simple web service for storing text log files

logpaste A minimalist web service for uploading and sharing log files. Run locally go run main.go Run in local Docker container The Docker container a

Michael Lynch 234 Aug 9, 2022
List files and their creation, modification and access time on android

andfind List files and their access, modification and creation date on a Android

Tek 2 Jan 5, 2022
BRUS - Parses your web server (e.g. nginx) log files and checks with GreyNoise how much noise your website is exposed to.

BRUS bbbbbb rrrrrr u u sssss b b r r u u s bbbbbb rrrrrr u u sssss b b r r u u s bbbbbb r r

dubs3c 1 May 29, 2022
A helper tool to work with profile.proto (pprof) files

qpprof qpprof complements the pprof tool. Commands Use qpprof command --help to get more information. Flat aggregation Alternative flat aggregations a

Iskander (Alex) Sharipov 54 Jun 24, 2022
A version control system to manage large files.

ArtiVC ArtiVC (Artifacts Version Control) is a handy command-line tool for data versioning on cloud storage. With only one command, it helps you neatl

InfuseAI 207 Aug 6, 2022
Package httpretty prints the HTTP requests you make with Go pretty on your terminal.

httpretty Package httpretty prints the HTTP requests of your Go programs pretty on your terminal screen. It is mostly inspired in curl's --verbose mod

Henrique Vicente 268 Jul 31, 2022
Structured logging package for Go.

Package log implements a simple structured logging API inspired by Logrus, designed with centralization in mind. Read more on Medium. Handlers apexlog

Apex 1.2k Jul 19, 2022
lumberjack is a log rolling package for Go

lumberjack Lumberjack is a Go package for writing logs to rolling files. Package lumberjack provides a rolling logger. Note that this is v2.0 of lumbe

Nate Finch 3.5k Aug 4, 2022
A Go (golang) package providing high-performance asynchronous logging, message filtering by severity and category, and multiple message targets.

ozzo-log Other languages 简体中文 Русский Description ozzo-log is a Go package providing enhanced logging support for Go programs. It has the following fe

Ozzo Framework 118 Jan 11, 2022
With this package you can create your own syslog server with your own handlers for different kind of syslog messages

Using this library you can easy implement your own syslog server that: Can listen on multiple UDP ports and unix domain sockets. Can pass parsed syslo

Michał Derkacz 68 Dec 15, 2021
Package zaperations provides a Google Cloud operations suite (formerly Stackdriver) compatible config for the uber-go/zap logger.

Package zaperations provides a Google Cloud Operations (formerly Stackdriver) compatible config for the excellent uber-go/zap logger. Example This exa

Wayne 0 Nov 6, 2021
Package logging implements a logging infrastructure for Go

Golang logging library Package logging implements a logging infrastructure for Go. Its output format is customizable and supports different logging ba

Luke Zhang 0 Nov 10, 2021
A golang package to communicate with HipChat over XMPP

hipchat This is a abstraction in golang to Hipchat's implementation of XMPP. It communicates over TLS and requires zero knowledge of XML or the XMPP p

Dane Harrigan 110 Jan 12, 2022
Package for easy logging to logstash http input from microservices

Micro Logger package for easy logging to logstash http input from microservices

null 0 Dec 28, 2021
Print build info from binary, using buildinfo package.

Go Build Info Print build info from binary, using buildinfo package. https://pkg.go.dev/debug/[email protected] Note: This was created to help me

null 0 Dec 30, 2021
This package enables json output, level logging and so on to standard go logger.

logplug This package enables json output, level logging and so on to standard logger. Usage log.SetOutput(logplug.NewJSONPlug(os.Stderr, logplug.LogF

Koumei Mikuni 0 Dec 27, 2021