xlog is a logger for net/context aware HTTP applications

Overview

⚠️ Check zerolog, the successor of xlog.

HTTP Handler Logger

godoc license Build Status Coverage

xlog is a logger for net/context aware HTTP applications.

Unlike most loggers, xlog will never block your application because one its outputs is lagging. The log commands are connected to their outputs through a buffered channel and will prefer to discard messages if the buffer get full. All message formatting, serialization and transport happen in a dedicated go routine.

Read more about xlog on Dailymotion engineering blog.

Features

  • Per request log context
  • Per request and/or per message key/value fields
  • Log levels (Debug, Info, Warn, Error)
  • Color output when terminal is detected
  • Custom output (JSON, logfmt, …)
  • Automatic gathering of request context like User-Agent, IP etc.
  • Drops message rather than blocking execution
  • Easy access logging thru github.com/rs/xaccess

Works with both Go 1.7+ (with net/context support) and Go 1.6 if used with github.com/rs/xhandler.

Install

go get github.com/rs/xlog

Usage

c := alice.New()

host, _ := os.Hostname()
conf := xlog.Config{
    // Log info level and higher
    Level: xlog.LevelInfo,
    // Set some global env fields
    Fields: xlog.F{
        "role": "my-service",
        "host": host,
    },
    // Output everything on console
    Output: xlog.NewOutputChannel(xlog.NewConsoleOutput()),
}

// Install the logger handler
c = c.Append(xlog.NewHandler(conf))

// Optionally plug the xlog handler's input to Go's default logger
log.SetFlags(0)
xlogger := xlog.New(conf)
log.SetOutput(xlogger)

// Install some provided extra handler to set some request's context fields.
// Thanks to those handler, all our logs will come with some pre-populated fields.
c = c.Append(xlog.MethodHandler("method"))
c = c.Append(xlog.URLHandler("url"))
c = c.Append(xlog.RemoteAddrHandler("ip"))
c = c.Append(xlog.UserAgentHandler("user_agent"))
c = c.Append(xlog.RefererHandler("referer"))
c = c.Append(xlog.RequestIDHandler("req_id", "Request-Id"))

// Here is your final handler
h := c.Then(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    // Get the logger from the request's context. You can safely assume it
    // will be always there: if the handler is removed, xlog.FromContext
    // will return a NopLogger
    l := xlog.FromRequest(r)

    // Then log some errors
    if err := errors.New("some error from elsewhere"); err != nil {
        l.Errorf("Here is an error: %v", err)
    }

    // Or some info with fields
    l.Info("Something happend", xlog.F{
        "user":   "current user id",
        "status": "ok",
    })
    // Output:
    // {
    //   "message": "Something happend",
    //   "level": "info",
    //   "file": "main.go:34",
    //   "time": time.Time{...},
    //   "user": "current user id",
    //   "status": "ok",
    //   "ip": "1.2.3.4",
    //   "user-agent": "Mozilla/1.2.3...",
    //   "referer": "http://somewhere.com/path",
    //   "role": "my-service",
    //   "host": "somehost"
    // }
}))
http.Handle("/", h)

if err := http.ListenAndServe(":8080", nil); err != nil {
    xlogger.Fatal(err)
}

Copy Logger

You may want to get a copy of the current logger to pass a modified version to a function without touching the original:

l := xlog.FromContext(ctx)
l2 := xlog.Copy(l)
l2.SetField("foo", "bar")

Make sure you copy a request context logger if you plan to use it in a go routine that may still exist after the end of the current request. Contextual loggers are reused after each requests to lower the pressure on the garbage collector. If you would use such a logger in a go routine, you may end up using a logger from another request/context or worse, a nil pointer:

l := xlog.FromContext(ctx)
l2 := xlog.Copy(l)
go func() {
    // use the safe copy
    l2.Info("something")
}()

Global Logger

You may use the standard Go logger and plug xlog as it's output as xlog implements io.Writer:

xlogger := xlog.New(conf)
log.SetOutput(xlogger)

This has the advantage to make all your existing code or libraries already using Go's standard logger to use xlog with no change. The drawback though, is that you won't have control on the logging level and won't be able to add custom fields (other than ones set on the logger itself via configuration or SetFields()) for those messages.

Another option for code you manage but which is outside of a HTTP request handler is to use the xlog provided default logger:

xlog.Debugf("some message with %s", variable, xlog.F{"and": "field support"})

This way you have access to all the possibilities offered by xlog without having to carry the logger instance around. The default global logger has no fields set and has its output set to the console with no buffering channel. You may want to change that using the xlog.SetLogger() method:

xlog.SetLogger(xlog.New(xlog.Config{
    Level: xlog.LevelInfo,
    Output: xlog.NewConsoleOutput(),
    Fields: xlog.F{
        "role": "my-service",
    },
}))

Configure Output

By default, output is setup to output debug and info message on STDOUT and warning and errors to STDERR. You can easily change this setup.

XLog output can be customized using composable output handlers. Thanks to the LevelOutput, MultiOutput and FilterOutput, it is easy to route messages precisely.

conf := xlog.Config{
    Output: xlog.NewOutputChannel(xlog.MultiOutput{
        // Send all logs with field type=mymodule to a remote syslog
        0: xlog.FilterOutput{
            Cond: func(fields map[string]interface{}) bool {
                return fields["type"] == "mymodule"
            },
            Output: xlog.NewSyslogOutput("tcp", "1.2.3.4:1234", "mymodule"),
        },
        // Setup different output per log level
        1: xlog.LevelOutput{
            // Send errors to the console
            Error: xlog.NewConsoleOutput(),
            // Send syslog output for error level
            Info: xlog.NewSyslogOutput("", "", ""),
        },
    }),
})

h = xlog.NewHandler(conf)

Built-in Output Modules

Name Description
OutputChannel Buffers messages before sending. This output should always be the output directly set to xlog's configuration.
MultiOutput Routes the same message to several outputs. If one or more outputs return error, the last error is returned.
FilterOutput Tests a condition on the message and forward it to the child output if true.
LevelOutput Routes messages per level outputs.
ConsoleOutput Prints messages in a human readable form on the stdout with color when supported. Fallback to logfmt output if the stdout isn't a terminal.
JSONOutput Serialize messages in JSON.
LogfmtOutput Serialize messages using Heroku like logfmt.
LogstashOutput Serialize JSON message using Logstash 2.0 (schema v1) structured format.
SyslogOutput Send messages to syslog.
UIDOutput Append a globally unique id to every message and forward it to the next output.

Third Party Extensions

Project Author Description
gRPClog Hugo González Labrador An adapter to use xlog as the logger for grpclog.
xlog-nsq Olivier Poitrey An xlog to NSQ output.
xlog-sentry trong An xlog to Sentry output.

Licenses

All source code is licensed under the MIT License.

Issues
  • Add option to opt-out of sync.Pool use

    Add option to opt-out of sync.Pool use

    When the http.Handler wrapped by the xlog middleware returns the internal state of the logger is cleared and the instance is returned to the pool. However, there are quite a few cases in which a reference to the logger escapes the context of the handler and ends up in a background task.

    When passing instances between goroutines we've established the habbit of making a copy to pass along to avoid this issue. However, when interacting with third party code we don't always have this as an option. For cases when it's absolutely necessary, we'd like to disable the sync.Pool optimisation in favour of persistent logger references.

    This patch adds the option in a backwards compatible way. The default behaviour is maintained and only adding a new, explicit option to disable the optimisation will trigger the new behaviour.

    opened by kevinconway 8
  • Fatal: Concurrent Map Writes

    Fatal: Concurrent Map Writes

    I'm experiencing fatal errors at 60k+ requests per second. It seems like the issue is in SetField since there's not a lock around it. The issue arrises when using the xlog.URLHandler("path") and xlog.MethodHandler("method") middleware.

    Any thoughts on the simplest fix?

    opened by eliquious 3
  • Simple code does not output anything

    Simple code does not output anything

    The code below does not output anything

    package main 
    
    import (
        "github.com/rs/xlog"
    )
    
    func main() {
        l := xlog.New(xlog.Config{})
        l.Infof("%s", "INFO")
    }
    

    Could you say - whats is wrong?

    opened by trong 2
  • Integrating error-tracking services like Rollbar

    Integrating error-tracking services like Rollbar

    Do you see xlog providing hooks for integrations with external services or do you rather see these as Outputs that can be configured under say MultiOutput?

    opened by joonathan 2
  • file field bad when using std logger

    file field bad when using std logger

    Example:

    package main
    
    import "github.com/rs/xlog"
    
    func main() {
        xlog.Error("what file is this?")
    }
    

    Output:

    2016/04/16 22:51:18 ERRO what file is this? file=std.go:44
    

    std.go:44 is xlog code: https://github.com/rs/xlog/blob/master/std.go#L44

    Logger assumes that it is being called by the function with the error and not an intermediate function. This causes the file and line number to be taken from std.go instead of the client code.

    opened by stephenmw 1
  • Clone logger

    Clone logger

    Sometimes I want to clone a logger - set some fields, give this prepared logger in a new context to a subfunction, but don't "pollute" the main logger, to be able to clone again.

    I don't see this possible ATM - I don't have the xlog.Config, just xlog.FromContext.

    Any idea? Or only a new function (xlog.Clone(xlog.Logger) ?

    opened by tgulacsi 1
  • Allow setting Console destination

    Allow setting Console destination

    os.Stderr would be a more traditional choice for NewConsoleOutput. NewJSONOutput does have such an argument, NewConsoleOutput does not...

    Possible solutions: a) change NewConsoleOutput to have an io.Writer argument, b) add NewConsoleOutputW(io.Writer) for using the given writer, c) do nothing, and document the "os.Stdout, os.Stderr = os.Stderr, os.Stdout; xlog.New(xlog.Config{Output:xlog.NewConsoleOutput()}); os.Stdout, os.Stderr = os.Stderr, os.Stdout" dance.

    I'd prefer b), if possible.

    opened by tgulacsi 1
  • Versioning of the project

    Versioning of the project

    Hi,

    first of all thanks for this great module.

    I think this module deserves some version control.

    BTW, I created an adapter to use xlog into gRPC world. https://github.com/clawio/grpcxlog

    opened by labkode 1
  • Add a LevelFromString method to avoid implementing the switch in the project using xlog

    Add a LevelFromString method to avoid implementing the switch in the project using xlog

    If the log level is a string in the package using xlog, it's easier to have a LevelFromString method to transform it into a proper xlog.Level without having to implement the switch itself.

    opened by asticode 0
  • README mixed reference to net/context, link to x/net/context

    README mixed reference to net/context, link to x/net/context

    This package supports both the X and the non-X context packages. The README ambiguously shows the net/context package and hyper-links to the x/net/context version.

    I expect the package to call-out its 1.7 support for x/net/context and link to the 1.8+ https://golang.org/pkg/context/ context home page.

    Text:

    xlog is a logger for net/context aware HTTP applications.

    Link:

    https://godoc.org/golang.org/x/net/context

    Markdown:

    `xlog` is a logger for [net/context](https://godoc.org/golang.org/x/net/context) aware HTTP applications.
    

    Thanks for the useful packages rs!

    opened by nrichards 2
  • Error field

    Error field

    Extract error typed field, so the error correct place can be determined, if the error was wrapped.

    So the error wrapper packages can be used in the output (eg.:github.com/pkg/errors https://github.com/juju/errors)

    opened by Ak-Army 0
  • fatal error: concurrent map writes

    fatal error: concurrent map writes

    fatal error: concurrent map writes
    
    goroutine 3765 [running]:
    runtime.throw(0xcf6ade, 0x15)
            /usr/local/go/src/runtime/panic.go:596 +0x95 fp=0xc42098f828 sp=0xc42098f808
    runtime.mapassign(0xc03600, 0xc4208f6030, 0xc42098f8f0, 0xba2b40)
            /usr/local/go/src/runtime/hashmap.go:499 +0x667 fp=0xc42098f8c8 sp=0xc42098f828
    github.com/rs/xlog.(*logger).SetField(0xc4208f6000, 0xceccc8, 0xa, 0xba2b40, 0xc4202aa600)
            /go/src/github.com/rs/xlog/xlog.go:234 +0x66 fp=0xc42098f910 sp=0xc42098f8c8
    
    opened by kalbasit 7
Owner
Olivier Poitrey
Director of Engineering at Netflix Co-Founder & ex-CTO of Dailymotion Co-Founder of NextDNS
Olivier Poitrey
Logger - Simple logger without written with std pkg

Go-Logger Simple usage is: package main

MaskedTrench 2 Jan 2, 2022
Logger - A thin wrapper of uber-go/zap logger for personal project

a thin wraper of uber-go/zap logger for personal project 0. thanks uber-go/zap B

tsingson 1 Jun 16, 2022
HTTP request logger for Golang

Horus ?? Introduction Horus is a request logger and viewer for Go. It allows developers log and view http requests made to their web application. Inst

Michael Okoh 78 Jun 5, 2022
SigNoz helps developer monitor applications and troubleshoot problems in their deployed applications

SigNoz helps developers monitor their applications & troubleshoot problems, an open-source alternative to DataDog, NewRelic, etc. ?? ??

SigNoz 7.3k Aug 2, 2022
Gin adapter for standard net/http middleware

midgin An adapter to use standard net/http middleware in Gin. Overview Gin is a very capable web framework, but it does not directly support standard

Mark Blythe 0 Feb 12, 2022
A logger, for Go

Go-Log A logger, for Go! It's sort of log and code.google.com/p/log4go compatible, so in most cases can be used without any code changes. Breaking cha

Ian Kent 38 May 12, 2022
Simple logger for Go programs. Allows custom formats for messages.

go-logger A simple go logger for easy logging in your programs. Allows setting custom format for messages. Preview Install go get github.com/apsdehal/

Amanpreet Singh 276 Jun 14, 2022
Loggly Hooks for GO Logrus logger

Loggly Hooks for Logrus Usage package main import ( "github.com/sirupsen/logrus" "github.com/sebest/logrusly" ) var logglyToken string = "YOUR_LOG

Sebest 27 Jul 27, 2021
A 12-factor app logger built for performance and happy development

logxi log XI is a structured 12-factor app logger built for speed and happy development. Simpler. Sane no-configuration defaults out of the box. Faste

Mario Gutierrez 348 Jun 25, 2022
Dead simple, super fast, zero allocation and modular logger for Golang

Onelog Onelog is a dead simple but very efficient JSON logger. It is one of the fastest JSON logger out there. Also, it is one of the logger with the

Francois Parquet 399 Jun 16, 2022
A logger for Go SQL database driver without modify existing *sql.DB stdlib usage.

SQLDB-Logger A logger for Go SQL database driver without modify existing *sql.DB stdlib usage. Colored console writer output above only for sample/dev

Sarjono Mukti Aji 230 Aug 2, 2022
Zero Allocation JSON Logger

Zero Allocation JSON Logger The zerolog package provides a fast and simple logger dedicated to JSON output. Zerolog's API is designed to provide both

Olivier Poitrey 6.7k Aug 5, 2022
A powerful zero-dependency json logger.

ZKits Logger Library About This package is a library of ZKits project. This is a zero-dependency standard JSON log library that supports structured JS

Qingshan Luo 16 Jun 15, 2022
Configurable Logger for Go

Timber! This is a logger implementation that supports multiple log levels, multiple output destinations with configurable formats and levels for each.

DeNA SF 66 Jun 28, 2022
A feature-rich and easy to use logger for golang

A feature-rich and easy to use logger for golang ?? Install ?? Common Logs lumber.Success() lumber.Info() lumber.Debug() lumber.Warning()

Matthew Gleich 50 Jul 20, 2022
A minimal and extensible structured logger

⚠️ PRE-RELEASE ⚠️ DO NOT IMPORT THIS MODULE YOUR PROJECT WILL BREAK package log package log provides a minimal interface for structured logging in ser

Go kit 111 Aug 2, 2022
Simple Yet Powerful Logger

sypl sypl provides a Simple Yet Powerful Logger built on top of the Golang sypl. A sypl logger can have many Outputs, and each Output is responsible f

Sauce Labs 9 Apr 28, 2022
simple concurrent logger

XMUS-LOGGER pure golang logger compatible with golang io standards. USAGE : logOptions := logger.LoggerOptions{ LogLevel: 6, // read more about lo

amupxm [amir hossein mokarrami far] 6 Aug 1, 2022
Binalyze logger is an easily customizable wrapper for logrus with log rotation

logger logger is an easily customizable wrapper for logrus with log rotation Usage There is only one function to initialize logger. logger.Init() When

Binalyze 26 Nov 18, 2021