A minimal and extensible structured logger

Overview

⚠️   PRE-RELEASE   ⚠️

DO NOT IMPORT THIS MODULE

YOUR PROJECT WILL BREAK

package log

package log provides a minimal interface for structured logging in services. It may be wrapped to encode conventions, enforce type-safety, provide leveled logging, and so on. It can be used for both typical application log events, and log-structured data streams.

Structured logging

Structured logging is, basically, conceding to the reality that logs are data, and warrant some level of schematic rigor. Using a stricter, key/value-oriented message format for our logs, containing contextual and semantic information, makes it much easier to get insight into the operational activity of the systems we build. Consequently, package log is of the strong belief that "the benefits of structured logging outweigh the minimal effort involved".

Migrating from unstructured to structured logging is probably a lot easier than you'd expect.

// Unstructured
log.Printf("HTTP server listening on %s", addr)

// Structured
logger.Log("transport", "HTTP", "addr", addr, "msg", "listening")

Usage

Typical application logging

w := log.NewSyncWriter(os.Stderr)
logger := log.NewLogfmtLogger(w)
logger.Log("question", "what is the meaning of life?", "answer", 42)

// Output:
// question="what is the meaning of life?" answer=42

Contextual Loggers

func main() {
	var logger log.Logger
	logger = log.NewLogfmtLogger(log.NewSyncWriter(os.Stderr))
	logger = log.With(logger, "instance_id", 123)

	logger.Log("msg", "starting")
	NewWorker(log.With(logger, "component", "worker")).Run()
	NewSlacker(log.With(logger, "component", "slacker")).Run()
}

// Output:
// instance_id=123 msg=starting
// instance_id=123 component=worker msg=running
// instance_id=123 component=slacker msg=running

Interact with stdlib logger

Redirect stdlib logger to Go kit logger.

import (
	"os"
	stdlog "log"
	kitlog "github.com/go-kit/log"
)

func main() {
	logger := kitlog.NewJSONLogger(kitlog.NewSyncWriter(os.Stdout))
	stdlog.SetOutput(kitlog.NewStdlibAdapter(logger))
	stdlog.Print("I sure like pie")
}

// Output:
// {"msg":"I sure like pie","ts":"2016/01/01 12:34:56"}

Or, if, for legacy reasons, you need to pipe all of your logging through the stdlib log package, you can redirect Go kit logger to the stdlib logger.

logger := kitlog.NewLogfmtLogger(kitlog.StdlibWriter{})
logger.Log("legacy", true, "msg", "at least it's something")

// Output:
// 2016/01/01 12:34:56 legacy=true msg="at least it's something"

Timestamps and callers

var logger log.Logger
logger = log.NewLogfmtLogger(log.NewSyncWriter(os.Stderr))
logger = log.With(logger, "ts", log.DefaultTimestampUTC, "caller", log.DefaultCaller)

logger.Log("msg", "hello")

// Output:
// ts=2016-01-01T12:34:56Z caller=main.go:15 msg=hello

Levels

Log levels are supported via the level package.

Supported output formats

Enhancements

package log is centered on the one-method Logger interface.

type Logger interface {
	Log(keyvals ...interface{}) error
}

This interface, and its supporting code like is the product of much iteration and evaluation. For more details on the evolution of the Logger interface, see The Hunt for a Logger Interface, a talk by Chris Hines. Also, please see #63, #76, #131, #157, #164, and #252 to review historical conversations about package log and the Logger interface.

Value-add packages and suggestions, like improvements to the leveled logger, are of course welcome. Good proposals should

  • Be composable with contextual loggers,
  • Not break the behavior of log.Caller in any wrapped contextual loggers, and
  • Be friendly to packages that accept only an unadorned log.Logger.

Benchmarks & comparisons

There are a few Go logging benchmarks and comparisons that include Go kit's package log.

Issues
  • Allow to configure allowed levels by string value

    Allow to configure allowed levels by string value

    This PR allows to configure the allowed log levels from a string.

    Closes: #18

    Use Case:

    Read the allowed log levels reading environment variable

    logger = level.NewFilter(logger, level.AllowByString(os.Getenv("LOG_LEVEL"), level.AllowInfo)) // <--
    
    opened by mcosta74 26
  • Updates for Go 1.17

    Updates for Go 1.17

    I want to update this package and tag a patch release before updating go-kit/kit in https://github.com/go-kit/kit/pull/1105. I've already updated both of this package's dependencies along the same lines. Just working my way up the tree to minimize the number of new releases and maximize the benefit of the Go 1.17 module pruning feature for consumers.

    opened by ChrisHines 15
  • Replacing

    Replacing "level" field

    Hi,

    logger := log.NewLogfmtLogger(log.NewSyncWriter(os.Stdout))
    logger.Log("msg", "foo")
    
    logger = level.Error(logger)
    logger.Log("msg", "foo")
    
    logger = level.Info(logger)
    logger.Log("msg", "foo")
    

    produces the following output playground:

    msg=foo
    level=error msg=foo
    level=info level=error msg=foo
    

    so level appears twice in last log.

    What do you think about adding a new function in log:

    func Replace(logger Logger, k, v interface{})
    

    which works as log.With but replace key k if it already exists, in logger, in the new returned Logger instance?

    So level.{Debug,Error,Info,Warn} can use it instead of log.WithPrefix as now.

    Replace could be variadic as With* functions, but is there a real need?

    If you are interested (variadic or not), I can provide a PR.

    opened by maxatome 5
  • consider providing a method to compute the log level based on a String

    consider providing a method to compute the log level based on a String

    For most of our apps we setup the log level based on environment variables. However go-kit/log currently has hardcoded internal levels which makes mapping from the env var setting to the real level more cumbersome.

    I checked the code since you already have a Value interface for the log levels which should be usable through this function, unfortunately it seems unused as here it is casted to levelValue instead of Value.

    Hope it helps

    opened by carocad 5
  • Update CI and add badges to README

    Update CI and add badges to README

    CI updates:

    • Update Go versions (1.17.x and 1.18.x)
    • Update actions versions
    • Use actions for staticcheck
    • Drop golint (deprecated)
    • Report test coverage
    opened by ChrisHines 3
  • Filtered-out log.Level lines still cost

    Filtered-out log.Level lines still cost

    I noticed a portion of our CPU was going into level.Debug(...).Log() lines, although that level wasn't enabled.

    I think what is happening is the Log() call goes into context.Log() first, which calls bindValues which is the expensive part, calling runtime.Caller, formatting a timestamp, etc., then after that it goes to level.Log() which discards everything.

    I looked around but didn't find any previous debate on this point. Am I holding it wrong? Is this expected?

    Since we are using a wrapper, something like https://github.com/go-kit/kit/pull/322 would let us work around the problem by checking before doing the work, but that was rejected.

    opened by bboreham 2
  • broken link for slides

    broken link for slides "The Hunt for a Logger Interface"

    Hello, from the README, following the link to the presentation slides "The Hunt for a Logger Interface" at http://go-talks.appspot.com/github.com/ChrisHines/talks/structured-logging/structured-logging.slide#1

    gives

    Error accessing api.github.com.
    403: (https://api.github.com/repos/ChrisHines/talks?client_id=XXX&client_secret=YYY)
    
    opened by marco-m 2
  • Logging error type that panic on Error() will break json format

    Logging error type that panic on Error() will break json format

    Hi, i found that logging error that cause panic when Error() func called, will cause json logger implementation to break. in contrast, logfmt format implementation safely output the log even if its failed to log the error description. i also put a sample code to replicate the issue on the bottom.

    i think the safeError() & safeString on json format implementation should always recover from panic and instead return the panic description that happen when its encode the value into string.

    i can help to provide a pr for it if you agree with my suggestion. thanks in advance!

    package main
    
    import (
    	kitlog "github.com/go-kit/log"
    	"os"
    )
    
    type myerror struct {
    }
    
    func main() {
    	// will safely output log even its failed
    	log := kitlog.NewLogfmtLogger(os.Stdout)
    	log.Log("err", myerror{})
    
    	// will panic
    	log = kitlog.NewJSONLogger(os.Stdout)
    	log.Log("err", myerror{})
    }
    
    func (e myerror) Error() string {
    	panic("fail retrurn error value")
    	return "test"
    }
    
    

    output:

    go run main.go
    err="PANIC:fail retrurn error value"
    panic: fail retrurn error value [recovered]
        panic: fail retrurn error value
    
    goroutine 1 [running]:
    github.com/go-kit/log.safeError.func1()
        /home/runner/go/pkg/mod/github.com/go-kit/[email protected]/json_logger.go:85 +0x134
    ...
    
    bug 
    opened by dwiyanr 1
  • feat: add a new zap logger adapter witch support github.com/go-kit/log/level

    feat: add a new zap logger adapter witch support github.com/go-kit/log/level

    i have edited the https://github.com/go-kit/kit/tree/master/log/zap adapter, and added a new feature to support github.com/go-kit/log/level, the new logger will try to parse kit-log's level into zap logger's level, if fail, it will use default log level instead

    opened by catusax 0
  • Simplify stack helpers and remove internal/stack package

    Simplify stack helpers and remove internal/stack package

    This PR simplifies the remaining stack trace helpers used in the tests that were extracted from github.com/go-stack/stack. The extracted code was more than we needed and also suffered from some historical baggage that go-stack/stack cares about but go-kit/log does not.

    opened by ChrisHines 0
  • is it possible to have a fixed key, such as 'msg'

    is it possible to have a fixed key, such as 'msg'

    A log message usually have a description, which is not key=value format, such as:

    successfully created an object, name=obj1

    Translate to go-kit log format:

    msg="successfully created an object" name=obj1

    The key "msg" can be fixed.

    Something like Log(msg string, keysAndValues ...interface{})

    In prometheus code: https://github.com/prometheus/prometheus/blob/main/tsdb/head.go#L466-L491

    level.Info(h.logger).Log("msg", "Replaying on-disk memory mappable chunks if any") level.Info(h.logger).Log("msg", "Chunk snapshot is enabled, replaying from the snapshot") level.Info(h.logger).Log("msg", "Chunk snapshot loading time", "duration", time.Since(start).String()) level.Error(h.logger).Log("msg", "Failed to load chunk snapshot", "err", err)

    The key "msg" is redundant here.

    opened by chenzhiwei 0
  • Idea: 'Helper' method for Caller

    Idea: 'Helper' method for Caller

    Currently, it is necessary to figure out how each log method will be called in order to specify a count of stack frames to Caller. https://github.com/go-kit/log/blob/5739c2646c7293c773ed50d2086e843e3330781c/value.go#L82-L84

    This makes functions that wrap or forward to go-kit/log hard to use.

    Suggestion: have a Helper method like in testing, which tells Caller to skip that function from the count.

    This would be relatively expensive – a stack walk, lock and map lookup each time the helper is entered, and more map lookups when actually logging – but much more pleasant to use.

    opened by bboreham 3
  • Documentation link broken

    Documentation link broken

    https://www.thoughtworks.com/radar/techniques/structured-logging in the README is no longer maintained:

    image

    We should either use a web archive link or link to a different article.

    documentation good first issue 
    opened by ShayNehmad-RecoLabs 0
Releases(v0.2.1)
  • v0.2.1(May 14, 2022)

    This release fixes a few small bugs and adds level.Parse which allows levels to be set by a string input from e.g. flags or environment variables. Thanks, @mcosta74!

    What's Changed

    • fix safeError & safeString for json format by @dwiyanr in https://github.com/go-kit/log/pull/20
    • Update CI and add badges to README by @ChrisHines in https://github.com/go-kit/log/pull/21
    • Allow to configure allowed levels by string value by @mcosta74 in https://github.com/go-kit/log/pull/22

    New Contributors

    • @dwiyanr made their first contribution in https://github.com/go-kit/log/pull/20
    • @mcosta74 made their first contribution in https://github.com/go-kit/log/pull/22

    Full Changelog: https://github.com/go-kit/log/compare/v0.2.0...v0.2.1

    Source code(tar.gz)
    Source code(zip)
  • v0.2.0(Sep 17, 2021)

    This release adds updates for Go 1.17, and reduces the set of dependencies.

    • Updates for Go 1.17 (#7)
    • Remove the stack library dependency (#10)
    • Simplify stack helpers and remove internal/stack package (#11)
    Source code(tar.gz)
    Source code(zip)
Owner
Go kit
A Go toolkit for microservices.
Go kit
Minimal structured logging library for Go

slog slog is a minimal structured logging library for Go. Install go get cdr.dev/slog Features Minimal API First class context.Context support First c

Coder 243 Jul 10, 2022
Convenient Logger interface and std logger wrapper

Convenient logger interface and wrapper around std logger Interface type Logger interface { Error(err error) Debugf(format string, args ...interface

Denis Mitrofanov 1 Nov 28, 2021
Logger - Simple logger without written with std pkg

Go-Logger Simple usage is: package main

MaskedTrench 2 Jan 2, 2022
Logger - A thin wrapper of uber-go/zap logger for personal project

a thin wraper of uber-go/zap logger for personal project 0. thanks uber-go/zap B

tsingson 1 Jun 16, 2022
Simple, configurable and scalable Structured Logging for Go.

log Log is a simple, highly configurable, Structured Logging library Why another logging library? There's allot of great stuff out there, but also tho

Go Playgound 280 Jun 22, 2022
Hierarchical, leveled, and structured logging library for Go

spacelog Please see http://godoc.org/github.com/spacemonkeygo/spacelog for info License Copyright (C) 2014 Space Monkey, Inc. Licensed under the Apach

Space Monkey Go 98 Apr 27, 2021
Simple and extensible monitoring agent / library for Kubernetes: https://gravitational.com/blog/monitoring_kubernetes_satellite/

Satellite Satellite is an agent written in Go for collecting health information in a kubernetes cluster. It is both a library and an application. As a

Teleport 194 Jul 10, 2022
Search and analysis tooling for structured logs

Zed The Zed system provides an open-source, cloud-native, and searchable data lake for semi-structured and structured data. Zed lakes utilize a supers

Brim 833 Jul 30, 2022
Structured logging package for Go.

Package log implements a simple structured logging API inspired by Logrus, designed with centralization in mind. Read more on Medium. Handlers apexlog

Apex 1.2k Jul 19, 2022
Structured log interface

Structured log interface Package log provides the separation of the logging interface from its implementation and decouples the logger backend from yo

teris.io 25 Jul 29, 2022
Structured, composable logging for Go

log15 Package log15 provides an opinionated, simple toolkit for best-practice logging in Go (golang) that is both human and machine readable. It is mo

Alan Shreve 1.1k Jul 23, 2022
Structured, pluggable logging for Go.

Logrus Logrus is a structured logger for Go (golang), completely API compatible with the standard library logger. Logrus is in maintenance-mode. We wi

Simon Eskildsen 21k Aug 1, 2022
Structured Logging Made Easy

Structured Logging Made Easy Features Dependency Free Simple and Clean Interface Consistent Writer IOWriter, io.Writer wrapper FileWriter, rotating &

phuslu 440 Aug 2, 2022
Blazing fast, structured, leveled logging in Go.

⚡ zap Blazing fast, structured, leveled logging in Go. Installation go get -u go.uber.org/zap Note that zap only supports the two most recent minor ve

Uber Go 16.6k Aug 9, 2022
Logrus is a structured, pluggable logging for Go.

Logrus is a structured logger for Go (golang), completely API compatible with the standard library logger.

Simon Eskildsen 274 May 25, 2021
Fully asynchronous, structured, pluggable logging for Go.

logr Logr is a fully asynchronous, contextual logger for Go. It is very much inspired by Logrus but addresses two issues: Logr is fully asynchronous,

Mattermost 14 Jul 12, 2022
structured logging helper

Logart Logart is a structured logging tool that aims to simplify logging to a database It is not yet in stable state, but is used in production and ac

Karitham 3 Apr 24, 2021
Log-structured virtual disk in Ceph

lsd_ceph Log-structured virtual disk in Ceph 1. Vision and Goals of the Project Implement the basic librbd API to work with the research block device

null 3 Dec 13, 2021
Go-metalog - Standard API for structured logging

Metalog is a standard API for structured logging and adapters for its implementa

Kirill 4 Jan 20, 2022