Gomol is a library for structured, multiple-output logging for Go with extensible logging outputs

Overview

gomol

GoDoc Build Status Code Coverage Go Report Card

Gomol (Go Multi-Output Logger) is an MIT-licensed structured logging library for Go. Gomol grew from a desire to have a structured logging library that could write to any number of outputs while also keeping a small in-band footprint.

Features

  • Attach meta-data to each log message with attributes
  • Multiple outputs at the same time
  • Pluggable Logger interface
  • Asynchronous logging so slow loggers won't slow down your application

Installation

Gomol can also be installed the standard way for Go:

go get github.com/aphistic/gomol
...
import "github.com/aphistic/gomol"

Vendoring is recommended!

Loggers

Gomol has a growing list of supported logging formats. The known loggers are listed below. If you have a logger you've written to support gomol and you'd like to add it to this list please either submit a pull request with the updated document or let me know and I can add it!

Other Usages

In addition to the loggers listed above, gomol can be used with other projects as well.

Examples

For brevity a lot of error checking has been omitted, be sure you do your checks!

This is a super basic example of adding a number of loggers and then logging a few messages:

package main

import (
	"github.com/aphistic/gomol"
	gc "github.com/aphistic/gomol-console"
	gg "github.com/aphistic/gomol-gelf"
)

func main() {
	// Add a console logger
	consoleCfg := gc.NewConsoleLoggerConfig()
	consoleLogger, _ := gc.NewConsoleLogger(consoleCfg)
	// Set the template to display the full message, including
	// attributes.
	consoleLogger.SetTemplate(gc.NewTemplateFull())
	gomol.AddLogger(consoleLogger)

	// Add a GELF logger
	gelfCfg := gg.NewGelfLoggerConfig()
	gelfCfg.Hostname = "localhost"
	gelfCfg.Port = 12201
	gelfLogger, _ := gg.NewGelfLogger(gelfCfg)
	gomol.AddLogger(gelfLogger)

	// Set some global attrs that will be added to all
	// messages automatically
	gomol.SetAttr("facility", "gomol.example")
	gomol.SetAttr("another_attr", 1234)

	// Configure gomol to add the filename and line number that the
	// log was generated from, the internal sequence number to help
	// with ordering events if your log system doesn't support a
	// small enough sub-second resolution, and set the size of the
	// internal queue (default is 10k messages).
	cfg := gomol.NewConfig()
	cfg.FilenameAttr = "filename"
	cfg.LineNumberAttr = "line"
	cfg.SequenceAttr = "sequence"
	cfg.MaxQueueSize = 50000
	gomol.SetConfig(cfg)

	// Initialize the loggers
	gomol.InitLoggers()
	defer gomol.ShutdownLoggers()

	// Create a channel on which to receive internal (asynchronous)
	// logger errors. This is optional, but recommended in order to
	// determine when logging may be dropping messages.
	ch := make(chan error)

	go func() {
		// This consumer is expected to be efficient as writes to 
		// the channel are blocking. If this handler is slow, the
		// user should add a buffer to the channel, or manually
		// queue and batch errors for processing.

		for err := range ch {
			fmt.Printf("[Internal Error] %s\n", err.Error())
		}
	}()

	gomol.SetErrorChan(ch)

	// Log some debug messages with message-level attrs
	// that will be sent only with that message
	for idx := 1; idx <= 10; idx++ {
		gomol.Dbgm(gomol.NewAttrsFromMap(map[string]interface{}{
			"msg_attr1": 4321,
		}), "Test message %v", idx)
	}
}

Fallback Logger

One feature gomol supports is the concept of a fallback logger. In some cases a logger may go unhealthy (a logger to a remote server, for example) and you want to log to a different logger to ensure log messages are not lost. The SetFallbackLogger method is available for these such instances.

A fallback logger will be triggered if any of the primary loggers goes unhealthy even if all others are fine. It's recommended the fallback logger not be added to the primary loggers or you may see duplicate messages. This does mean if multiple loggers are added as primary loggers and just one is unhealthy the fallback logger logger will be triggered.

To add a fallback logger there are two options. One is to use the default gomol instance (gomol.SetFallbackLogger()) and the other is to use the method on a Base instance:

import (
	"github.com/aphistic/gomol"
	"github.com/aphistic/gomol-console"
	"github.com/aphistic/gomol-json"
)

func main() {
	// Create a logger that logs over TCP using JSON
	jsonCfg := gomoljson.NewJSONLoggerConfig("tcp://192.0.2.125:4321")
	// Continue startup even if we can't connect initially
	jsonCfg.AllowDisconnectedInit = true
	jsonLogger, _ := gomoljson.NewJSONLogger(jsonCfg)
	gomol.AddLogger(jsonLogger)

	// Create a logger that logs to the console
	consoleCfg := gomolconsole.NewConsoleLoggerConfig()
	consoleLogger, _ := gomolconsole.NewConsoleLogger(consoleCfg)

	// Set the fallback logger to the console so if the
	// TCP JSON logger is unhealthy we still get logs
	// to stdout.
	_ = gomol.SetFallbackLogger(consoleLogger)

	gomol.InitLoggers()
	defer gomol.ShutdownLoggers()

	gomol.Debug("This is my message!")
}
Issues
  • Can't set caller offset

    Can't set caller offset

    If filename and line numbers are turned on, but I have a wrapper around a the log calls, then the line number is always at this abstraction level. I'd like to be able to do something like gomol.SetCallerOffset(1) which would look one more call up the stack before returning.

    Otherwise, since it doesn't look like your caller information is based on index but on strings, provide a way to blacklist packages so that it will skip gomol and this particular filename or package.

    opened by efritz 3
  • Address unbounded queue (Issue #20)

    Address unbounded queue (Issue #20)

    Here is a solution, but still needs some input. In short, the queue worker was rewritten so that it doesn't need to shovel values off of a channel into a slice (I'm not sure the reason for this in the first place, if you're only reading off the slice in order in another worker). I'm assuming that this was an attempt to stop from blocking the application thread.

    Instead, all coordination is now done through a single message channel. If the channel is full when trying to enqueue a message, an old one is popped off and the enqueue is retried. We've been doing similar things in the Riemann and AMQP clients.

    Two things we need to discuss before this would be mergeable:

    1. How do you want to set a max queue size, as it's required when creating the queue (we could move this to initialization). I'm not super familiar with how the base logger is initialized, so maybe just supplying a package function SetMaxQueueCapacity may work as long as you call it before you call InitializeLoggers (and we move newQueue to InitLoggers instead of NewBase).
    2. We need to make the log dropping event observable by the user. This could be a callback that can be registered or a channel. Maybe we should talk about Issue #12 before finalizing this, as that would likely be the same solution.
    opened by efritz 3
  • demo failed.

    demo failed.

    I run the example but get "cannot use map[string]interface {} literal (type map[string]interface {}) as type *gomol.Attrs in argument to gomol.Dbgm".

    On the other hand, it seems the configuration for line number and filename is not working. My code is as below:

    	config := gomol.NewConfig()
    	config.FilenameAttr = "filename"
    	config.LineNumberAttr = "line"
    	b := gomol.NewBase()
    	b.SetConfig(config)
    
    	// Add a console logger
    	consoleCfg := gc.NewConsoleLoggerConfig()
    	consoleLogger, _ := gc.NewConsoleLogger(consoleCfg)
    	b.AddLogger(consoleLogger)
    
    	// Set some global attrs that will be added to all
    	// messages automatically
    	b.SetAttr("another_name", "ele")
    
    	// Initialize the loggers
    	b.InitLoggers()
    	defer b.ShutdownLoggers()
    
    	// Log some debug messages with message-level attrs
    	// that will be sent only with that message
    	for i := 0; i < 10; i++ {
    		b.Infom(gomol.NewAttrsFromMap(map[string]interface{}{"test": 1}),
    			"msg string: %d", i)
    	}
    

    and the output is

    [INFO] msg string: 0
    [INFO] msg string: 1
    [INFO] msg string: 2
    [INFO] msg string: 3
    [INFO] msg string: 4
    [INFO] msg string: 5
    [INFO] msg string: 6
    [INFO] msg string: 7
    [INFO] msg string: 8
    [INFO] msg string: 9
    

    no linenumber, no filename as well as attr I set.

    opened by ictar 2
  • Create some sort of channel that users of the library can listen to for errors that happen asynchronously

    Create some sort of channel that users of the library can listen to for errors that happen asynchronously

    For example, when a message is originally logged and is successfully added to the queue, but when the message is actually "sent" (via whatever logger) it fails. Adding this channel will allow applications to look into the errors that are happening "out of band" to improve troubleshooting.

    opened by aphistic 2
  • Add a utility method to get a log level from a string

    Add a utility method to get a log level from a string

    In most cases people won't want to write their own methods to translate from a string to a concrete log level, so make a function that does it for them for standard values.

    opened by aphistic 1
  • Gomol internal message queue grows unbounded

    Gomol internal message queue grows unbounded

    The internal message queue used before messages are written to the loggers has no upper bounds, so if one or more loggers are being bad citizens it could cause the queue to grow unbounded. There should be a default upper limit that can be configured by the user to allow them to handle this case.

    opened by aphistic 1
  • add filename and line number

    add filename and line number

    I'm not sure if this should be on by default or not, but it'd be really handy if we could get the filename and line number of the log message as attributes.

    See https://godoc.org/runtime#Caller for how to do it.

    opened by grimmy 1
  • Base cannot be re-initialized

    Base cannot be re-initialized

    Init/Shutdown/Init will cause a panic due to use of closed channels in the queue. This can be fixed simply with the following change in base.go:

            if b.queue != nil {
                    b.queue.stopWorker()
                    b.queue = nil // Line added here
            }
    

    This allows the queue to be re-initialized on a subsequent call to Initialize.

    opened by efritz 0
  • Add support for a fallback logger

    Add support for a fallback logger

    Adds a fallback logger that will be logged to if any of the primary loggers go unhealthy. This is to avoid a situation where you don't want to log to something most of the time (console, for example) but you do when your primary logger is unavailable (such as a network partition).

    opened by aphistic 0
  • Give more context for the filename in runtime.go

    Give more context for the filename in runtime.go

    It would be nice to see the package that includes the filename instead of just the basename. Zap does this and seems rather nice (without needing the entire path nor the from-gopath path): https://github.com/uber-go/zap/blob/e15639dab1b6ca5a651fe7ebfd8d682683b7d6a8/zapcore/entry.go#L101

    opened by efritz 0
  • It's possible to make gomol stop logging without an error

    It's possible to make gomol stop logging without an error

    If envelope.TaskID is a function in adapter.SetAttr("task_id", envelope.TaskID) gomol will silently stop logging on that adapter. The console and json loggers were added in the case this is failing in.

    opened by aphistic 0
  • Allow log adapters to log at a lower level than the base log level

    Allow log adapters to log at a lower level than the base log level

    Right now the log level set on the base object set the lowest level anything can be logged at. It would be nice to be able to have a log adapter set to debug while the base is set to info, for example.

    v2 feature 
    opened by aphistic 0
  • Allow setting log level per logger

    Allow setting log level per logger

    Allow users to specify a log level per logger added to gomol. For example, debug can be logged to the console and info can be logged to a remote aggregator.

    v2 feature 
    opened by aphistic 0
  • Make sure gomol isn't preventing Loggers from being used with multiple gomol.Base instances

    Make sure gomol isn't preventing Loggers from being used with multiple gomol.Base instances

    It appears that when adding a single Logger to multiple gomol Base instances it might be mixing up the attrs that are being logged. Explore if this is a possibility and if not be sure to document it and possibly disallow it somehow.

    opened by aphistic 0
Owner
Kristin Davidson
Kristin Davidson
A minimal and extensible structured logger

⚠️ PRE-RELEASE ⚠️ DO NOT IMPORT THIS MODULE YOUR PROJECT WILL BREAK package log package log provides a minimal interface for structured logging in ser

Go kit 102 Jun 8, 2022
Hierarchical, leveled, and structured logging library for Go

spacelog Please see http://godoc.org/github.com/spacemonkeygo/spacelog for info License Copyright (C) 2014 Space Monkey, Inc. Licensed under the Apach

Space Monkey Go 98 Apr 27, 2021
Minimal structured logging library for Go

slog slog is a minimal structured logging library for Go. Install go get cdr.dev/slog Features Minimal API First class context.Context support First c

Coder 243 Jun 19, 2022
Structured logging package for Go.

Package log implements a simple structured logging API inspired by Logrus, designed with centralization in mind. Read more on Medium. Handlers apexlog

Apex 1.2k Jun 28, 2022
Simple, configurable and scalable Structured Logging for Go.

log Log is a simple, highly configurable, Structured Logging library Why another logging library? There's allot of great stuff out there, but also tho

Go Playgound 280 Jun 22, 2022
Structured, composable logging for Go

log15 Package log15 provides an opinionated, simple toolkit for best-practice logging in Go (golang) that is both human and machine readable. It is mo

Alan Shreve 1.1k Jun 24, 2022
Structured, pluggable logging for Go.

Logrus Logrus is a structured logger for Go (golang), completely API compatible with the standard library logger. Logrus is in maintenance-mode. We wi

Simon Eskildsen 20.7k Jun 20, 2022
Structured Logging Made Easy

Structured Logging Made Easy Features Dependency Free Simple and Clean Interface Consistent Writer IOWriter, io.Writer wrapper FileWriter, rotating &

phuslu 427 Jun 20, 2022
Blazing fast, structured, leveled logging in Go.

⚡ zap Blazing fast, structured, leveled logging in Go. Installation go get -u go.uber.org/zap Note that zap only supports the two most recent minor ve

Uber Go 16.2k Jun 30, 2022
Logrus is a structured, pluggable logging for Go.

Logrus is a structured logger for Go (golang), completely API compatible with the standard library logger.

Simon Eskildsen 274 May 25, 2021
Fully asynchronous, structured, pluggable logging for Go.

logr Logr is a fully asynchronous, contextual logger for Go. It is very much inspired by Logrus but addresses two issues: Logr is fully asynchronous,

Mattermost 13 Mar 3, 2022
structured logging helper

Logart Logart is a structured logging tool that aims to simplify logging to a database It is not yet in stable state, but is used in production and ac

Karitham 3 Apr 24, 2021
Go-metalog - Standard API for structured logging

Metalog is a standard API for structured logging and adapters for its implementa

Kirill 4 Jan 20, 2022
Yandex Cloud Logging output for Fluent Bit

Fluent Bit plugin for Yandex Cloud Logging Fluent Bit output for Yandex Cloud Logging. Configuration parameters Key Description group_id (optional) Lo

Yandex.Cloud 14 Jun 11, 2022
This package enables json output, level logging and so on to standard go logger.

logplug This package enables json output, level logging and so on to standard logger. Usage log.SetOutput(logplug.NewJSONPlug(os.Stderr, logplug.LogF

Koumei Mikuni 0 Dec 27, 2021
Simple and extensible monitoring agent / library for Kubernetes: https://gravitational.com/blog/monitoring_kubernetes_satellite/

Satellite Satellite is an agent written in Go for collecting health information in a kubernetes cluster. It is both a library and an application. As a

Teleport 193 Mar 26, 2022
A Go (golang) package providing high-performance asynchronous logging, message filtering by severity and category, and multiple message targets.

ozzo-log Other languages 简体中文 Русский Description ozzo-log is a Go package providing enhanced logging support for Go programs. It has the following fe

Ozzo Framework 118 Jan 11, 2022
A simple logging module for go, with a rotating file feature and console logging.

A simple logging module for go, with a rotating file feature and console logging. Installation go get github.com/jbrodriguez/mlog Usage Sample usage W

Juan B. Rodriguez 25 Jun 11, 2022
FactorLog is a logging infrastructure for Go that provides numerous logging functions for whatever your style may be

FactorLog FactorLog is a fast logging infrastructure for Go that provides numerous logging functions for whatever your style may be. It could easily b

Kevin Darlington 54 Oct 12, 2021