Simple Proof of Concept REST event logger.

Overview

REST Event Logger PoC

I am working on this project intermittently. I have set myself a time limit of ~3hrs which includes the time to acquire and adapt to Golang. This is an extremely quick Proof of Concept for demonstration purposes only. It shall additionally serve as a refresher to Golang.

Gorilla MUX library is used for concurrent multiplexed HTTP request handling.

Live demo

Please note that Heroku wipes all stored files every hour so updates to logs might vanish at the top of the hour.

Local host testing will attempt to read the environment variable PORT, and if not found will set the listening port to 45456.

E.g.: http://localhost:45456

TODO's and Shortcuts

  • Git Flow: No development branching for features. This is not good in a production environment.
  • Test Suite: Since this is not meant for production and is a very quick exercise I have not had the time to build a test suite. This is a poor practice and does not fly in production environments.
  • Code Coverage: Time constrained.
  • Benchmarking: Time constrained.
  • Profiling: Time constrained.
  • HTTP Codes: Appropriate HTTP response codes should be sent for errors and success.
  • Write Chunking: Log entries should be buffered and then written to disk to improve performance.
  • Error Handling: Better and more careful error handling.
  • Request Validation: Validate the inbound requests and reject any that are malformed.

Event Tuple

  • ServiceName: Reporting service's name.
  • ServerID: Server's unique identifier.
  • Date: Date of event. ddmmyyyy
  • Time: Time of event. hhmmss
  • Level: Level of event: Critical, Warn, Info, etc.
  • EventType: Type of event is specific to service.
  • Description: Details of the event.

Details

Statelessness

The design outlined below is not stateless which will lead to high availability and fault-tolerance issues. One solution to this issue is to use a NoSQL database like Cassandra for data storage as this is a write-intensive service. Another solution is to use object stores (Ceph or AWS S3) or NFS as the backing storage. This might require local storage replicated and then written at time intervals to object stores.

Storage

Events are tail appended to files stored on disk in the file structure service_name/server_id/date.log.

Using a file on a disk that is tail appended too will improve memory consumption but result in reduced performance due to writes to disk. Ideally, writes would be chunked in blocks and tail-appended to files on disk. Tail-appending in blocks also improves concurrency as we will not require implicitly locking of the file to append. This is a similar scheme to that which is used in Apache Kafka.

The file structure above is efficient for ETL/ELT jobs (Spark etc.) when moving the logs to a data warehouse (OLAP) for analysis. Loglines are comma seperated and contain the service name, server name, and date despite it being contained in the folder structure; this is for big data jobs.

Event Logger Server Logs

You may view the activity of the Event Logger by visiting /logs

Example

server_address/logs

Submission

Events are submitted to the server via the body of an HTTP: put with the details of the event structured in JSON. The url would be server_address/append/.

Examples
curl -X POST http://localhost:45456/append \
   -H "Content-Type: application/json" \
   -d '{"service_name": "serviceA", "server_id": "server001", "date": "09022022", "time": "000102", "level": "INFO", "event_type": "Account Created", "description": "New user Bilbo Baggins"}'
curl -X POST http://localhost:45456/append \
   -H "Content-Type: application/json" \
   -d '{"service_name": "serviceD", "server_id": "server045", "date": "09022022", "time": "000102", "level": "SEVERE", "event_type": "Hullaballoo", "description": "Some strange stuff happened here."}'

Retrieval

Events are retrieved via an HTTP: get on the url structure server_address/service_name/server_id/date/. This will return all events on the specific date for a specific service's server. In a production system the log would be read and written back to the client in chunks as they can get very large.

Example

server_address/logs/serviceA/server001/19092022

Owner
Saad Ur Rahman
Distributed & Data Systems.
Saad Ur Rahman
Convenient Logger interface and std logger wrapper

Convenient logger interface and wrapper around std logger Interface type Logger interface { Error(err error) Debugf(format string, args ...interface

Denis Mitrofanov 1 Nov 28, 2021
Logger - A thin wrapper of uber-go/zap logger for personal project

a thin wraper of uber-go/zap logger for personal project 0. thanks uber-go/zap B

tsingson 1 Jun 16, 2022
Simple logger for Go programs. Allows custom formats for messages.

go-logger A simple go logger for easy logging in your programs. Allows setting custom format for messages. Preview Install go get github.com/apsdehal/

Amanpreet Singh 276 Jun 14, 2022
Dead simple, super fast, zero allocation and modular logger for Golang

Onelog Onelog is a dead simple but very efficient JSON logger. It is one of the fastest JSON logger out there. Also, it is one of the logger with the

Francois Parquet 399 Jun 16, 2022
Simple Yet Powerful Logger

sypl sypl provides a Simple Yet Powerful Logger built on top of the Golang sypl. A sypl logger can have many Outputs, and each Output is responsible f

Sauce Labs 9 Apr 28, 2022
simple concurrent logger

XMUS-LOGGER pure golang logger compatible with golang io standards. USAGE : logOptions := logger.LoggerOptions{ LogLevel: 6, // read more about lo

amupxm [amir hossein mokarrami far] 6 Aug 1, 2022
🪵 A dead simple, pretty, and feature-rich logger for golang

?? lumber ?? A dead simple, pretty, and feature-rich logger for golang ?? Install ?? Logging Functions lumber.Success() lumber.Info() lumber.Debug() l

Matt Gleich 50 Jul 20, 2022
A simple logger API.

flog a simple logger API for Go program that save logs into a file. NOTE: This package is provided "as is" with no guarantee. Use it at your own risk

Iss Meftah 3 May 14, 2022
A simple Go JSON logger.

logger A simple JSON logger for Go. It uses a context.Context to store values which will then be logged along with each message. It is possible to rec

Shawn Milochik 2 Jul 25, 2022
A simple and super power logger for golang

The most powerfull and faster logger for golang powered by DC ?? What is this? W

Teo 6 Jun 26, 2022
A Simple logger for golang

go-logger Installation go get github.com/mo-taufiq/go-logger Quickstart package main import ( gologger "github.com/mo-taufiq/go-logger" ) func main

Moh Taufiqur Rohman 1 Jan 15, 2022
Golog is a logger which support tracing and other custom behaviors out of the box. Blazing fast and simple to use.

GOLOG Golog is an opinionated Go logger with simple APIs and configurable behavior. Why another logger? Golog is designed to address mainly two issues

Damiano Petrungaro 34 Jul 26, 2022
A logger, for Go

Go-Log A logger, for Go! It's sort of log and code.google.com/p/log4go compatible, so in most cases can be used without any code changes. Breaking cha

Ian Kent 38 May 12, 2022
Loggly Hooks for GO Logrus logger

Loggly Hooks for Logrus Usage package main import ( "github.com/sirupsen/logrus" "github.com/sebest/logrusly" ) var logglyToken string = "YOUR_LOG

Sebest 27 Jul 27, 2021
A 12-factor app logger built for performance and happy development

logxi log XI is a structured 12-factor app logger built for speed and happy development. Simpler. Sane no-configuration defaults out of the box. Faste

Mario Gutierrez 348 Jun 25, 2022
A logger for Go SQL database driver without modify existing *sql.DB stdlib usage.

SQLDB-Logger A logger for Go SQL database driver without modify existing *sql.DB stdlib usage. Colored console writer output above only for sample/dev

Sarjono Mukti Aji 257 Aug 16, 2022
xlog is a logger for net/context aware HTTP applications

⚠️ Check zerolog, the successor of xlog. HTTP Handler Logger xlog is a logger for net/context aware HTTP applications. Unlike most loggers, xlog will

Olivier Poitrey 135 Nov 22, 2021
Zero Allocation JSON Logger

Zero Allocation JSON Logger The zerolog package provides a fast and simple logger dedicated to JSON output. Zerolog's API is designed to provide both

Olivier Poitrey 6.7k Aug 17, 2022
A powerful zero-dependency json logger.

ZKits Logger Library About This package is a library of ZKits project. This is a zero-dependency standard JSON log library that supports structured JS

Qingshan Luo 18 Aug 10, 2022