A simple web service for storing text log files

Overview

logpaste

CircleCI Docker Pulls License

A minimalist web service for uploading and sharing log files.

Run locally

go run main.go

Run in local Docker container

The Docker container automatically replicates to an AWS S3 bucket

AWS_ACCESS_KEY_ID=YOUR-ACCESS-ID
AWS_SECRET_ACCESS_KEY=YOUR-SECRET-ACCESS-KEY
AWS_REGION=YOUR-REGION
DB_REPLICA_URL=s3://your-bucket-name/db

docker run \
  -e "AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}" \
  -e "AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}" \
  -e "AWS_REGION=${AWS_REGION}" \
  -e "DB_REPLICA_URL=${DB_REPLICA_URL}" \
  -e "CREATE_NEW_DB='true'" `# change to false after first run` \
  --name logpaste \
  mtlynch/logpaste

Run with custom site settings

SITE_TITLE="My Cool Log Pasting Service"
SITE_SUBTITLE="Upload all your logs for FooBar here"
SITE_SHOW_DOCUMENTATION="false" # Hide usage information from homepage

AWS_ACCESS_KEY_ID=YOUR-ACCESS-ID
AWS_SECRET_ACCESS_KEY=YOUR-SECRET-ACCESS-KEY
AWS_REGION=YOUR-REGION
DB_REPLICA_URL=s3://your-bucket-name/db

docker run \
  -e "AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}" \
  -e "AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}" \
  -e "AWS_REGION=${AWS_REGION}" \
  -e "DB_REPLICA_URL=${DB_REPLICA_URL}" \
  -e "SITE_TITLE=${SITE_TITLE}" \
  -e "SITE_SUBTITLE=${SITE_SUBTITLE}" \
  -e "SITE_SHOW_DOCUMENTATION=${SITE_SHOW_DOCUMENTATION}" \
  -e "CREATE_NEW_DB='true'" `# change to false after first run` \
  --name logpaste \
  mtlynch/logpaste
Issues
  • APIkey error on non-aws bucket

    APIkey error on non-aws bucket

    Trying to use filebase as bucket results in this error, using it with other programs I get no such error(s3fs)

    monitor error: cannot lookup bucket region: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
    

    My litestream.yml

    access-key-id: ${LITESTREAM_ACCESS_KEY_ID}
    secret-access-key: ${LITESTREAM_SECRET_ACCESS_KEY}
    dbs:
      - path: ${DB_PATH}
        replicas:
          - url: s3://mypaste-new.filebase.com
            sync-interval: ${DB_SYNC_INTERVAL}
    
    opened by Dark-Matter7232 4
  • Add litestream health check

    Add litestream health check

    If logpaste expects litestream to be replicating data, and it's not, it should fail hard so that users stop uploading logs that will never be replicated.

    Maybe the Go app can have a periodic check similar to what's mentioned in this Kubernetes example and panic if litestream is no longer replicating.

    opened by mtlynch 4
  • Feature request: Dark mode.

    Feature request: Dark mode.

    Hello!

    I recently discovered logpaste, and it's a pleasure to use so far. Easy to set up, self host, and use! However, the text isn't always the easiest to read because of the white background on the home page. May can I ask, would it be possible to add dark mode, which can be enabled using a docker environment value?

    opened by KoxSosen 3
  • Limit maximum request size in pastePut

    Limit maximum request size in pastePut

    Without this, clients can upload arbitrarily-large pastes. Such pastes will be caught later by validatePaste, but in the interim, the entire paste is held in memory, so a malicious client could trivially pipe /dev/urandom into curl and eventually cause your server to OOM.

    opened by lukechampine 3
  • Do not assume S3

    Do not assume S3

    I successfully deployed logpaste to Google Cloud on Google Cloud Run and Google Cloud Storage

    I had the change the code in the following way:

    Then, I built with Cloud Build gcloud builds submit . --tag gcr.io/<PROJECT>/logpaste:latest

    And deployed to Cloud Run: gcloud beta run deploy logpaste --image gcr.io/<PROJECT>/logpaste:latest --set-env-vars DB_REPLICA_URL=gcs://<BUCKET>/db --execution-environment gen2 --no-cpu-throttling

    I can try to send a PR: we can enable litestream if DB_REPLICA_URL is set, but I'm not sure about the best way handle the translation of envvars to litestream.yml

    opened by steren 2
  • Switch to native HTTP muxer

    Switch to native HTTP muxer

    We're using the third-party gorilla mux package, but I don't think we really need it. We can probably get away with the standard library implementation.

    wontfix 
    opened by mtlynch 1
  • Update fly.io.md

    Update fly.io.md

    Thanks for making this great doc -- I just went through it and wanted to suggest some edits here:

    1/ You don't need docker installed if you are just deploying following these instructions -- I say this because I deployed WITHOUT having docker installed. 2/ When I ran fly init "${APP_NAME}" --nowrite - the fly cli said Error We need your payment information to continue! Add a credit card or buy credit: https://fly.io/organizations/personal

    opened by wasauce 1
  • Restore fails on Google Cloud Run

    Restore fails on Google Cloud Run

    Works fine in a local container, but fails for some reason on Google Cloud Run

    + litestream restore -v /app/data/store.db 
    2021/02/22 15:11:08 /app/data/store.db(s3): starting restore: generation 68ce1ab8c28d0111, index 00000000-00000000 
    2021/02/22 15:11:08 /app/data/store.db(s3): restoring snapshot 68ce1ab8c28d0111/00000000 to /app/data/store.db.tmp 
    cannot restore wal: disk I/O error: invalid argument 
    
    opened by mtlynch 1
  • Use DB_REPLICA_URL to determine if Litestream is enabled

    Use DB_REPLICA_URL to determine if Litestream is enabled

    We were previously using environment variables related to Litestream's S3 replica, but that's a brittle way to check, since sftp doesn't rely on any S3 variables. Using DB_REPLICA_URL gives a more reliable answer as to whether the configuration relies on Litestream.

    Related: #133

    opened by mtlynch 0
  • Exercise litestream in e2e test

    Exercise litestream in e2e test

    #129 slipped through because logpaste doesn't use litestream in the e2e test. This is only a light test on litestream as it doesn't verify that backups are actually working, but it's at least good to verify that we can launch the binary successfully.

    opened by mtlynch 0
  • Add option to force HTTPS

    Add option to force HTTPS

    One small thing to mention: When using curl to upload text, the output URL defaults to http. Most webservers should already redirect from port 80 to port 443, but I think it would be nice to add an option to enable https everywhere instead of the http output.

    Originally posted by @KoxSosen in https://github.com/mtlynch/logpaste/issues/121#issuecomment-963351527

    opened by mtlynch 0
Releases(0.2.7)
  • 0.2.7(Jul 25, 2022)

  • 0.2.6(Jul 25, 2022)

    • Add deployment instructions for Google Cloud Run by @mtlynch in https://github.com/mtlynch/logpaste/pull/135
    • Add litestream's recommended pragmas for SQLite by @mtlynch in https://github.com/mtlynch/logpaste/pull/139
    • Upgrade to go v1.18.4 by @mtlynch in https://github.com/mtlynch/logpaste/pull/145
    • Upgrade to Litestream v0.3.9 by @mtlynch in https://github.com/mtlynch/logpaste/pull/144

    Full Changelog: https://github.com/mtlynch/logpaste/compare/0.2.5...0.2.6

    Source code(tar.gz)
    Source code(zip)
  • 0.2.5(Dec 5, 2021)

  • 0.2.4(Nov 9, 2021)

    • Adds support for dark mode theme on homepage (#121)
      • To use it, add the -darkmode flag when launching the binary directly or set an environment variable to SITE_DARK_MODE="true" when launching via Docker.
    Source code(tar.gz)
    Source code(zip)
  • 0.2.3(Oct 13, 2021)

  • 0.2.2(Aug 7, 2021)

    • Updates to litestream 0.3.5
    • Use litestream's -exec flag to run logpaste as a child process in Docker to minimize the risk of data loss. (#115)
    Source code(tar.gz)
    Source code(zip)
  • 0.2.1(Apr 30, 2021)

    • Fixes a Denial of Service vulnerability (#111, #113)
      • Fixes an issue where a malicious user could send LogPaste a very large request to exhaust memory or disk space.

    Thanks to @lukechampine for spotting this issue and providing a fix!

    Source code(tar.gz)
    Source code(zip)
  • 0.2.0(Apr 28, 2021)

    • Upgrades to Litestream v0.3.4
    • You no longer have to deploy twice to initialize a new database

    Breaking changes

    • All AWS_ environment variables have been renamed so that the prefix is LITESTREAM_ instead of AWS_
    Source code(tar.gz)
    Source code(zip)
  • 0.1.1(Apr 2, 2021)

  • 0.1.0(Apr 2, 2021)

Distributed-Log-Service - Distributed Log Service With Golang

Distributed Log Service This project is essentially a result of my attempt to un

Hamza Yusuff 6 Jun 1, 2022
An golang log lib, supports tracking and level, wrap by standard log lib

Logex An golang log lib, supports tracing and level, wrap by standard log lib How To Get shell go get gopkg.in/logex.v1 source code import "gopkg.in/

chzyer 39 Apr 15, 2022
Nginx-Log-Analyzer is a lightweight (simplistic) log analyzer for Nginx.

Nginx-Log-Analyzer is a lightweight (simplistic) log analyzer, used to analyze Nginx access logs for myself.

Mao Mao 22 Jul 23, 2022
Log-analyzer - Log analyzer with golang

Log Analyzer what do we have here? Objective Installation and Running Applicatio

Lawrence Agbani 0 Jan 27, 2022
BRUS - Parses your web server (e.g. nginx) log files and checks with GreyNoise how much noise your website is exposed to.

BRUS bbbbbb rrrrrr u u sssss b b r r u u s bbbbbb rrrrrr u u sssss b b r r u u s bbbbbb r r

dubs3c 1 May 29, 2022
Simple log parser written in Golang

Simple log parser written in Golang

Matteo Baiguini 0 Oct 31, 2021
Log-generator - A simple CLI tool that generates near real logs for testing

Log-generator - A simple CLI tool that generates near real logs for testing

RexWu 2 Jan 22, 2022
A simple to use log system, minimalist but with features for debugging and differentiation of messages

A simple to use log system, minimalist but with features for debugging and differentiation of messages

structy 4 Jan 27, 2022
Simple text-line analog clock

anaclock anaclock prints a simple analog clock as a line of text. Demo $ anaclock 23 . : .| 00 anaclock is easy to use in CLI prompts or anywhere

null 0 Dec 31, 2021
a golang log lib supports level and multi handlers

go-log a golang log lib supports level and multi handlers Use import "github.com/siddontang/go-log/log" //log with different level log.Info("hello wo

siddontang 31 Jun 15, 2022
Structured log interface

Structured log interface Package log provides the separation of the logging interface from its implementation and decouples the logger backend from yo

teris.io 25 Jul 29, 2022
lumberjack is a log rolling package for Go

lumberjack Lumberjack is a Go package for writing logs to rolling files. Package lumberjack provides a rolling logger. Note that this is v2.0 of lumbe

Nate Finch 3.5k Aug 4, 2022
CoLog is a prefix-based leveled execution log for Go

What's CoLog? CoLog is a prefix-based leveled execution log for Go. It's heavily inspired by Logrus and aims to offer similar features by parsing the

null 156 Aug 2, 2022
OpenTelemetry log collection library

opentelemetry-log-collection Status This project was originally developed by observIQ under the name Stanza. It has been contributed to the OpenTeleme

OpenTelemetry - CNCF 88 Jul 11, 2022
exo: a process manager & log viewer for dev

exo: a process manager & log viewer for dev exo- prefix – external; from outside. Features Procfile compatible process manager.

Deref 327 Aug 1, 2022
Write log entries, get X-Ray traces.

logtoxray Write to logs, get X-Ray traces. No distributed tracing instrumenation library required. ?? ?? ?? THIS PROJECT IS A WORK-IN-PROGRESS PROTOTY

JBD 27 Apr 24, 2022
Binalyze logger is an easily customizable wrapper for logrus with log rotation

logger logger is an easily customizable wrapper for logrus with log rotation Usage There is only one function to initialize logger. logger.Init() When

Binalyze 26 Nov 18, 2021
Log-structured virtual disk in Ceph

lsd_ceph Log-structured virtual disk in Ceph 1. Vision and Goals of the Project Implement the basic librbd API to work with the research block device

null 3 Dec 13, 2021
Multi-level logger based on go std log

mlog the mlog is multi-level logger based on go std log. It is: Simple Easy to use NOTHING ELSE package main import ( log "github.com/ccpaging/lo

null 0 May 18, 2022