A lightweight event collection system.

Overview

Honeypot

agplv3 GitHub tag (latest SemVer)

Honeypot

A self-contained, multi-protocol streaming event collection system with ambitions to be as boring as benthos.

Honeypot is primarily built for flexibility, scalability, and speed.

Secondarily for configuration, deployment, and management ease.

Honeypot Quickstart

Quickstart documentation for setting up an end-to-end streaming analytics stack with Honeypot, Redpanda, Materialize, and Kowl can be found here.

Honeypot Documentation

Documentation can be found here. It is a WIP!

Comments
  • Short-circuit and return 400 when snowplow requests are invalid

    Short-circuit and return 400 when snowplow requests are invalid

    Right now, sometimes sending unexpected payloads can cause 500 responses with panics:

    import requests
    payload = {
      "schema": "iglu:com.snowplowanalytics.snowplow/unstruct_event/jsonschema/1-0-0",
      "data": {
        "schema": "iglu:com.my_company/viewed_product/jsonschema/1-0-0",
        "data": {
          "product_id": "ASO01043",
          "price": 49.95
        }
      }
    }
    requests.post('http://localhost:8080/com.snowplowanalytics.snowplow/tp2', json=payload)
    
    2022/09/06 23:08:07 [Recovery] 2022/09/06 - 23:08:07 panic recovered:
    POST /com.snowplowanalytics.snowplow/tp2 HTTP/1.1
    Host: localhost:8080
    Accept: */*
    Accept-Encoding: gzip, deflate
    Connection: keep-alive
    Content-Length: 208
    Content-Type: application/json
    User-Agent: python-requests/2.27.1
    
    
    runtime error: invalid memory address or nil pointer dereference
    /usr/local/Cellar/go/1.19/libexec/src/runtime/panic.go:260 (0x103b9e4)
            panicmem: panic(memoryError)
    /usr/local/Cellar/go/1.19/libexec/src/runtime/signal_unix.go:835 (0x1053a1c)
            sigpanic: panicmem()
    /Users/jpcassil/buz/pkg/snowplow/eventBuilder.go:169 (0x1bf16f9)
            setTstamps: e.DvceCreatedTstamp = *getTimeParam(params, "dtm")
    /Users/jpcassil/buz/pkg/snowplow/eventBuilder.go:355 (0x1bf3ee7)
            BuildEventFromMappedParams: setTstamps(&event, params)
    /Users/jpcassil/buz/pkg/envelope/builderSnowplow.go:106 (0x1bf70fe)
            BuildSnowplowEnvelopesFromRequest: spEvent := snowplow.BuildEventFromMappedParams(c, event.Value().(map[string]interface{}), *conf)
    /Users/jpcassil/buz/pkg/handler/snowplow.go:21 (0x2d9da2e)
            SnowplowHandler.func1: envelopes := envelope.BuildSnowplowEnvelopesFromRequest(c, h.Config, h.CollectorMeta)
    /Users/jpcassil/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:168 (0x181145c)
            (*Context).Next: c.handlers[c.index](c)
    /Users/jpcassil/buz/pkg/middleware/identity.go:50 (0x2da8e5c)
            Identity.func1: c.Next()
    /Users/jpcassil/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:168 (0x181145c)
            (*Context).Next: c.handlers[c.index](c)
    /Users/jpcassil/buz/pkg/middleware/cors.go:28 (0x2da899a)
            CORS.func1: c.Next()
    /Users/jpcassil/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:168 (0x181145c)
            (*Context).Next: c.handlers[c.index](c)
    /Users/jpcassil/go/pkg/mod/github.com/gin-gonic/[email protected]/recovery.go:99 (0x18220d7)
            CustomRecoveryWithWriter.func1: c.Next()
    /Users/jpcassil/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:168 (0x181145c)
            (*Context).Next: c.handlers[c.index](c)
    /Users/jpcassil/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:555 (0x181fe93)
            (*Engine).handleHTTPRequest: c.Next()
    /Users/jpcassil/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:511 (0x181f964)
            (*Engine).ServeHTTP: engine.handleHTTPRequest(c)
    /usr/local/Cellar/go/1.19/libexec/src/net/http/server.go:2947 (0x14ad873)
            serverHandler.ServeHTTP: handler.ServeHTTP(rw, req)
    /usr/local/Cellar/go/1.19/libexec/src/net/http/server.go:1991 (0x14a73db)
            (*conn).serve: serverHandler{c.server}.ServeHTTP(w, w.req)
    /usr/local/Cellar/go/1.19/libexec/src/runtime/asm_amd64.s:1594 (0x10705a0)
            goexit: BYTE    $0x90   // NOP
    

    We should consider building in some nice mechanisms for graceful failures and helpful responses, where applicable.

    I'm not sure what about this payload in particular caused a panic.

    opened by johncassil 2
  • Snowplow example not working.

    Snowplow example not working.

    Perhaps I misunderstood this section: https://github.com/silverton-io/honeypot/blob/main/website/docs/examples/quickstart.md#4-send-events-to-honeypot

    but it indicates that the page it serves up at 8080 has snowplow integrated for easy testing. I don't think that's the case... I just get 404 page not found and I also get 404s when trying to POST or GET anything from localhost or where I have this served up in GCP right now. Not sure if I'm doing something wrong...

    opened by jpcassil 2
  • Add configurable sink-level retries

    Add configurable sink-level retries

    Right now sinks do not retry if they fail. Having that level of guarantee is important.

    Sinks should be independently configurable to retry yes/no, and it would be cool to configure the retry strategy. Exponential decay with min/max? Something else?

    opened by jakthom 2
  • Make application secrets more flexible

    Make application secrets more flexible

    Currently there's one way of providing secrets to honeypot - a file titled config.yml in the same directory as the binary. This is obviously non-ideal.

    Options (probably all of them are ideal)

    1. Add env var specifying config file location override
    2. Add env-var-based config overrides: LEVEL1_LEVEL2_VARNAME
    3. Add default config.yml to the docker build
    opened by jakthom 2
  • Add tf outputs, add system/env flexibility

    Add tf outputs, add system/env flexibility

    This PR:

    1. Shuffles the terraform directory under deploy
    2. Adds system and env vars to multiple environments can be rapidly provisioned using the same code
    3. Shuffles variable building from main to locals
    4. Autogenerate cloud run service revision names
    5. Adds outputs
    6. Make bucket naming be user-flexible (admittedly this might be a step backward, since the using project id's in bucket names to keep them globally unique is 🔥 )
    opened by jakthom 1
  • 391: GCP Terraform

    391: GCP Terraform

    Issue 391

    Terraform for the GCP deploy as outlined by the GCP documentation steps. First ever Terraform for GCP so may not be following all the best standards. Main hurdle is that it has to be applied in two steps because of the manual process of uploading the image. Otherwise seems to work.

    opened by dehume 1
  • Make sink config simpler

    Make sink config simpler

    It would be cool if this thing just split events into tables without being configured to do so. But could be configured in reverse to say "no don't do that".

    opened by jakthom 0
  • Create swagger documentation for endpoints

    Create swagger documentation for endpoints

    buz endpoints like /stats, /schemas etc right now are kind of 'need to know', but ideally we have an auto generated documentation for them with descriptions of what they do and what we can expect back when we hit them.

    opened by johncassil 0
Releases(v0.12.7)
Owner
silverton
Data stuff
silverton
Govent is an event bus framework for DDD event source implement

Govent is an event bus framework for DDD event source implement. Govent can also solve the package circular dependency problem.

Michaelyn 2 Jan 28, 2022
Event-planning-go - GRAPHQL Project for Event Planning

About The Project GRAPHQL Project for Event Planning Building the project with l

Muhamad Hilmi Hibatullah 2 Mar 13, 2022
:incoming_envelope: A fast Message/Event Hub using publish/subscribe pattern with support for topics like* rabbitMQ exchanges for Go applications

Hub ?? A fast enough Event Hub for go applications using publish/subscribe with support patterns on topics like rabbitMQ exchanges. Table of Contents

Leandro Lugaresi 125 Dec 17, 2022
Easy to use distributed event bus similar to Kafka

chukcha Easy to use distributed event bus similar to Kafka. The event bus is designed to be used as a persistent intermediate storage buffer for any k

Yuriy Nasretdinov 81 Dec 30, 2022
Simple, high-performance event streaming broker

Styx Styx is a simple and high-performance event streaming broker. It aims to provide teams of all sizes with a simple to operate, disk-persisted publ

Dataptive 48 Nov 24, 2022
Discrete-event simulation in Go using goroutines

SimGo SimGo is a discrete event simulation framework for Go. It is similar to SimPy and aims to be easy to set up and use. Processes are defined as si

Felix Schütz 24 Sep 6, 2022
Simple synchronous event pub-sub package for Golang

event-go Simple synchronous event pub-sub package for Golang This is a Go language package for publishing/subscribing domain events. This is useful to

itchyny 19 Jun 16, 2022
pubsub controller using kafka and base on sarama. Easy controll flow for actions streamming, event driven.

Psub helper for create system using kafka to streaming and events driven base. Install go get github.com/teng231/psub have 3 env variables for config

Te Nguyen 6 Sep 26, 2022
POC of an event-driven Go implementation

Event Driven example in Golang This POC shows an example of event-driven architecture with a working domain event broker, an event producer and a cons

Fede Barcelona 0 Nov 2, 2021
Go library to build event driven applications easily using AMQP as a backend.

event Go library to build event driven applications easily using AMQP as a backend. Publisher package main import ( "github.com/creekorful/event" "

Aloïs Micard 0 Dec 5, 2021
Example Golang Event-Driven with kafka Microservices Choreography

Microservices Choreography A demonstration for event sourcing using Go and Kafka example Microservices Choreography. To run this project: Install Go I

Muhammad Nasrul 0 Dec 2, 2021
A basic event queue (and publisher/subscriber) in go

queue A basic event queue (and publisher/subscriber) in go. Installation go get github.com/jimjibone/queue Queue Usage Queue is a channel-based FIFO q

James Reuss 0 Dec 17, 2021
Basic Event Streaming - Fundamentals of Kafka Studies (BESt-FunKS)

Apache Kafka My study repo for Apache Kafka. Based on this tutorial. Contents Overview Key Terms Event Topic Producer Consumer Partition Getting Start

João Saraceni 2 Mar 2, 2022
CQRS & Event-Sourcing Framework for Go.

goes - Event-Sourcing Framework goes is a collection of interfaces, tools, and backend implementations that allow you to write event-sourced applicati

modernice.design 65 Dec 27, 2022
[Go] Lightweight eventbus with async compatibility for Go

EventBus Package EventBus is the little and lightweight eventbus with async compatibility for GoLang. Installation Make sure that Go is installed on y

Alex Saskevich 1.3k Jan 1, 2023
RapidMQ is a pure, extremely productive, lightweight and reliable library for managing of the local messages queue

RapidMQ RapidMQ is a pure, extremely productive, lightweight and reliable library for managing of the local messages queue in the Go programming langu

Vadim Shakun 65 Sep 27, 2022
⚡️ A lightweight service that will build and store your go projects binaries, Integrated with Github, Gitlab, Bitbucket and Bitbucket Server.

Rabbit A lightweight service that will build and store your go projects binaries. Rabbit is a lightweight service that will build and store your go pr

Ahmed 196 Nov 19, 2022
GTA(Go Task Async) is a lightweight reliable asynchronous task and transaction message library for Golang

GTA (Go Task Async) is a lightweight and reliable asynchronous task and transaction message library for by golang.

Kevin Su 12 Jun 4, 2022
A lightweight, distributed and reliable message queue based on Redis

nmq A lightweight, distributed and reliable message queue based on Redis Get Started Download go get github.com/inuggets/nmq Usage import "github.com

Nuggets 2 Nov 22, 2021