golang long polling library. Makes web pub-sub easy via HTTP long-poll server :smiley: :coffee: :computer:

Related tags

Messaging golongpoll
Overview

golongpoll Build Status codecov GoDoc Go Report Card

Golang long polling library. Makes web pub-sub easy via an HTTP long-poll server.

New in v1.1

  • Deprecated CreateManager and CreateCustomManager in favor of StartLongpoll
    • The deprecated functions still work but they log warnings when called.
    • StartLongpoll takes an Options struct that allows you to configure more behavior. See [Options] (#options)
  • Event expiration via the EventTimeToLiveSeconds option.
  • Bug fixes for client disconnects

Basic usage

To use, create a LongpollManager and then use it to publish events and expose an HTTP handler to subscribe to events.

import	"github.com/jcuga/golongpoll"

// This launches a goroutine and creates channels for all the plumbing
manager, err := golongpoll.StartLongpoll(golongpoll.Options{})  // default options

// Pass the manager around or create closures and publish:
manager.Publish("subscription-category", "Some data.  Can be string or any obj convertable to JSON")
manager.Publish("different-category", "More data")

// Expose events to browsers
// See subsection on how to interact with the subscription handler
http.HandleFunc("/events", manager.SubscriptionHandler)
http.ListenAndServe("127.0.0.1:8081", nil)

For a working demo, see basic.go.

Note that you can add extra access-control, validation, or other behavior on top of the manager's SubscriptionHandler. See the advanced example. This example also shows how to publish a more complicated payload JSON object.

You can also configure the LongpollManager by defining values in the golongpoll.Options param passed to StartLongpoll(opts)

Options

The default options should work for most people's needs. However, if you are worried about old events sitting around taking up space and want to fine tune how long events last inside the internal buffers you can roll up your sleeves and configure the LongpollManager. This applies most to programs that are going to deal with a 'large' number of events over a 'long' period of time.

The following options are now available.

  • LoggingEnabled bool: Whether or not to output logs about events, subscriptions, and clients. Defaults to false.
  • MaxLongpollTimeoutSeconds int: Max amount of time clients are allowed to keep a longpoll connection. This is used against the timeout: Query param sent to the SubscriptionHandler. Defaults to 120.
  • MaxEventBufferSize int: The max number of events kept in a given subscription category before the oldest events are discarded even if they're not expired yet. Buffering is important if you want to let clients request events from the past, or if you have high-volume events that occur in spurts and you don't clients to miss anything. Defaults to 250.
  • EventTimeToLiveSeconds int: How long events can remain in the internal buffer before they're discarded for being too old. This determines how long in the past clients can see events (they can provide a since_time query param to request old events). Defaults to the constant golongpoll.FOREVER which means they never expire. If you're concerned about old events sitting around in the internal buffers wasting space (especially if you're generating a large number of events and running the program for long periods of time), you can set a reasonable expiration and they automatically be removed once expired.
  • DeleteEventAfterFirstRetrieval bool: Whether or not an event is immediately deleted as soon as it is retrieved by a client (via the SubscriptionHandler). This is useful if you only have one client per subscription category and you only care to see event's once. This may be useful for a notification type scenario where each client has a subscription category and you only see events once. Alternatively, clients could just update their since_time param in their longpoll request to be that of the timestamp of their most recent notification, thus causing previously seen notifications to not be retrieved. But there may be other scenarios where this option is more desirable since people have asked for this. Defaults to false.

These are set on the golongpoll.Options struct that gets passed to golongpoll.StartLongpoll. For example:

	manager, err := golongpoll.StartLongpoll(golongpoll.Options{
		LoggingEnabled:                 true,
		MaxLongpollTimeoutSeconds:      60,
		MaxEventBufferSize:             100,
		EventTimeToLiveSeconds:         60 * 2, // Event's stick around for 2 minutes
		DeleteEventAfterFirstRetrieval: false,
	})

Or if you want the defauls, just provide an empty struct, or a struct that only defines the options you want to override.

// all default options:
manager, err := golongpoll.StartLongpoll(golongpoll.Options{})

// Default options with EventTimeToLiveSeconds override:
manager, err := golongpoll.StartLongpoll(golongpoll.Options{
			EventTimeToLiveSeconds:         60 * 2,
	})

HTTP Subscription Handler

The LongpollManager has a field called SubscriptionHandler that you can attach as an http.HandleFunc.

This HTTP handler has the following URL query params as input.

  • timeout number of seconds the server should wait until issuing a timeout response in the event there are no new events during the client's longpoll. The default manager has a max timeout of 120 seconds, but you can customize this by using Options.MaxLongpollTimeoutSeconds
  • category the subscription category to subscribe to. When you publish an event, you publish it on a specific category.
  • since_time optional. the number of milliseconds since epoch. If not provided, defaults to current time. This tells the longpoll server to only give you events that have occurred since this time.

The response from this HTTP handler is one of the following application/json responses:

  • error response: {"error": "error message as to why request failed."}
    • Perhaps you forgot to include a query param? Or an invalid timeout?
  • timeout response: {"timeout":"no events before timeout","timestamp":1450827183289}
    • This means no events occurred within the timeout window. (also given your since_time param)
    • The timestamp is the server time when it issued a timeout response, so you can use this value as since_time in your next request.
  • event(s) response: {"events":[{"timestamp":1447218359843,"category":"farm","data":"Pig went 'Oink! Oink!'"}]}
    • includes one or more event object. If no events occurred, you should get a timeout instead.

To receive a continuous stream of chronological events, you should keep hitting the http handler after each response, but with an updated since_time value equal to that of the last event's timestamp.

You can see how to make these longpoll requests using jquery by viewing the example programs' code.

What is longpolling

Longpolling is a way to get events/data "pushed" to the browser as soon as they occur* (with a usually very small delay). Longpolling is an option to consider when you want updates to be sent from the webserver to a browser as they occur. This is a one-way communication path. If you need full-duplex communication, consider an alternative like websockets.

To better understand longpolling, let's consider what it improves upon. A naive way to get updates as soon as possible from a webserver is to continuously make AJAX requests from a webpage asking the server if there is anything new.

polling diagram

The problem with this approach is that when there are no updates, you are continuously spamming the webserver with new requests. An alternative approach is to have the webserver wait until there is actually data before responding to your request.

longpolling diagram

This is an improvement since both the client and the server aren't setting up and tearing down connections so quickly. But you can't just wait forever to hear from the server. So longpolling has the concept of a timeout. If the server waits too long and there are no new events, the server responds to the client that there's nothing new. The client can then initiate a new longpoll request.

longpolling diagram

Essentially, longpolling is a much more sane version of spamming the server with a bunch of requests asking for new data.

Why not just use websockets instead? Websockets are great if you need to push data in both directions. But if you're really interested in pushing data from the server to the client and not vice-versa, then longpolling may be a viable option for a number of reasons.

  • longpolling is just simple, plain old HTTP. The server is just... slow to respond at times.
    • This means much wider range of browser support, especially the older ones
    • Will work over infrastructure that uses proxies that only allow port 80/443
    • Also works well through VPN webclient type products that do "magic" to web traffic
      • As a general rule, the closer to traditional HTTP you are, the wider support you have.

Why does everyone run to websockets even when they only need server-to-client pushing? Probably because it's difficult to get longpolling right. By this I mean handling the subtleties on the server end to make sure that any events that occur in the small window between the time that a client gets a response and before they make a new request, handling disconnects, and buffering older events in case clients went offline. There is a plethora of posts on the internet to make a half-baked longpoll server, but few if any posts outline how to make a robust one. (that's why you should use golongpoll--it will do this for you!).

Also, depending on what language you're writing the webserver in, longpolling might be more difficult. Think python running in a WSGI container. Without the flexibility of golang and it's channels, such implementations could be quite the headache.

Included examples

There are two fully-functional example programs provided. Basic

This program creates a default LongpollManager, shows how a goroutine can generate some events, and how to subscribe from a webpage. See basic.go

To run this example

go build examples/basic/basic.go
./basic
OR: ./basic.exe

Then visit:

http://127.0.0.1:8081/basic

And observe the events appearing every 0-5 seconds.

Advanced

This program creates a custom LongpollManager, shows how an http handler can publish events, and how to subscribe from a webpage. See advanced.go

To run this example

go build examples/advanced/advanced.go
./advanced
OR: ./advanced.exe

Then visit:

http://127.0.0.1:8081/advanced

Try clicking around and notice the events showing up in the tables. Try opening multiple windows as different users and observe events. Toggle whether user's events are public or private.

More advanced use

All of the below topics are demonstrated in the advanced example:

Events with JSON payloads

Try passing any type that is convertible to JSON to Publish(). If the type can be passed to encoding/json.Marshal(), it will work.

Wrapping subscriptions

You can create your own HTTP handler that calls LongpollManager.SubscriptionHandler to add your own layer of logic on top of the subscription handler. Uses include: user authentication/access-control and limiting subscriptions to a known set of categories.

Publishing events via the web

You can create a closure that captures the LongpollManager and attach it as an http handler function. Within that function, simply call Publish().

Comments
  • how to delete buffer after certain time?

    how to delete buffer after certain time?

    Hi, i'm using this package to implement notifications (it's awesome by the way) i use one category for each user after the user gets the notifications i'm not interested in keeping them in the buffer, so i'm deleting the user buffer after every publish (i did this already, i would do a pull request if you like)

    also i don't want to keep the buffer if the user left the site for a long time so, how i could do that?

    opened by marwan-nwh 18
  • High CPU load?

    High CPU load?

    I'm using this library for publishing events of multiple connected meters (constantly being polled at 9600 baud). I've noticed that just publishing those events longpoll manager- even without connected clients- raises CPU load to > 50% on Raspi3:

    func (f *Firehose) Run() {
        for {
            select {
            case snip := <-f.in:
                f.lpManager.Publish("meterupdate", snip)
            case statupdate := <-f.statstream:
                f.lpManager.Publish("statusupdate", statupdate)
            }
        }
    }
    ...
    go firehose.Run()
    

    Does this behaviour make sense to you or should a low CPU load be expected when publishing without clients attached?

    opened by andig 5
  • question about Http Server integration

    question about Http Server integration

    This looks pretty useful for my use case.

    I have a restful interface at the moment, and need to push things to the Web Client. I want to avoid Web sockets and too much complexity.

    So, i am wondering about If the Restful Server i have , (based on Gin) needs to be a separate service from the Long Poller. I ask this because for my Restful interface i have alot of routing URLS already setup, and i am not sure if i can just all the Long Poller as a path on the standard Gin Router or not.

    I know this is maybe hard to answer and i should just give it a go, but thought it was worth asking first.

    thanks in advance.

    opened by joeblew99 5
  • Handler keeps on waiting for a timeout even though the client has been disconnected.

    Handler keeps on waiting for a timeout even though the client has been disconnected.

    Won't it be better for the handler function to stop waiting for a timeout if the client has been disconnected?

    Steps to reproduce the issue:

    1. Create a new manager and register the handler:

      manager, _ = golongpoll.CreateCustomManager(120, 1, true)
      http.HandleFunc("/events", func(w http.ResponseWriter, r *http.Request) {
        manager.SubscriptionHandler(w, r)
        fmt.Println("client disconnected")
      })
      
    2. Connect to the server from a browser (it will start waiting for the response)

    3. Manager's log: Adding Client (Category: "abc112" Client: c6650762-f3d7-4876-410c-c6df1ea1dfc0)

    4. Without publishing anything disconnect the client (close browser's window or something)

    5. The SubscriptionHandler is still running

    I guess it'd be logical to terminate the handler's execution at once after the disconnect event.

    opened by alex-ant 5
  • Return timestamp in timeout response

    Return timestamp in timeout response

    I can implement it if it makes sense.

    I have an application that can be offline sometimes, so I'm passing the timestamp to get past events, but in this scenario it fails:

    • The application starts and connects to the server, there is no event, so it gets one or more timeouts.
    • Then the connection drops and while the client is offline the server receives some new events.
    • When the client reconnects, there is no events as it did not pass a since_time value because it didn't have a timestamp from the last event (as it did not receive any).

    I could calculate the timestamp on the client, but I don't think it would solve this, the date/time of the client could be wrong for example.

    It would work better if the timeout reply returned a timestamp, in this case:

    {
      "timeout": "no events before timeout",
      "timestamp": 1447218359843
    }
    

    Then I could update the timestamp after the first timeout, and even if the client loses the connection It will be able to get past events since the last timeout.

    What do you think?

    opened by fabioxgn 5
  • condition when user leaving page

    condition when user leaving page

    Hi, as I understand..long polling need to timeout the polling so the server could free up the holding request. How do you handle user when user leave page without information (page crash or internet down) then user refresh the page and re-request the long polling? You hold previous request for nothing.

    opened by goldalworming 5
  • Please replace deprecated CloseNotify with Context().Done()

    Please replace deprecated CloseNotify with Context().Done()

    disconnectNotify := r.Context().Done()

    --- a/longpoll.go
    +++ b/longpoll.go
    @@ -270,7 +270,7 @@ func getLongPollSubscriptionHandler(maxTimeoutSeconds int, subscriptionRequests
                    // event that a client crashes or the connection goes down.  We don't
                    // need to wait around to fulfill a subscription if no one is going to
                    // receive it
    -               disconnectNotify := w.(http.CloseNotifier).CloseNotify()
    +               disconnectNotify := r.Context().Done()
                    select {
                    case <-time.After(time.Duration(timeout) * time.Second):
                            // Lets the subscription manager know it can discard this request's
    
    opened by kondratev 3
  • Events published in same millisecond aren't seen by clients

    Events published in same millisecond aren't seen by clients

    When two events are published to the LongpollManager in the same millisecond, the behavior I observed is that clients will receive the first event only. The other events in the same millisecond can be seen by trying a request again with the same since_time, but if clients listen again using the timestamp received (as advised in https://github.com/jcuga/golongpoll#http-subscription-handler), they will never see those events.

    For example, here is sample output from a client listening to a simple golongpoll app that is hardcoded to publish two events in quick succession every 10s:

    # Initial call
    > curl -i "localhost:8081/events?category=connectivity&timeout=60&since_time=1588816900233"
    HTTP/1.1 200 OK
    Cache-Control: no-cache, no-store, must-revalidate
    Content-Type: application/json
    Expires: 0
    Pragma: no-cache
    Date: Thu, 07 May 2020 02:01:50 GMT
    Content-Length: 97
    
    {"events":[{"timestamp":1588816910237,"category":"connectivity","data":{"message":"message 1"}}]}
    
    # If calling again with the same since_time, I see the event with duplicate timestamp:
    > curl -i "localhost:8081/events?category=connectivity&timeout=60&since_time=1588816900233"
    HTTP/1.1 200 OK
    Cache-Control: no-cache, no-store, must-revalidate
    Content-Type: application/json
    Expires: 0
    Pragma: no-cache
    Date: Thu, 07 May 2020 02:01:51 GMT
    Content-Length: 182
    
    {"events":[{"timestamp":1588816910237,"category":"connectivity","data":{"message":"message 1"}},{"timestamp":1588816910237,"category":"connectivity","data":{"message":"message 2"}}]}                         
    
    # If calling again with the received timestamp, the "message 2" message is never received.
    > curl -i "localhost:8081/events?category=connectivity&timeout=60&since_time=1588816910237"
    HTTP/1.1 200 OK
    Cache-Control: no-cache, no-store, must-revalidate
    Content-Type: application/json
    Expires: 0
    Pragma: no-cache
    Date: Thu, 07 May 2020 02:02:00 GMT
    Content-Length: 97
    
    {"events":[{"timestamp":1588816920241,"category":"connectivity","data":{"message":"message 1"}}]}
    

    Is this behavior expected? I am planning to work around this by forcing a 1ms delay between publishes, but perhaps this could be handled in the LongpollManager by aggregating all events in the same millisecond and sending them together?

    bug 
    opened by concreted 3
  • client js

    client js

    Hello,

    I really like the project, and I am using it for some internal tools. I wanted to share the little js client "library" I have created over the weekend. Maybe you can use it.

    https://gist.github.com/bluebrown/53e5144ea5c7e8fb1c77c20187844273

    Anyway, thank you for the package.

    Regards, Nico

    opened by bluebrown 3
  • Added control headers so demos work out of box

    Added control headers so demos work out of box

    Issue Attempting to run the demos out of box will fail due to errors with the AJAX request (tested with Chrome 52)

    Sample error

    XMLHttpRequest cannot load http://127.0.0.1:8081/basic/events?
    timeout=45&category=farm&since_time=1471491258748. No 'Access-Control-Allow-Origin' header is 
    present on the requested resource. Origin 'http://localhost:8081' is therefore not allowed access.
    

    Solution Add CORS headers to resolve

    opened by astraldawn 3
  • How do you make sure the concurrent issue?

    How do you make sure the concurrent issue?

    https://github.com/jcuga/golongpoll/blob/033d1fa3098d8aed25a1f9dd9a02df93341de62a/longpoll.go#L652

    I see there is no lock when remove and add item from map, this will case crash.

    opened by dawei101 2
Releases(v1.3.0)
  • v1.3.0(Apr 29, 2021)

    • Fixes #19
    • Adds an official js client (#17)
    • replace glp-client with golongpoll.client
    • AddOn interface and golongpoll.Options.AddOn for customizing longpoll server behavior.
    • Adds FilePersistorAddOn for persisting events to disk/across runs. (#10)
    • More examples
    • Updated README/docs.
    Source code(tar.gz)
    Source code(zip)
  • v1.2.0(Apr 24, 2021)

  • v1.1.0(Feb 5, 2016)

    New in v1.1

    • Deprecated CreateManager and CreateCustomManager in favor of StartLongpoll
      • The deprecated functions still work but they log warnings when called.
      • StartLongpoll takes an Options struct that allows you to configure more behavior.
    • Event expiration via the EventTimeToLiveSeconds option.
    • Bug fixes for client disconnects
    Source code(tar.gz)
    Source code(zip)
  • v1.0.0(Nov 11, 2015)

Owner
J Cuga
J Cuga
Simple synchronous event pub-sub package for Golang

event-go Simple synchronous event pub-sub package for Golang This is a Go language package for publishing/subscribing domain events. This is useful to

itchyny 19 Jun 16, 2022
A simple in-process pub/sub for golang

go-pub-sub A simple in-process pub/sub for golang Motivation Call it somewhere between "I spent no more than 5 minutes looking for one that existed" a

Ryan Catherman 0 Jan 25, 2022
nanoQ — high-performance brokerless Pub/Sub for streaming real-time data

nanoQ — high-performance brokerless Pub/Sub for streaming real-time data nanoQ is a very minimalistic (opinionated/limited) Pub/Sub transport library.

Aigent 149 Nov 9, 2022
A basic pub-sub project using NATS

NATS Simple Pub-Sub Mechanism This is a basic pub-sub project using NATS. There is one publisher who publishes a "Hello World" message to a subject ca

Prosonul Haque 1 Dec 13, 2021
Simple-messaging - Brokerless messaging. Pub/Sub. Producer/Consumer. Pure Go. No C.

Simple Messaging Simple messaging for pub/sub and producer/consumer. Pure Go! Usage Request-Response Producer: consumerAddr, err := net.ResolveTCPAddr

IchHabeKeineNamen 1 Jan 20, 2022
Kafka, Beanstalkd, Pulsar Pub/Sub framework

go-queue Kafka, Beanstalkd, Pulsar Pub/Sub framework.

chenquan 3 Sep 17, 2022
CLI tool for generating random messages with rules & publishing to the cloud services (SQS,SNS,PUB/SUB and etc.)

Randomsg A CLI tool to generate random messages and publish to cloud services like (SQS,SNS,PUB/SUB and etc.). TODO Generation of nested objects is no

Kerem Dokumacı 6 Sep 22, 2022
Cadence is a distributed, scalable, durable, and highly available orchestration engine to execute asynchronous long-running business logic in a scalable and resilient way.

Cadence Visit cadenceworkflow.io to learn about Cadence. This repo contains the source code of the Cadence server. To implement workflows, activities

Uber Open Source 6.5k Jan 9, 2023
A simple queueing system for long-running commands

qme (queue me) A simple queueing system for long-running commands. It allows you to queue up shell commands from anywhere, and run them in order. This

Abdussamet Koçak 25 Nov 18, 2022
⚡ HTTP/2 Apple Push Notification Service (APNs) push provider for Go — Send push notifications to iOS, tvOS, Safari and OSX apps, using the APNs HTTP/2 protocol.

APNS/2 APNS/2 is a go package designed for simple, flexible and fast Apple Push Notifications on iOS, OSX and Safari using the new HTTP/2 Push provide

Adam Jones 2.7k Jan 1, 2023
golang client library to Viessmann Vitotrol web service

Package go-vitotrol provides access to the Viessmann™ Vitotrol™ cloud API for controlling/monitoring boilers. See https://www.viessmann.com/app_vitoda

Maxime Soulé 21 Nov 16, 2022
It's client library written in Golang for interacting with Linkedin Cruise Control using its HTTP API.

go-cruise-control It's client library (written in Golang) for interacting with Linkedin Cruise Control using its HTTP API. Supported Cruise Control ve

Banzai Cloud 1 Jan 10, 2022
🐇 Easy to use socket lib for Golang

Hare Sockets ?? Hare is a user-friendly lib for sockets in Golang. You can send and listen to TCP connections with a few lines of code. Contents Insta

Leonardo Lima 52 Dec 26, 2022
An easy-to-use CLI client for RabbitMQ.

buneary, pronounced bun-ear-y, is an easy-to-use RabbitMQ command line client for managing exchanges, managing queues and publishing messages to exchanges.

Dominik Braun 56 Sep 3, 2022
Easy to use distributed event bus similar to Kafka

chukcha Easy to use distributed event bus similar to Kafka. The event bus is designed to be used as a persistent intermediate storage buffer for any k

Yuriy Nasretdinov 81 Dec 30, 2022
pubsub controller using kafka and base on sarama. Easy controll flow for actions streamming, event driven.

Psub helper for create system using kafka to streaming and events driven base. Install go get github.com/teng231/psub have 3 env variables for config

Te Nguyen 6 Sep 26, 2022
Golang push server cluster

gopush-cluster gopush-cluster is a go push server cluster. Features light weight high performance pure golang implementation message expired offline m

Terry.Mao 2.1k Dec 28, 2022
A push notification server written in Go (Golang).

gorush A push notification micro server using Gin framework written in Go (Golang) and see the demo app. Contents gorush Contents Support Platform Fea

Bo-Yi Wu 6.7k Jan 8, 2023
websocket based messaging server written in golang

Guble Messaging Server Guble is a simple user-facing messaging and data replication server written in Go. Overview Guble is in an early state (release

Sebastian Mancke 154 Oct 19, 2022