A realtime distributed messaging platform

Last update: Dec 29, 2022

Overview

Source: https://github.com/nsqio/nsq
Issues: https://github.com/nsqio/nsq/issues
Mailing List: [email protected]
IRC: #nsq on freenode
Docs: https://nsq.io
Twitter: @nsqio

NSQ is a realtime distributed messaging platform designed to operate at scale, handling billions of messages per day.

It promotes distributed and decentralized topologies without single points of failure, enabling fault tolerance and high availability coupled with a reliable message delivery guarantee. See features & guarantees.

Operationally, NSQ is easy to configure and deploy (all parameters are specified on the command line and compiled binaries have no runtime dependencies). For maximum flexibility, it is agnostic to data format (messages can be JSON, MsgPack, Protocol Buffers, or anything else). Official Go and Python libraries are available out of the box (as well as many other client libraries) and, if you're interested in building your own, there's a protocol spec.

We publish binary releases for linux, darwin, freebsd and windows as well as an official Docker image.

NOTE: master is our development branch and may not be stable at all times.

In Production

Code of Conduct

Help us keep NSQ open and inclusive. Please read and follow our Code of Conduct.

Authors

NSQ was designed and developed by Matt Reiferson (@imsnakes) and Jehiah Czebotar (@jehiah) but wouldn't have been possible without the support of Bitly, maintainers (Pierce Lopez), and all our contributors.

Logo created by Wolasi Konu (@kisalow).

Comments

nsqadmin: refactoring
nsqadmin is a Go web app, which has some tradeoffs. It's great because it's dead simple to deploy. It's terrible because it's a PITA writing Go based web apps.

I think an ideal middle ground is:

nsqadmin becomes a JSON API with convenience endpoints to query the cluster and aggregate data

nsqadmin UI is a client-side app

all the static assets for (2) are bundled into the binary so it's still easy to deploy (this is already the case, just stating the obvious)

Simultaneously, it could use a facelift, but this refactoring alone makes things much easier to extend and for others to contribute.

cc @visionmedia
feature
opened by mreiferson 52
nsqd: remove --worker-id, replace with --node-id

This is one of the for-1.0 changes contemplated in #741

This is the minimal-effort approach. If --worker-id is specified with an int, parsing options will fail, and the user will hopefully notice the change instead of just specifying "true" or "false" to work-around the failure. Or is backwards-compatibility (with just a "deprecated" warning) desired?
1.0 chore

opened by ploxiln 51
nsqd: support deferred PUB

This came up on the mailing list: https://groups.google.com/forum/#!topic/nsq-users/HkoOMygaHK0 - just opening the issue here for others to weigh in and gauge interest.

This would allow you to PUB a message that goes directly to the deferred priority queue with the specified delay.

Also, It's complicated a bit by #34.
feature request

opened by mreiferson 42
nsqd: track stats of end-to-end message processing time
As per #268, this pull request adds the ability to track statistics about the time it takes for messages to get .finished(). This is done by maintaining a probabilistic percentile calculation over each channel (using the perks package) and merging them when topic quantiles are requested.

This data is surfaced in:

[x] The /stats call

[x] statsd

[x] admin interface

Still to do:

[x] Before/after timings

[x] Docs

Sample data from /stats call:

$ curl "localhost:4151/stats" nsqd v0.2.23 (built w/go1.1.2) [test ] depth: 0 be-depth: 0 msgs: 591 e2e%: 13.5s, 13.5s, 683.9us, 375.1us, 269.6us [test_chan ] depth: 0 be-depth: 0 inflt: 0 def: 0 re-q: 0 timeout: 0 msgs: 591 e2e%: 13.5s, 13.5s, 1.1ms, 394.7us, 268.8us [V2 muon:55712 ] state: 3 inflt: 0 rdy: 62 fin: 591 re-q: 0 msgs: 591 connected: 7m18s
feature
opened by mynameisfiber 42
up-to-date dockerfiles

The official Docker builds have fallen out of date. It's also unfortunate that they can't take advantage of new Go speed improvements over time without manual intervention.

Would it be possible to include a Dockerfile in the repo for each app, based off google/golang, that built the apps from source on top of it? That would, at least, keep the Dockerfiles in source control. It also permits you to use the latest version of Go automatically. It can also be built into an automated pipeline, then, that updates the Docker Hub when a release is made.
feature docs 1.0 request

opened by paddycarver 41
nsqd: not properly creating channels registered in lookupd on seeing a new topic
I have a bunch of topics that are produced on a variety of nodes. It's not always clear to me ahead of time which nodes will be producing which topics, but it is certainly some overlapping subset. I have two consumer channels: "archive," for nsq_to_file, and "events," for part of the application. The problem is bringing up a new node: I don't know exactly which topics to create in nsqd, but I do know which topics it might produce, and of those, which channels they should go to.

As it stands, the first message on a topic will get to nsqd, and whichever consumer happens to find it first will see that message on its channel. Since nsqd doesn't yet know about the other channel, it will drop the message before the second consumer has connected.

Some thoughts on how to potentially deal with this:

The lookupd http apis seemed promising (though missing some documentation). If information created there were to flow in reverse back to nsqd, say, when a topic/channel is created, all the nsqd's peering with that lookupd could know about that topic-channel association. A fresh nsqd won't register itself as publishing the topic, but if the does get a message to publish on that topic, it could create the known channels immediately to ensure a consumer on those channels will definitely get that first message.

As an alternative solution: on a new topic, when lookupd is configured, nsqd could wait until lookupd has (somehow) confirmed all known peers have seen the new producer information and had a chance to connect on any additional channels before dropping messages on the new topic.

Let me know if there's something I'm misunderstanding! Thanks!
bug
opened by stephensearles 37
nsqd authorization

This is an attempt at creating an authorization protocol as originaly requested in #224

Specification:

nsqd TCP protocol will add a AUTH command with a json body that MUST be called before SUB or PUB when nsqd has authorization enabled.

Nsqd will make authorization requests against any auth server configured until it gets a valid response. It will pass along the metadata from the client, and the clients remote address. The auth server will reply with a list of topic/channels that the connection is valid for. The topic and channel names returned will be interpreted as a regex which will allow for the auth server to give blanket auth response in the form of topic=.*, channels=[.*] response and will specify a TTL for how long the authorization is good for. The auth request will be cached and re-tried upon expiration of the authorization metadata. (this will ensure that when authorization is removed from a auth server, you can ensure that it no longer applies after a specified ttl).

Authorization information will be stored on the connection for the life of the connection or until the TTL expires, and will be checked at SUB or PUB time.

It is expected that when using an authorization server, the nsqd HTTP endpoints are not exposed to untrusted clients.

A simple auth server will be included that gives a UI to manage authorizations based on login, tls, remote_ip and optionally an oauth2 endpoint used to lookup login values. It is expected that other auth servers will be written for different authorization backends (like LDAP, ssl certificate checking, etc). Connections between nsqd and the authorization server are expected to be over a trusted network.

cc: @mreiferson
feature 1.0

opened by jehiah 30
nsqd: per-topic message IDs

(Pushing up rebased branches I've had laying around locally for ages)

This modifies message ID generation such that each topic maintains a monotonic/atomic counter to generate message IDs, a more scalable and performant approach.
1.0 perf

opened by mreiferson 26
nsqadmin: work-around bug for raw ipv6 addresses

Ipv6 address in broadcast_address should be returned with leading and trilling square bracket trimmed.

When we use go-nsq connect to nsqd by using lookupd's lookup api, go-nsq will compose the address by calling net.JoinHostPort. net.JoinHostPort will assume that if it is an ipv6 address, there should be no square brackets as net.JoinHostPort will add square brackets when it detects an ipv6 address is added.
bug

opened by andyxning 25
nsqd: (optionally) require client certs

This is a first pass at client certificate requirement, optional verification, and ability to load custom certificate authority (for self signed certificates). Closing the client connection if not upgraded to TLS still needs to be implemented, as well as the test coverage. Was hoping to get any feedback on current implementation (even if it's just a thumbs up) before I finish it out. Thanks!
feature request

opened by chrisroberts 25

nsqd: client error logged even when cleanly closed

nsqd always print log as follow on our production environment:

[nsqd] 2015/01/08 22:49:51.130538 TCP: new client(10.6.28.53:22505)
[nsqd] 2015/01/08 22:49:51.130847 CLIENT(10.6.28.53:22505): desired protocol magic '  V2'
[nsqd] 2015/01/08 22:49:51.137175 PROTOCOL(V2): [10.6.28.53:22505] exiting ioloop
[nsqd] 2015/01/08 22:49:51.137273 ERROR: client(10.6.28.53:22505) - failed to read command - EOF
[nsqd] 2015/01/08 22:49:51.137310 PROTOCOL(V2): [10.6.28.53:22505] exiting messagePump

my environment like this:

CentOS release 6.6 (Final)                                                                                                 
Kernel \r on an \m                                                                                                
2.6.32-504.3.3.el6.x86_64
nsqd v0.3.0 (built w/go1.3.3)

and my nsq client used nsqphp

so, Any idea?

bug help wanted

opened by detailyang 24

Ability to bind unix sockets for nsqd
Hello, this PR provided unix socket supports.

nsqd may started with

/build/nsqd --use-unix-sockets --tcp-address /var/run/nsqd.sock --http-address /var/run/nsqd-http.sock --data-path /var/local/nsqd/

I've adopted pynsq and go-nsq for connecting to unix sockets as well. I'll create PRs after resolution.
opened by telepenin 0

defer is not working as expected

Based on the document, we only need pass defer parameter in ms to achieve message delay delivery. https://nsq.io/components/nsqd.html#post-pub

consumer

const nsq = require('nsqjs')
const moment = require('moment')

// reader
const reader = new nsq.Reader('sample_topic', 'sample_topic', {
  lookupdHTTPAddresses: 'nsqlookupd:4161'
})

reader.connect()

reader.on('message', msg => {
  console.log('Received message [%s] [%s]: %s', moment.utc().format('YYYY-MM-DD HH:mm:ss:SSS'), msg.id, msg.body.toString())
  setTimeout(() => {
    msg.finish()
  }, 1000)
});

producer, just call the API with postman http://localhost:4151/pub?defer=5000&topic=sample_topic

pre-request part, will generate the current time in ms.

var moment = require('moment');
var date = moment.utc().format('YYYY-MM-DD HH:mm:ss:SSS');
pm.environment.set("now", date);

in the request

{
    "text": "some message",
    "message  ": [{{now}}]
}

output

nsq-demo-client-1      | Received message [2022-12-13 14:37:15:684] [11bc612bb5c88000]: {
nsq-demo-client-1      |     "text": "some message",
nsq-demo-client-1      |     "message  ": [2022-12-13 14:37:12:336]
nsq-demo-client-1      | }

expect the Received message should be 5 seconds later comparing with the timestamp in the message body.

actual the Received message is always within the defer ms.

needs info

opened by cychub 1

nsqd: optimize the performance of httpServer's doPub

When a dense request is encountered and a single message is large, the use of ioutil.ReadAll will have an impact on the performance, causing a certain degree of memory leak. I have encountered serious cases, resulting in OOM. Because ioutil.ReadAll is to read out the information at one time. A large amount of information will also cause the expansion of the slice and affect the performance. At the same time, it will cause memory escape and increase the burden of GC.

Using io.Copy() avoids reading out messages at one time, and using pool improves memory utilization and reduce the burden of GC.
perf

opened by guozhao-coder 6
nsqd: ephemeral channels don't work with --mem-queue-size=0

When mem-queue-size= 0 is set, the message is sent to a temporary channel (#ephemeral), and it is found that the message can never be consumed. Is it because the mem-queue-size is exceeded and the message is directly discarded?
bug

opened by herry-go 9
nsqadmin: non admin user can delete the topic on a node

Hi Team

On the nsqadmin UI page, Non admin user can see the delete icon and delete the topic on a node successfully

tombstoneNodeForTopicHandler is not protected by isAuthorizedAdminRequest

Thanks /Joe
bug help wanted

opened by guishoudaoge 5
*: Support NSQ in ArgoLabs Dataflow
In Dataflow, we currently have support for Kafka, Jetstream, NATS Streaming and other sources and sinks. It would be amazing to add support for NSQ too. We know that we plenty of use cases in the OSS community and this would get a lot of love.

The Dataflow team are not NSQ experts. My ask is - would anyone be interested in implementing the changes needed?

Here is a pull request for Jetstream, breaking it down:

Add manifests to install a dev NSQ into Kubernetes cluster.

Add APIs to specify a NSQ source and sink (e.g. the URL, authentication).

Implement the source and sink.

Write test infrastructure (e.g. check the right number of messages get written to a sink).

Write tests (e.g. does the right number of message get written, even if the pod is deleted).

Write examples.

Write docs.

I'll raise a ticket against the main repo, and the Go client repo (there might be engineers there with the exact skills needed).
request help wanted
opened by alexec 0

Releases(v1.2.1)

v1.2.1(Aug 16, 2021)
Upgrading

#1227 - bump dependencies, officially drop dep support, drop Go 1.9 support

Features:

#1347 - nsqadmin: switch to go:embed for static assets

#1355 / #1364 - arm64 builds (thanks @danbf)

#1346 - nsqd: ability to skip ephemeral topics/channels in statsd output

#1336 / #1341 / #1343 - nsqd: ability to configure auth endpoint path (thanks @tufanbarisyildirim)

#1307 - remove Context to use stdlib context

#1295 / #1296 - switch to GitHub Actions CI

#1292 - nsqd: minimize allocations on message send (thanks @imxyb)

#1289 - optimize uniq (thanks @donutloop)

#1230 / #1232 - nsqd: ability to omit memory stats from /stats (thanks @creker)

#1226 - nsqd: only update internal RDY count for client when it changes (thanks @andyxning)

#1221 / #1363 - test against more recent versions of Go

#1209 - nsqd: bump go-diskqueue (interface change) (thanks @bitpeng)

#1206 - prefer idiomatic sort.Ints over sort.Sort (thanks @lelenanam)

#1197 / #1362 - Dockerfile: update Alpine base image, use /data by default

#1178 - nsqd: configurable queue scan worker pool (thanks @andyxning)

#1159 - nsqd: don't buffer messages when --mem-queue-size=0 (thanks @bitpeng)

#1073 / #1297 - nsqd: support separate broadcast ports for TCP and HTTP (thanks @shyam-king)

Bugs:

#1347 - nsqadmin: fix graphite key for ephemeral topics/channels

#765 / #1195 / #1203 / #1205 - fix build on illumos (thanks @i-sevostyanov)

#1333 - fix race detector tests on non-bash shells

#1330 - fix log_level support in configuration file (thanks @edoger)

#1319 / #1331 / #1361 - nsqd: handle SIGTERM

#1287 - nsqadmin: fix --proxy-graphite support (thanks @fanlix)

#1270 / #1271 - nsqlookupd: fix incorrect error message for HTTP listener (thanks @TangDH03)

#1264 - fix benchmark script

#1251 / #1314 / #1327 - nsqd: fix live lock for high churn ephemeral topic/channel reconnections (thanks @slayercat)

#1237 - Dockerfile: add nsswitch.conf to ensure go resolver uses /etc/hosts first

#1217 / #1220 - nsqd: improve error message when --data-path does not exist (thanks @mdh67899)

#1198 / #1190 / #1262 - synchronize close of all connections on Exit (thanks @benjsto)

#1188 / #1189 - nsqadmin: fix channel delete, fix source-maps in Firefox (thanks @avtrifanov)

#1186 - nsqadmin: fix nodes list with ipv6 addresses (thanks @andyxning)

Source code(tar.gz)
Source code(zip)
nsq-1.2.1.darwin-amd64.go1.16.6.tar.gz(20.96 MB)
nsq-1.2.1.darwin-arm64.go1.16.6.tar.gz(20.22 MB)
nsq-1.2.1.freebsd-amd64.go1.16.6.tar.gz(20.37 MB)
nsq-1.2.1.linux-amd64.go1.16.6.tar.gz(20.32 MB)
nsq-1.2.1.linux-arm64.go1.16.6.tar.gz(18.42 MB)
nsq-1.2.1.windows-amd64.go1.16.6.tar.gz(20.47 MB)
v1.2.0(Aug 28, 2019)
Upgrading

#1055 - nsqd: removed support for old metadata scheme used in v0.3.8 and earlier

you cannot upgrade directly from v0.3.8 to v1.2.0, you must go through v1.0.0-compat or v1.1.0

#1115 - manage dependencies with go modules

dep support still present for now, but deprecated

Features:

#1136 - nsqd: add --max-channel-consumers (default unlimited) (thanks @andyxning)

#1133 - nsqd: add --min-output-buffer-timeout (default 25ms) to limit how low a timeout a consumer can request

and raise default --max-output-buffer-timeout to 30 seconds (lower timeout, more cpu usage)

#1127 - nsqd: add topic total message bytes to stats (thanks @andyxning)

#1125 - nsqd: add flag to adjust default --output-buffer-timeout (thanks @andyxning)

#1163 - nsqd: add random load balancing for authd requests (thanks @shenhui0509)

#1119 - nsqd: include client TLS cert CommonName in authd requests

#1147 - nsq_to_file: include topic/channel in most log messages

#1117 - nsq_to_file: add --log-level and --log-prefix flags

#1117/#1120/#1123 - nsq_to_file: big refactor, more robust file switching and syncing and error handling

#1118 - nsqd: add param to /stats endpoint to allow skipping per-client stats (much faster if many clients)

#1118 - nsqadmin, nsq_stat: use include_clients param for /stats for a big speedup for big clusters

#1110 - nsq_to_file: support for separate working directory with --work-dir (thanks @mccutchen)

#856 - nsqadmin: add --base-path flag (thanks @blinklv)

#1072 - nsq_to_http: add --header flag (thanks @alwindoss)

#881 - nsqd: add producer client tcp connections to stats (thanks @sparklxb)

#1071/#1074 - nsq_to_file: new flag --sync-interval (default same as previous behavior, 30 seconds) (thanks @alpaker)

Bugs:

#1153 - nsqd: close connections that don't send "magic" header (thanks @JoseFeng)

#1140 - nsqd: exit on all fatal Accept() errors - restart enables better recovery for some conditions (thanks @mdh67899)

#1140 - nsqd, nsqlookupd, nsqadmin: refactor LogLevel, general refactor to better exit on all fatal errors

#1140 - nsqadmin: switch to using judwhite/go-svc like nsqd and nsqadmin do

#1134 - nsqadmin: fix clients count and channel total message rate (new bugs introduced in this cycle)

#1132 - nsqd, nsqlookupd, nsqadmin: fix http error response unreliable json serialization

#1116 - nsqlookupd: fix orphaned ephemeral topics in registration DB

#1109 - nsqd: fix topic message mis-counting if there are backend write errors (thanks @SwanSpouse)

#1099 - nsqlookupd: optimize /nodes endpoint, much better for hundreds of nsqd (thanks @andyxning)

#1085 - switch GOFLAGS to BLDFLAGS in Makefile now that GOFLAGS is automatically used by go

#1080 - nsqadmin: eslint reported fixes/cleanups

Source code(tar.gz)
Source code(zip)
nsq-1.2.0.darwin-amd64.go1.12.9.tar.gz(24.86 MB)
nsq-1.2.0.freebsd-amd64.go1.12.9.tar.gz(23.47 MB)
nsq-1.2.0.linux-amd64.go1.12.9.tar.gz(23.39 MB)
nsq-1.2.0.windows-amd64.go1.12.9.tar.gz(23.04 MB)
v1.1.0(Aug 19, 2018)
Upgrading from 1.0.0-compat: Just a few backwards incompatible changes:

#1056 - Removed the nsq_pubsub utility

#873 - nsqd flags --msg-timeout and --statsd-interval only take duration strings

plain integer no longer supported

#921 - nsqd: http /mpub endpoint binary param interprets "0" or "false" to mean text mode

previously any value meant to use binary mode instead of text mode - (thanks @andyxning)

The previous release, version "1.0.0-compat", was curiously-named to indicate an almost (but not quite) complete transition to a 1.0 api-stable release line. Confusingly, this follow-up release which completes the transition comes more than a year later. Because there have been a fair number of changes and improvements in the past year, an additional minor version bump seems appropriate.

Features:

#874 - nsqd: add memory stats to http /stats response (thanks @sparklxb)

#892 - nsqd, nsqlookupd, nsqadmin: add --log-level option (deprecating --verbose) (thanks @antihax)

#898 - nsqd, nsqlookupd, nsqadmin: logging refactor to use log levels everywhere

#914 - nsqadmin: X-Forwarded-User based "admin" permission (thanks @chen-anders)

#929 - nsqd: add topic/channel filter to /stats, use in nsqadmin and nsq_stat for efficiency (thanks @andyxning)

#936 - nsq_to_file: refactor/cleanup

#945 - nsq_to_nsq: support multiple --topic flags (thanks @jlr52)

#957 - nsq_tail: support multiple --topic flags (thanks @soar)

#946 - nsqd, nsqadmin: update internal http client with new go http.Transport features (keepalives, timeouts, dualstack)

affects metadata/stats requests between nsqadmin, nsqd, nsqlookupd

#954 - manage dependencies with dep (replacing gpm) (thanks @judwhite)

#957 - multi-stage docker image build (thanks @soar)

#996 - nsqd: better memory usage when messages have different sizes (thanks @andyxning)

#1019 - nsqd: optimize random channel selection in queueScanLoop (thanks @vearne)

#1025 - nsqd: buffer and spread statsd udp sends (avoid big burst of udp, less chance of loss)

#1038 - nsqlookupd: optimize for many producers (thousands) (thanks @andyxning)

#1050/#1053 - nsqd: new topic can be unlocked faster after creation

#1062 - nsqadmin: update JS deps

Bugs:

#753 - nsqadmin: fix missing channels in topic list

#867 - to_nsq: fix divide-by-zero issue when --rate not specified (thanks @adamweiner)

#868 - nsqd: clamp requeue timeout to range instead of dropping connection (thanks @tsholmes)

#891 - nsqd: fix race when client subscribes to ephemeral topic or channel while it is being cleaned up (reported by @slayercat)

#927 - nsqd: fix deflate level handling

#934 - nsqd: fix channel shutdown flush race

#935 - nsq_to_file: fix connection leaks when using --topic-pattern (thanks @jxskiss)

#951 - mention docker images and binaries for additional platforms in README (thanks @DAXaholic)

#950 - nsqlookupd: close connection when magic read fails (thanks @yc90s)

#971 - nsqd: fix some races getting ChannelStats (thanks @daroot)

#988 - nsqd: fix e2e timings config example, add range validation (thanks @protoss-player)

#991 - nsq_tail: logging to stderr (only nsq messages to stdout)

#1000 - nsq_to_http: fix http connect/request timeout flags (thanks @kamyanskiy)

#993/#1008 - nsqd: fix possible lookupd-identify-error busy-loop (reported by @andyxning)

#1005 - nsqadmin: fix typo "Delfate" in connection attrs list (thanks @arussellsaw)

#1032 - nsqd: fix loading metadata with messages queued on un-paused topic with multiple channels (thanks @michaelyou)

#1004 - nsqlookupd: exit with error when failed to listen on ports (thanks @stephens2424)

#1068 - nsqadmin: fix html escaping for large_graph url inside javascript

misc test suite improvements and updates (go versions, tls certs, ...)

Source code(tar.gz)
Source code(zip)
nsq-1.1.0.darwin-amd64.go1.10.3.tar.gz(18.96 MB)
nsq-1.1.0.freebsd-amd64.go1.10.3.tar.gz(17.80 MB)
nsq-1.1.0.linux-amd64.go1.10.3.tar.gz(17.77 MB)
nsq-1.1.0.windows-amd64.go1.10.3.tar.gz(17.55 MB)
v1.1.0-rc1(Jul 31, 2018)

See release v1.1.0
Source code(tar.gz)
Source code(zip)
nsq-1.1.0-rc1.darwin-amd64.go1.10.3.tar.gz(18.96 MB)
nsq-1.1.0-rc1.freebsd-amd64.go1.10.3.tar.gz(17.80 MB)
nsq-1.1.0-rc1.linux-amd64.go1.10.3.tar.gz(17.77 MB)
nsq-1.1.0-rc1.windows-amd64.go1.10.3.tar.gz(17.55 MB)
v1.0.0-compat(Mar 22, 2017)
Upgrading from 0.3.8: Numerous backwards incompatible changes:

Deprecated nsqd features removed:

Pre-V1 HTTP endpoints / response format:

/{m,}put (use /{m,}pub)

/{create,delete,empty,pause,unpause}_{topic,channel} (use /{topic,channel}/<operation>)

--max-message-size flag (use --max-msg-size)

V2 protocol IDENTIFY command short_id and long_id properties (use client_id, hostname, and user_agent)

/stats HTTP response name property (use client_id)

Deprecated nsqlookupd features removed:

Pre-V1 HTTP endpoints / response format:

/{create,delete}_{topic,channel} (use /{topic,channel}/<operation>)

/tombstone_topic_producer (use /topic/tombstone)

Deprecated nsqadmin features removed:

--template-dir flag (not required, templates are compiled into binary)

--use-statsd-prefixes flag (use --statsd-counter-format and --statsd-gauge-format)

nsq_stat --status-every flag (use --interval)

--reader-opt on all binaries that had this flag (use --consumer-opt)

nsq_to_file --gzip-compression flag (use --gzip-level)

nsq_to_http --http-timeout and --http-timeout-ms flags (use --http-connect-timeout and --http-request-timeout)

nsq_to_http --round-robin flag (use --mode=round-robin)

nsq_to_http --max-backoff-duration flag (use --consumer-opt=max_backoff_duration,X)

nsq_to_http --throttle-fraction flag (use --sample=X)

nsq_to_nsq --max-backoff-duration flag (use --consumer-opt=max_backoff_duration,X)

nsqd --worker-id deprecated in favor of --node-id (to be fully removed in subsequent release)

This is a compatibility release that drops a wide range of previously deprecated features (#367) while introducing some new deprecations (#844) that we intend to fully remove in a subsequent 1.0 release.

Of note, all of the pre-1.0 HTTP endpoints (and response formats) are gone. Any clients or tools that use these endpoints/response formats won't work with this release. These changes have been available since 0.2.29 (released in July of 2014). Clients wishing to forwards-compatibly upgrade can either use the new endpoints or send the following header:

Accept: application/vnd.nsq version=1.0

Also, many command line flags have been removed — in almost all cases an alternative is available with a (hopefully) more obvious name. These changes have the same affect on config file option names.

On Linux, this release will automatically migrate nsq.<worker-id>.dat named metadata files to nsq.dat in a way that allows users to seamlessly downgrade from this release back to 0.3.8, if necessary. A subsequent release will clean up these convenience symlinks and observe only nsq.dat. See the discussion in #741 and the changes #844 for more details.

Performance wise, #741 landed which significantly reduces global contention on internal message ID generation, providing a ~1.75x speed improvement on multi-topic benchmarks.

Finally, a number of minor issues were resolved spanning contributions from 9 community members! Thanks!

Features:

#766 - use alpine base image for official Docker container (thanks @kenjones-cisco)

#775 - nsqadmin: /config API (thanks @kenjones-cisco)

#776 - nsqadmin, nsq_stat, nsq_to_file, nsq_to_http: HTTP client connect/request timeouts (thanks @kenjones-cisco)

#777/#778/#783/#785 - improve test coverage (thanks @kenjones-cisco)

#788 - to_nsq: add --rate message throttling option

#367 - purge deprecated features (see above)

#741 - nsqd: per-topic message IDs (multi-topic pub benchmarks up to ~1.75x faster)

#850 - nsqd, nsqlookupd, nsqadmin: add --log-prefix option (thanks @ploxiln)

#844 - nsqd: deprecate --worker-id for --node-id and drop ID from nsqd.dat file (thanks @ploxiln)

Bugs:

#787 - nsqlookupd: properly close TCP connection in IOLoop (thanks @JackDrogon)

#792 - nsqdmin: fix root CA verification (thanks @joshuarubin)

#794 - nsq_to_file: require --topic or --topic-pattern (thanks @judwhite)

#816/#823 - nsqadmin: fix handling of IPv6 broadcast addresses (thanks @magnetised)

#805/#832 - nsqd: fix requeue and deferred message accounting (thanks @sdbaiguanghe)

#532/#830 - nsqd: switch to golang/snappy to fix snappy deadlock

#826/#831/#837/#839 - nsqd: fix default --broadcast-address and error when nsqlookupd reqs fail (thanks @ploxiln @stephensearles)

#822/#835 - nsqd: prevent panic in binary /mpub (thanks @yangyifeng01)

#841 - nsqadmin: allow ctrl/meta+click to open a new tab

#843 - nsqd: check for exit before requeing

Source code(tar.gz)
Source code(zip)
nsq-1.0.0-compat.darwin-amd64.go1.8.tar.gz(19.70 MB)
nsq-1.0.0-compat.freebsd-amd64.go1.8.tar.gz(18.50 MB)
nsq-1.0.0-compat.linux-amd64.go1.8.tar.gz(18.47 MB)
nsq-1.0.0-compat.windows-amd64.go1.8.tar.gz(18.31 MB)
v0.3.8(May 26, 2016)
Upgrading from 0.3.7: Binaries contain no backwards incompatible changes.

This release fixes a critical regression in 0.3.7 that could result in message loss when attempting to cleanly shutdown nsqd by sending it a SIGTERM. The expected behavior was for it to flush messages in internal buffers to disk before exiting. See #757 and #759 for more details.

A few performance improvements landed including #743, which improves channel throughput by ~17%, and #740, which reduces garbage when reading messages from disk.

We're now stripping debug info, reducing binary size, in the official binary downloads and Windows binaries are now bundled with the appropriate .exe extension (#726 and #751).

Features:

#743 - nsqd: remove channel messagePump

#751 - strip debug info from binaries (thanks @ploxiln)

#740 - nsqd: reduce garbage when reading from diskqueue (thanks @dieterbe)

Bugs:

#757/#759 - nsqd: properly handle SIGTERM (thanks @judwhite)

#738 - updates for latest go-options

#730 - nsqd: diskqueue sync count on both read/write

#734 - nsqadmin: make rate column work without --proxy-graphite (thanks @ploxiln)

#726 - add .exe extension to Windows binaries (thanks @ploxiln)

#722 - nsqadmin: fix connected duration > 1hr

Source code(tar.gz)
Source code(zip)
nsq-0.3.8.darwin-amd64.go1.6.2.tar.gz(22.13 MB)
nsq-0.3.8.freebsd-amd64.go1.6.2.tar.gz(20.87 MB)
nsq-0.3.8.linux-amd64.go1.6.2.tar.gz(20.87 MB)
nsq-0.3.8.windows-amd64.go1.6.2.tar.gz(20.84 MB)
v0.3.7(Feb 24, 2016)
Upgrading from 0.3.6: Binaries contain no backwards incompatible changes.

This release has been built with Go 1.6.

Highlights include the various work done to reduce nsqd lock contention, significantly improving the impact of high load on the /stats endpoint, addressing issues with timeouts and failures in nsqadmin (#700, #701, #703, #709).

Thanks to @judwhite, nsqd and nsqlookupd now natively support being run as a Windows service (#718). We're also now publishing official Windows releases.

nsqd will now flock its data directory on linux, preventing two nsqd from running simultaneously pointed at the same path (#583).

On the bugfix side, the most noteworthy change is that nsqd will now correctly reset health state on a successful backend write (#671).

Features:

#700/#701/#703/#709 - nsqd: reduce lock contention (thanks @zachbadgett @absolute8511)

#718 - nsqd/nsqlookupd: support running as a windows service (thanks @judwhite)

#706 - nsqd: support enabling/disabling block profile via HTTP (thanks @absolute8511)

#710 - nsqd: support POST /debug/pprof/symbol (thanks @absolute8511)

#662 - nsqadmin: add flags for formatting statsd keys (thanks @kesutton)

#583 - nsqd: flock --data-path on linux

#663 - nsqd: optimize GUID generation (thanks @ploxiln)

Bugs:

#672 - nsqd: fix max size accounting in diskqueue (thanks @judwhite)

#671 - nsqd: reset health on successful backend write (thanks @judwhite)

#615 - nsqd: prevent OOM when reading from nsqlookupd peer

#664/#666 - dist.sh/Makefile cleanup (thanks @ploxiln)

Source code(tar.gz)
Source code(zip)
nsq-0.3.7.darwin-amd64.go1.6.tar.gz(28.00 MB)
nsq-0.3.7.freebsd-amd64.go1.6.tar.gz(27.76 MB)
nsq-0.3.7.linux-amd64.go1.6.tar.gz(27.78 MB)
nsq-0.3.7.windows-amd64.go1.6.tar.gz(27.70 MB)
v0.3.6(Sep 25, 2015)
Upgrading from 0.3.5: Binaries contain no backwards incompatible changes.

We've adopted the Contributor Covenant 1.2 Code of Conduct (#593). Help us keep NSQ open and inclusive by reading and following this document.

We closed a few longstanding issues related to nsqadmin, namely (#323, et al.) converting it to an API and single-page app (so that it is much easier to develop), displaying fine-grained errors (#421, #657), and enabling support for --tls-required configurations (#396).

For nsqd, we added support for deferred publishing aka DPUB (#293), which allows a producer to specify a duration of time to delay initial delivery of the message. We also addressed performance issues relating to large numbers of topics/channels (#577) by removing some per-channel goroutines in favor of a centralized, periodic, garbage collection approach.

In order to provide more flexibility when deploying NSQ in dynamically orchestrated topologies, nsqd now supports the ability to configure nsqlookupd peers at runtime via HTTP (#601), eliminating the need to restart the daemon.

As part of the large nsqadmin refactoring, we took the opportunity to cleanup the internals for all of the daemon's HTTP code paths (#601, #610, #612, #641) as well as improving the test suite so that it doesn't leave around temporary files (#553).

Features:

#593 - add code of conduct

#323/#631/#632/#642/#421/#649/#650/#651/#652/#654 - nsqadmin: convert to API / single-page app

#653 - nsqadmin: expand notification context

#293 - nsqd: add deferred pub (DPUB)

#577 - nsqd: drop per-channel queue workers in favor of centralized queue GC

#584 - nsqlookupd: improve registration DB performance (thanks @xiaost)

#601 - nsqd: HTTP endpoints to dynamically configure nsqlookupd peers

#608 - nsqd: support for filtering /stats to topic/channel (thanks @chrusty)

#601/#610/#612/#641 - improved HTTP internal routing / log HTTP requests

#628 - nsqd: clarify help text for --e2e-processing-latency-percentile

#640 - switch --{consumer,producer}-opt to nsq.ConfigFlag

Bugs:

#656 - nsqadmin: update statsd prefix to stats.counters

#421/#657 - nsqadmin: display upstream/partial errors

#396 - nsqdamin/nsqd: support for --tls-required

#558 - don't overwrite docker root FS

#582 - nsqd: ignore benign EOF errors

#587 - nsqd: GUID error handling / catch errors if GUID goes backwards (thanks @mpe)

#586 - nsqd: fix valid range for --worker-id

#550/#602/#617/#618/#619/#620/#622 - nsqd: fix benchmarks (thanks @Dieterbe)

#553 - cleanup test dirs

#600 - nsqd: enforce diskqueue min/max message size (thanks @twmb)

Source code(tar.gz)
Source code(zip)
nsq-0.3.6.darwin-amd64.go1.5.1.tar.gz(25.05 MB)
nsq-0.3.6.linux-amd64.go1.5.1.tar.gz(24.87 MB)
v0.3.5(Apr 26, 2015)

Upgrading from 0.3.4: Binaries contain no backwards incompatible changes.

This is another quick bug fix release to address the broken nsqadmin binary in the v0.3.4 distribution (see #578).

See the release notes for v0.3.3 for more info.
Source code(tar.gz)
Source code(zip)
nsq-0.3.5.darwin-amd64.go1.4.2.tar.gz(20.98 MB)
nsq-0.3.5.linux-amd64.go1.4.2.tar.gz(20.99 MB)
v0.3.4(Apr 26, 2015)

WARNING: please upgrade to v0.3.5 to address the broken nsqadmin binary.

Upgrading from 0.3.3: Binaries contain no backwards incompatible changes.

This is a quick bug fix release to fix the outdated go-nsq dependency in v0.3.3 for the bundled utilities (see 6e8504e).
Source code(tar.gz)
Source code(zip)
nsq-0.3.4.darwin-amd64.go1.4.2.tar.gz(18.64 MB)
nsq-0.3.4.linux-amd64.go1.4.2.tar.gz(18.64 MB)
v0.3.3(Apr 26, 2015)
WARNING: please upgrade to v0.3.5 to address the outdated go-nsq dependency for the bundled utilities and the broken nsqadmin binary.

Upgrading from 0.3.2: Binaries contain no backwards incompatible changes.

This release is primarily a bug fix release after cleaning up and reorganizing the codebase. nsqadmin is now importable, which paves the way for completing #323. The bundled utilities received a few feature additions and bug fixes (mostly from bug fixes on the go-nsq side).

Features:

#569 - nsqadmin: re-org into importable package

#562 - nsq_to_{nsq,http}: add epsilon-greedy mode (thanks @twmb)

#547 - nsqd: adds start_time to /stats (thanks @ShawnSpooner)

#544 - nsq_to_http: accept any 200 response as success (thanks @mikedewar)

#548 - nsq_to_http: read entire request body (thanks @imgix)

#552/#554/#555/#556/#561 - code cleanup and /internal package re-org (thanks @cespare)

Bugs:

#573 - nsqd: don't persist metadata upon startup (thanks @xiaost)

#560 - nsqd: do not print EOF error when client closes cleanly (thanks @twmb)

#557 - nsqd: fix --tls-required=tcp-https with --tls-client-auth-policy (thanks @twmb)

#545 - enable shell expansion in official Docker image (thanks @paddyforan)

NOTE: the bundled utilities are built against go-nsq v1.0.4 and include all of those features/fixes.
Source code(tar.gz)
Source code(zip)
nsq-0.3.3.darwin-amd64.go1.4.2.tar.gz(18.60 MB)
nsq-0.3.3.linux-amd64.go1.4.2.tar.gz(18.61 MB)
v0.3.2(Feb 8, 2015)
Upgrading from 0.3.1: Binaries contain no backwards incompatible changes however as of this release we've updated our official Docker images.

We now provide a single Docker image nsqio/nsq that includes all of the NSQ binaries. We did this for several reasons, primarily because the tagged versions in the previous incarnation were broken (and did not actually pin to a version!). The new image is an order of magnitude smaller, weighing in around 70mb.

In addition, the impetus for this quick release is to address a slew of reconnect related bug fixes in the utility apps (nsq_to_nsq, nsq_to_file, etc.), for details see the go-nsq v1.0.3 release notes.

Features:

#534/#539/#540 - single Dockerfile approach (thanks @paddyforan)

Bugs:

#529 - nsqadmin: fix more #ephemeral topic deletion issues

#530 - nsqd: fix the provided sample config file (thanks @jnewmano)

#538 - nsqd: fix orphaned ephemeral channels (thanks @adamsathailo)

Source code(tar.gz)
Source code(zip)
nsq-0.3.2.darwin-amd64.go1.4.1.tar.gz(20.85 MB)
nsq-0.3.2.linux-amd64.go1.4.1.tar.gz(20.86 MB)
v0.3.1(Jan 22, 2015)
Upgrading from 0.3.0: No backwards incompatible changes.

This release contains minor bug fixes and feature additions.

There are a number of functionality improvements to the nsq_stat and nsq_to_file helper applications (and general support for #ephemeral topics, broken in 0.2.30).

Additionally, the TLS options continue to improve with support for setting --tls-min-version and a work-around for a bug relating to TLS_FALLBACK_SCSV (to be fixed in Go 1.5).

Features:

#527 - nsq_stat: deprecate --status-every in favor of --interval

#524 - nsq_stat: add --count option (thanks @nordicdyno)

#518 - nsqd: set defaults for --tls-min-version and set TLS max version to 1.2

#475/#513/#518 - nsqd: --tls-required can be disabled for HTTP / add --tls-min-version (thanks @twmb)

#496 - nsq_to_file: add <PID> to filename and rotation by size/interval (thanks @xiaost)

#507 - nsq_stat: add rates (thanks @xiaost)

#505 - nsqd: speed up failure path of BytesToBase10 (thanks @iand)

Bugs:

#522 - nsqadmin: fix #ephemeral topic deletion issues

#509 - nsqd: fix diskqueue atomic rename on Windows (thanks @allgeek)

#479 - nsqd: return output_buffer_* resolved settings in IDENTIFY response (thanks @tj)

Source code(tar.gz)
Source code(zip)
nsq-0.3.1.darwin-amd64.go1.4.1.tar.gz(20.84 MB)
nsq-0.3.1.linux-amd64.go1.4.1.tar.gz(20.85 MB)
v0.3.0(Nov 18, 2014)
Upgrading from 0.2.31: No backwards incompatible changes.

This release includes a slew of bug fixes and few key feature additions.

The biggest functional change is that nsqd no longer decrements its RDY count for clients. This means that client libraries no longer have to periodically re-send RDY. For some context, nsqd already provided back-pressure due to the fact that a client must respond to messages before receiving new ones. The decremented RDY count only made the implementation of the server and client more complex without additional benefit. Now the RDY command can be treated as an "on/off" switch. For details see #404 and the associated changes in bitly/go-nsq#83 and bitly/pynsq#98.

The second biggest change (and oft-requested feature!) is #ephemeral topics. Their behavior mirrors that of channels. This feature is incredibly useful for situations where you're using topics to "route" messages to consumers (like RPC) or when a backlog of messages is undesirable.

There are now scripts in the bench directory that automate the process of running a distributed benchmark. This is a work-in-progress, but it already provides a closer-to-production setup and therefore more accurate results. There's much work to do here!

A whole bunch of bugs were fixed - notably all were 3rd-party contributions! Thanks!

#305 - #ephemeral topics

#404/#459 - don't decr RDY / send RDY before FIN/REQ

#472 - improve nsqd diskqueue sync strategies

#488 - ability to filter topics by regex in nsq_to_file (thanks @lxfontes)

#438 - distributed pub-sub benchmark scripts

#448 - better nsqd IOLoop logging (thanks @rexposadas)

#458 - switch to gpm for builds

Bugs:

#493 - ensure all nsqd Notify() goroutines have exited prior to shutdown (thanks @allgeek)

#492 - ensure diskqueue syncs at end of benchmarks (thanks @Dieterbe)

#490 - de-flake TestPauseMetadata (thanks @allgeek)

#486 - require ports to be specified for daemons (thanks @jnewmano)

#482 - better bash in dist.sh (thanks @losinggeneration)

#480 - fix panic when nsqadmin checks stats for missing topic (thanks @jnewmano)

#469 - fix panic when misbehaving client sends corrupt command (thanks @prio)

#461 - fix panic when nsqd decodes corrupt message data (thanks @twmb)

#454/#455 - fix 32-bit atomic ops in nsq_to_nsq/nsq_to_http (thanks @leshik)

#451 - fix go get compatibility (thanks @adams-sarah)

Source code(tar.gz)
Source code(zip)
nsq-0.3.0.darwin-amd64.go1.3.3.tar.gz(20.11 MB)
nsq-0.3.0.linux-amd64.go1.3.3.tar.gz(20.09 MB)
v0.2.31(Aug 26, 2014)
Upgrading from 0.2.30: No backwards incompatible changes.

This release includes a few key changes. First, we improved feedback and back-pressure when nsqd writes to disk. Previously this was asynchronous and would result in clients not knowing that their PUB had failed. Interestingly, this refactoring improved performance of PUB by 41%, by removing the topic's goroutine responsible for message routing in favor of N:N Go channel communication. For details see #437.

@paddyforan contributed official Dockerfiles that are now built automatically via Docker Hub. Please begin to use (and improve these) as the various older images we had been maintaining will be deprecated. See the updated docs.

The utility apps deprecated the --reader-opt flag in favor of --consumer-opt and nsq_to_nsq and to_nsq received a --producer-opt flag, for configuring details of the connection publishing to nsqd. Additionally, it is now possible to configure client side TLS certificates via tls_cert and tls_key opts.

As usual, we fixed a few minor bugs, see below for details.

New Features / Enhancements:

#422/#437 - nsqd: diskqueue error feedback/backpressure (thanks @boyand)

#412 - official Dockerfiles for nsqd, nsqlookupd, nsqadmin (thanks @paddyforan)

#442 - utilities: add --consumer-opt alias for --reader-opt and add --producer-opt to nsq_to_nsq (also support configuration of tls_cert and tls_key)

#448 - nsqd: improve IOLoop error messages (thanks @rexposadas)

Bugs:

#440 - nsqd: fixed statsd GC stats reporting (thanks @jphines)

#434/#435 - refactored/stabilized tests and logging

#429 - nsqd: improve handling/documentation of --worker-id (thanks @bschwartz)

#428 - nsqd: IDENTIFY should respond with materialized msg_timeout (thanks @visionmedia)

Source code(tar.gz)
Source code(zip)
nsq-0.2.31.darwin-amd64.go1.2.2.tar.gz(21.88 MB)
nsq-0.2.31.darwin-amd64.go1.3.1.tar.gz(20.10 MB)
nsq-0.2.31.linux-amd64.go1.2.2.tar.gz(21.99 MB)
nsq-0.2.31.linux-amd64.go1.3.1.tar.gz(20.08 MB)
v0.2.30(Jul 29, 2014)
Upgrading from 0.2.29: No backwards incompatible changes.

IMPORTANT: this is a quick bug-fix release to address a panic in nsq_to_nsq and nsq_to_http, see #425.

New Features / Enhancements:

#417 - nsqadmin/nsqd: expose TLS connection state

#425 - nsq_to_nsq/nsq_to_file: display per-destination-address timings

Bugs:

#425 - nsq_to_nsq/nsq_to_file: fix shared mutable state panic

Source code(tar.gz)
Source code(zip)
nsq-0.2.30.darwin-amd64.go1.0.3.tar.gz(13.21 MB)
nsq-0.2.30.darwin-amd64.go1.2.2.tar.gz(21.86 MB)
nsq-0.2.30.darwin-amd64.go1.3.tar.gz(20.07 MB)
nsq-0.2.30.linux-amd64.go1.0.3.tar.gz(13.82 MB)
nsq-0.2.30.linux-amd64.go1.2.2.tar.gz(21.97 MB)
nsq-0.2.30.linux-amd64.go1.3.tar.gz(20.05 MB)
v0.2.29(Jul 25, 2014)
Upgrading from 0.2.28: No backwards incompatible changes.

This release includes a slew of new features and bug fixes, with contributions from 8 members of the community, thanks!

The most important new feature is authentication (the AUTH command for nsqd), added in #356. When nsqd is configured with an --auth-http-address it will require clients to send the AUTH command. The AUTH command body is opaque to nsqd, it simply passes it along to the configured auth daemon which responds with well formed JSON, indicating which topics/channels and properties on those entities are accessible to that client (rejecting the client if it accesses anything prohibited). For more details, see the spec or the nsqd guide.

Additionally, we've improved performance in a few areas. First, we refactored in-flight handling in nsqd to reduce garbage creation and improve baseline performance 6%. End-to-end processing latency calculations are also significantly faster, thanks to improvements in the perks package.

HTTP response formats have been improved (removing the redundant response wrapper) and cleaning up some of the endpoint namespaces. This change is backwards compatible. Clients wishing to move towards the new response format can either use the new endpoint names or send the following header:

Accept: application/vnd.nsq; version=1.0

Other changes including officially bumping the character limit for topic and channel names to 64 (thanks @svmehta), making the REQ timeout limit configurable in nsqd (thanks @AlphaB), and compiling static asset dependencies into nsqadmin to simplify deployment (thanks @crossjam).

Finally, to_nsq was added to the suite of bundled apps. It takes a stdin stream and publishes to nsqd, an extremely flexible solution (thanks @matryer)!

As for bugs, they're mostly minor, see the pull requests referenced in the section below for details.

New Features / Enhancements:

#304 - apps: added to_nsq for piping stdin to NSQ (thanks @matryer)

#406 - nsqadmin: embed external static asset dependencies (thanks @crossjam)

#389 - apps: report app name and version via user_agent

#378/#390 - nsqd: improve in-flight message handling (6% faster, GC reduction)

#356/#370/#386 - nsqd: introduce AUTH

#358 - increase topic/channel name max length to 64 (thanks @svmehta)

#357 - remove internal go-nsq dependencies (GC reduction)

#330/#366 - version HTTP endpoints, simplify response format

#352 - nsqd: make REQ timeout limit configurable (thanks @AlphaB)

#340 - nsqd: bump perks dependency (E2E performance improvement, see 25086e4)

Bugs:

#384 - nsqd: fix statsd GC time reporting

#407 - nsqd: fix double TOUCH and use of client's configured msg timeout

#392 - nsqadmin: fix HTTPS warning (thanks @juliangruber)

#383 - nsqlookupd: fix race on last update timestamp

#385 - nsqd: properly handle empty FIN

#365 - nsqd: fix IDENTIFY msg_timeout response (thanks @visionmedia)

#345 - nsq_to_file: set proper permissions on new directories (thanks @bschwartz)

#338 - nsqd: fix windows diskqueue filenames (thanks @politician)

Source code(tar.gz)
Source code(zip)
nsq-0.2.29.darwin-amd64.go1.0.3.tar.gz(13.21 MB)
nsq-0.2.29.darwin-amd64.go1.2.2.tar.gz(21.85 MB)
nsq-0.2.29.darwin-amd64.go1.3.tar.gz(20.06 MB)
nsq-0.2.29.linux-amd64.go1.0.3.tar.gz(13.81 MB)
nsq-0.2.29.linux-amd64.go1.2.2.tar.gz(21.95 MB)
nsq-0.2.29.linux-amd64.go1.3.tar.gz(20.04 MB)
v0.2.28(Apr 29, 2014)
Upgrading from 0.2.27: No backwards incompatible changes. We've deprecated the short_id and long_id options in the IDENTIFY command in favor of client_id and hostname, which more accurately reflect the data typically used.

This release includes a few important new features, in particular enhanced nsqd TLS support thanks to a big contribution by @chrisroberts.

You can now require that clients negotiate TLS with --tls-required and you can configure a client certificate policy via --tls-client-auth-policy (require or require-verify):

require - the client must offer a certificate, otherwise rejected

require-verify - the client must offer a valid certificate according to the default CA or the chain specified by --tls-root-ca-file, otherwise rejected

This can be used as a form of client authentication.

Additionally, nsqd is now structured such that it is importable in other Go applications via github.com/bitly/nsq/nsqd, thanks to @kzvezdarov.

Finally, thanks to @paddyforan, nsq_to_file can now archive multiple topics or optionally archive all discovered topics (by specifying no --topic params and using --lookupd-http-address).

New Features / Enhancements:

#334 - nsq_to_file can archive many topics (thanks @paddyforan)

#327 - add nsqd TLS client certificate verification policy, ability to require TLS, and HTTPS support (thanks @chrisroberts)

#325 - make nsqd importable (github.com/bitly/nsq/nsqd) (thanks @kzvezdarov)

#321 - improve IDENTIFY options (replace short_id and long_id with client_id and hostname)

#319 - allow path separator in nsq_to_file filenames (thanks @jsocol)

#324 - display memory depth and total depth in nsq_stat

Bug Fixes:

bitly/go-nsq#19 and bitly/go-nsq#29 - fix deadlocks on nsq.Reader connection close/exit, this impacts the utilities packaged with the NSQ binary distribution such as nsq_to_file, nsq_to_http, nsq_to_nsq and nsq_tail.

#329 - use heartbeat interval for write deadline

#321/#326 - improve benchmarking tests

#315/#318 - fix test data races / flakiness

Source code(tar.gz)
Source code(zip)
nsq-0.2.28.darwin-amd64.go1.0.3.tar.gz(11.63 MB)
nsq-0.2.28.darwin-amd64.go1.2.1.tar.gz(19.27 MB)
nsq-0.2.28.linux-amd64.go1.0.3.tar.gz(12.16 MB)
nsq-0.2.28.linux-amd64.go1.2.1.tar.gz(19.37 MB)
v0.2.27(Feb 17, 2014)
Upgrading from 0.2.26: No backwards incompatible changes. We deprecated --max-message-size in favor of --max-msg-size for consistency with the rest of the flag names.

IMPORTANT: this is another bug-fix release to address an issue in nsqadmin where templates were incompatible with older versions of Go (pre-1.2).

#306 - fix nsqadmin template compatibility (and formatting)

#310 - fix nsqadmin behavior when E2E stats are disabled

#309 - fix nsqadmin INVALID_ERROR on node page tombstone link

#311/#312 - fix nsqd client metadata race condition and test flakiness

#314 - fix nsqd test races (run w/ -race and GOMAXPROCS=4) deprecate --max-message-size

Source code(tar.gz)
Source code(zip)
nsq-0.2.27.darwin-amd64.go1.0.3.tar.gz(11.60 MB)
nsq-0.2.27.darwin-amd64.go1.1.2.tar.gz(13.85 MB)
nsq-0.2.27.darwin-amd64.go1.2.tar.gz(19.23 MB)
nsq-0.2.27.linux-amd64.go1.0.3.tar.gz(12.13 MB)
nsq-0.2.27.linux-amd64.go1.1.2.tar.gz(13.86 MB)
nsq-0.2.27.linux-amd64.go1.2.tar.gz(19.32 MB)
v0.2.26(Feb 7, 2014)
Upgrading from 0.2.25: No backwards incompatible changes.

IMPORTANT: this is a quick bug-fix release to address a regression identified in 0.2.25 where statsd prefixes were broken when using the default (or any) prefix that contained a %s for automatic host replacement.

#303 - fix nsqd --statsd-prefix when using %s host replacement

Source code(tar.gz)
Source code(zip)
nsq-0.2.26.darwin-amd64.go1.0.3.tar.gz(11.60 MB)
nsq-0.2.26.darwin-amd64.go1.1.2.tar.gz(13.85 MB)
nsq-0.2.26.darwin-amd64.go1.2.tar.gz(19.23 MB)
nsq-0.2.26.linux-amd64.go1.0.3.tar.gz(12.13 MB)
nsq-0.2.26.linux-amd64.go1.1.2.tar.gz(13.86 MB)
nsq-0.2.26.linux-amd64.go1.2.tar.gz(19.32 MB)
v0.2.25(Feb 6, 2014)
IMPORTANT: please upgrade to v0.2.26 to resolve a potentially important regression.

Upgrading from 0.2.24: No backwards incompatible changes.

This release adds several commonly requested features.

First, thanks to @elubow you can now configure your clients to sample the stream they're subscribed to. To read more about the details of the implementation see #286 and the original discussion in #223. Eric also contributed an improvement to nsq_tail to add the ability to tail the last N messages and exit.

We added config file support (TOML) for nsqd, nsqlookupd, and nsqadmin - providing even more deployment flexibility. Example configs are in the contrib directory. Command line arguments override the equivalent option in the config file.

We added the ability to pause a topic (it is already possible to pause individual channels). This functionality stops all message flow from topic to channel for all channels of a topic, queueing at the topic level. This enables all kinds of interesting possibilities like atomic channel renames and trivial infrastructure wide operations.

Finally, we now compile the static assets used by nsqadmin into the binary, simplifying deployment. This means that --template-dir is now deprecated and will be removed in a future release and you can remove the templates you previously deployed and maintained.

New Features / Enhancements:

#286 - add client IDENTIFY option to sample a % of messages

#279 - add TOML config file support to nsqd, nsqlookupd, and nsqadmin

#263 - add ability to pause a topic

#291 - compile templates into nsqadmin binary

#285/#288 - nsq_tail support for -n # to get recent # messages

#287/#294 - display client IDENTIFY attributes in nsqadmin (sample rate, TLS, compression)

#189/#296 - add client user agent to nsqadmin

#297 - add nsq_to_nsq JSON message filtering options

Source code(tar.gz)
Source code(zip)
v0.2.24(Dec 7, 2013)
Upgrading from 0.2.23: No backwards incompatible changes. However, as you'll see below, quite a few command line flags to the utility apps (nsq_to_http, nsq_to_file, nsq_to_http) were deprecated and will be removed in the next release. Please use this release to transition over to the new ones.

NOTE: we are now publishing additional binaries built against go1.2

The most prominent addition is the tracking of end-to-end message processing percentiles. This measures the amount of time it's taking from PUB to FIN per topic/channel. The percentiles are configurable and, because there is some overhead in collecting this data, it can be turned off entirely. Please see the section in the docs for implementation details. It looks like this:

Additionally, the utility apps received comprehensive support for all configurable reader options (including compression, which was previously missing). This necessitated a bit of command line flag cleanup, as follows:

nsq_to_file

deprecated --gzip-compression in favor of --gzip-level

deprecated --verbose in favor of --reader-opt=verbose

nsq_to_http

deprecated --throttle-fraction in favor of --sample

deprecated --http-timeout-ms in favor of --http-timeout (which is a duration flag)

deprecated --verbose in favor of --reader-opt=verbose

deprecated --max-backoff-duration in favor of --reader-opt=max_backoff_duration=X

nsq_to_nsq

deprecated --verbose in favor of --reader-opt=verbose

deprecated --max-backoff-duration in favor of --reader-opt=max_backoff_duration=X

New Features / Enhancements:

#280 - add end-to-end message processing latency metrics

#267 - comprehensive reader command line flags for utilities

Source code(tar.gz)
Source code(zip)
nsq-0.2.24.darwin-amd64.go1.0.3.tar.gz(11.47 MB)
nsq-0.2.24.darwin-amd64.go1.1.2.tar.gz(13.70 MB)
nsq-0.2.24.darwin-amd64.go1.2.tar.gz(19.03 MB)
nsq-0.2.24.linux-amd64.go1.0.3.tar.gz(11.99 MB)
nsq-0.2.24.linux-amd64.go1.1.2.tar.gz(13.71 MB)
nsq-0.2.24.linux-amd64.go1.2.tar.gz(19.13 MB)
v0.2.23(Oct 23, 2013)
Upgrading from 0.2.22: No backwards incompatible changes.

We now use godep in order to achieve reproducible builds with pinned dependencies. If you're on go1.1+ you can now just use godep get github.com/bitly/nsq/....

This release includes nsqd protocol compression feature negotiation. Snappy and Deflate are supported, clients can choose their preferred format.

--statsd-prefix can now be used to modify the prefix for the statsd keys generated by nsqd. This is useful if you want to add datacenter prefixes or remove the default host prefix.

Finally, this release includes a "bug" fix that reduces CPU usage for nsqd with many clients by choosing a more reasonable default for a timer used in client output buffering. For more details see #236.

New Features / Enhancements:

#266 - use godep for reproducible builds

#229 - compression (Snappy/Deflate) feature negotiation

#241 - binary support for HTTP /mput

#269 - add --statsd-prefix flag

Bug Fixes:

#278 - fix nsqd race for client subscription cleanup (thanks @simplereach)

#277 - fix nsqadmin counter page

#275 - stop accessing simplejson internals

#274 - nsqd channel pause state lost during unclean restart (thanks @hailocab)

#236 - reduce "idle" CPU usage by 90% with large # of clients

Source code(tar.gz)
Source code(zip)
nsq-0.2.23.darwin-amd64.go1.0.3.tar.gz(11.32 MB)
nsq-0.2.23.darwin-amd64.go1.1.2.tar.gz(13.60 MB)
nsq-0.2.23.linux-amd64.go1.0.3.tar.gz(11.84 MB)
nsq-0.2.23.linux-amd64.go1.1.2.tar.gz(13.61 MB)
v0.2.22(Aug 26, 2013)
Upgrading from 0.2.21: message timestamps are now officially nanoseconds. The protocol docs always stated this however nsqd was actually sending seconds. This may cause some compatibility issues for client libraries/clients that were taking advantage of this field.

This release also introduces support for TLS feature negotiation in nsqd. Clients can optionally enable TLS by using the appropriate handshake via the IDENTIFY command. See #227.

Significant improvements were made to the HTTP publish endpoints and in flight message handling to reduce GC pressure and eliminate memory abuse vectors. See #242, #239, and #245.

This release also includes a new utility nsq_to_nsq for performant, low-latency, copying of an NSQ topic over the TCP protocol.

Finally, a whole suite of debug HTTP endpoints were added (and consolidated) under the /debug/pprof namespace. See #238, #248, and #252. As a result nsqd now supports direct profiling via Go's pprof tool, ie:

$ go tool pprof --web http://ip.address:4151/debug/pprof/heap

New Features / Enhancements:

#227 - TLS feature negotiation

#238/#248/#252 - support for more HTTP debug endpoints

#256 - nsqadmin single node view (with GC/mem graphs)

#255 - nsq_to_nsq utility for copying a topic over TCP

#230 - nsq_to_http takes --content-type flag (thanks @michaelhood)

#228 - nsqadmin displays tombstoned topics in the /nodes list

#242/#239/#245 - reduced GC pressure for inflight and /mput

Bug Fixes:

#260 - tombstone_topic_producer action in nsqadmin missing node info

#244 - fix 64bit atomic alignment issues on 32bit platforms

#251 - respect configured limits for HTTP publishing

#247 - publish methods should not allow 0 length messages

#231/#259 - persist nsqd metadata on topic/channel changes

#237 - fix potential memory leaks with retained channel references

#232 - message timestamps are now nano

#228 - nsqlookupd/nsqadmin displays inactive nodes in /nodes

Source code(tar.gz)
Source code(zip)
nsq-0.2.22.darwin-amd64.go1.0.3.tar.gz(11.14 MB)
nsq-0.2.22.darwin-amd64.go1.1.2.tar.gz(13.40 MB)
nsq-0.2.22.linux-amd64.go1.0.3.tar.gz(11.66 MB)
nsq-0.2.22.linux-amd64.go1.1.2.tar.gz(13.40 MB)
v0.2.21(Jul 2, 2013)
Upgrading from 0.2.20: there are no backward incompatible changes in this release.

This release introduces a significant new client feature as well as a slew of consistency and recovery improvements to diskqueue.

First, we expanded the feature negotiation options for clients. There are many cases where you want different output buffering semantics from nsqd to your client. You can now control both output buffer size and the output buffer timeout via new fields in the IDENTIFY command. You can even disable output buffering if low latency is a priority.

You can now specify a duration between fsyncs via --sync-timeout. This is a far better way to manage when the process fsyncs messages to disk (vs the existing --sync-every which is based on # of messages). --sync-every is now considered a deprecated option and will be removed in a future release.

Finally, 0.2.20 introduced a significant regression in #176 where a topic would not write messages to its channels. It is recommended that all users running 0.2.20 upgrade to this release. For additional information see #217.

New Features / Enhancements:

#214 - add --sync-timeout for time based fsync, improve when diskqueue syncs

#196 - client configurable output buffering

#190 - nsq_tail generates a random #ephemeral channel

Bug Fixes:

#218/#220 - expose --statsd-interval for nsqadmin to handle non 60s statsd intervals

#217 - fix new topic channel creation regression from #176 (thanks @elubow)

#212 - dont use port in nsqadmin cookies

#214 - dont open diskqueue writeFile with O_APPEND

#203/#211 - diskqueue depth accounting consistency

#207 - failure to write a heartbeat is fatal / reduce error log noise

#206 - use broadcast address for statsd prefix

#205 - cleanup example utils exit

Source code(tar.gz)
Source code(zip)
nsq-0.2.21.darwin-amd64.go1.0.3.tar.gz(9.90 MB)
nsq-0.2.21.darwin-amd64.go1.1.1.tar.gz(11.91 MB)
nsq-0.2.21.linux-amd64.go1.0.3.tar.gz(10.36 MB)
nsq-0.2.21.linux-amd64.go1.1.1.tar.gz(11.92 MB)
v0.2.20(Jul 2, 2013)
Upgrading from 0.2.19: there are no backward incompatible changes in this release.

This release adds a couple of convenient features (such as adding the ability to empty a topic) and continues our work to reduce garbage produced at runtime to relieve GC pressure in the Go runtime.

nsqd now has two new flags to control the max value clients can use to set their heartbeat interval as well as adjust a clients maximum RDY count. This is all set/communicated via IDENTIFY.

nsqadmin now displays nsqd -> nsqlookupd connections in the "nodes" view. This is useful for visualizing how the topology is connected as well as situations where --broadcast-address is being used incorrectly.

nsq_to_http now has a "host pool" mode where upstream state will be adjusted based on successful/failed requests and for failures, upstreams will be exponentially backed off. This is an incredibly useful routing mode.

As for bugs, we fixed an issue where "fatal" client errors were not actually being treated as fatal. Under certain conditions deleting a topic would not clean up all of its files on disk. There was a reported issue where the --data-path was not writable by the process and this was only discovered after message flow began. We added a writability check at startup to improve feedback. Finally. deferred_count was being sent as a counter value to statsd, it should be a gauge.

New Features / Enhancements:

#197 - nsqadmin nodes list improvements (show nsqd -> lookupd conns)

#192 - add golang runtime version to daemon version output

#183 - ability to empty a topic

#176 - optimizations to reduce garbage, copying, locking

#184 - add table headers to nsqadmin channel view (thanks @elubow)

#174/#186 - nsq_to_http hostpool mode and backoff control

#173/#187 - nsq_stat utility for command line introspection

#175 - add nsqd --max-rdy-count configuration option

#178 - add nsqd --max-heartbeat-interval configuration option

Bug Fixes:

#198 - fix fatal errors not actually being fatal

#195 - fix delete topic does not delete all diskqueue files

#193 - fix data race in channel requeue

#185 - ensure that --data-path is writable on startup

#182 - fix topic deletion ordering to prevent race conditions with lookupd/diskqueue

#179 - deferred_count as gauge for statsd

#173/#188/#191 - fix nsqadmin counter template error; fix nsqadmin displaying negative rates

Source code(tar.gz)
Source code(zip)
v0.2.19(Jul 2, 2013)
Upgrading from 0.2.18: there are no backward incompatible changes in this release.

This release is a small release that introduces one major client side feature and resolves one critical bug.

nsqd clients can now configure their own heartbeat interval. This is important because as of 0.2.18 all clients (including producers) received heartbeats by default. In certain cases receiving a heartbeat complicated "simple" clients that just wanted to produce messages and not handle asynchronous responses. This gives flexibility for the client to decide how it would like behave.

A critical bug was discovered where emptying a channel would leave client in-flight state inconsistent (it would not zero) which limited deliverability of messages to those clients.

New Features / Enhancements:

#167 - 'go get' compatibility

#158 - allow nsqd clients to configure (or disable) heartbeats

Bug Fixes:

#171 - fix race conditions identified testing against go 1.1 (scheduler improvements)

#160 - empty channel left in-flight count inconsistent (thanks @dmarkham)

Source code(tar.gz)
Source code(zip)
v0.2.18(Jul 2, 2013)
Upgrading from 0.2.17: all V2 clients of nsqd now receive heartbeats (previously only clients that subscribed would receive heartbeats, excluding TCP producers).

Upgrading from 0.2.16: follow the notes in the 0.2.17 changelog for upgrading from 0.2.16.

Beyond the important note above regarding heartbeats this release includes nsq_tail, an extremely useful utility application that can be used to introspect a topic on the command line. If statsd is enabled (and graphite in nsqadmin) we added the ability to retrieve rates for display in nsqadmin.

We resolved a few critical issues with data consistency in nsqlookupd when channels and topics are deleted. First, deleting a topic would cause that producer to disappear from nsqlookupd for all topics. Second, deleting a channel would cause that producer to disappear from the topic list in nsqlookupd.

New Features / Enhancements:

#131 - all V2 nsqd clients get heartbeats

#154 - nsq_tail example reader

#143 - display message rates in nsqadmin

Bug Fixes:

#148 - store tombstone data per registration in nsqlookupd

#153 - fix large graph formulas in nsqadmin

#150/#151 - fix topics disappearing from nsqlookupd when channels are deleted

Source code(tar.gz)
Source code(zip)
v0.2.17(Jul 2, 2013)
Upgrading from 0.2.16: IDENTIFY and SUB now return success responses (they previously only responded to errors). The official Go and Python libraries are forwards/backwards compatible with this change however 3rd party client libraries may not be.

Upgrading from 0.2.15: in #132 deprecations in SUB were removed as well as support for the old, line oriented, nsqd metadata file format. For these reasons you should upgrade to 0.2.16 first.

New Features / Enhancements:

#119 - add TOUCH command to nsqd

#142 - add --broadcast-address flag to nsqd/nsqadmin (thanks @dustismo)

#135 - atomic MPUB

#133 - improved protocol fatal error handling and responses; IDENTIFY/SUB success responses

#118 - switch nsqadmin actions to POST and require confirmation

#117/#147 - nsqadmin action POST notifications

#122 - configurable msg size limits

#132 - deprecate identify in SUB and old nsqd metadata file format

Bug Fixes:

#144 - empty channel should clear inflight/deferred messages

#140 - fix MPUB protocol documentation

#139 - fix nsqadmin handling of legacy statsd prefixes for graphs

#138/#145 - fix nsqadmin action redirect handling

#134 - nsqd to nsqlookupd registration fixes

#129 - nsq_to_file gzip file versioning

#106 - nsqlookupd topic producer tombstones

#100 - sane handling of diskqueue read errors

#123/#125 - fix notify related exit deadlock

Source code(tar.gz)
Source code(zip)
v0.2.16(Jul 2, 2013)
Upgrading from 0.2.15: there are no backward incompatible changes in this release.

However, this release introduces the IDENTIFY command (which supersedes sending metadata along with SUB) for clients of nsqd. The old functionality will be removed in a future release.

#114 persist paused channels through restart

#121 fix typo preventing compile of bench_reader (thanks @datastream)

#120 fix nsqd crash when empty command is sent (thanks @michaelhood)

#115 nsq_to_file --filename-format --datetime-format parameter and fix

#101 fix topic/channel delete operations ordering

#98 nsqadmin fixes when not using lookupd

#90/#108 performance optimizations / IDENTIFY protocol support in nsqd. For a single consumer of small messages (< 4k) increases throughput ~400% and reduces # of allocations ~30%.

#105 strftime compatible datetime format

#103 nsq_to_http handler logging

#102 compatibility with Go tip

#99 nsq_to_file --gzip flag

#95 proxy graphite requests through nsqadmin

#93 fix nqd API response for no topics

#92 graph rendering options

#86 nsq_to_http Content-Length headers

#89 gopkg doc updates

#88 move pynsq to it's own repo

#81/#87 reader improvements / introduced MPUB. Fix bug for mem-queue-size < 10

#76 statsd/graphite support

#75 administrative ability to create topics and channels

Source code(tar.gz)
Source code(zip)