RabbitMQ wire tap and swiss army knife

Overview

rabtap - RabbitMQ wire tap

rBuild Status Coverage Status Go Report Card

Swiss army knife for RabbitMQ. Tap/Pub/Sub messages, create/delete/bind queues and exchanges, inspect broker.

Contents

Features

  • tap to messages being sent to exchanges using RabbitMQ exchange-to-exchange bindings without affecting actual message delivery (aka tapping)
  • display broker related information using the RabbitMQ REST management API
  • save messages and meta data for later analysis and replay
  • publish messages to exchanges
  • consume messages from a queue (subscribe)
  • supports TLS
  • no runtime dependencies (statically linked golang single file binary)
  • simple to use command line tool
  • runs on Linux, Windows, Mac and wherever you can compile go

Screenshots

Show broker topology

Output of rabtap info command:

info mode

Output of rabtap info --stats command, showing additional statistics:

info mode

Visualize broker topology with graphviz

Using the --format=dot option, the info command can generate output in the dot format, which can be visualized using graphviz, e.g. rabtap info --show-default --format dot | dot -T svg > mybroker.svg. The resulting SVG file can be visualized with a web browser.

info mode

Tap messages

Output of rabtap in tap mode, showing message meta data and the message body:

info mode

Installation

Pre-compiled binaries

Pre-compiled binaries can be downloaded for multiple platforms from the releases page.

Arch Linux

Rabtap can be installed from the Arch Linux User Repository (AUR):

$ yay -S rabtap-bin

Installation from source

See the build from source section if you prefer to compile from source.

Usage

rabtap - RabbitMQ wire tap.                    github.com/jandelgado/rabtap

Usage:
  rabtap -h|--help
  rabtap info [--api=APIURI] [--consumers] [--stats] [--filter=EXPR] [--omit-empty] 
              [--show-default] [--mode=MODE] [--format=FORMAT] [-knv]
              [--tls-cert-file=CERTFILE] [--tls-key-file=KEYFILE] [--tls-ca-file=CAFILE]
  rabtap tap EXCHANGES [--uri=URI] [--saveto=DIR] [--format=FORMAT] [-jknsv]
                       [--tls-cert-file=CERTFILE] [--tls-key-file=KEYFILE] [--tls-ca-file=CAFILE]
  rabtap (tap --uri=URI EXCHANGES)... [--saveto=DIR] [--format=FORMAT] [-jknsv]
                                      [--tls-cert-file=CERTFILE] [--tls-key-file=KEYFILE] [--tls-ca-file=CAFILE]
  rabtap sub QUEUE [--uri URI] [--saveto=DIR] [--format=FORMAT] [--no-auto-ack] [-jksvn]
                   [--tls-cert-file=CERTFILE] [--tls-key-file=KEYFILE] [--tls-ca-file=CAFILE]
  rabtap pub [--uri=URI] [SOURCE] [--exchange=EXCHANGE] [--routingkey=KEY] [--format=FORMAT] 
             [--delay=DELAY | --speed=FACTOR] [-jkv]
             [--tls-cert-file=CERTFILE] [--tls-key-file=KEYFILE] [--tls-ca-file=CAFILE]
  rabtap exchange create EXCHANGE [--uri=URI] [--type=TYPE] [-adkv]
                                  [--tls-cert-file=CERTFILE] [--tls-key-file=KEYFILE] [--tls-ca-file=CAFILE]
  rabtap exchange rm EXCHANGE [--uri=URI] [-kv] [--tls-cert-file=CERTFILE] [--tls-key-file=KEYFILE] [--tls-ca-file=CAFILE]
  rabtap queue create QUEUE [--uri=URI] [-adkv] [--tls-cert-file=CERTFILE] [--tls-key-file=KEYFILE] [--tls-ca-file=CAFILE]
  rabtap queue bind QUEUE to EXCHANGE --bindingkey=KEY [--uri=URI] [-kv]
                                                       [--tls-cert-file=CERTFILE] [--tls-key-file=KEYFILE] [--tls-ca-file=CAFILE]
  rabtap queue unbind QUEUE from EXCHANGE --bindingkey=KEY [--uri=URI] [-kv]
                                                           [--tls-cert-file=CERTFILE] [--tls-key-file=KEYFILE] [--tls-ca-file=CAFILE]
  rabtap queue rm QUEUE [--uri=URI] [-kv] [--tls-cert-file=CERTFILE] [--tls-key-file=KEYFILE] [--tls-ca-file=CAFILE]
  rabtap queue purge QUEUE [--uri=URI] [-kv] [--tls-cert-file=CERTFILE] [--tls-key-file=KEYFILE] [--tls-ca-file=CAFILE]
  rabtap conn close CONNECTION [--api=APIURI] [--reason=REASON] [-kv] [--tls-cert-file=CERTFILE] [--tls-key-file=KEYFILE] [--tls-ca-file=CAFILE]
  rabtap --version

Arguments and options:
 EXCHANGES            comma-separated list of exchanges and binding keys,
                      e.g. amq.topic:# or exchange1:key1,exchange2:key2.
 EXCHANGE             name of an exchange, e.g. amq.direct.
 SOURCE               file or directory to publish in pub mode. If omitted, stdin will be read.
 QUEUE                name of a queue.
 CONNECTION           name of a connection.
 DIR                  directory to read messages from.
 -a, --autodelete     create auto delete exchange/queue.
 --api=APIURI         connect to given API server. If APIURI is omitted,
                      the environment variable RABTAP_APIURI will be used.
 -b, --bindingkey=KEY binding key to use in bind queue command.
 --by-connection      output of info command starts with connections.
 --consumers          include consumers and connections in output of info command.
 --delay=DELAY        Time to wait between sending messages during publish.
                      If not set then messages will be delayed as recorded. 
                      The value must be suffixed with a time unit, e.g. ms, s etc.
 -d, --durable        create durable exchange/queue.
 --exchange=EXCHANGE  Optional exchange to publish to. If omitted, exchange will
                      be taken from message being published (see JSON message format).
 --filter=EXPR        Predicate for info command to filter queues [default: true]
 --format=FORMAT      * for tap, pub, sub command: format to write/read messages to console
                        and optionally to file (when --saveto DIR is given). 
                        Valid options are: "raw", "json", "json-nopp". Default: raw
                      * for info command: controls generated output format. Valid 
                        options are: "text", "dot". Default: text
 -h, --help           print this help.
 -j, --json           Deprecated. Use "--format json" instead.
 -k, --insecure       allow insecure TLS connections (no certificate check).
 --tls-cert-file=CERTFILE A Cert file to use for client authentication.
 --tls-key-file=KEYFILE   A Key file to use for client authentication.
 --tls-ca-file=CAFILE     A CA Cert file to use for client authentication.
 --mode=MODE          mode for info command. One of "byConnection", "byExchange".
                      [default: byExchange].
 -n, --no-color       don't colorize output (also environment variable NO_COLOR).
 --no-auto-ack        disable auto-ack in subscribe mode. This will lead to
                      unacked messages on the broker which will be requeued
                      when the channel is closed.
 --omit-empty         don't show echanges without bindings in info command.
 --reason=REASON      reason why the connection was closed [default: closed by rabtap].
 -r, --routingkey=KEY routing key to use in publish mode. If omitted, routing key
                      will be taken from message being published (see JSON 
                      message format).
 --saveto=DIR         also save messages and metadata to DIR.
 --show-default       include default exchange in output info command.
 -s, --silent         suppress message output to stdout.
 --speed=FACTOR       Speed factor to use during publish [default: 1.0].
 --stats              include statistics in output of info command.
 -t, --type=TYPE      exchange type [default: fanout].
 --uri=URI            connect to given AQMP broker. If omitted, the
                      environment variable RABTAP_AMQPURI will be used.
 -v, --verbose        enable verbose mode.
 --version            show version information and exit.

Examples:
  rabtap tap --uri amqp://guest:[email protected]/ amq.fanout:
  rabtap tap --uri amqp://guest:[email protected]/ amq.topic:#,amq.fanout:
  rabtap pub --uri amqp://guest:[email protected]/ amq.topic message.json -j
  rabtap info --api http://guest:[email protected]:15672/api

  # use RABTAP_AMQPURI environment variable to specify broker instead of --uri
  export RABTAP_AMQPURI=amqp://guest:[email protected]:5672/
  rabtap queue create JDQ
  rabtap queue bind JDQ to amq.topic --bindingkey=key
  echo "Hello" | rabtap pub amq.topic --routingkey "key"
  rabtap sub JDQ
  rabtap queue rm JDQ

  # use RABTAP_APIURI environment variable to specify mgmt api uri instead of --api
  export RABTAP_APIURI=http://guest:[email protected]:15672/api
  rabtap info
  rabtap info --filter "binding.Source == 'amq.topic'" --omit-empty
  rabtap conn close "172.17.0.1:40874 -> 172.17.0.2:5672"

  # use RABTAP_CERTFILE | RABTAP_KEYFILE | RABTAP_CAFILE environments variables
  # instead of specify --tls-cert-file=CERTFILE --tls-key-file=KEYFILE --tls-ca-file=CAFILE

Basic commands

Rabtap understands the following commands:

  • tap - taps to an exchange and transparently receives messages sent to the exchange, without affecting actual message delivery (using exchange-to-exchange binding). Simulatanous
  • sub - subscribes to a queue and consumes messages sent to the queue (acts like a RabbitMQ consumer)
  • pub - publish messages to an exchange, optionally with the timing as recorded.
  • info - show broker related info (exchanges, queues, bindings, stats). The features of an exchange are displayed in square brackets with D (durable), AD (auto delete) and I (internal). The features of a queue are displayed in square brackets with D (durable), AD (auto delete) and EX (exclusive). If --statistics option is enabled, basic statistics are included in the output. The --filter option allows to filter output. See filtering section for details. Use the --by-connection to sort output by connection (implies --consumers)
  • queue - create/bind/unbind/remove/purge queue
  • exchange - create/remove exchange
  • connection - close connections

See the examples section for further information.

Broker URI specification

The specification of the RabbitMQ broker URI follows the AMQP URI specification as implemented by the go RabbitMQ client library.

Examples:

Note that according to RFC3986 it might be necessary to escape certain characters like e.g. ? (%3F) or # (%23) as otherwise parsing of the URI may fail with an error.

Format specification for tap and sub command

The --format=FORMAT option controls the format of the tap and sub commands when writing messages to the console and optionally to the filesystem (i.e. when --saveto is set).

The FORMAT parameter has the following effect on the output:

FORMAT Format on console Format of saved messages (--saveto DIR)
raw (default) Pretty-printed metadata + raw Message body Metadata as JSON-File + Body as-is
json Pretty-printed JSON wiht base64 encoded body Pretty-printed JSON with base64 encoded body
json-nopp Single line JSON wiht base64 encoded body Pretty-printed JSON with base64 encoded body

Notes:

  • the --json option is now deprecated. Use --format=json instead
  • nopp stands for no pretty-print

Environment variables

Use environment variables to specify standard values for broker and api endpoint.

Default RabbitMQ broker

In cases where the URI argument is optional, e.g. rabtap tap [-uri URI] exchange ..., the URI of the RabbitMQ broker can be set with the environment variable RABTAP_AMQPURI. Example:

$ export RABTAP_AMQPURI=amqp://guest:[email protected]:5672/
$ rabtap tap amq.fanout:
...

Default RabbitMQ management API endpoint

The default RabbitMQ management API URI can be set using the RABTAP_APIURI environment variable. Example:

$ export RABTAP_APIURI=http://guest:[email protected]:15672/api
$ rabtap info
...

Default RabbitMQ TLS config

The default TLS config certificates path can be set using the RABTAP_TLS_CERTFILE and RABTAP_TLS_KEYFILE and RABTAP_TLS_CAFILE environments variables. Example:

$ export RABTAP_TLS_CERTFILE=/etc/rabbitmq/ssl/cert.pem
$ export RABTAP_TLS_KEYFILE=/etc/rabbitmq/ssl/key.pem
$ export RABTAP_TLS_CAFILE =/etc/rabbitmq/ssl/ca.pem
$ echo "Hello" | rabtap pub amq.topic --routingkey "key"
...

Disable color output

Set environment variable NO_COLOR to disable color output.

Examples

The following examples assume a RabbitMQ broker running on localhost:5672 and the management API available on port 15672. Easiest way to start such an instance is by running docker run -ti --rm -p 5672:5672 -p 15672:15672 rabbitmq:3-management or similar command to start a RabbitMQ container.

Broker info

The info command uses the REST API of RabbitMQ to gather and display topolgy related information from the broker.

The --mode MODE option controls how the output is structured. Valid options for MODE are byExchange (default) or byConnection.

The --format=FORMAT option controls the format of generated output. Valid options are text for console text format (default) or dot to output the tree structure in dot format for visualization with graphviz.

Examples (assume that RABTAP_APIURI environment variable is set):

  • rabtap info --consumers - shows virtual hosts exchanges, queues and consumers of given broker in a tree view (see screenshot).
  • rabtap info --mode=byConnection - shows virtual hosts, connections, consumers and queues of given broker in an tree view.
  • rabtap info --format=dot | dot -T svg > broker.svg - renders broker info into dot format and uses graphviz to render a SVG file for final visualization.

Wire-tapping messages

The tap command allows to tap exchanges and transparently receives messages sent to the exchanges. Rabtap automatically reconnects on connections failures. The syntax of the tap command is rabtap tap [--uri URI] EXCHANGES where the EXCHANGES argument specifies the exchanges and binding keys to use. The EXCHANGES argument is of the form EXCHANGE:[KEY][,EXCHANGE:[KEY]]*. If the exchange name contains a colon, use \\: to escape it, e.g. myexchange\\:with\\:colons:KEY.

The acutal format of the binding key depends on the exchange type (e.g. direct, topic, headers) and is described in the RabbitMQ documentation.

Examples for binding keys used in tap command:

  • # on an exchange of type topic will make the tap receive all messages on the exchange.
  • a valid queue name for an exchange of type direct binds exactly to messages destined for this queue
  • an empty binding key for exchanges of type fanout or type headers will receive all messages published to these exchanges

Note: on exchanges of type headers the binding key is currently ignored and all messages are received by the tap.

The following examples assume that the RABTAP_AMQPURI environment variable is set, otherwise you have to pass the additional --uri URI parameter to the commands below.

  • $ rabtap tap my-topic-exchange:#
  • $ rabtap tap my-fanout-exchange:
  • $ rabtap tap my-headers-exchange:
  • $ rabtap tap my-direct-exchange:binding-key

The following example connects to multiple exchanges:

  • $ rabtap tap my-fanout-exchange:,my-topic-exchange:#,my-other-exchange:binding-key
Tap all messages published or delivered

The RabbitMQ Firehose Tracer allows to "see" every message that is published or delivered. To use it, the firehose tracer has to be enabled first:

$ rabbitmqctl rabbitmqctl trace_on 

Then every message published or delivered will be CC'd to the topic exhange amq.rabbitmq.trace. At this exchange, the messages can now be tapped with rabtap:

$ rabtap --uri amqp://guest:[email protected]:5672/ tap amq.rabbitmq.trace:#
Connect to multiple brokers

Rabtap allows you also to connect simultaneously to multiple brokers and exchanges:

  • $ rabtap tap --uri amqp://broker1 amq.topic:# tap --uri amqp://broker2 amq.fanout:

The example connects to broker1 and taps to the amq.topic exchange and to the amq.fanout exchange on broker2.

Message recorder

All tapped messages can be also be saved for later analysis or replay. Rabtap supports saving of messages in two formats: raw body and metadata in separate files or JSON message format with embedded metadata and message the body base64 encode. Examples:

  • $ rabtap tap amq.topic:# --saveto /tmp - saves messages as pair of files consisting of raw message body and JSON meta data file to /tmp directory.
  • $ rabtap tap amq.topic:# --saveto /tmp --format json - saves messages as JSON files to /tmp directory.

Files are created with file name rabtap-+<Unix-Nano-Timestamp>+ . + <extension>.

Messages consumer (subscribe)

The sub command reads messages from a queue. Note that unlike tap, sub will consume messages that are in effect removed from the specified queue. Example:

  • $ rabtap sub somequeue --format json

Will consume messages from queue somequeue and print out messages in JSON format (this is equivalent to using the now deprecated --json option). The Example assumes that RABTAP_AMQPURI environment variable is set, as the --uri AMQPURI parameter is omitted.

Publish messages

The pub command is used to publish messages to an exchange with a routing key. The messages to be published are either read from a file, or from a directory which contains previously recorded messages (e.g. using the --saveto option of the tap command). Messages can be published either in raw format, in which they are send as-is, or in JSON-format, as described here, which includes message metadata and the body in a single JSON document.

The general form of the pub command is

rabtap pub [--uri=URI] [SOURCE] [--exchange=EXCHANGE] [--routingkey=KEY] [--format=FORMAT] 
           [--delay=DELAY | --speed=FACTOR] [-jkv]
  • $ echo hello | rabtap pub amq.fanout - publish "hello" to exchange amqp.fanout
  • $ rabtap pub messages.json --format=json - messages are read from file messages.json in raptab JSON format. Target exchange and routing keys are read from the messages meta data. The messages.json file can contain multiple JSON documents as it is treated as a JSON stream. Rabtap will honor the XRabtapReceived timestamps of the messages and by default delay the messages as they were recorded. This behaviour can be overridden by the --delay and --speed options.
  • $ rabtap pub amq.direct -r myKey --format=json messages.json --delay=0s - as before, but publish messages always to exchange amq.direct with routing key myKey and without any delays.
  • $ rabtap pub amq.direct -r myKey --format=raw somedir --delay=0s - as before, but assuming that somedir is a directory, the messages are read from message files previously recorded to this directory and replayed in the order they were recorded.

Poor mans shovel

Rabtap instances can be connected through a pipe and messages will be read on one side and published to the other. Note that for publish to work in streaming mode, the JSON mode (--format json) must be used on both sides, so that messages are encapsulated in JSON messages.

The example taps messages on broker1 and publishes the messages to the amq.direct exchange on broker2

$ rabtap tap --uri amqp://broker1 my-topic-exchange:# --format json | \
  rabtap pub --uri amqp://broker2 amq.direct -r routingKey --format json

Close connection

The conn command allows to close a connection. The name of the connection to be closed is expected as parameter. Use the info command with the --consumers option to find the connection associated with a queue. Example:

$ rabtap info --consumers
http://localhost:15672/api (broker ver='3.6.9', mgmt ver='3.6.9', cluster='[email protected]')
└── Vhost /
    ├── amq.direct (exchange, type 'direct', [D])
    :
    └── test-topic (exchange, type 'topic', [AD])
        ├── test-q-test-topic-0 (queue, key='test-q-test-topic-0', running, [])
        │   └── __rabtap-consumer-4823a3c0 (consumer user='guest', chan='172.17.0.1:59228 -> 172.17.0.2:5672 (1)')
        │       └── '172.17.0.1:59228 -> 172.17.0.2:5672' (connection client='https://github.com/streadway/amqp', host='172.17.0.2:5672', peer='172.17.0.1:59228')
        ├── test-q-test-topic-1 (queue, key='test-q-test-topic-1', running, [])
        :
$ rabtap conn close '172.17.0.1:59228 -> 172.17.0.2:5672'

Queue commands

The queue command can be used to easily create, remove, bind or unbind queues:

$ rabtap queue create myqueue
$ rabtap info --show-default
http://localhost:15672/api (broker ver='3.7.8', mgmt ver='3.7.8', cluster='[email protected]')
└── Vhost /
    ├── (default) (exchange, type 'direct', [D])
    │   └── myqueue (queue, key='myqueue', idle since 2018-12-07 20:46:15, [])
    :
    └── amq.topic (exchange, type 'topic', [D])
$ rabtap queue bind myqueue to amq.topic --bindingkey hello
$ rabtap info --show-default
http://localhost:15672/api (broker ver='3.7.8', mgmt ver='3.7.8', cluster='[email protected]')
└── Vhost /
    ├── (default) (exchange, type 'direct', [D])
    │   └── myqueue (queue, key='myqueue', idle since 2018-12-07 20:46:15, [])
    :
    └── amq.topic (exchange, type 'topic', [D])
        └── myqueue (queue, key='hello', idle since 2018-12-07 20:46:15, [])
$ rabtap queue unbind myqueue from amq.topic --bindingkey hello
$ rabtap info --show-default
http://localhost:15672/api (broker ver='3.7.8', mgmt ver='3.7.8', cluster='[email protected]')
└── Vhost /
    ├── (default) (exchange, type 'direct', [D])
    │   └── myqueue (queue, key='myqueue', idle since 2018-12-07 20:46:15, [])
    :
    └── amq.topic (exchange, type 'topic', [D])
$ rabtap queue rm myqueue
$ raptap info
http://localhost:15672/api (broker ver='3.7.8', mgmt ver='3.7.8', cluster='[email protected]')
└── Vhost /
    :
    └── amq.topic (exchange, type 'topic', [D])

Additionally use the purge command to remove all elements from a queue, e.g.

$ rabtap queue purge myqueue

JSON message format

When using the --format json option, messages are print/read as a stream of JSON messages in the following format:

...
{
  "ContentType": "text/plain",
  "ContentEncoding": "",
  "DeliveryMode": 0,
  "Priority": 0,
  "CorrelationID": "",
  "ReplyTo": "",
  "Expiration": "",
  "MessageID": "",
  "Timestamp": "2017-11-10T00:13:38+01:00",
  "Type": "",
  "UserID": "",
  "AppID": "rabtap.testgen",
  "DeliveryTag": 27,
  "Redelivered": false,
  "Exchange": "amq.topic",
  "RoutingKey": "test-q-amq.topic-0",
  "XRabtapReceivedTimestamp": "2019-06-13T19:33:51.920711583+02:00",
  "Body": "dGhpcyB0ZXN0IG1lc3NhZ2U .... IGFuZCBoZWFkZXJzIGFtcXAuVGFibGV7fQ=="
}
...

Note that in JSON mode, the Body is base64 encoded.

Filtering output of info command

When your brokers topology is complex, the output of the info command can become very bloated. The --filter helps you to narrow output to the desired information.

Filtering expressions

A filtering expression is a function that evaluates to true or false (i.e. a predicate). Rabtap allows the specification of predicates to be applied when printing queues using the info command. The output will only proceed if the predicate evaluates to true.

Rabtap uses the govalute to evaluate the predicate. This allows or complex expressions.

See official govaluate documentation for further information.

Note: currently the filter is ignored when used in conjunction with --by-connection.

Evaluation context

During evaluation the context (i.e. the current exchange, queue and binding) is available in the expression as variables:

  • the current exchange is bound to the variable exchange
  • the current queue is bound to the variable queue
  • the curren binding is bound to the variable binding

Examples

The examples assume that RABTAP_APIURI environment variable points to the broker to be used, e.g. http://guest:[email protected]:15672/api).

  • rabtap info --filter "exchange.Name == 'amq.direct'" --omit-empty - print only queues bound to exchange amq.direct and skip all empty exchanges.
  • rabtap info --filter "queue.Name =~ '.*test.*'" --omit-empty - print all queues with test in their name.
  • rabtap info --filter "queue.Name =~ '.*test.*' && exchange.Type == 'topic'" --omit-empty - like before, but consider only exchanges of type topic.
  • rabtap info --filter "queue.Consumers > 0" --omit --stats --consumers - print all queues with at least one consumer

Type reference

The types reflect more or less the JSON API objects of the REST API of RabbitMQ transformed to golang types.

Exchange type

Definition of the Exchange type
type Exchange struct {
    Name       string
    Vhost      string
    Type       string
    Durable    bool
    AutoDelete bool
    Internal   bool
    MessageStats struct {
        PublishOut
        PublishOutDetails struct {
            Rate float64
        }
        PublishIn        int
        PublishInDetails struct {
            Rate float64
        }
    }
}

Queue type

Definition of the Queue type
type Queue struct {
    MessagesDetails struct {
        Rate float64
    }
    Messages
    MessagesUnacknowledgedDetails struct {
        Rate float64
    }
    MessagesUnacknowledged int
    MessagesReadyDetails   struct {
        Rate float64
    }
    MessagesReady     int
    ReductionsDetails struct {
        Rate float64
    }
    Reductions int
    Node       string
    Exclusive            bool
    AutoDelete           bool
    Durable              bool
    Vhost                string
    Name                 string
    MessageBytesPagedOut int
    MessagesPagedOut     int
    BackingQueueStatus   struct {
        Mode string
        Q1   int
        Q2   int
        Q3  int
        Q4  int
        Len int
        NextSeqID         int
        AvgIngressRate    float64
        AvgEgressRate     float64
        AvgAckIngressRate float64
        AvgAckEgressRate  float64
    }
    MessageBytesPersistent     int
    MessageBytesRAM            int
    MessageBytesUnacknowledged int
    MessageBytesReady          int
    MessageBytes               int
    MessagesPersistent         int
    MessagesUnacknowledgedRAM  int
    MessagesReadyRAM           int
    MessagesRAM                int
    GarbageCollection          struct {
        MinorGcs        int
        FullsweepAfter  int
        MinHeapSize     int
        MinBinVheapSize int
        MaxHeapSize     int
    }
    State string
    Consumers int
    IdleSince string
    Memory    int
}

Binding type

Definition of the Binding type
type Binding struct {
    Source          string
    Vhost           string
    Destination     string
    DestinationType string
    RoutingKey      string
    PropertiesKey string
}

Build from source

Download and build using go get

$ GO111MODULE=on go get github.com/jandelgado/rabtap/cmd/rabtap

Build using Makefile and tests

To build rabtap from source, you need go (version >= 1.12) and golangci-lint installed.

$ git clone https://github.com/jandelgado/rabtap && cd rabtap
$ make test  -or- make short-test
$ make

In order to run all tests (make test) an instance of RabbitMQ is expected to run on localhost. Easiest way to start one is running make run-broker, which will start a RabbitMQ docker container (i.e. docker run -ti --rm -p 5672:5672 -p 15672:15672 rabbitmq:3-management).

Test data generator

A simple test data generator tool for manual tests is included in the cmd/testgen directory.

Contributing

  • fork this repository
  • create your feature branch
  • add code
  • add tests and make sure test coverage does not fall (make test)
  • make sure pre-commit hook does not fail (./pre-commit)
  • add documentation
  • commit changes
  • submit a PR

Author

Jan Delgado (jdelgado at gmx dot net)

Copyright and license

Copyright (c) 2017-2020 Jan Delgado. rabtap is licensed under the GPLv3 license.

Issues
  • Replay a directory

    Replay a directory

    A feature I would find useful would be to "replay" all the messages recorded in a directory at a speed comparable to their original. I could build this on top of rabtap by using a shell script or a Python script to examine the messages and the .json metadata files, but a built-in facility would be convenient.

    It could look something like the following:

    rabtap replay <directoryName> [--speed=1x]
    

    where directoryName would be a directory containing previously recorded messages and their metadata. The speed parameter would specify how fast to replay the messages and would require that the timestamp had been added when the messages were originally sent. (Or the recording could be modified to automatically add a current timestamp as an option.) If no timestamps are present the replay would always be "as fast as possible".

    The default speed of 1x would replay at real time as follows. The first message would be sent immediately and its timestamp would then be matched with the current timestamp. For all subsequent messages we would compute the duration between the timestamp and the first timestamp, add that duration to our starting timestamp, and pause until that time is reached. 2x, 3x, and so on (including 0.5x and other <1 numbers) would modify the duration appropriately. A value of "max" or perhaps "99x" or something similar would be used to indicate "as fast as possible", i.e. publish as soon as read.

    The exchange and routing key for each message would by default be the same as when it was recorded, but could be overridden via the command line.

    The idea of this feature is to allow a significant recording over time to be replayed in a manner similar to the original. This is useful for:

    1. assisting developers to make changes for their components without having to have full end-to-end installations in place
    2. allowing recordings to be used for automated testing
    3. allowing recordings to be used for performance testing (i.e. can our new version keep up with the rates of the old version).

    If this is deemed useful, our software team may be willing to perform the work.

    enhancement 
    opened by stevenklassen8376 12
  • rabtap info fails

    rabtap info fails

    > rabtap info --verbose --api <URL>/api
    
    ERROR[0001] failed retrieving info from rabbitmq REST api: json: cannot unmarshal string into Go struct field RabbitConsumer.channel_details of type int
    
    > rabtap --version
    
    v1.24
    
    bug workaround 
    opened by yashbhutwala 10
  • Using wrong exchange name doesn't result in any error/warning message when publishing message

    Using wrong exchange name doesn't result in any error/warning message when publishing message

    Using rabtap v1.27 to publish a message with the command like below. The raptap output doesn't indicate whether publishing was actually successful, i.e. it states that message got published while RabbitMQ states an error because the used exchange is unknown.

    $ RABTAP_AMQPURI=amqp://guest:[email protected]:5672/
    $ RABTAP_APIURI=http://guest:[email protected]:15672/api
    $ rabtap --verbose pub --exchange non_existing_exchange --routingkey foo --format json rabtap-1628256743950000000.json
    INFO[0000] waiting for new session ...
    INFO[0000] sleeping for 0s
    DEBUG[0000] publishing message to exchange 'non_existing_exchange' with routing key 'foo'
    INFO[0000] got new amqp session ...
    INFO[0000] publishing channel closed.
    INFO[0000] session: shutting down factory (cancel)
    

    While RabbitMQ server indicates an error on its console output regarding an unknown exchange:

    rabbitmq              | 2021-08-06 13:39:44.940 [info] <0.2930.0> accepting AMQP connection <0.2930.0> (172.21.0.1:62628 -> 172.21.0.3:5672)
    rabbitmq              | 2021-08-06 13:39:44.947 [info] <0.2930.0> connection <0.2930.0> (172.21.0.1:62628 -> 172.21.0.3:5672): user 'guest' authenticated and granted access to vhost '/'
    rabbitmq              | 2021-08-06 13:39:44.951 [error] <0.2939.0> Channel error on connection <0.2930.0> (172.21.0.1:62628 -> 172.21.0.3:5672, vhost: '/', user: 'guest'), channel 1:
    rabbitmq              | operation basic.publish caused a channel exception not_found: no exchange 'non_existing_exchange' in vhost '/'
    rabbitmq              | 2021-08-06 13:39:44.953 [warning] <0.2930.0> closing AMQP connection <0.2930.0> (172.21.0.1:62628 -> 172.21.0.3:5672, vhost: '/', user: 'guest'):
    rabbitmq              | client unexpectedly closed TCP connection
    
    enhancement 
    opened by manuel-koch 8
  • Is it possible to tap queue on default exchange?

    Is it possible to tap queue on default exchange?

    When attempting to run rabtap tap -v "":<my queue name>. I receive

    INFO[0000] waiting for new session ...
    INFO[0000] got new session ...
    INFO[0000] tap: bind to exchange  failed with Exception (403) Reason: "ACCESS_REFUSED - operation not permitted on the default exchange"
    INFO[0000] delete exchange: <nil>
    DEBUG[0000] subscribe: cancel
    INFO[0000] session: shutting down factory (cancel)
    ERROR[0000] tap failed with Exception (403) Reason: "ACCESS_REFUSED - operation not permitted on the default exchange
    

    I believe the issue may be we are trying to bind the queue to the default exchange. Am I completely misunderstanding? I am new to rabbit so any help would be appreciated.

    Also let me know if I can assist in anyway.

    question 
    opened by andrewmweaver89 8
  • Publish a Docker image?

    Publish a Docker image?

    It would be awesome to have rabtap available as a Docker image. I don't know if it's better to run a Go build inside the Dockerfile or just to download/extract the prebuilt binary.

    enhancement question 
    opened by AnthonyMastrean 7
  • Index out of range panic upon encountering an empty string

    Index out of range panic upon encountering an empty string

    Hi, I hope you will find this patch useful/acceptable. It looks like there's no handling, at least in master of empty string values. I observed a panic like this one, only to later realize I was producing an empty value. It feels like the program needs to be more robust to this sort of thing.

    DEBUG[0010] subscribe: messageReceiveLoop: new message rabtap.TapMessage{AmqpMessage:(*amqp.Delivery)(0xc000282140), ReceivedTimestamp:time.Time{wall:0xbf58d195a9233448, ext:10030160565, loc:(*time.Location)(0x17d4a60)}}
    panic: runtime error: index out of range
    
    goroutine 55 [running]:
    main.JSONMessageFormatter.Format(0xc000282140, 0xbf58d195a9233448, 0x255d81ab5, 0x17d4a60, 0xc0000ba5a0, 0xc000080310)
    	/Users/szaydel/go/src/github.com/jandelgado/cmd/rabtap/json_message_formatter.go:31 +0x4ba
    main.PrettyPrintMessage(0x14f6740, 0xc0000aa000, 0xc000282140, 0xbf58d195a9233448, 0x255d81ab5, 0x17d4a60, 0x0, 0xc00007e478, 0x1052fe0)
    	/Users/szaydel/go/src/github.com/jandelgado/cmd/rabtap/message_printer.go:72 +0x10f
    main.createMessageReceiveFuncRaw.func1(0xc000282140, 0xbf58d195a9233448, 0x255d81ab5, 0x17d4a60, 0x0, 0x0)
    	/Users/szaydel/go/src/github.com/jandelgado/cmd/rabtap/subscribe.go:76 +0x8a
    main.messageReceiveLoop.func1(0xc00007e420, 0xc0001bc000)
    	/Users/szaydel/go/src/github.com/jandelgado/cmd/rabtap/subscribe.go:39 +0x77
    created by main.messageReceiveLoop
    	/Users/szaydel/go/src/github.com/jandelgado/cmd/rabtap/subscribe.go:36 +0x243
    
    bug 
    opened by szaydel 6
  • UUID Api update: Multiple-value uuid.NewV4() in single-value context

    UUID Api update: Multiple-value uuid.NewV4() in single-value context

    Unable build pkg because of uuid api update, uuid.NewV4() now returns two values, uuid and error. can you please update the code accordingly.

    opened by krishnakairi 6
  • PowerShell: First publish lost and for subsequent publishes missing end brackets '}' when using PowerShell to receive output

    PowerShell: First publish lost and for subsequent publishes missing end brackets '}' when using PowerShell to receive output

    Summary: I am using PowerShell to create jobs to set up the taps. Then I publish messages and receive job output.

    So, If I have three files to iterate from a folder to publish messages; What I see is that:

    1. First files messages are "Lost"
    2. Second files messages are "Received" but with a missing "}" for the last JSON msg.
    3. Third (and subsequent) file, starts with "}{" and the last line misses "}"

    Note that this is not an issue when I run another PowerShell instance to Set up a tap and receive the output (i.e. only happens when I use the job mechanism). Due to this, I am not sure if this is entirely a PowerShell issue or something that you can fix in rabtap? In any case, I bring this to your notice so that you can try it at your end and atleast are aware of this issue. If something can be done to fix this, that would ofcourse be awesome!

    Steps: The code that I am using is as below:

    Step01: Set up tap using PowerShell jobs

    $pingblockForRawMessagesTap = { rabtap tap "Name.A.B.MyGateway:GatewayMessage:" --json }

        Start-Job $pingblockForRawMessagesTap -Name tapGatewayMsg
    

    Step02: Publish messages by reading JSON files.

    Get-Content $inputMessages | rabtap pub Name.A.B.MyGateway:GatewayMessage --json

    Step03: Receive output

    Receive-Job -Name tapGatewayMsg | Out-File $outputRawMessages

    #Output of the first file will be lost. Second have a missing '}' in end. and the next file starts as '}{' with each file missing '{' in the last line.

    opened by PramodKumarYadav 6
  • Non-beautified JSON output

    Non-beautified JSON output

    For automation & further processing of messages it's somewhat annoying that the JSON-formatted output is beautified. Therefore you can't parse rabtap's output line by line. You need to assemble the whole output line by line until the JSON message is completed.

    It would be awesome if there would be a flag to disable beautification of the JSON-output (or to opt-in to it).

    enhancement question 
    opened by thannaske 5
  • support header based routing

    support header based routing

    header based routing is now supported in pub, queue bind and queue unbind commands.

    opened by jandelgado 1
  • Feature Request: Add

    Feature Request: Add "json" as valid output for "info" command

    To ease parsing of rabtap output, it would be nice if "json" would be a valid output format, using it like rabtap info --stats --format=json

    enhancement 
    opened by manuel-koch 2
  • Feature Request: Terminate after all messages are consumed from queue

    Feature Request: Terminate after all messages are consumed from queue

    I like to use rabtap to consume all messages from a DeadLetter/ParkinLot queue to inspect them and later maybe re-publish them.

    Currently rabtap seem to continue to consume message from given queue without ever terminating ( unless I hit Ctrl-C ).

    Would be nice if rabtap would terminate ( after a given timeout ) when no more messages where consumed from given queue.

    enhancement 
    opened by manuel-koch 1
  • WIP: add max messages counter

    WIP: add max messages counter

    opened by jandelgado 1
  • Feature Request: Limit number of messages consumed

    Feature Request: Limit number of messages consumed

    Thanks for this really useful tool! I'd like to be able to consume n messages per invocation. Additionally I'd like to be able to use the --delay option when consuming to limit the rate at which messages are consumed.

    eg rabtap sub somequeue --limit 10 --delay 30s would consume 1 message every 30s until 10 messages had been consumed.

    Is this something you might consider implementing? Thanks!

    enhancement 
    opened by hrobertson 1
  • WIP: Show DLX in info command

    WIP: Show DLX in info command

    If a DLX is configured for an exchange, show it in the output of the info command.

    • [x] add output to text rendering
    • [x] add output to dot rendering
    • [ ] Update SVG image in README.md
    opened by jandelgado 1
Releases(v1.29)
A wrapper of streadway/amqp that provides reconnection logic and sane defaults

go-rabbitmq Wrapper of streadway/amqp that provides reconnection logic and sane defaults. Hit the project with a star if you find it useful ⭐ Supporte

Lane Wagner 111 Oct 15, 2021
An AMQP 0-9-1 Go client maintained by the RabbitMQ team. Originally by @streadway: `streadway/amqp`

Go RabbitMQ Client Library This is a Go AMQP 0.9.1 client maintained by the RabbitMQ core team. It was originally developed by Sean Treadway. Differen

RabbitMQ 109 Oct 19, 2021
An easy-to-use CLI client for RabbitMQ.

buneary, pronounced bun-ear-y, is an easy-to-use RabbitMQ command line client for managing exchanges, managing queues and publishing messages to exchanges.

Dominik Braun 46 Jul 4, 2021
A user friendly RabbitMQ library written in Golang.

TurboCookedRabbit A user friendly RabbitMQ library written in Golang to help use streadway/amqp. Based on my work found at CookedRabbit. Work Recently

Tristan (HouseCat) Hyams 90 Oct 11, 2021
High level manegment for rabbitmq.

High level manegment for rabbitmq. Features Simple configuration bootstrap. Gracefully shutting down. Consume messages in parallel specifying a number

Diego 4 Aug 13, 2021
A high-level RabbitMQ driver for Golang.

grabbitmq A high-level RabbitMQ driver for Golang. Import in your project: go get github.com/shaswata56/grabbitmq Usage Demo: package main import (

Shaswata Das 16 Sep 27, 2021
Machinery is an asynchronous task queue/job queue based on distributed message passing.

Machinery Machinery is an asynchronous task queue/job queue based on distributed message passing. V2 Experiment First Steps Configuration Lock Broker

Richard Knop 5.6k Oct 14, 2021
🚀 Golang, Go Fiber, RabbitMQ, MongoDB, Docker, Kubernetes, GitHub Actions and Digital Ocean

Bookings Solução de cadastro de usuários e reservas. Tecnologias Utilizadas Golang MongoDB RabbitMQ Github Actions Docker Hub Docker Kubernetes Digita

Jailton Junior 6 Jul 20, 2021
Abstraction layer for simple rabbitMQ connection, messaging and administration

Jazz Abstraction layer for quick and simple rabbitMQ connection, messaging and administration. Inspired by Jazz Jackrabbit and his eternal hatred towa

SOCIFI Ltd. 13 Aug 10, 2021
Declare AMQP entities like queues, producers, and consumers in a declarative way. Can be used to work with RabbitMQ.

About This package provides an ability to encapsulate creation and configuration of RabbitMQ([AMQP])(https://www.amqp.org) entities like queues, excha

Alex 69 Oct 5, 2021
A tiny wrapper over amqp exchanges and queues 🚌 ✨

Rabbus ?? ✨ A tiny wrapper over amqp exchanges and queues. In memory retries with exponential backoff for sending messages. Protect producer calls wit

Rafael Jesus 92 Oct 4, 2021
Testing message queues with RabbitMQ

Rabbit-MessageQueue Just a repository of RabbitMQ simple usage for queueing messages. You can use this as a sender or a receiver. More information is

Jawady Muhammad Habib 1 Oct 16, 2021
Declarative streaming ETL for mundane tasks, written in Go

Benthos is a high performance and resilient stream processor, able to connect various sources and sinks in a range of brokering patterns and perform h

Ashley Jeffs 3.5k Oct 18, 2021
:incoming_envelope: A fast Message/Event Hub using publish/subscribe pattern with support for topics like* rabbitMQ exchanges for Go applications

Hub ?? A fast enough Event Hub for go applications using publish/subscribe with support patterns on topics like rabbitMQ exchanges. Table of Contents

Leandro Lugaresi 111 Sep 30, 2021
RabbitMQ Reconnection client

rmqconn RabbitMQ Reconnection for Golang Wrapper over amqp.Connection and amqp.Dial. Allowing to do a reconnection when the connection is broken befor

Babiv Sergey 14 Oct 9, 2021
Queue with NATS Jetstream to remove all the erlangs from cloud

Saf in Persian means Queue. One of the problems, that we face on projects with queues is deploying RabbitMQ on the cloud which brings us many challenges for CPU load, etc. I want to see how NATS with Jetstream can work as the queue to replace RabbitMQ.

Parham Alvani 5 Sep 22, 2021
GopherSay allow you to display a message said by a cute random Gopher.

GopherSay About Welcome in GopherSay! GopherSay is inspired by Cowsay program. GopherSay allow you to display a message said by a cute random Gopher.

Aurelie Vache 12 Oct 1, 2021
💨A well crafted go packages that help you build robust, reliable, maintainable microservices.

Hippo A Microservices Toolkit. Hippo is a collection of well crafted go packages that help you build robust, reliable, maintainable microservices. It

Ahmed 138 Aug 6, 2021
Kafka implemented in Golang with built-in coordination (No ZooKeeper, single binary install, Cloud Native)

Jocko Distributed commit log service in Go that is wire compatible with Kafka. Created by @travisjeffery, continued by nash. Goals: Protocol compatibl

Nash.io 104 Aug 9, 2021