: RabbitMQ Utility – amqpfind
- Basic Information
- SSEC Satellite Data Services (SDS) is using RabbitMQ servers to provide events to users signaling when real-time and archive data become available.
- The utility python script amqpfind creates connections to the RabbitMQ servers and listens for events published by SDS ingest processes.
- Three different methods of listening for events. Example code and commands can be found at the following:
- Multiple Servers using Race Criteria
- Amqpfind monitors two or more servers in a race criteria …..
- Servers include:
- mq1.ssec.wisc.edu
- mq2.ssec.wisc.edu
- mq3.ssec.wisc.edu
Download
You can download from the SDS website. The package also contains examples using the amqpfind library to customize scripts.
Python requirements
Multi-server: Python 2.7 and greater
Tested operating systems
Redhat/CentOS 6,7 Rocky 8 and 9
OS X 10.0
Syntax
amqpfind -H -u -p [-t] [-X] [-j'{key} {key}' | '{?}'] [-C key | key]
| -H | host ip address of the Satellite Data Services RabbitMQ server mq1(2).ssec.wisc.edu | |||
| -u | user for SDS RabbitMQ server (sdsuser) | |||
| -p | password for SDS RabbitMQ server (sdsmq) | |||
| -j | outputs subset of json key values of the message; must be a single quoted string, each key must be surrounded by curly braces; for example: -j '{start_time} {band} {status}' | |||
| -j '?' | lists all keys and values from message, curly brackets surrounding question mark may be necessary for some shell types | |||
| -k | (multi-server only) keys checked to see is a message, sent from multiple servers, is considered a duplicate | |||
| -w | (multi-server only) time window, in second, used to watch for duplicate messages | |||
| -t | sets timeout in seconds before amqpfind ends | |||
| -X | SDS RabbitMQ server exchange; valid options are satellite or himawari | |||
| -C | period-separated list of nine subscription keys; specific values or wild card must be specified for all nine keys; options shown below | |||
| -v | debug level | |||
Subscription keys (-C)
type.family.satellite_id.satellite_instrument.medium.server_type.format.classification.status
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Basic Examples
Outputs a user friendly list all messages and content when a GOES-R series satellite abi instrument completes an image:
amqpfind -H mq1.ssec.wisc.edu -u sdsuser -p sdsmq -X satellite -j '?' -C geo.goes.*.abi.*.*.*.image.complete
The intended use of amqpfind is subscribe to one or more message topics and start a script using xargs:
amqpfind -H mq1.ssec.wisc.edu -u sdsuser -p sdsmq -j '{adde_dataset} {start_time}' \
-X satellite -C geo.goes.g16.abi.adde.sdi.ncdf.image.complete | xargs -P1 -n3 myscript.bash
The command started looks as follows:
myscript.bash EASTA/M1 2018-01-23 21:54:29.8 8
In the above xargs command, -P1 limits xargs to use a single processor. If your processing script takes a significant amount of time to complete, you may need to increase the number of processes available to xargs. As an example, the above amqpfind command returns an message for all GOES-16 images, if your processing for a FD image takes longer than 30 seconds, Mesoscale images must wait until the FD processing completes.
Detailed Information
The RabbitMQ utility amqpfind runs in either single server or multi-server mode and makes the necessary connections to RabbitMQ server(s) allowing clients to listen for event messages (e.g., image complete or band end). When a client finds a specific event, a script can be launched and use a message's metadata as input. Metadata in the messages include values such as image start time, coverage, band, server IP and much more. All events may not be of interest so a key based subscription can filter out unwanted events. SDS ingestors publishe message to multiple RabbitMQ, so in the case of one RabbitMQ server failing a user has a backup source of events. SDS maintains the key names and message content. As new uses, new instruments and new facilities are rolled out, expect incremental changes in both message types and content. The amqpfind utility provides basic connections to the SDS RabbitMQ servers using the Advanced Message Queuing Protocol (AMQP). Custom scripts using this protocol or others can be used to subscribe to the SDS RabbitMQ servers. At no time will users be allowed to publish messages to the SDS RabbitMQ servers.
