1. Basic Information
  • SSEC Satellite Data Services (SDS) is using RabbitMQ servers to provide events to users signaling when real-time and archive data become available.
  • The utility python script amqpfind creates connections to the RabbitMQ servers and listens for events published by SDS ingest processes.
  • Three different methods of listening for events. Example code and commands can be found at the following:

Download

You can download from the SDS website. The package also contains examples using the amqpfind library to customize scripts.

Python requirements

Multi-server: Python 2.7 and greater

Tested operating systems

Redhat/CentOS 6,7 Rocky 8 and 9
OS X 10.0

Syntax

amqpfind -H -u -p [-t] [-X] [-j'{key} {key}' | '{?}'] [-C key | key]

-H host ip address of the Satellite Data Services RabbitMQ server mq1(2).ssec.wisc.edu
-u user for SDS RabbitMQ server (sdsuser)
-p password for SDS RabbitMQ server (sdsmq)
-j outputs subset of json key values of the message; must be a single quoted string, each key must be surrounded by curly braces; for example: -j '{start_time} {band} {status}'
-j '?' lists all keys and values from message, curly brackets surrounding question mark may be necessary for some shell types
-k (multi-server only) keys checked to see is a message, sent from multiple servers, is considered a duplicate
-w (multi-server only) time window, in second, used to watch for duplicate messages
-t sets timeout in seconds before amqpfind ends, when using -t 0, amqpfind will try to reconnect to the server if the connection is lost
-X SDS RabbitMQ server exchange; valid options are satellite or himawari
-C period-separated list of nine subscription keys; specific values or wild card must be specified for all nine keys; options shown below
-i identification value to help distinguish between multiple amqpfind processes; value is not used by amqpfind
-v debug level

Subscription keys (-C)

type.family.satellite_id.satellite_instrument.medium.server_type.format.classification.status

The options for each of the keys can best be seen using the Satellite Data Service RabbitMQ monitoring site (links below require access to SSEC network). The keys can be filtered to help users derive the options for –C argument to amqpfind. Many users trigger their scripts to run when an image is complete (all bands have been ingested), here is an example for GOES-16

-C geo.goes.g16.abi.adde.realtime.ncdf.image.complete

or a combination of GOES-16 and GOES-18.

-C geo.goes.*.abi.adde.realtime.ncdf.image.complete

Similar messages can be viewed for Himawari, Meteosat, COMS, FY

Messages are also published for many of the available instruments

-C geo.goes.g16.*.*.*.*.*.*

Satellite Data Service is striving to publish messages for all incoming data types.

-C *.*.*.*.*.*.*.*.*

 Basic Examples

Outputs a user friendly list all messages and content when a GOES-R series satellite abi instrument completes an image:

amqpfind -H mq1.ssec.wisc.edu -u sdsuser -p sdsmq -X satellite -j '?' -C geo.goes.*.abi.*.*.*.image.complete

The intended use of amqpfind is subscribe to one or more message topics and start a script using xargs:

amqpfind -H mq1.ssec.wisc.edu -u sdsuser -p sdsmq -j '{adde_dataset} {start_time}' \
-X satellite -C geo.goes.g16.abi.adde.sdi.ncdf.image.complete | xargs -P1 -n3 myscript.bash

The command started looks as follows:

myscript.bash EASTA/M1 2018-01-23 21:54:29.8 8

In the above xargs command, -P1 limits xargs to use a single processor. If your processing script takes a significant amount of time to complete, you may need to increase the number of processes available to xargs. As an example, the above amqpfind command returns an message for all GOES-16 images, if your processing for a FD image takes longer than 30 seconds, Mesoscale images must wait until the FD processing completes.

Detailed Information

The RabbitMQ utility amqpfind runs in multi-server mode and makes the necessary connections to RabbitMQ server(s) allowing clients to listen for event messages (e.g., image complete or band end).  When a client finds a specific event, a script can be launched and use a message's metadata as input. Metadata in the messages include values such as image start time, coverage, band, server IP and much more. All events may not be of interest so a key based subscription can filter out unwanted events. SDS ingestors, publishes message to multiple RabbitMQ, so in the case of one RabbitMQ server failing a user has a backup source of events. SDS maintains the key names and message content. As new uses, new instruments and new facilities are rolled out, expect incremental changes in both message types and content. The amqpfind utility provides basic connections to the SDS RabbitMQ servers using the Advanced Message Queuing Protocol (AMQP). Custom scripts using this protocol or others can be used to subscribe to the SDS RabbitMQ servers. At no time will users be allowed to publish messages to the SDS RabbitMQ servers.