Basic Information


  • SSEC Satellite Data Services (SDS) is using RabbitMQ servers to provide events to users signaling when real-time and archive data become available.
  • The utility python script amqpfind creates connections to the RabbitMQ servers and listens for events published by SDS ingest processes.
  • Currently, events are only available for GOES-East and Himawari satellites
  • Three different methods of listening for events. Example code and commands can be found at the following:

Download


You can download amqpfind from the SDS website. The package also contains examples using the amqpfind library to customize scripts.

Syntax


amqpfind -H -u -p [-t] [-X] [-j'{key} {key}’ | ‘{?}’] [-C key | key]

-H host ip address of the Satellite Data Services RabbitMQ server mq1(2).ssec.wisc.edu
-u user for SDS RabbitMQ server (sdsuser)
-p password for SDS RabbitMQ server (sdsmq)
-j outputs subset of json key values of the message; must be a single quoted string, each key must be surrounded by curly braces; for example: -j ‘{start_time} {band} {status}’
-j ‘?’ lists all keys and values from message, curly brackets surrounding question mark may be necessary for some shell types
-k (multi-server only) keys checked to see is a message, sent from multiple servers, is considered a duplicate
-w (multi-server only) time window, in second, used to watch for duplicate messages
-t sets timeout in seconds before amqpfind ends
-X SDS RabbitMQ server exchange; valid options are satellite or himawari
-C period-separated list of nine subscription keys; specific values or wild card must be specified for all nine keys; options shown below
-v debug level

Subscription keys (-C)

satellite_type.satellite_family.satellite_id.satellite_instrument.medium.server_type.format.classification.status

Satellite
satellite_type
satellite_family
satellite_id
satellite_instrument
medium
server_type
format
classification
status
GOES-R Series
geo
goes
g16 or g17
abi glm exis magnetometer seiss suvi
files or adde file or adde files file file file
filesystem or adde filesystem or adde filesystem filesystem filesystem filesystem
ncdf
band, image or product point image image image image
start, end or complete complete complete complete complete complete
Himawari
geo
himawari
h8 or 8; h9 or 9
ahi
file or adde
satbuf, arcdata or cluster
hsf
granule, band or image
complete

 Basic Examples


Outputs a user friendly list all messages and content when a GOES-R series satellite abi instrument completes an image:

amqpfind -H mq1.ssec.wisc.edu -u sdsuser -p sdsmq -X satellite -j ‘?’ -C geo.goes.*.abi.*.*.*.image.complete

The intended use of amqpfind is subscribe to one or more message topics and start a script using xargs:

amqpfind -H mq1.ssec.wisc.edu -u sdsuser -p sdsmq -j ‘{adde_dataset} {start_time}’ \
-X satellite -C geo.goes.g16.abi.adde.sdi.ncdf.image.complete | xargs -P1 -n3 myscript.bash

The command started looks as follows:

myscript.bash EASTA/M1 2018-01-23 21:54:29.8 8

In the above xargs command, -P1 limits xargs to use a single processor. If your processing script takes a significant amount of time to complete, you may need to increase the number of processes available to xargs. As an example, the above amqpfind command returns an message for all GOES-16 images, if your processing for a FD image takes longer than 30 seconds, Mesoscale images must wait until the FD processing completes.

Detailed Information


The RabbitMQ utility amqpfind runs in either single server or multi-server mode and makes the necessary connections to RabbitMQ server(s) allowing clients to listen for event messages (e.g., image complete or band end).¬† When a client finds a specific event, a script can be launched and use a message’s metadata as input. Metadata in the messages include values such as image start time, coverage, band, server IP and much more. All events may not be of interest so a key based subscription can filter out unwanted events. SDS ingestors publishe message to multiple RabbitMQ, so in the case of one RabbitMQ server failing a user has a backup source of events. SDS maintains the key names and message content. As new uses, new instruments and new facilities are rolled out, expect incremental changes in both message types and content. The amqpfind utility provides basic connections to the SDS RabbitMQ servers using the Advanced Message Queuing Protocol (AMQP). Custom scripts using this protocol or others can be used to subscribe to the SDS RabbitMQ servers. At no time will users be allowed to publish messages to the SDS RabbitMQ servers.