Satellite Data Services

-H	host ip address of the Satellite Data Services RabbitMQ server mq1(2).ssec.wisc.edu
-u	user for SDS RabbitMQ server (sdsuser)
-p	password for SDS RabbitMQ server (sdsmq)
-j	outputs subset of json key values of the message; must be a single quoted string, each key must be surrounded by curly braces; for example: -j ‘{start_time} {band} {status}’
-j	‘{?}’ – lists all keys and values from message
-t	sets timeout in seconds before amqpfind ends, default = 0, no timeout
-X	SDS RabbitMQ server exchange; valid options are satellite or himawari
-C	period-separated list of nine subscription keys; specific values or wild card must be specified for all nine keys
-k	(multi-server only) keys checked to see is a message sent from multiple servers should be processed
-w	(multi-server only) window, in second, used to watch for duplicate messages
-v	debug level

Examples

---

amqpfind -H mq1.ssec.wisc.edu -u sdsuser -p sdsmq -X satellite -C geo.goes.g16.abi.*.*.*.band.end

Output
geo.goes.g16.abi.adde.sdi.ncdf.band.end: ‘{“message_type”: “band”, “status”: “end”, “satellite_family”: “GOES”, “satellite_ID”: “G16”, “medium”: “adde”, “start_time”: “2018-02-02 16:20:34.6”, “coverage”: “Mesoscale-1”, “satellite_location”: “GOES-East”, “create_time”: “2018-02-02 16:20:44.4”, “data_type”: “RadM1”, “instrument”: “ABI”, “end_time”: “2018-02-02 16:20:40.3”, “signal_type”: “grb”, “server_ip”: “satbuf1.ssec.wisc.edu”, “mode”: “3”, “path”: “/data/goes/grb/goes16/2018/2018_02_02_033/abi/L1b/RadM1/OR_ABI-L1b-RadM1-M3C14_G16_s20180331620346_e20180331620403_c20180331620444.nc”, “adde_dataset”: “EASTA/M1”, “title”: “ABI L1b Radiances”, “server_type”: “sdi”, “band”: 14}’

amqpfind -H mq1.ssec.wisc.edu -u sdsuser -p sdsmq -X satellite -j ‘{adde_dataset} {start_time} {band}’ -C geo.goes.abi.*.*.*.band.end

Output
EASTA/M1 2018-01-23 21:54:29.8 8

Scripting

---

One method to implement amqpfind is to write a wrapper script. The script can set an environment and start anqpfind. One advantage of using this method is that your processing script can be updated without having to stop and restart the wrapper script.

Wrapper Script code

#! /bin/bash

#
# Script that starts up bash script based on amqpfind messages
#

# Note – when using Full Disk make sure -n is set to 7. Otherwise, ‘Disk’ will be passed to the next call to the script.
# The example xargs command is limited to using one processor (-P1). If your processing script takes a significant amount of time to complete,
# you may need to increase the number of processes available to xargs. In the case below, if processing for a CONUS image takes longer than
# 5 minutes, subsequent processes are forced to wait until the previous process has completed.

export COVERAGE=CONUS
export PATH=$HOME/amqpfind:$PATH

while true; do

amqpfind -H mq1.ssec.wisc.edu -X satellite -j ‘{start_time} {adde_dataset} {server_ip} {satellite_location} {coverage}’ -p sdsmq -u sdsuser /
-C geo.goes.*.abi.*.*.*.image.complete | grep –line-buffered -i ${COVERAGE} | xargs -P1 -n6 $PWD/remap.bash

echo $(date -u) amqpfind died in $0
done

---

Processing Script code

#!/bin/ksh

# Basic script that takes in arguments from an amqpfind command and runs McIDAS commands

ROOT_DIR=$HOME/remap
export ROOT_DIR
if [[ ! -d “${ROOT_DIR}” ]];then
mkdir -p ${ROOT_DIR}
fi

LOG_DIR=${ROOT_DIR}/remap/log
export LOG_DIR
if [[ ! -d “${LOG_DIR}” ]];then
mkdir -p ${LOG_DIR}
fi

export LOG=${LOG_DIR}/remap_script.log
exec >>$LOG
exec 2>>$LOG

DATA_DIR=${ROOT_DIR}/remap/data
export DATA_DIR
if [[ ! -d “${DATA_DIR}” ]];then
mkdir -p ${DATA_DIR}
fi

MCPATH=${DATA_DIR}:/home/mcidas/data
export MCPATH

export LOG=$HOME/remap/log/remap_script.log
exec >>$LOG
exec 2>>$LOG

DAY=$1
TIME=$2
ADDE_DATASET=$3
ADDE_SERVER=$4
SATELLITE=$5
COVERAGE=$6

# McIDAS does not deal with fractional times
TIME=”$(date -d ${TIME} +%H:%M:%S)”
export DAY TIME BAND ADDE_DATASET ADDE_SERVER SATELLITE

# Split apart the adde data
ADDE_GROUP=”$(echo ${ADDE_DATASET}| cut -d’/’ -f1)”
ADDE_DESCRIPTOR=”$(echo ${ADDE_DATASET} | cut -d’/’ -f2)”
export ADDE_DESCRIPTOR ADDE_GROUP

# Define a local ADDE datasets
IMGCOPY_DATASET=IMGCOPY/AREAS
IMGREMAP_DATASET=IMGREMAP/AREAS
export IMGCOPY_DATASET IMGREMAP_DATASET

# Make the script a little easier to update if you need to update center point and size
REMAP_LAT=’45:30:00′
REMAP_LON=’085:30:00′
RES=’3.5′
REMAP_LINES=970
REMAP_ELES=1100
PROJECTION=’PS’
BAND=13
export REMAP_LAT REMAP_LON RES REMAP_LINES REMAP_ELES PROJECTION BAND

# Generate a day/time based file name – first remove quotes from time
FILE_TIME=”$(echo ${TIME} | tr -d ‘:’)”
export FILE_TIME
FILENAME=${SATELLITE}_sector_001_${DAY}_${FILE_TIME}.gif
export FILENAME

# SSEC needs user initials and project number
MCUSER=USER
MCPROJ=PROJ
export MCUSER MCPROJ

# Startup McIDAS
mcenv -i 200 -f 1@${REMAP_LINES}x${REMAP_LINES} << ‘EOF’

logon.k ${MCUSER} ${MCPROJ}

# Define local dataset
dsserve.k ADD ${IMGCOPY_DATASET} AREA 1001 1010
dsserve.k ADD ${IMGREMAP_DATASET} AREA 1011 1011

# Setup MCTABLE entry
dataloc.k ADD $ADDE_GROUP ${ADDE_SERVER}

# One way of guarding agains processing the data more than one time is to first use IMGCOPY
# to copy down a tiny 4×4 part of the dataset into the LOCAL_DATASET. If the date/time already
# exists, you don’t process the data.
mceval.k IMGCOPY ${ADDE_DATASET} ${IMGCOPY_DATASET} SIZE=4 4 BAND=${BAND}
ret=$?

if [[ $ret -ne 0 ]]; then
exit
fi

mceval.k IMGREMAP ${ADDE_DATASET} ${IMGREMAP_DATASET}.1 LATLON=${REMAP_LAT} ${REMAP_LON} SIZE=${REMAP_LINES} ${REMAP_ELES} PRO=${PROJECTION} RES=${RES} BAND=${BAND}
mceval.k IMGDISP ${IMGREMAP_DATASET}.1
mceval.k MAP
mceval.k FRMLABEL \”${SATELLITE} \(BAND\) SECTOR 001 \(DAY\) \(HHMMSS\)
mceval.k FRMSAVE X ${FILENAME}

exit 0
EOF

exit 0