{"id":4388,"date":"2018-05-08T16:31:01","date_gmt":"2018-05-08T16:31:01","guid":{"rendered":"http:\/\/www.ssec.wisc.edu\/datacenter\/?p=4388"},"modified":"2018-05-09T21:37:59","modified_gmt":"2018-05-09T21:37:59","slug":"rabbitmq-utility-amqpfind-single-server","status":"publish","type":"post","link":"https:\/\/www.ssec.wisc.edu\/datacenter\/2018\/05\/08\/rabbitmq-utility-amqpfind-single-server\/","title":{"rendered":"amqpfind – Single Server"},"content":{"rendered":"
\n
\n

Usage<\/h2>\n
\n

In single server mode, amqpfind creates a connection to a single RabbitMQ server and returns messages for the topics specified. Satellite Data Services ingests data from multiple sources. Each of these ingest process may publish identical messages. Using ampfind in this mode requires users’s scripts to be written to guard against the possibility of processing the data multiple times.<\/p>\n

amqpfind -H -u -p [-t] [-X] [-j'{key} {key}’ | ‘{?}’] [-C key | key]<\/p>\n<\/div>\n<\/div>\n\n\n\n\n\n\n\n\n\n\n\n
-H<\/td>\nhost ip address of the Satellite Data Services RabbitMQ server mq1(2).ssec.wisc.edu<\/td>\n<\/tr>\n
-u<\/td>\nuser for SDS RabbitMQ server (sdsuser)<\/td>\n<\/tr>\n
-p<\/td>\npassword for SDS RabbitMQ server (sdsmq)<\/td>\n<\/tr>\n
-j<\/td>\noutputs subset of json key values of the message; must be a single quoted string, each key must be surrounded by curly braces; for example: -j ‘{start_time} {band} {status}’<\/td>\n<\/tr>\n
-j<\/td>\n‘{?}’ – lists all keys and values from message<\/td>\n<\/tr>\n
-t<\/td>\nsets timeout in seconds before amqpfind ends, default = 0, no timeout<\/td>\n<\/tr>\n
-X<\/td>\nSDS RabbitMQ server exchange; valid options are satellite or himawari<\/td>\n<\/tr>\n
-C<\/td>\nperiod-separated list of nine subscription keys; specific values or wild card must be specified for all nine keys<\/td>\n<\/tr>\n
-v<\/td>\ndebug level<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Examples<\/h2>\n
\n

amqpfind -H mq1.ssec.wisc.edu -u sdsuser -p sdsmq -C geo.goes.g16.abi.*.*.*.band.end<\/p>\n

Output
\n<\/strong>geo.goes.g16.abi.adde.sdi.ncdf.band.end: ‘{“message_type”: “band”, “status”: “end”, “satellite_family”: “GOES”, “satellite_ID”: “G16”, “medium”: “adde”, “start_time”: “2018-02-02 16:20:34.6”, “coverage”: “Mesoscale-1”, “satellite_location”: “GOES-East”, “create_time”: “2018-02-02 16:20:44.4”, “data_type”: “RadM1”, “instrument”: “ABI”, “end_time”: “2018-02-02 16:20:40.3”, “signal_type”: “grb”, “server_ip”: “satbuf1.ssec.wisc.edu”, “mode”: “3”, “path”: “\/data\/goes\/grb\/goes16\/2018\/2018_02_02_033\/abi\/L1b\/RadM1\/OR_ABI-L1b-RadM1-M3C14_G16_s20180331620346_e20180331620403_c20180331620444.nc”, “adde_dataset”: “EASTA\/M1”, “title”: “ABI L1b Radiances”, “server_type”: “sdi”, “band”: 14}’<\/p>\n

amqpfind -H mq1.ssec.wisc.edu -u sdsuser -p sdsmq -j ‘{adde_dataset} {start_time} {band}’ -C geo.goes.abi.*.*.*.band.end<\/p>\n

Output
\n<\/strong>EASTA\/M1 2018-01-23 21:54:29.8 8<\/p>\n

 <\/p>\n

Scripting<\/h2>\n
\n

One method to implement amqpfind is to write a wrapper script. The script can set an environment and start anqpfind. One advantage of using this method is that your processing script can be updated without having to stop and restart the wrapper script.<\/p>\n

Wrapper Script code<\/a><\/h3>\n

#! \/bin\/bash<\/p>\n

#
\n# Script that starts up bash script based on amqpfind messages
\n#<\/p>\n

# Note – when using Full Disk make sure -n is set to 7. Otherwise, ‘Disk’ will be passed to the next call to the script.<\/p>\n

export COVERAGE=CONUS
\nexport PATH=$HOME\/remap\/bin:$HOME\/amqpfind:$PATH
\nexport MCPATH=$HOME\/remap\/data\/home\/mcidas\/data<\/p>\n

while true; do<\/p>\n

amqpfind -H mq1.ssec.wisc.edu -X satellite -j ‘{start_time} {adde_dataset} {server_ip} {satellite_location} {coverage}’ -p sdsmq -u sdsuser -C geo.goes.*.abi.*.*.*.image.complete | grep –line-buffered -i ${COVERAGE} | xargs -P1 -n6 $HOME\/remap\/bin\/remap-script.bash<\/p>\n

echo $(date -u) amqpfind died in $0
\ndone<\/p>\n

\n

Processing Script code<\/a><\/h3>\n

#!\/bin\/ksh<\/p>\n

# Basic script that takes in arguments from an amqpfind command and runs McIDAS commands<\/p>\n

export LOG=$HOME\/remap\/log\/remap_script.log
\nexec >>$LOG
\nexec 2>>$LOG<\/p>\n

echo ‘started script’<\/p>\n

DAY=$1
\nTIME=$2
\nADDE_DATASET=$3
\nADDE_SERVER=$4
\nSATELLITE=$5
\nCOVERAGE=$6<\/p>\n

# McIDAS does not deal with fractional times
\nTIME=”$(date -d ${TIME} +%H:%M:%S)”
\nexport DAY TIME BAND ADDE_DATASET ADDE_SERVER SATELLITE<\/p>\n

# Split apart the adde data
\nADDE_GROUP=”$(echo ${ADDE_DATASET}| cut -d’\/’ -f1)”
\nADDE_DESCRIPTOR=”$(echo ${ADDE_DATASET} | cut -d’\/’ -f2)”
\nexport ADDE_DESCRIPTOR ADDE_GROUP<\/p>\n

# Define a local ADDE datasets
\nIMGCOPY_DATASET=IMGCOPY\/AREAS
\nIMGREMAP_DATASET=IMGREMAP\/AREAS
\nexport IMGCOPY_DATASET IMGREMAP_DATASET<\/p>\n

# Make the script a little easier to update if you need to update center point and size
\nREMAP_LAT=’45:30:00′
\nREMAP_LON=’085:30:00′
\nRES=’3.5′
\nREMAP_LINES=970
\nREMAP_ELES=1100
\nPROJECTION=’PS’
\nBAND=13
\nexport REMAP_LAT REMAP_LON RES REMAP_LINES REMAP_ELES PROJECTION BAND<\/p>\n

# Generate a day\/time based file name – first remove quotes from time
\nFILE_TIME=”$(echo ${TIME} | tr -d ‘:’)”
\nexport FILE_TIME
\nFILENAME=${SATELLITE}_sector_001_${DAY}_${FILE_TIME}.gif
\nexport FILENAME<\/p>\n

# SSEC needs user initials and project number
\nMCUSER=USER
\nMCPROJ=PROJ
\nexport MCUSER MCPROJ<\/p>\n

# Startup McIDAS
\nmcenv -i 200 -f 1@${REMAP_LINES}x${REMAP_LINES} << ‘EOF’<\/p>\n

logon.k ${MCUSER} ${MCPROJ}<\/p>\n

# Define local dataset
\ndsserve.k ADD ${IMGCOPY_DATASET} AREA 1001 1010
\ndsserve.k ADD ${IMGREMAP_DATASET} AREA 1011 1011<\/p>\n

# Setup MCTABLE entry
\ndataloc.k ADD $ADDE_GROUP ${ADDE_SERVER}<\/p>\n

# One way of guarding agains processing the data more than one time is to first use IMGCOPY
\n# to copy down a tiny 4×4 part of the dataset into the LOCAL_DATASET. If the date\/time already
\n# exists, you don’t process the data.
\nmceval.k IMGCOPY ${ADDE_DATASET} ${IMGCOPY_DATASET} SIZE=4 4 BAND=${BAND}
\nret=$?<\/p>\n

if [[ $ret -ne 0 ]]; then
\nexit
\nfi<\/p>\n

mceval.k IMGREMAP ${ADDE_DATASET} ${IMGREMAP_DATASET}.1 LATLON=${REMAP_LAT} ${REMAP_LON} SIZE=${REMAP_LINES} ${REMAP_ELES} PRO=${PROJECTION} RES=${RES} BAND=${BAND}
\nmceval.k IMGDISP ${IMGREMAP_DATASET}.1
\nmceval.k MAP
\nmceval.k FRMLABEL \\”${SATELLITE} \\(BAND\\) SECTOR 001 \\(DAY\\) \\(HHMMSS\\)
\nmceval.k FRMSAVE X ${FILENAME}<\/p>\n

exit 0
\nEOF<\/p>\n

exit 0<\/p>\n

 <\/p>\n","protected":false},"excerpt":{"rendered":"

Usage In single server mode, amqpfind creates a connection to a single RabbitMQ server and returns messages for the topics specified. Satellite Data Services ingests data from multiple sources. Each of these ingest process may publish identical messages. Using ampfind in this mode requires users’s scripts to be written to guard against the possibility of […]<\/p>\n","protected":false},"author":23,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-4388","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.ssec.wisc.edu\/datacenter\/wp-json\/wp\/v2\/posts\/4388","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.ssec.wisc.edu\/datacenter\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.ssec.wisc.edu\/datacenter\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.ssec.wisc.edu\/datacenter\/wp-json\/wp\/v2\/users\/23"}],"replies":[{"embeddable":true,"href":"https:\/\/www.ssec.wisc.edu\/datacenter\/wp-json\/wp\/v2\/comments?post=4388"}],"version-history":[{"count":9,"href":"https:\/\/www.ssec.wisc.edu\/datacenter\/wp-json\/wp\/v2\/posts\/4388\/revisions"}],"predecessor-version":[{"id":4398,"href":"https:\/\/www.ssec.wisc.edu\/datacenter\/wp-json\/wp\/v2\/posts\/4388\/revisions\/4398"}],"wp:attachment":[{"href":"https:\/\/www.ssec.wisc.edu\/datacenter\/wp-json\/wp\/v2\/media?parent=4388"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.ssec.wisc.edu\/datacenter\/wp-json\/wp\/v2\/categories?post=4388"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.ssec.wisc.edu\/datacenter\/wp-json\/wp\/v2\/tags?post=4388"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}