TPSS Operation Guide

Overview

The Thermodynamic Profiling Software System is a software package that processes raw AERI data and optional meteorological surface observations in order to produce thermodynamic profiles and uncertainties using an optimal estimation method.  The principal components of TPSS are

  • rsync data transfer
    • transfer the raw AERI data (.RNC and .SUM files) from an AERI or other source into the \raw directory
    • (optional) transfer the MET and ceilometer data into the \met and \ceil directories
    • specified by \config\rsync_update.sh, see TPSS Customization Options.
  • Ingest of AERI data and (optional) MET and ceilometer data
    • Converts raw AERI data into NetCDF files and stores them in the \aerich1 and \aerich2 directories
  • Quality control of data
  • Noise filtering of AERI data
    • Note:  4-7 days of ingested data must already exist in the \aerich1 and \aerich2 directories in order for the noise filter to get generated
  • Application of optimal estimation algorithm
  • Output of results and quicklooks

 

 

Setup and Customization

Prior to running TPSS, the retrieval configuration file (vip.txt), climatology and data transfer need to be configured.  See TPSS Customization Options.

 

 

TPSS File Structure

The files related to TPSS are in the directory:  C:\Users\AERI\docker

  • Configuration files (see TPSS Customization Options)
    • \config
  • Input files
    • \raw: raw AERI data, directories of the form AEYYMMDD e.g. AE180301
    • \ceil: (optional) ceilometer data for cloud base height determination
    • \met: (optional) ground-based met data (temperature, pressure, relative humidity)
  • Executable scripts
    • \run: tpss.bat & tpss_stop.bat
  • Intermediary files
    • \aerich1: ingested AERI ch1 data
    • \aerich2: ingested AERI ch2 data
    • \aerinfch1: noise filtered AERI ch1 data
    • \aerinfch2: noise filtered AERI ch2 data
    • \aerisum: ingested AERI summary data
  • Diagnostic files
    • \logs: log files
    • \runflags: flags indicating running processes
  • Output files
    • \aerioe: output files and quicklooks

 

 

Initialization

A recent noise filter file must exist in order for TPSS to run.  The noise filter is generated automatically, but in general 4-7 days of recent ingested data must exist to generate the noise filter.  Run TPSS in reprocessing mode on 4-7 days of recent data:  this will run ingest, but not the retrieval algorithm.  After 4-7 days of data have been ingested, TPSS can run normally in either reprocessing or real-time mode.

e.g.

cd c:\Users\AERI\docker\run
.\tpss.bat 20180301
.\tpss.bat 20180302
.\tpss.bat 20180303
.\tpss.bat 20180304

TPSS will run for 20180304, after enough data have been ingested from the previous days to generate the noise filter.

 

 

Running TPSS in Reprocessing Mode

In reprocessing mode, TPSS is run on a given day YYYYMMDD specified by the user.  The desired temporal resolution and other parameters can be set in vip.txt.  To run TPSS in reprocessing mode:

  • Run Docker for Windows.  Note:  this is configured to start automatically at startup.  Ensure that Docker is running.
  • Open Command Prompt and type

cd c:\Users\AERI\docker\run
.\tpss.bat YYYYMMDD

Where YYYYMMDD is the specified date to reprocess (e.g. 20180301).

  • Progress on the processing will be displayed in the Command Prompt window.
  • By default, TPSS will always append to existing output data for the specified day.  If brand new output is desired, then ensure that previous NetCDF files for the specified day have been deleted.
  • TPSS will stop and shut down automatically when processing is complete.
  • Quicklooks will be generated at the end after the processing is complete.
  • If processing needs to be stopped before the process has completed, type <ctrl>+C, <ctrl>+D in the Command Prompt with the running process.  This may need to be repeated until the process is stopped.

 

 

Running TPSS in Real-time Mode

In real-time mode, TPSS is run continuously in order to output results and quicklooks of the latest available AERI data.  It is recommended to select an appropriate temporal resolution (tres in vip.txt) to match the processing speed of the hardware being used.  To run TPSS in real-time mode:

  • Run Docker for Windows.  Note:  This is configured to start automatically at startup.  Ensure that Docker is running.
  • Click on the tpss icon on the desktop

Or

  • Open Command Prompt and type

cd c:\Users\AERI\docker\run
.\tpss.bat

  • Progress on the processing can be inspected in the log files in \logs (not the Command Prompt window).
  • By default, TPSS will always append to existing output data for the current UTC day.

 

 

Stopping TPSS in Real-time Mode

  • Click on the tpss_stop icon on the desktop

Or

  • Open Command Prompt and type

cd c:\Users\AERI\docker\run
.\tpss_stop.bat

 

 

TPSS Output

When TPSS is run in either real-time or reprocessing mode, output files and quicklooks will be incrementally generated in the output directory:  C:\Users\AERI\docker\aerioe

  • Output NetCDF files *.cdf are self-describing files that contain the full results of the retrieval. e.g.
    • height: height above ground level in km
    • temperature: retrieved temperature in Celsius
    • waterVapor: retrieved water vapor mixing ratio in g/kg
    • sigma_temperature: 1-sigma uncertainty in retrieved temperature in Celsius
    • sigma_waterVapor: 1-sigma uncertainty in retrieved water vapor mixing ratio in g/kg
  • Quicklooks
    • \aerioe.skewt.YYYMMDD.HHMM: Skew-T plots at ~3-hour intervals
    • \aerioe.RH.YYYYMMDD.HHMM: Relative humidity time-height cross section
    • \aerioe.temperature.YYYYMMDD.HHMM: Temperature time-height cross section
    • \aerioe.theta.YYYYMMDD.HHMM: Potential temperature time-height cross section
    • \aerioe.wvmr.YYYYMMDD.HHMM: Water vapor mixing ratio time-height cross section

 

 

 

Troubleshooting

In rare cases TPSS may not start correctly and crash.  Ensure that a noise filter file is present (see Initialization section above).  Reboot the hardware.