GIPS Data Processing Software
G. Martin, M. Smuga-Otto, R. Garcia
Copyright 2005, University of Wisconsin Space Science & Engineering Center.

-----------
Background:
-----------

The Geosynchronous Imaging Fourier Transform 
Spectrometer (GIFTS) instrument is expected to 
generate data at rates exceeding 1TB/day. This codebase
is intended as a prototype for processing data generated 
during ground tests, as well as for testing algorithm 
variations, design alternatives, and implementation 
strategies. It embodies the core philosophy and 
architecture ideas from design studies carried out 
previously for a production-level GIFTS data system. 
These include a component approach which uses 
UML-RT to capture and communicate design 
decisions, a processing pipeline with pluggable 
algorithm stages, a flexible architecture 
implementable on a variety of platforms but targeted 
mainly for a distributed computing facility such as a 
cluster, and focus on the careful management and 
annotation of metadata with a view on complete 
reproducibility of processing and on fault tracing. 

---------
Overview: 
---------

This release includes a test patterns and software designs for "Level 1" radiometric calibration of GIFTS simulated data cubes being provided by UW-SSEC for purpose of algorithm and systems development.

-----------------------------
Simulation data availability: 
-----------------------------

The simulation data used to test this demonstration can be obtained via anonymous ftp from: 

    ftp.ssec.wisc.edu/pub/gifts/data/bb_20051207

----------------------
Software dependencies:
----------------------

This software is dependent on the following libraries:
    Boost C++ library 1.32 or newer (http://www.boost.org)
    Unidata NetCDF library 3.6.0p1 or newer (http://www.unidata.ucar.edu/software/netcdf)
    FFTW2 library version 2.1.5 or newer (http://www.fftw.org)
    A modern C++ compiler (e.g. GCC 3.3 or newer) with STL.

------------------------
Implementation overview:
------------------------

The primary demonstration application is built from SimCal2Test.cc as simcal2. Unit tests are included for the sub-modules.

The data processing takes place as a synchronous pipeline of data processing capsules, including IfgToSpectrum (itself containing IfgShift, CplxFFT, and SpectrumChop sub-capsules), and a three-body Radiometric Calibration. 

I/O is handled using a "BindCdf" toolkit which moves data between memory and NetCDF files. These files are specified using "CDL" format (refer to ncgen and ncdump manual pages in the NetCDF library). 

The main flow of the demonstration includes:
    - build workspace and configuration information for capsules operating in a 'TestBench" environment. (As this implementation matures, this becomes the foundation for a science pipeline capsule.)
    - map input spectra for Hot, Ambient, Space, and Earth interferograms from simulated files
    - for each pixel
        - map in new input data and new output buffers, flushing data to disk as needed
        - invoke interferogram-to-spectrum and three-body radiometric calibration
        
-----------
Walkthrough
-----------

This walkthrough has only been tested on a Linux system. 

The file SimCal2Test.cc contains a reference pipeline implementation that runs IfgToSpectrum and ThreeBodyRadiometricCalibration on longwave data from GIFTS data cubes files. To run SimCal2Test, you must have the following data files:

    - GIFTS NetCDF data cubes for hot, ambient and space, 20051207 release version. Currently there are no simulated Earth files in the correct format, but any blackbody file can be used instead for purposes of testing calibration.
    - template file for NetCDF output 'template.cdf' (distributed with the source and located in CalRad/test)

Before building the application you must install the libraries Blitz, Boost, FFTW and NetCDF, and set corresponding environment variables as described in CalRad/Makefile. 

To build the application, go to the directory CalRad and type "make app". This will create the executable simcal2, which can be called from the command line with the following syntax:

    simcal2 [filetype] [filename] [filetype] [filename] ...

where [filetype] is one of the values:

    H     -- hot blackbody data cube
    W     -- warm blackbody data cube
    S     -- space blackbody data cube
    E     -- observation (Earth) data cube

and [filename] is the name of the data file specified by [filetype], including the path. You must specify one of each blackbody file and one Earth file. The 20051207 version of the data files does not contain temperatures, so those values are currently hard-coded.

The output data will be written in NetCDF format to two files, located in CalRad/test:

    [filename].uncal.nc      -- spectral data written after IfgToSpectrum 
    [filename].cal.nc        -- spectral data written after ThreeBodyRadiometricCalibration (final output)

where [filename] is the name of the Earth data file.

In each output file, the real component of the spectrum is in the variable signalReal_lw, and the imaginary component is on signalImag_lw. Due to a bug in the file output code the output spectra currently have 2048 data points instead of 1024; the second half of the spectrum can be considered garbage and ignored. In the current implementation values are not written to any of the other variables, other than a "generated by" tag.
