Purpose: 
This file describes the code and test data provided to test the spectral scale resampling stage
of the GIFTS information processing pipeline over an entire datacube.

Authors: Ray Garcia and Maciek Smuga-Otto, Space Science and Engineering Center, UW-Madison

Note: Both the code and data are very preliminary, and only have value as demonstrations
of the principles involved.

Directory structure:
. src:                                  source code to run the tests
    |/c++                               the resampler code, written in C++
        |/dfft                          double FFT (heritage) algorithm implementation
        |/sinc                          alternative sinc matrix resampler implementation
    |/python                            python scripts for running tests, 
                                        plus any other required python libraries.

. data:                                 data directory (all data in Flat Binary Format (FBF))
     |/specQC_1800_2_2_LW_on            directory containing on-axis "truth" data
                                        (i.e. without the simulated off-axis effect)
     |/specQC_1800_2_2_LW_off           directory containing data with simulated off-axis
                                        effect
     |/specQC_1800_2_2_LW               test result directory - this is created in the course
                                        of running the tests. An example output is provided.

Overview:
The GIFTS data cube represents the (pixel) data coming from a single "snapshot" by the instrument:
it's a 128 x 128 array (each point of which corresponds to a detector pixel on the detector array)
each element containing a 1025 point spectrum for the LW band (the LW band is the case covered by these tests. By comparison, the SMW band has 2049 points)

Due to the optics of the instrument and the positioning of the detector array on the focal plane, different pixels obtain differently scaled versions of the perceived spectrum.  The correction back to a common wavenumber scale is called spectral resampling, and can be carried out by either of the two algorithms exercised here:

- The double FFT approach first transforms each pixel's data into interferogram space, zero-fills the resulting interferogram and transforms back to an upsampled spectrum - then obtains the resampling by linear interpolation of the upsampled spectrum.

- The sinc matrix approach avoids the FFT's by convolving the spectrum against a particular sinc function(computationally accomplished by multiplying the spectrum against a specially prepared matrix). If this matrix is prepared beforehand and cached for the given pixel it applies to, significant savings in time may be obtained.

Both algorithms require a 128 x 128 table of values which specifies the amount of spectral shift, which may be given as a ratio of the input to output wavenumber scales. This table is provided here in the flat binary file data/specQC_1800_2_2_LW_off/wnRatio.real8.128.128 (here, "rolled" refers to the rolling-off of each point's spectrum to cut out-of-band noise, and to minimize false ringing in any subsequent FFT's or convolutions against a Sinc).

The actual test runs both algorithms on the simulated off-axis effect file (data/specQC_1800_2_2_LW/spectrumOffAxisRolled.real8.1025) to produce corrected data for each algorithm  (data/specQC_1800_2_2_LW/spectrumCorrectedDFFT.real8.1025 data/specQC_1800_2_2_LW/spectrumCorrectedSINC.real8.1025)

these files are then compared against "truth" (data/specQC_1800_2_2_LW/spectrumTruthRolled.real8.1025)

The comparison generates a raw error spectrum (data/specQC_1800_2_2_LW/errorDFFT.real8.1025, data/specQC_1800_2_2_LW/errorSINC.real8.1025) for each pixel as well as a summary per-point max error (data/specQC_1800_2_2_LW/maxErrorDFFT.real8, data/specQC_1800_2_2_LW/maxErrorSINC.real8) and rms error (data/specQC_1800_2_2_LW/rmsErrorDFFT.real8, data/specQC_1800_2_2_LW/rmsErrorSINC.real8) - all these files are vector views of the array - that is, the data for the pixel [1,1] is at index 0, while data for the pixel [128,128] is at index 16383.

Compiling code and running the tests:
C++ source for both resamplers is provided, along with the makefiles. The sinc resampler requires the blas and atlas C++ linear algebra libraries for compilation, while the double FFT resampler requires the FFTW2 library. Some editing of the Makefiles will be necessary to adjust for local environments.

The python test code was tested with Python 2.3, but should work with Python 2.2.
The main test script is src/python/runCubeResampleTest.py - which should go through all the steps 
from creating the test output directory to running both resamplers to genrating the error summaries.
Please remove (or rename) the output directory (data/specQC_1800_2_2_LW) before running the test, to avoid getting an error.

The other important python script is src/python/resampleWrapper2.py, which gives (by way of example) the call interface to the C++ resamplers (both have identical command and data interfaces). 

The test currently uses only a single machine to serially process all pixels in a cube. This takes a few minutes for the double FFT resampler, and significantly longer for the sinc resampler which is asked to generate the matrices - a time-costly step - before applying them to the pixel spectra. In production, such matrices would be cached across a cluster, making this the ultimately more efficient resampling method. 

Since the C++ resamplers run as persistent executables which can be commanded and fed data as demonstrated in resampleWrapper2.py, other wrapping scripts may be developed which maintain a pool of the resampler executables across a cluster, and farm out pixel processing tasks to meet throughput and latency requirements.

Copyright:
Text, code and data are Copyright 2004 University of Wisconsin, Madison