This chapter provides the information you will need to access and use McIDAS-X data. You'll learn:
This chapter is organized into these sections:
Reading and writing data to and from disk is fundamental to many applications programs. A McIDAS-X disk file stores information that applications can randomly access by byte address using standard system library calls.
McIDAS-X disk file utilities have several characteristics that distinguish them from other file system utilities.
The table below describes the McIDAS-X library functions that you will use when programming with disk files. In McIDAS-X, a disk file is called an LW (Large Word) array file. You will notice that many of the Fortran APIs below begin with the letters lw .
Each function is described in the sections below, along with sample code illustrating its use.
The McIDAS-X library provides the Mcread and Mcwrite functions for reading and writing disk files in C. The comparable functions in Fortran are lbi and lbo , which are shown in the code fragment below. The lwi and lwo functions read and write words (4-byte groups) from a disk file.
If your application uses Fortran or C library functions for disk I/O, you must use the volnam or Mcpathname function to convert the name of the disk file into a fully qualified, system pathname/filename . This pathname is essential for locating and working on a file.
The code fragment below illustrates the use of volnam with a Fortran OPEN statement. The maximum number of characters allowed for the fully-qualified pathname is stored in the constant MAXPATHLENGTH in the Fortran INCLUDE file, fileparm.inc . The limit for C is in the file /usr/include/sys/limits.h .
Use the function lwfile to determine if a file with a given name already exists. If the file does not exist, lwfile returns a zero; it does not create the file for you. The following code fragment illustrates how to use lwfile .
c --- find out if file 'mystuff' exists status = lwfile('mystuff') if (status .ne. 0) then call sdest('The file is there!',0) else call edest('The file is NOT there!!!',0) endif ... |
To delete the contents of a disk file but not the file itself, use the lwtrunc or Mctruncate function. These functions remove the contents of the file, leaving one word (4 bytes) of 0x80808080 at the beginning of the file.
Deleting only the contents of a file and not the file itself is important if the file name appears in more than one location in the MCPATH tree. For example, if you delete the file ABC from a directory where you have write permissions but the file also exists further down your MCPATH in a directory where you have only read permissions, you won't be able to create a writable file by that name. If you use lwtrunc or Mctruncate to delete only the file's contents, you can write into it again in the future, since it resides in the original, writable directory.
The code fragment below uses the lwtrunc function to delete the contents of a file.
Use the lwd or Mcremove functions to delete a disk file, as shown in the code fragment below.
As long as a user has read and write permissions, the McIDAS-X disk file I/O subsystem will open all files and permit simultaneous access to these files for both reading and writing. If you write applications that update information in disk files, you must synchronize access to the file to avoid potential file collisions.
McIDAS-X has a locking mechanism called lock for coordinating file updates between applications or between copies of the same application. The lock function acquires exclusive use of a unique lock. If your application is using the lock, another application trying to lock the file must wait until you free the lock with the unlock function before using it.
The lock and unlock functions do not prevent other applications from reading a file or writing to it. They simply ensure an orderly means of updating a file without losing or overwriting information.
The code fragment below shows how to lock and unlock a file to protect its integrity during updating.
Use the lwcopy function to copy one disk file into another, as shown in the code fragment below.
McIDAS-X images are typically composed of atmospheric and oceanographic data, which are measured from remote sensing platforms such as satellites and radar. Images may contain any data that can be represented in a two-dimensional matrix.
Image data has several unique attributes that determine how an image is displayed on the McIDAS-X Image Window. The resolution of the image, the size of the data points and the number of spectral bands all influence the final product.
A satellite observes features by scanning small slices of the earth's surface with each pass of the sensors. The geographic width of each image line helps determine the size of the smallest surface feature the satellite can detect. This concept is called resolution .
Image resolution refers to the number of satellite image lines represented in each data point of an image line. If the image resolution is one, the image is stored at full resolution . This means that one image data point represents one satellite sensor data point. If the image resolution is four, either sampling or averaging was performed on the image so that one data point in the image represents 16 satellite scan data points. Each satellite has its own scan resolution, so an image resolution of one will mean different geographic resolutions from one satellite to another.
When copying or displaying an image, a user can modify its resolution. Image resolution can be artificially increased, or blown up , by replicating data point values, much like enlarging a 3 x 5 photograph to 8 x 10. Image resolution can be decreased, or blown down, by sampling or averaging the image data points. For example, if a blowdown factor of two is applied to a McIDAS-X image, every other data point along the line and every other line in the image is dropped out. Each data point on the displayed image represents four data points from the original, scanned image.
Data-point size refers to the number of bytes needed to accurately represent the value of a data point. Although the size varies among sensor sources, data is one, two or four bytes. For example, Meteosat-7 data is 1-byte per data point while GOES-12 data is 2-byte.
A spectral band is the wavelength in which a scanning instrument measures data; for example, band 4 for the GOES-8 Imager senses 10.7 micron wavelength radiation. These wavelengths are specific to the measuring instrument. Most satellites can measure radiation from many wavelengths simultaneously over the same geographic location.
|
For more information about the bands for satellite imagery, see Appendix B, Satellite Information . |
Image data is quantitatively useless unless it is transformed into physical units (calibration) and oriented relative to time and physical space (navigation). In addition, it is often necessary to know when and how an image was collected and processed. The actual image, along with these ancillary data , is collectively called an image object . Each image object in McIDAS-X is composed of the following blocks of information:
The API functions and the procedures for reading, writing and deleting image objects follow.
Most applications for reading image objects will do one of the following:
The table below lists alphabetically the McIDAS-X library functions for performing these tasks.
These functions are described below along with sample code.
|
See the online man pages provided with the McIDAS-X software for detailed information about any of the API functions discussed in this section. |
In ADDE, a client has the ability to request only directory blocks from a server. This allows the client to sample information on a server without transferring large amounts of image data that may not be necessary for an application. For example, the McIDAS-X IMGLIST command reads only directory blocks.
The ADDE interface to the image directory is through mcadir , which opens a connection based on a set of selection clauses for a given dataset name. The valid selection conditions are provided in the table below.
|
Selection clause |
Description |
|---|---|
|
appends center latitude/longitude, resolution, and calibration types to the comment block (default=YES) |
|
Selection clauses can restrict the search based on the image day, image start time and SSEC sensor source number. With the exception of AUX, you must specify these selection clauses as a range of values.
Note that in a mcadir call the dataset name does not contain the position field. The position, if known, is specified with the SUBSET selection clause. For example, if the application requires the first 10 images for a given dataset, the selection clause is SUBSET 1 10. If the application requires all the image directories in a dataset, the selection clause is SUBSET ALL.
If you include the selection clause AUX YES in the condition list, the image object directory server will append comment entries describing the latitude and longitude of the center element of the image, the earth area (in latitude and longitude resolution) covered by the center element of the image, and the valid calibration types for the image. These values can subsequently be parsed out with the mcpnav and mcpcal functions, as shown in the code fragment on the next page.
Once the connection is opened with mcadir , the application makes repeated calls to mcadrd until all image object directory blocks are retrieved. The mcadir call must precede the call to mcadrd , as shown below.
The directory block contains a list of ancillary information about the image. The entries in the directory block are described in the table below.
IMPORTANT: When using mcadir to read the dataset, a 65 word block is returned, which includes the absolute position number. When using mcaget, the directory is 64 words long, including only words 1-64 in the the table below.
|
Entry |
Description |
|---|---|
|
actual image start time, hhmmss; in milliseconds for POES data |
|
|
For more information about the directory block, see the AREAnnnn data structure in Chapter 6, Format of the Data Files. |
In ADDE, a client can request an entire image object, including the directory, data, line prefix, navigation, calibration and comment blocks, from a server. For example, the McIDAS-X IMGDISP and IMGCOPY commands typically request entire image objects.
To open a connection to read an image object's data block, an application must perform these two steps:
Applications use selection clauses to specify the spatial, temporal and spectral limits of the transaction with the server. This eliminates the need for the application to scan the dataset for a particular image object and also limits the size of the transmission to only that needed. The number and format of selection clauses are strictly regulated. Below is a list of the valid selection clause formats for the mcaget interface, which passes the request for an image sector from the client to the server. Additional information for each selection clause follows.
|
Selection clause |
Description |
|---|---|
|
includes the documentation from the line prefix (default=NO) |
|
|
sets the coordinate type and the coordinate positions relative to the coordinate type (default=AU 0 0) |
|
AUX YES
-- Data server: use this clause to insert the unit and scale factor of the image data into entries 58 and 59 of the directory block.
-- Directory server: use this clause to get center point and calibration information from the server as additional comment cards.
BAND --use this clause to identify a spectral band of image data. Specify BAND ALL to return all spectral bands in the source image.
CAL (obsolete) --use this clause only when the source data type is TIRO. Use CAL QTIR for quick calibration.
DAY --use this clause to specify the day of the source. Using DAY implies that the image object position (POS) is not specified in the connection request. If POS is specified, the DAY clause is ignored.
DOC YES --use this clause to include the line prefix's documentation when reading an image object.
LOCATE --use this clause to position the request spatially. LOCATE specifies a reference point from which sector bounds are determined. To specify the reference point of the image sector, use one of three coordinate systems (I=satellite, A=array or file, E=earth) with one of two standard offsets (U=upper-left, C=center); for example, EC=earth center, IU=satellite upper-left. Following the reference point are two values (ycor and xcor ) that identify the absolute position of the reference point in the chosen coordinate system. For earth coordinates, the values are latitude and longitude; for satellite coordinates, the values are line and element; for array coordinates, the values are array row and column. If you don't specify LOCATE, the default is a satellite upper-left (IU) reference point of the first scan line and pixel of the source image object.
MAG --use this clause to specify the resolution magnification factor for the line and element dimensions of the image object. Enter a negative integer for a blowdown; enter a positive integer for a blowup. For example, a magnification factor of -4 will use every fourth data point on the line and every fourth line in the image. A magnification factor of 16 for both line and element dimensions will duplicate each data point in the image 256 times (16 x 16). The mcaget function performs all line and element duplication. The application must reduce the size of the data request to allow for a blowup factor. To use a magnification factor other than one, modify the input for SIZE accordingly. For example, to get an image that is 500 x 1000 with a MAG value of 2 2, you must specify SIZE 250 500. You cannot store images with non-integer line/element resolutions in the McIDAS-X image object data structure.
POS --use this clause to specify the position of the image in the dataset. If POS isn't specified, the server uses the most current (time-relative) image. Specify POS as either absolute or time relative. Numbers less than one imply time-relative position, with zero as the most current image.
SIZE --use this clause to specify the image line and data element limits of the transaction. Specify SIZE ALL to read the entire source image object.
SU --use this clause to specify the name of an image data stretching table. These tables are generated with the McIDAS-X SU command.
TIME --use this clause to specify a range of image start times identifying specific images in a dataset. Using this clause implies that the image object position (POS) is not part of the connection request. If a POS clause does exist, the TIME clause is ignored.
|
Server-specific selection clause |
Description |
|---|---|
|
Coverage; accesses data for specified time range (realtime POES server) |
|
Once the selection conditions for the image sector are defined, the mcaget function passes the request for the image sector from the client to the server. The return status shows if the connection is open and if the request can be completed.
The mcaget function lets the application specify the units and format of the data points. Since these parameters are necessary to any data transaction, they are specified as separate parameters to the mcaget function and are not entered as selection clauses. Units may be any unit identifier valid for the image object type. A list of valid unit identifiers is available to an application through the mcadir function by specifying the AUX YES selection clause. Use the format parameter to specify the bytes-per-data point in the return array. The valid formats are I1 (1 byte per data point), I2 (2 bytes per data point) and I4 (4 bytes per data point).
If you request that 2-byte data be returned as a 1-byte representation, without going through a calibration process that converts the 2-byte data to 1-byte, the server will truncate the least significant byte.
Because mcaget requires many selection conditions to request image data, the mcasort function can be called to translate command line keyword parameters into equivalent mcaget selection clauses. Any application-level program may call mcasort to retrieve these keywords and return them as mcaget selection clauses. The table below lists the keywords for accessing image objects and their equivalent selection clauses.
If an image sector in the dataset satisfies the client request, a connection is established between the server and the client and the transaction proceeds. Because applications manipulating image data must sometimes sample data from different sources simultaneously, ADDE allows an application to receive data from multiple datasets in any order required for that application. When a request for data is initiated with mcaget , a handle identifying the request is returned. This handle is subsequently used to retrieve each line of data for the request. If your application requires accessing data from multiple requests simultaneously, you only have to manage a handle for each request.
This section describes the functions you should use to get the requested image data from the server and to read the image object's line prefix, navigation, calibration and comment blocks.
The mcalin function reads the requested image data from the server one line at a time. When all the data is read, the connection is closed. Then the mcafree function readies the environment for the next request by freeing the image handle and the memory allocated to store the image data from the previous request.
If you request multispectral data, each data point in the line will contain the measurements for the bands arranged consecutively. For example, Figure 5-1 shows the arrangement of data points for an image line of 1-byte data containing three spectral bands.
Figure 5-1. An image line may contain multispectral data stored in bands arranged consecutively.
|
For more information about manipulating data at the byte level, see the section titled Conversion utilities in Chapter 4, McIDAS-X Utilities. |
Below is a code fragment showing the steps for reading an image object data block. Note that the mcaget call must precede a call to mcalin .
If processing the image requires the line prefix, the application must get the prefix using the mcapfx function. The line prefix is the set of information that may precede the data on an image line; its maximum size is 1000 bytes. As shown in Figure 5-2, an image line consists of an optional line prefix and the actual data values.
Figure 5-2. Each image line contains an optional line prefix and data values.
The line prefix is divided into four regions:
The line prefix is the same length for each line in an image. The sensor source determines the size and content of the line prefix and which of its four regions are present.
As shown in the sample code below, the call to mcapfx must occur immediately after the call to mcalin so that the prefix and data are for the same image line.
The mcanav function reads an image object's navigation block. At any point after the connection is opened by mcaget , the application may retrieve the navigation block using the handle returned by the preceding mcaget call. See the code segment below. To eliminate the risk of buffer overflow, dimension the nav_buffer to 4096 bytes.
The mcacal function reads the calibration block of an image object. The call to mcacal can occur any time after the connection is opened by the mcaget call. The handle returned by mcaget is passed to mcacal , which returns the associated calibration block. To eliminate the risk of buffer overflow, dimension the cal_buffer to 40,000 bytes.
To read the comment block, use mcacrd and the handle returned by mcaget . The mcacrd function returns the entire comment block to the application. The call to mcacrd can occur only after the calls to mcalin are done.
Below is an example using mcaget , mcalin and mcacrd to read a comment block. To eliminate the risk of buffer overflow, dimension the comment_buffer to 40,000 bytes.
To write an image object to an image dataset, the application must identify a dataset and position, and open a connection with the server. Use the API functions below to write image objects to a dataset.
|
Function |
Description |
|---|---|
|
opens a connection to write the directory, navigation and calibration blocks of an image object |
|
The request to open a connection is performed by the mcaput function, which requires the following:
Because mcaput does not return an object handle, only one image object is written at a time.
The only valid selection clause for writing image objects is POS, which defines the location of the image object in the dataset. You must specify this clause or the request to open a connection will fail.
Once the connection is open, the server expects to transfer the number of bytes defined by the entries in the directory block. Transferring too few or too many bytes results in an error. All data block write transactions are performed by mcaout , which is called as many times as necessary to transfer the bytes. The mcaout function has only one argument, which is an array of data points to write to the data block. The call to mcaput must occur prior to mcaout .
The comment block is written after the last byte of the data block is transferred. The number of comment entries is defined in word 64 of the directory block. If this entry is nonzero, the specified number of 80-byte entries is transferred. The mcacou function transfers the comment block. It has only one argument, which is an array holding the entire comment block.
The sample code fragment below writes image objects to a dataset. For another mcaput code example, see the function imgcopy.f
c --- initialize the directory block call zeros(directory_block, 64) c --- create a comment card call getday( day ) call gettim( time ) cday = cfu( day ) ctime = cfu( time ) comment = cday(1:5)//' '//ctime(1:6)//' This is a comment ' ncard = ( len_trim(comment) / 80 ) + 1 c --- fill the essential directory block entries directory_block(2) = 4 ! version directory_block(3) = sss ! satellite number directory_block(4) = jday ! Julian day of image directory_block(5) = time ! nominal start time of image directory_block(6) = start_line ! starting image line number directory_block(7) = start_elem ! starting image element number directory_block(9) = num_lines ! number of lines of image data directory_block(10)= num_elems ! number of data points/line directory_block(11)= num_bytes ! number of bytes/data element directory_block(12)= line_res ! line resolution directory_block(13)= elem_res ! element resolution directory_block(14)= num_bands ! number of bands directory_block(19)= 2**(band-1) ! band map call movcw(memo, directory_block(25)) ! memo field directory_block(34)= data_offset ! byte offset to the data block directory_block(35)= nav_offset ! byte offset to the nav block directory_block(49)= doc_length ! length of prefix doc section directory_block(50)= cal_length ! length of prefix cal section directory_block(51)= lev_length ! length of prefix lev section directory_block(52)= lit( stype ) ! sensor source type directory_block(53)= lit( ctype ) ! calibration type directory_block(63)= cal_offset ! byte offset to the cal block directory_block(64)= ncard ! number of comment cards c --- initialize the navigation block call zeros(nav_block, nav_size) c --- fill the navigation block entries c Note: "navigation_params" is an array of navigation parameters c that describes the geo-location of the elements of the c image object. do 10 i = 1,nav_size nav_block(i) = navigation_params(i) 10 continue c --- initialize the calibration block call zeros(cal_block, cal_size) c --- fill the calibration block entries c Note: "calibration_parms" is an array of calibration parameters c that transforms the data elements to physical units. do 20 i = 1,cal_size cal_block(i) = calibration_params(i) 20 continue c --- fill the selection array nselect = 1 selects(nselect) = 'POS '//cfu(position) c --- open a connection to write the image object if( mcaput( image, nselect, selects, directory_block, nav_block, & cal_block).ne.0 ) then call edest('Unable to initialize image ='//image,0) return endif c --- loop to write image lines to the image object do 100 line = 1,num_lines c --- pack the data array c "data_array" is a (num_lines) by (num_elems) array of data c elements each of which is (num_bytes) long. The elements c represent data for (band) from the sensor numbered (sss) c on (jday) at (time). "data_buffer" is a one-dimension array c sized to (num_elems) c Note: this assumes a 4 byte to 1 byte compression of the data. call pack( num_elems, data_array(line, 1), data_buffer) c --- write a line of data to the image object if( mcaout( data_buffer ).ne.0 ) then call edest('failed to write image line=',line) return endif 100 continue c --- write the comment block if( mcacou( comment ).ne.0 ) then call edest('failed to write comment block',0) return endif
You can delete image objects from a dataset using mcadel, as long as you have write permission. The mcadel function has one valid selection clause, SUBSET, which you must specify during the connection phase of the transaction or the server request will fail. The code fragment below deletes the image objects at locations pos1 through pos2 from the dataset.
McIDAS-X grids are typically composed of atmospheric and oceanographic data, which are produced by numerical models or derived from observational data using an objective analysis scheme. Grids may contain any data that can be represented in a two-dimensional matrix.
Because grid data and image data can both be represented in two-dimensional matrices, it's important to know how they're different. The attributes that distinguish them are listed in the table below.
|
Attribute |
Grid data |
Image data |
|---|---|---|
Grid data has five unique attributes: valid time, level, parameter, origin and navigation. Each is described below.
Grid data is often used to store output from numerical models that simulate the atmosphere and ocean. Because these models predict what the atmosphere or ocean will look like at some time in the future, grids must contain two different time attributes when representing model forecasts:
For example, if a model run on 17 January at 00 UTC generates forecast fields valid 36 hours later, the primary time attribute is 17 January at 00 UTC and the secondary time attribute is 18 January at 12 UTC. If the grid does not represent a model forecast, no secondary time attribute is needed.
Grids store information that represents data at some vertical location in the atmosphere. The level can be the constant height above the surface, such as 100 meters or 32000 feet; the constant pressure surface, such as 500 millibars; or some other meteorological surface, such as the level of the tropopause or isentropic surface.
The grid parameter is the data type the grid represents. Parameters can be any field that can be stored as a number, such as temperature, wind speed or dew point. Grids must also contain the stored units used in the grid.
A grid's origin refers to the process that created the grid. For example, if a grid is created by a forecast model, the origin is the name of the model, such as ECMWF or GFS. If the grid is created from another objective analysis process, that name can appear as the origin.
Navigation is the process of determining the latitude and longitude location of data points on a grid. This information is needed when colocating data points from a grid with another data type. For example, you can use this information to contour the gridded field on a satellite image displayed on the McIDAS-X Image Window.
McIDAS-X currently recognizes these grid projection formats:
The diagrams below show how a three-dimensional Earth is represented on a two-dimensional surface for the pseudo-Mercator, polar stereographic and Lambert conformal projection formats, and the orientation of the data points on these projections. A sixth grid projection format is also available, but since it has no navigation, the data points can't be converted to planetary coordinates.
|
For more information about grid projections and McIDAS-X navigation, see the section titled McIDAS navigation later in this chapter. |
Figure 5-3. Pseudo-Mercator projection.
Figure 5-4. Polar stereographic projection .
Figure 5-5. Lambert conformal secant cone (left) and tangent cone (right) projections
Gridded data is two-dimensional data representing an atmospheric or oceanic parameter along an evenly spaced matrix. For the matrix to be useful, ancillary information about the grid must also be known. This ancillary information, along with the gridded data, is collectively called a grid object . Grid objects in McIDAS-X contain two blocks of information.
The API functions and procedures for reading and writing grid objects are described below.
Most applications for reading grid objects will do one of the following:
The table below lists alphabetically the McIDAS-X library functions for performing these tasks.
These functions are described below along with sample code, following an explanation of the selection conditions for requesting grid objects.
Applications use selection clauses to restrict the information sent from the server to the client. For example, selection clauses can restrict the search based on grid time attributes, the parameter type and level, or the process that generated the grid. Below is a list of the valid grid selection clauses as sent to the server. Additional information for each follows.
|
Selection clause |
Description |
|---|---|
DAY --use this clause to identify a list of primary days that the grids represent. For model forecast grids, these values will be the day the model was initialized. Otherwise, DAY is the Julian day the data represents.
DRANGE --use this clause to identify a range of primary days that the grid represents. Enter the beginning and ending day numbers and the increment between days in the range. The increment default is one day.
FDAY --use this clause to identify the secondary day attribute for requested forecast grids. For example, if you want only the forecast fields valid for day 1997017, specify FDAY=1997017.
FHOUR --use this clause to identify a list of secondary forecast hours that the grids represent. For example if you want only the 12-, 24- and 48-hour forecasts from a model run, specify FHOUR=12 24 48.
FRANGE --use this clause to identify a range of forecast hours that the grids represent. Enter the beginning and ending forecast hours and the increment between hours in the range. The increment default is one hour.
FTIME --use this clause to identify the secondary time attribute for the grids requested. Use this field with the FDAY clause to isolate grids that are valid at a particular time. The format for the arguments is hhmmss . For example, to request all grids valid at 12 UTC on day 96017, specify FDAY=96017 FTIME=120000.
GRIB --use this clause to select grids with the GRIB values specified for the geographic region, the parameter, the model, or the level. This is a useful way to find grids when other sort conditions provide insufficient information to differentiate between grids.
GRID --use this clause to access grids based on their position in specific grid files. Since this is an artifact of previous McIDAS-X API functions, avoid using this clause if a practical alternative exists.
LEV --use this clause to identify a list (not a range) of levels in the atmosphere or ocean that this data represents. This field is typically filled with height in millibars or words such as SFC, MSL or TRO. To retrieve data for several levels, enumerate them individually.
NUM --use this clause to specify the maximum number of grids returned from the server. The default is one grid. To receive all grids matching the selection conditions, use NUM=ALL.
