This chapter describes the procedures that you will use to build data servers for McIDAS-X ADDE applications. You'll learn:
This chapter is organized into the following sections:
ADDE (Abstract Data Distribution Environment) is the client/server mechanism for distributing data in McIDAS-X. In ADDE, an application sends a request for data through an API routine to a server. The server is the software running on a machine in a distributed system that stores data and supplies it to the client upon request.
When the application reads the data sent from the server, the physical location of the original data and its stored format are transparent.
McIDAS-X has two server categories: primary and secondary.
In short, a user enters a data request based on the ADDE group and descriptor name. mcserv starts the appropriate primary server based on the request type. The primary server extracts information about the requested dataset from the server mapping table, including the stored format of the data. If the stored data format is different from the standard McIDAS-X stored formats, the primary server starts a secondary server to convert the data to the format the client expects.
Before writing an ADDE server, you should understand some basic concepts about servers in McIDAS-X. This section describes the following:
The ADDE design is stream-oriented, so both the client and server can work simultaneously. An ADDE server reads a data request sent from the client via stdin (standard input) and sends data back to the client using stdout (standard output) through a pipe.
Although the size of the data sent from the server may be many megabytes, intermediate data storage on the server or client is not needed. Since the pipe is a finite size, the server will wait to write if the pipe is full. The client will wait for up to two minutes if the pipe is empty. If no activity takes place on the pipe after two minutes, the process stops. This is important for requests that take a long time to fulfill due to extensive searching. An environment variable, ADDETIMEOUT, can be set on the server to change this timeout length.
The rules below apply to all data transmissions between the server and client.
Primary servers, along with the client APIs, define the client selection syntax and the transmission format between the server and the client. The selection syntax and transmission format are different for the various data types in McIDAS-X. The table below lists the current primary servers provided in McIDAS-X.
|
Client APIs |
Client request type |
Data type |
Server name |
Description |
Secondary
|
|---|---|---|---|---|---|
|
retrieves the image header, navigation, calibration and data; data is returned line by line |
|||||
The ADDE communications module, mcserv , starts the primary servers based on the server's IP address, which tells mcserv if the request will be fulfilled locally or remotely. If the request is handled locally, mcserv finds the ADDE request type and runs the appropriate server process. The server process reads the body of the client request, acquires the data requested and sends the data back to the client.
If the request is handled remotely, mcserv opens a connection to the remote server and acts as a TCP-to-pipe bridge, sending out the request. On the server machine, inetd receives the connection and creates a child process running the same mcserv module with slightly different command arguments. Like local requests, this version reads the beginning of the client request, determines which server process is needed and runs it. The server processes the request and sends the data requested back to the client.
In ADDE, the client requesting data doesn't care about the file format of the stored data. It only cares that the data is delivered in a well-known, predefined format. For example, if you have gridded data stored in a flat ASCII file, the McIDAS-X GRDDISP command can contour that data as long as the server delivers the gridded data to the client in the appropriate format. To accomplish this, you need a secondary server.
The secondary server's job is to:
The example below shows the steps that a primary and secondary server will use to fulfill a data request. You should assume that the server administrator inserted the two commands below into the server mapping table:
DSSERVE ADD ETA/00 GRID 101 110 TYPE=GRID "00Z ETA Model Run DSSERVE ADD LOCAL/MODEL FLAT TYPE=GRID "Local Model Grids |
The first entry created the grid dataset ETA/00, which is stored in the standard McIDAS-X grid format in grid file numbers 101 to 110. The second entry created the grid dataset LOCAL/MODEL. The grids for this dataset aren't stored in the McIDAS-X grid format, but in a format called FLAT. If a user enters a GRDDISP command to display a grid from each of these datasets, the steps taken by the server to fulfill these requests are different.
If the user enters the GRDDISP command below, the steps that follow are performed once the appropriate machine is identified.
If the user enters the command below, the steps that follow are performed.
Whether you're creating primary or secondary ADDE servers, they must perform these seven steps:
Steps 1, 2, 3 and 7 are the same for all data types; steps 4, 5 and 6 are different among the data types. All seven steps are described in this section, along with the function calls required to perform them.
This section is organized into these parts:
The McIDAS-X library functions that you will use when creating an ADDE server are listed alphabetically in the table below.
Your server must be able to access McIDAS-X disk files, since most servers are built using disk file utilities. Use the initblok function to initialize the McIDAS-X disk file system so the server can recognize MCPATH and the redirection table, as shown below. Note that the parameter passed into initblok must be a 2-byte integer.
integer*2 init_stat : ok = initblok (init_stat) |
short init_stat; : ok = initblok_ (&init_stat); |
|
For more information about disk files, see the section titled McIDAS disk files in Chapter 5, Accessing Data . For more information about file redirection and MCPATH, see the section titled McIDAS user applications in Chapter 2, Learning the Basics . |
Each ADDE client request received by the server contains a 160-byte request header , which contains system-level information built for the server.
The table below shows the input components included in the request header. The third column contains the starting word locations if the server is written in Fortran; the fourth column contains the variable names for the C structure, servacct , if the server is written in C.
|
Request header components |
Length, in bytes |
Word number
|
servacct structure names in C |
|---|---|---|---|
The last 120 characters of the request header is a buffer that may hold the client request string. The request string contains the group and descriptor names, and any selection conditions the server may need to fulfill the client request. If the request string is too long to fit in this buffer, the request string variable will contain a value of zero in bytes 4-7; bytes 0-3 will contain the actual length of the request string, which the server then must read from stdin .
The process for reading the ADDE client request is performed with one of two functions, depending on whether the server reading the request is a primary or secondary server.
Words 41 to 64 are for sending a return packet. The section Ending the transaction in this chapter provides more information.
To read the client request from a primary server, call M0sxGetClientRequest or m0sxgetclientrequest , as shown in the code fragments below.
include 'fileparm.inc' character*(MAXPATHLENGTH) request integer request_block(64) : : ok = m0sxgetclientrequest(request_block, request) |
#include "mcidas.h" : servacct request_block; char *request; int ok; ok = M0sxGetClientRequest (&request_block, &request); |
Upon successful completion, request_block contains the request header information sent to the server, and request contains the request string, including the group and descriptor names and selection conditions specified.
To read the client request from a secondary server, call the function M0InitLocalServer, as shown in the code fragment below.
const char Server[] = {"TESTSERV"}; servacct request_block; char *request; : : ok = M0InitLocalServer (Server, &request_block, &request); |
Upon successful completion, request_block contains the request header information sent to the server, and request contains the request string, including the group and descriptor names and selection conditions specified.
If the dataset requested by the client is stored in a non-standard McIDAS-X format on the server, the primary server starts the secondary server with a call to m0subserv . m0subserv builds an argument list for the secondary server and starts it with a call to execlp . The argument list contains both information from the client request block and the actual client request string. The M0InitLocalServer function extracts values from the secondary server's argument list and puts them in the client request block.
Once the server successfully receives the client's request, it must extract information from the server mapping table, which is a database that provides the server with information needed to fulfill a user request. The server mapping table is manipulated by the DSSERVE command and contains information about the location and format of the data on the server machine. This information is accessed by the server based on the group and descriptor names included in the client request.
The function M0sxdatasetinfo or m0sxdatasetinfo extracts information that the server needs from the server mapping table, including:
The code fragment below demonstrates the use of M0sxdatasetinfo . It assumes that the server administrator entered the following DSSERVE command, which created the ADDE dataset GMS/IR:
Upon successful completion, the output variables will contain the following values:
|
Variable |
Value |
|---|---|
After the server reads the client request and extracts the dataset information from the server mapping table, it parses the client request so the data can be located and the request fulfilled.
Since client requests are similar in form to a McIDAS-X command, it is easiest to use the McIDAS-X command line parsing routines to extract information from the request. To use common command line data retrievers, such as Mccmdstr and Mccmdint , you must initialize the command line subsystem. You only need to do this for primary servers. The M0InitLocalServer function automatically performs these steps for secondary servers.
The code fragments below demonstrate how to initialize the command line subsystem in both Fortran and C.
character*256 request integer len_req : c--- request already contains the request string to be parsed call m0cmdput (m0cmdparse (request, len_req)) |
char *request; int len_req; int stat; : /* request already contains the request string to be parsed */ stat = M0cmdput (M0cmdparse (request, &len_req)) ; |
Once the request string is parsed by the command line subsystem, it is your job, as the server developer, to find the data matching the request. The section titled Request syntax and data transmission formats later in this chapter explains the request syntax for each of the McIDAS-X data types.
The method of retrieving data matching the client request will vary from file format to file format.
When determining the location of data files, you should be aware that the server mapping table may not have enough entries available for an individual dataset to fully explain the filing format. If this happens, use a configuration file and store the name of that file in the INFO field of the server mapping table. Then the server can get additional file location information there.
When the request is parsed and the appropriate data is located, it can be sent back to the client. All data transactions are done in pairs (count + data) using the function M0sxsend or m0sxsend. First, a 4-byte value containing the length of the data being sent to the client is transmitted. Then that many bytes of data are transmitted.
To ensure that data sent between the server and the client is in network-byte-order (big-endian), include calls to M0swbyt4 or swbyt4 for all integer values. This function flips 4-byte memory segments on little-endian machines; it has no effect on big-endian machines.
The first value sent from the server to the client is read internally by the client function M0cxreq or m0cxreq before any API-level reading routines are called. For example, when serving image data, this value is the total number of bytes the server will send to completely transmit the data object. Some ADDE clients expect a dummy, nonzero value in this location. If a zero is sent back, M0cxreq or m0cxreq assumes the server was unable to fulfill the request. The complete transmission format for each McIDAS-X data type is described later in this chapter in the section titled Request syntax and data transmission formats.
The code fragment below demonstrates the appropriate way to send data from the server to the client. It assumes the client wants to receive a 4-byte dummy value of one, which will be read by m0cxreq, followed by two 40-byte records. Words 2 through 4 in the record are character values; the remainder of the record contains integer values.
After the data is transmitted to the client, the server ends the transaction by calling M0sxdone or m0sxdone . This function sends a 96-byte trailer, which is filled on the server and sent to the client at the end of the transaction. Its contents are described in the table below.
|
Trailer description |
Length,
|
Word number
|
servacct structure names in C |
|---|---|---|---|
Before the transaction ends, you must set the return status code and any error messages that may have voided the transaction. The only successful return code is zero, which appears in word 43 of the client request block. If you send a nonzero value to the client, also put an error string in words 44-61 so the client API will print a message telling the user why the request couldn't be fulfilled. If your site uses the ADDE accounting software, also fill word 41 with the total number of bytes transmitted to the client.
The code fragment below shows the calling sequence that a server should include for a request that can't be fulfilled.
Using this example, the message below is printed by the function m0cxreq if the server encounters a syntax error.
COMMAND: Syntax error in the user request -1001 |
Because ADDE servers read from stdin and write to stdout , you can't call the sdest or Mcprintf function to trace the progress of a server for two reasons:
The solution is to dump messages needed for tracing errors to a file. If errors are reported and tracing is activated, the function M0sxtrce or m0sxtrce will write trace messages to a file named trce . If tracing is not activated, M0sxtrce or m0sxtrce does nothing.
The keyword TRACE is appended to each client request. If TRACE=0, which is the default, tracing is not performed. If TRACE=nonzero value, tracing is activated. The function M0IsTraceSet or m0istraceset automatically activates tracing on the server if the value for TRACE in the client request string is a nonzero number. If you write secondary servers, a call to M0IsTraceSet is made within the function M0InitLocalServer . To activate tracing within a server on your own, call M0sxSetTraceOn or m0sxsettraceon . To turn it off, call M0sxSetTraceOff or m0sxsettraceoff .
Since each account has only one trace file, prefix each line of your tracing strings with the name of the server generating that string. When you look through the contents of the trace file, you can easily find the trace information generated by the problem server.
You can also use the Mctrace or mctrace function to automatically append the server name to the trace message and set a flag for each message. The flag restricts tracing to only selected values of the keyword TRACE.
The sample code fragment below is from the server TOMGSERV.
If the user enters the following client request:
Upon completion, the file trace will contain this line:
TOMGSERV: GMS AUST TIME=12 DAY=1996352 ID=YSSY TRACE=1 |
This section is organized into the following topics:
In ADDE, the client will manipulate data regardless of the format in which it is stored on the server. For this to happen, however, you must follow a specified set of rules for processing data and delivering it to the client. This section describes the client request syntax and data transmission formats for each data type supported in McIDAS-X. The table below provides a summary of the necessary components for each request type, in alphabetical order.
|
Request type |
Data type |
Description |
Client requester |
Client reader |
Sample McIDAS-X command |
|---|---|---|---|---|---|
|
image header, navigation, calibration and data; data is returned line by line; comments |
|||||
All secondary servers must perform the four basic tasks below to deliver the appropriate data to the client.
Since ADDE client requests are sent as character strings with a format similar to McIDAS-X commands, it is easiest to use the McIDAS-X command line retrieving routines to extract information from the client request. The request syntax description for each data type described in this section uses the McIDAS-X command line notation.
Additionally, remember that transactions of integer values between the client and the server are performed in network-byte-order (big-endian). Unless otherwise noted, assume that all strings sent back to the client are blank padded.
The ADDE server, agetserv , processes a client request and returns a complete image object to the client. An image object includes an image header and the image lines, and may also contain a line prefix for each line, calibration, navigation and comment cards. Due to the volume of data associated with image objects, the server delivers the data portion of the object to the client one line at a time. The McIDAS-X command IMGDISP accesses image objects.
The table below lists the client request syntax options for an image object. Note that keywords 1 through 9 are placed in the request block as positional parameters.
|
Server-specific keywords |
Description |
|---|---|
|
Coverage; accesses data for specified time range (realtime POES server) |
|
You must adhere to the rules below when transmitting an image object. The first five are specific to the client request; the last three are specific to the transmission format sent back to the client.
The image object contains the image header and the image lines. It may also contain line prefixes, navigation, calibration and comment cards. Each of these components is created by the server. Their formats are described in Chapter 6 of this manual in the AREAnnnn data file documentation. Below are the common modifications needed for the image header.
Once the data is formatted correctly, the image object can be sent to the client. Because the transmission protocol is count+data, the server will send the following information to the client:
The sample code below shows you how to set up your server to transmit image objects.
integer image_header(64) integer nav_block(1024) integer cal_block(1024) integer line(2000) integer prefix(100) integer comment(20) integer total_bytes integer nav_size integer cal_size integer pre_size integer n_comments integer n_lines integer n_elements integer data_size integer n_send integer n_send_nbo c--- assume the image_header, nav_block, cal_block have already been loaded n_lines = image_header(9) n_elements = image_header(10) data_size = image_header(11) nav_size = image_header(63) - image_header(35) cal_size = image_header(34) - image_header(63) pre_size = image_header(15) n_comments = image_header(64) total_bytes= (n_lines * n_elements * data_size) + & (n_rows * pre_size) + & (n_comments * 80) & (nav_size + cal_size) c--- send the total number of bytes in the image object n_send_nbo = total_bytes call swbyt4(n_send_nbo, 1) call m0sxsend(4, n_send_nbo) c--- send the image header; assume imageheadernbo converts the header c--- to network-byte-order call imageheadernbo(image_header) call m0sxsend(256, image_header) c--- send the nav block; assume navnbo converts the nav block to c--- network-byte order if (nav_size .gt. 0)then call navnbo(nav_block) call m0sxsend(nav_size, nav_block) endif c--- send the cal block; assume calnbo converts the cal block to c--- network-byte order if (cal_size .gt. 0)then call calnbo(cal_block) call m0sxsend(cal_size, cal_block) endif c--- now we will send the image data lines, one line at a time; c--- assume getoneline retrieves one line of data as packed bytes do 10 i = 1 , n_lines ok = getoneline(i, line, prefix) if (pre_size .gt. 0)then call m0sxsend(pre_size, prefix) endif call m0sxsend(n_elements, line) 10 continue c--- send the comments; assume readcomment reads one comment card if (n_comment .gt. 0)then do 20 i = 1 , n_comments call readcomment(i, comment) call m0sxsend(80, comment) 20 continue endif
The ADDE server, adirserv , processes a client request and returns image directories and comment cards to the client. The McIDAS-X command IMGLIST accesses image directories.
