VisAD / Java Design Bill Hibbard 12 May 97 The three primary design activities are: A. Integrating metadata into the data model. B. Preparing to exploit JDK 1.1, including RMI for distributed computing. C. Designing the display architecture. Here are these three activities in greater detail: A. Integrating metadata into the data model. Metadata integrated into the data model include: 1. Mathematical types (MathType) for all Data objects, including names for real values. Every real value in a Data object has an application-specific name such as 'Latitude'. 2. Unit-s for real values. Every real value in a Data object may have a Unit (from Steve Emerson's Unit package). 3. ErrorEstimate-s for real values. Every real value in a Data object may have a variance of expected error, or variances may be averages for columns in a FlatField. 4. CoordinateSystem-s for tuples (including 1-tuples) of real values. For example, the mathematical transform between (Latitude, Longitude, Altitude) and (Latitude, Longitude, Pressure). 5. Set-s (finite samplings) for tuples (including 1-tuples) of real values. For example, satellite images are finite samplings (perhaps in the (Latitude, Longitude) CoordinateSystem) of continuous radiance functions. Set-s are full-fledged Data objects in their own right but are also metadata for Field objects. 6. Audit objects that document the sequences of operations that create Data objects. Metadata integration means that metadata are built into each Data object and metadata operations are implicit in Data operations. Interesting issues include: 1. Interactions between different kinds of metadata are very complex. 2. The need to maintain computational efficiency, particularly for Data objects that do not include optional metadata. 3. MathType-s define default metadata for Data objects. The issue is when Data objects may override those default metadata and what consistency constraints are enforced with default metadata. For example, Data can override the default Unit-s, but the new Unit-s must be convertable with the default Unit-s. 4. The key issue for ErrorEstimate-s is finding the right balance between accuracy of estimates, computational efficiency, and the sanity of the programmer (i.e., me). To help find this balance, there are five different modes for ErrorEstimate-s. First, there may be no ErrorEstimate-s. If there are, they may be attached to each real value or may be averages over columns in FlatField-s, and they may be combined assuming independent or dependent error distributions. Assuming dependent error distributions is a worst case analysis and produces error growth similar to interval arithmetic. 5. Audit trails have not been implemented yet. The key issue is to make sure that Audit objects do not become larger than the Data objects they are attached to. B. Preparing to exploit JDK 1.1, including RMI for distributed computing. The new JDK 1.1 features addressed by this design are: 1. RMI (remote method invocation) which supports: a. invoking mehods of objects on other JVMs (Java virtual machines), which may be running on other physical machines. b. sending copies of objects to other machines (either as arguments to remote methods, or return values from remote methods). 2. The Serializable interface, which must be supported by any object copied to another machine. It is also required for copying objects to and from persistent storage. 3. Versioning for Serializable classes. This allows compatibility between new versions of classes and persistent storage of objects instantiated using old versions of those classes. 4. The blank final declaration to enforce the immutability of variables set in constructors. 5. JDK 1.1 includes improved UI support, but since VisAD defers UI to higher layers, this design does not address the UI improvements. Java is the first mainstream programming language that enables objects to migrate around the network (e.g., returned from remote method calls), executing methods wherever they go. This has lots of consequences that are being addressed in the VisAD design: 1. Static variables are generally useless unless they have the same value on all JVMs. For example, there is no way to enforce that all RealType objects have unique name String-s (in a single JVM this could be done with a static Vector of names), so the design does not depend on RealType objects having unique names. In general, equals() methods should take the consequences of migrating objects into account. 2. Object locking by synchronized methods should work, but any mechanism to lock objects across sequences of mathod calls will have serious problems (e.g., remote lock-holder dies). As one speaker at JavaOne 97 said, immutable objects are a great way to avoid the need to lock objects. This is especially true in a distributed computing environment. Thus most VisAD objects have been made immutable. Special cases are: a. Field-s have immutable MathType-s, domain Set-s, range Unit-s and and range CoordinateSystem-s. However, their range values and range ErrorEstimate-s may change. If two threads make sequences of method calls on the same Field, they may create inconsistent values but will not create inconsistent structure that could cause an Exception. b. DataReference objects have flexible (i.e., mutable) bindings to Data objects. [Note - DataReference objects play the role of variable names in a programming language, and are necessary for implementing computational UIs on top of VisAD. They are also necessary for linking Display-s to Data objects]. c. Display objects have many mutable variables, in order to support interactive control of data display, and all its methods are synchronized. The development of collaborative UIs will require careful attention to the issue of simultaneous access to Display objects by multiple threads. The primary issue for exploiting JDK 1.1 is adapting the VisAD class hierarchy to RMI. In order to do this, the basic Data class hierarchy is replicated three times: as interfaces, as local implementations, and as remote implementations (which are adapters for local implementations). For convenience, the interface and local implementation hierarchies are collapsed together for Tuple, Set and Scalar. Furthermore, the remote hierarchy is limited to a single class, RemoteFieldImpl, a remote adapter for local Field-s and FlatField-s. Other Data classes cannot be remote (but DataReference-s bound to them can be - see below). If we decide that other Data classes need to be remote, the basic design pattern of three parallel hierarchies can be elaborated to other Data classes. The interface hierarchy includes both Field (which is implemented by the local FieldImpl) and RemoteField (which extends Remote and is implemented by the remote RemoteFieldImpl). The rationale is to allow any Field reference to be either local or remote, but not to force access to local objects to go through the Remote interface. This pattern is also used for DataReference and Display. Note, special thought must be given to calls to hashCode, equals, toString and clone for remote implementations - these methods apply to the local stubs rather than the remote objects. In particular, the remote implementation hierarchy should not adapt these methods from the local implementation hierarchy. Here is how the Data class hierarchy is adapted to RMI: // interface hierarchy public interface Data extends Cloneable; public interface Function extends Data; public interface Field extends Function; public interface RemoteField extends Field, Remote; // local implementation hierarchy // (collapsed with interface hierarchy for Tuple, Set and Scalar) public class DataImpl implements Data; public class FieldImpl extends DataImple implements Field; public class FlatFieldImpl extends FieldImple; public class Tuple extends DataImpl; etc public class Set extends DataImpl; etc public class Scalar extends DataImple; etc // remote implementation hierarchy // (adapter for local implementation hierarchy) public class RemoteFieldImpl extends UnicastRemoteObject implements RemoteField; // adapter for FieldImpl and FlatFieldImpl VisAD also supports remote DataReference objects and remote Action and Display objects. DataReference plays the role of Observable and Action/Display plays the role of Observer. However, since Observable is a class rather than an interface, and RemoteDataReferenceImpl needs to extend UnicastRemoteObject, Observable and Observer cannot be used. In any case DataReference and Action/Display need to use slightly different semantics than Observable and Observer. Specifically, the link between Action/Display 'observers' and DataReference 'observables' may behave in one of two different ways, determined by arguments to the Action/Display constructors: 1. A DataReference 'observable' can notify an Action/Display 'observer' every time it changes. 2. An Action/Display 'observer' may periodically poll DataReference 'observables' to see if they have changed. The choice of these two modes of operation is made independently for local (i.e., DataReference and Action/Display on same JVM) and remote (i.e., DataReference and Action/Display on different JVMs) cases. The Action/Display constructor also defines the polling interval in ms. The goal of the polling mode is to avoid the overhead of remote notification every time a remote DataReference 'observable' changes. The RemoteDataReferenceImpl adapter for getData() decides whether to return Field-s by copy (FieldImpl, FlatFieldImpl) or remote reference (RemoteFieldImpl. All other Data classes are returned by copy. The Field interface defines a method local(): local Field-s return 'this' and remote Field-s return their AdaptedField (i.e., a local copy is returned). Thus the decision to replace remote access to a Field or FlatField object with local access (by sending a copy to the client) can be made by either the client or the server. Here is how the DataReference class is adapted to RMI: // interface hierarchy public interface DataReference public interface RemoteDataReference extends DataReference, Remote; // local implementation hierarchy public class DataReferenceImpl implements DataReference; // remote implementation hierarchy // (adapter for local implementation hierarchy) public class RemoteDataReferenceImpl extends UnicastRemoteObject implements RemoteDataReference; // adapter for DataReferenceImpl Here is how the Action and Display classes are adapted to RMI: // interface hierarchy public interface Action extends Runnable; public interface RemoteAction extends Action, Remote; public interface Display extends Action; public interface RemoteDisplay extends Display, Remote // local implementation hierarchy public class ActionImpl implements Action; public class DisplayImpl extends ActionImpl implements Display; // remote implementation hierarchy // (adapters for local implementation hierarchy) class RemoteActionImpl extends UnicastRemoteObject implements RemoteAction; // adapter for ActionImpl class RemoteDisplayImpl extends RemoteActionImpl implements RemoteDisplay; // adapter for DisplayImpl C. Designing the display architecture. The VisAD display architecture will use the Java3D API for basic graphics rendering. This will provide the greatest portability, not only to different platforms but to different graphics media. For example, the Java3D API is being designed to support immersive virtual reality as well as ordinary workstation screens. The Java3D API spec has not yet been published for public comment, and the publication date has been delayed three times (first it was Q4 96, then Q1 97, then May 97, now Q3 97). However, the Java3D API was demo'ed at JavaOne 97 so there is evidence that it exists. The delay in the Java3D API is delaying the VisAD display architecture. However, VisAD has difficult design goals for its display architecture which are taking time to meet. Here are the design goals quoted from the 12 Feb 97 VisAD design review: 1. Display data according to a set of mappings from RealType's (e.g., Latitude, Time, Pressure) to DisplayRealType's (e.g., xAxis, RGB, Animation). 2. Allow user extensions, to define new DisplayRealType's, new DisplayRealTuple's (and hence new display CoordinateSsystem's), and new display transformation and rendering algorithms. 3. Support direct manipulation, where users modify data objects by manipulating their graphical depictions. At this time, the best description of the VisAD display architecture is this snapshot the (very raw) working design document: DISPLAY ARCHITECTURE 1. Display class Runnable ordered list of DataReference objects each has a list of ConstantMap objects for Offset, default Color, etc list of ScalarMap objects ordered list of DataRenderer objects these transform Data objects into Renderable objects, must look at samplings in addition to types to determine feasibility adaptable level-of-detail defered to DataRenderer Renderable objects are Java3D scene graphs one DisplayRenderer object merges Renderable objects does color compositing marks merged scene graph as retained or compiled-retained scene graph compression? metadata visualization (spatial axes, skewt background, etc) DisplayRealType values determined by (in priority order): a. Data object RealType values, via ScalarMap-s b. ConstantMap-s in DataDisplayLink c. ConstantMap-s in Display d. DisplayRealType.DefaultValue runs 3-D UI components embedded in 3-D display 2. No direct manipulation instead, embed 3-D UI components embedded in display these may return complex Data objects (e.g., FlatField-s) use ScalarMap-s to determine MathType-s and values of return Data 3. Display events changes to Control-s may trigger re-transform or re-render delegation event model (JDK 1.1) 4. Metadata visualization 5. User extensibility render per Display (e.g., render metadata) render per DataReference DisplayRealType-s DisplayTupleType-s 6. Clipping 7. Spatial cells Set Cell type Dimension point 0 line 1 triangle 2 rectangle 2 tetrahedron 3 rectangular solid 3 a. If Field Domain is mapped to Spatial then may use Field sampling Cells for Spatial Field b. Contour, Flow and Shape create derived cells c. Otherwise, use points (i.e., 0-D Cells) Need Set method(s) to enumerate Cells, but not one-at-a-time. Rendering algorithms to distinguish different kinds of sets (e.g., GriddedSet, etc). 8. Color RealType to 1-D color component via ScalarMap.function RealType to 3-D color via ColorControl compositing multiple color layers ordered layers a. ordered by Zaxis and using Alpha compositing? b. DataRenderer-s create PixelArray2D & PixelArray3D arrays of color DisplayRenderer composites these by Data object order pan layers by per object XAxisOffset and YAxisOffset ConstantMap Fulker wants consistent color tables across multiple objects McIDAS supports independent color table per frame used for navigation landmark images emulate this on-the-fly via computation 9. Pan / scroll and Zoom (spatial) support very large images can PixelArray2D & PixelArray3D do this? links from UI embedded in Display to ProjectionControl scroll control via scroll bar zoom control via rubberband box re-center via mouse click auto-scale and auto-center ** FlatField-s and Field-s carry ranges and other statistics? useful for auto-scale and for Accuracy perhaps only computed by (recursive) Data method invoked when Data is added to Display or when FlatField.MissingFlag goes true -> false and FlatField (or a super-object????) is selected for Display automatic only if Data is selected for Display or if Accuracy is requested NEEDS MORE THOUGHT ** 10. Animation animation scroll and zoom divisible real line UI for animationi, link to AnimationControl auto-scale and -center some Data objects animate by scene graph events, some need re-transform 11. Pay attention to deadlock avoidance 12. How to combine (lat, lon)->pres with (lat, lon)->(alt->temp)? do it in computational model? 13. Make sure this list addresses each DisplayRealType RETHINKING THE ROLE OF Renderer-s 1. Renderer-s may have multiple Data object inputs can combine (lat, lon)->pres with (lat, lon)->(alt->temp) 2. User may specify binding of Renderer-s to DataReference-s 3. Direct manipulation UI components are special Renderer-s bound to DataReference-s