Re: General Advice

Bill , et alia -

Thanks for all the good advice, I'm copying our  last interaction offlist
(because I replied to the wrong message) with Bill here just for the archives
and anyone else interested in the same things I'm pursuing.

I'm to my hilt of understanding at this point and have plenty of homework to do
now to check things out.

- Steve

> Hi Steve,
> > Bill and all the rest who responded on-list and privately:
> First, note that you sent your reply to me only, not to
> visad-list.
> > >VisAD is specifically designed for dealing with the issue of
> > >the great diversity of data. The VisAD data model can deal with
> > >virtually any numerical data (plus text fields), and integrates
> > >metadata about Units, CoordinateSystems, regular and irregular
> > >sampling topologies, missing data and error estimates into
> > >computational and visualization operations on data.
> > >
> > This is exactly why I'm interested.   However, one of my potential
> > detractors here is in a significant collaboration with the Kitware Crue
> > and is promoting VTK for exactly the same reasons.  Without suggesting a
> > pithy contest here, I am hoping you know enough about VTK to provide a
> > little contrast.   I seem to remember that VTK, in fact, has Java
> > bindings but is obviously not implemented there nor is likely to be
> > compatible with Java3D.   These people are C++/OpenGL developers to
> > start with.
> VTK is an excellent system - I asked Bill Lorensen to contribute
> a recent VisFiles column. VTK supports a greater variety of
> specialized topologies, but VisAD's IrregularSets do support
> arbitrary topologies. I don't think VTK integrates Units and
> CoordinateSystems in the way that VisAD does. And I don't think
> it supports distributed objects the way that VisAD does.
> > >VisAD includes classes for supporting general irregular
> > >topologies, in its Irregular2DSet and Irregular3DSet
> > >classes. These are not optimized for any partial regularities
> > >that may exist in topologies (e.g., nested grids), but will
> > >handle arbitrary topologies.
> > >
> > In my career I've worked primarily with 2D/3D grids for simulating
> > continuous physical systems.  I'm now in the somewhat different domain
> > of agent based, discrete-event modeling where at best continuous fields
> > are a contrivance or a boundary condition in some since.   I understand
> > that both Irregular Grids and Sets would provide a base level data
> > structure but I'm fishing for even more insight than that.
> >
> > I was hoping that someone on this list had already used an "irregular
> > grid" in what I'd characterize as "edge/node" centric rather than
> > "cell/node" centric as most continuous problems seem to.    If anyone
> > has experience or just some more insight into how the tools and
> > techniques of mesh/grid structures can be adapted in more than a mundane
> > way to network/graph problems, I'd be obliged to hear about them.
> VisAD MathTypes provide very flexible schemas for describing
> arbitrary grouping of and functional dependencies among data.
> Lots of applications customize display by manipulating data
> structures. This can especially be used to create interactive
> displays where users click and drag data depictions to
> interact with calculations.
> > >You are right to focus on performance as the main current
> > >problem with Java, Java3D and VisAD. But there are numerous
> > >efforts to address performance.
> > >
> > >1. Java3D is making progress to reduce memory use.
> > >
> > I'm assuming this is mostly avoidable by not using too many scene graph
> > objects and by using "shared" objects...  I haven't done more than read
> > the performance guidelines offered by members here... they were very
> > helpful but I guess I'll need to actually write some tests and such to
> > really understand how the Scene Graph becomes geometry and where it is
> > and isn't fully effecient.
> VisAD enables you to get down to the Java3D scene graph
> level by extending the DataRenderer class.
> > Can you address how effeciently (or not) the Irregular Grid and Set
> > objects in Visad translate into OpenGl?
> VisAD makes Java3D scene graphs. Its up to Java3D to
> translate that to OpenGL. The speed of VisAD creating
> scene graphs depends on exactly what is onvolved. It
> has a very fast iso-surface algorithm. On the other hand
> implicit coordinate transforms cna slow displays down.
> > >2. JDK 1.4 supports 64-bit addresses (to escape the 2 GB
> > >limit on program size).
> > >
> > And does anyone know of the equivalent of "memory mapping" from a file
> > in C?   I sometimes use this for outrageously large  data structures
> > which are neccesarrily addressed sequentially many times to derive
> > other, more compact data structures...  in C it is much more effecient
> > than "rereading" the file all the time and is near  the effeciency of
> > swapping best we can tell.
> This is being added in JDK 1.4. But for VisAD we rely
> on the next point, which is our version of this:
> > >3. VisAD supports transparent caching of files (applications
> > >see them as a data object, but data moves between disk and
> > >memory).
> > >
> > Like that... I'll look for this.  What is the Java idiom for doing this?
> We didn't follow any Java interface for this. We were largely
> building on our experience with a similar function in Vis5D.
> > >4. We are working on support in VisAD for multi-resolution
> > >displays (see visad/examples/Test71 for an example).
> > >
> > I'd expected to have  to have multiple correlated data structures which
> > I manually/explicitely keep synchronized.   I'll look at the example.
> >
> > >5. We are working on support in VisAD for visualzing large
> > >data objects distributed across the processors of a cluster.
> > >
> > This is where VTK shines for us right now... they've been parallelizing
> > it pretty effectively under a big contract with us (ASCI).   This
> > project is going to be very "cluster" oriented I expect.  This addresses
> > data object management/processing, right?  But does it address parallel
> > rendering?  Does anyone have any insight around paralell rendering in
> > Java3D?   We run various powerwalls and a cave here, some of which are
> > driven by multiple boxes instead of multiple pipes/channels from one box.
> VisAD supports distributed transform of data into Java3D
> scene graphs. You'd have to ask the Java3D folks about
> their plans for using the Chromium disatributed OpenGL.
> > >6. VisAD includes custom DataRenderers, and special case
> > >logic in default DataRenderers, for more efficient display
> > >of common data structures.
> > >
> > I'll look into it.
> >
> > >
> > >7. IBM and others are developing Java implementations that
> > >run just as fast as C and C++.
> > >
> > I'm confident this will come, in the meantime we just have to swat away
> > the naysayers who build really bad counterexamples to flog us with...
> > "see how stupid I can use Java and then blame Java for being lame?".  I
> > lived through this in the early eighties around Assembly vs C and then
> > again in the 90's with C vs C++ so I know how they turn out in the long
> > run... unfortunately few of the detractors have that good or long of a
> > memory.  Or perhaps mine is just selective.   Whatever happened to
> > Objective C and NeWS an NeXT anyway <grin>?
> Yeah, what us old guys know is that you have to be able
> to pick the winners out of all the cool new things. I
> think Java is a winner (and thought so in January 1996).
> > >I think an infrastructure for solving the problem of mutliple
> > >special incompatible applications justifies the effort of
> > >solving these performance issues.
> > >
> > >
> > I agree and thanks for all you past, present and future hard work and
> > good advice...
> Good luck,
> Bill