[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDV #IYO-548353]: Request for links to hurricane bundles



> Hi Tom and Yaun,
> 
> Just wanted to catch up on a few things:
> 
> OCEANS:
> 
> The Aggregate grids by time didn't link Paul Holland's grid to his model 
> output. IDV says 'AggregationExisiting: no coordinate variable for agg 
> dimension= Time
> 
> I've contacted Paul Holland about his Antarctic Ocean model netCDF files - I 
> think our computer is ultimately too underpowered to handle this (see next 
> item) and I've asked if he can create a smaller complete file or even make a 
> visualisation himself.
> 

I tried here to download these data, and I had to give up after a couple hours 
waiting, the ftp is to slow.
> 
> I downloaded a set of month averages (theta) from the ECCO2 server.
> 
> Ecco2.jpl.nasa.gov/data1/cube/cube92/lat_lon/quart_90S_90N/THETA_monthly.nc/
> 
> Each one is about 200 MB. I can load one it and it looks good, but a second 
> time step doesn't seem to load in. No error comes up, but I suspect the 
> computer can't handle rendering the volume of data. I've experimented with 
> adjusting the stride to less levels (every other fifth point) so that there's 
> less for the computer to deal with, but this doesn't even produce an image. I 
> should point out that I switched to a newer macbook pro with the i7 core 
> (still 4 GB ram).
> 

This dataset is more interesting to me. My IDV has 10G memory and no problem to 
deal with the volume display, such as isosurface.
If you see any problem with the memory, you can do the strike.

> I got the same aggregation error trying to link up the different months into 
> one time frame - is this because they're netCDF and IDV prefers doing this 
> with GRIB?
> 

I think this is the problem of the IDV, we probably check the time dimension 
variable in lower case, but this dataset using upper case for the time 
dimension variable, I attached a ncml file you can reference. If you want to 
load a series of file, you can modify this ncml file, it should be easy to 
understand and change its content, otherwise, let me know.

> Also I select to make a 3D isosurface (the idea being to pick out the dense 
> bottom water which is at about -1.5 degrees C and below) but it produces a 
> 'flat' surface, it's still undulating in 3D but doesn't have a thickness. 
> This is presumably because that's how the data works, but is there a way of 
> selecting an isosurface over a range, i.e -1.5 degrees C and below?
> 


You need to change the vertical scaling to see the 3D effect, the default scale 
is 16km, you can try to use 3000m, the change icon is in the viewpoint tool bar 
on the left side of the map view window.

> I'm currently getting access to another model that gives a years worth of 
> data at 80 MB, hopefully our Mac will be able to handle this.
> 
> DUST:
> 
> I'm really keen to try and make a better visualisation of this, any further 
> advice on how to adjust the scale to get more detail?
> 
I would suggest you to check a few sample bundles already available in the IDV:

Data Choosers > catalogs 

select or enter:  http://www.unidata.ucar.edu/georesources/idvcatalog.xml

select IDV catalog > Unidata's RAMADDA Server > IDV Community Resources > 
Bundles > NOAA CSI

and find a few examples, and double click on it, when the open bundle window 
popup, click ok, the IDV will bring up the display to play with.


Yuan



> Also, do you know what might be wrong to stop it making a nice isosurface, or 
> is just the nature of the data (very variable over a logarithmic scale!)
> 
> AIR CURRENTS
> 
> Thanks for pointing me towards NCDC, should hopefully be able to make some 
> nice yearly visualisations with their archived data (if the computer can 
> handle it!)
> 
> COMPUTER:
> 
> Any tips on improving performance? I've got a feeling the Mac isn't liking 
> the JAVA script (is IDV running on a JAVA background, I seen a JAVA error 
> come up a few times it's crashed?)
> 
> Would it be more stable on a PC?
> 
> Thanks,
> 
> Dom
> 
> 
> 
> 
> 
> 
> 
> -----Original Message-----
> From: Unidata IDV Support [mailto:address@hidden]
> Sent: 01 October 2011 17:30
> To: Rees-Roberts, Dominic
> Cc: address@hidden; Rees-Roberts, Dominic
> Subject: [IDV #IYO-548353]: Request for links to hurricane bundles
> 
> > Hi Dom,
> >
> > re:
> > > Thanks for the guide, the precipitable water, when displayed as described,
> > > looks great. I've been trying look at the various settings as I've gone
> > > along to pick up a few hints and tips.
> >
> > > Do you know off hand where this data comes from?
> >
> > The data is from the Global Forecast System (GFS) model.  The display I
> > outlined used the 0.5 degree (55 km) resolution.
> >
> > re:
> > > It would be great to get precipitable water with a longer time frame.
> >
> > One can get LOTS of this data at the U.S. National Climactic Data Center
> > (NCDC).  The process is straightforward:  register an email address
> > and then select the type of data desired, parameters in that data (subset
> > of the full set of parameters that are possible), and times.  The files
> > will be put on disk for you to FTP.
> >
> > re:
> > > I've also been busy trying to find and visualise various models and couple
> > > of issues/questions have popped up.
> > >
> > > There are four models I'm keen to try and display at the moment:
> > >
> > > 1) The Antarctic current model generated by Paul Holland at the British
> > >    Antarctic Survey, that I mentioned before
> > > 2) The Antarctic atmospheric model (AMPS) produced by Jordan Powers at 
> > > UCAR
> > > 3) Global dust output produced by Natalie Mahowald at Cornell
> > > 4) Global current model ECCO2 generated at JPL NASA
> >
> > OK.
> >
> > re:
> > > 1) I'm still waiting to find out from Paul what the original output was. 
> > > He
> > > did say that the 10 GB file didn't include grid points and sent through
> > > another file which he uses to map the data out (grid file). Below are the
> > > details for our FTP site, Paul's files are saved into the 'Holland' 
> > > folder.
> > > As you mentioned, downloading a 10 GB file probably isn't the best, the 
> > > grid
> > > file is about 1 GB.
> >
> > This is easily doable.
> >
> > re:
> > > It maybe that I just need to link the two files - they're both .nc?
> >
> > This should be possible using the "aggregate by time" feature.
> >
> > re:
> > > To upload/download files externally: 
> > > ftp://Earthfromspace5575:address@hidden
> > >
> > > FTP client details:
> > >
> > > username: xxxx
> > > password: xxxx
> > > URL: ftp.endemoluk.com
> >
> > Thanks.
> >
> > re:
> > > 2) Jordan has uploaded a bit of the AMPS output for me to look at, it's 
> > > only
> > > a portion (1.7 GB) of the total output that we'd be interested in and came
> > > as a .tar.gz file. Jordan has since taken the file off of his server, but
> > > it's in the Powers folder of our FTP if you fancy taking a look.
> > >
> > > After unzipping I end up with a list of .nc files for hourly time points
> > > which seem to load up fine. I was thinking of using isosurfaces set to a
> > > particular wind speed to highlight the polar jet, I think I've seen
> > > something similar done by one of the visualisation teams at UCAR. How do I
> > > load multiple time points into IDV and tell IDV to put them into the same
> > > timeline?
> >
> > The current output of AMPS and 4 days previous is made available online on
> > our motherlode.ucar.edu demonstration server:
> >
> > motherlode.ucar.edu
> >
> > RAMADDA Data Repository
> > http://motherlode.ucar.edu/repository
> >
> > Projects
> > http://motherlode.ucar.edu/repository/entry/show/RAMADDA/Projects?entryid=1c0cb8d1-5177-4028-b685-577cbbf44ef6
> >
> > Antarctic IDD
> > http://motherlode.ucar.edu/repository/entry/show/RAMADDA/Projects/Antarctic+IDD?entryid=2c55cb2c-82ff-4c1f-8215-1fdb14ae46da
> >
> > The model output is available in GRIB in the folders under AMPS Model 
> > Output.
> > This is available directly in the IDV through the Catalogs selection.
> >
> > re:
> > > 3) Natalie has shared her global dust model output. These are relatively
> > > small files and can be found here:
> > >
> > > http://www.geo.cornell.edu/eas/PeoplePlaces/Faculty/mahowald/fordominic/
> > >
> > > I'm particularly interested in DSTQ.00.nc as this contains data in 3
> > > dimensions. What I'd like to do is try and select an isosurface that shows
> > > how the dust rises and travels across the Atlantic (and other locations).
> > > The problem is that IDV doesn't seem to identify the Z component and gives
> > > it a null value. I can display each level individually though.
> >
> > We will take a look to see what my be missing from the file(s).
> >
> The dataset is 280m in size, I will check it out next week from office.
> 
> > re:
> > > Also, the dust range is fairly logarithmic, at the lower levels there are
> > > very small differences in dust concentration (e-12 to e-11) and when the
> > > default colour bar is used (in this range) you see nice eddies in the dust
> > > movement. However at higher levels the dust is in the e-8 range and the
> > > standard linear colour bar doesn't resolve the fine detail in the very low
> > > concentrations.
> >
> > Yuan will need to comment on this.
> >
> 
> The color table is linear, however, we should be able to pick up some details 
> in the dataset through applying the certain scaling factor.
> 
> > re:
> > > Failing being able to visualise this in 3D I was thinking of displaying 
> > > each
> > > level individually, which our CGI team can stack and fill in the gaps.
> > > Ideally the colour scale would be consistent through each level and be 
> > > able
> > > to show the fine detail throughout the e-12 to e-4 range.
> > >
> > > I'm guessing if isosurfaces are possible it's going to be limited to the
> > > higher dust concentrations and the detail would be lost anyway. Also, it
> > > might be a bit of a cheat showing values on the logarithmic scale, the 
> > > finer
> > > stuff you wouldn't see!
> > >
> > > On a side note, how does IDV decide on the default values for the colour
> > > scale?
> >
> > A default is assigned for each dataset, and this can be changed by the user.
> > I will let Yuan lead you through this.
> >
> 
> Basically, there is a default xml table to assign the default color table and 
> range. You can always make the change to this table, or select different 
> table after the image being created.
> 
> > re:
> > > And is there a way of browsing the values or a summary of the values
> > > in the data - like maximum and minimum values, mean distribution?
> >
> > Again, this is a question for Yuan.
> >
> The IDV do have a formula to find the max and min in the grid dataset, but 
> only in the time dimension if I remember correctly.
> 
> 
> 
> Yuan
> > re:
> > > 4) The ECCO2 directory can be found here:
> > > http://ecco2.jpl.nasa.gov/data1/cube/cube92/
> > >
> > > I need to double check with the PI which are the most relevant data sets 
> > > to
> > > work with (is THETA commonly temperature?). In any case the data are
> > > presented in individual time steps (like Jordan's AMPS output) and are 
> > > .data
> > > files. Are these recognised by IDV?
> > >
> > > Sorry - loads of questions! I'm going to hold off finding more data sets, 
> > > I
> > > think I'm better off getting to grips with these for the time being.
> >
> > I agree.  It is best to learn enough to be able to use the IDV effectively
> > before drowning oneself in data.
> >
> > re:
> > > The Mac is struggling a bit, when I try to do a 'flythrough' it freezes 
> > > and
> > > the various displays start going wrong. Even if Paul's 10 GB current file
> > > could be loaded I suspect it would crash. I guess it's a case of finding
> > > more RAM (8GB?)
> >
> > Since everything in the IDV is inherently 3D, one needs to have a LOT of 
> > memory
> > if one is trying to use large datasets.  Our development machines here in
> > the UPC typically have 24 GB or more of RAM and we don't find that to be
> > too much :-)
> >
> > re:
> > > Many thanks,
> >
> > No worries.
> > Cheers,
> >
> > Tom
> > --
> > ****************************************************************************
> > Unidata User Support                                    UCAR Unidata Program
> > (303) 497-8642                                                 P.O. Box 3000
> > address@hidden                                   Boulder, CO 80307
> > ----------------------------------------------------------------------------
> > Unidata HomePage                       http://www.unidata.ucar.edu
> > ****************************************************************************
> >
> 
> 
> Ticket Details
> ===================
> Ticket ID: IYO-548353
> Department: Support IDV
> Priority: Normal
> Status: Open
> 
> 
> 
> 


Ticket Details
===================
Ticket ID: IYO-548353
Department: Support IDV
Priority: Normal
Status: Open