[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [thredds] NetCDF Subset Service - allow only 'as point dataset'


Steve Ansari wrote:
> Hi John,
> 
> Yes - we are running into memory issues when multiple people hit the
> grid subsetting for some large NetCDF files: 
> 
> Check out
> http://www.ncdc.noaa.gov/thredds/catalog/radar/StIV/catalog.html .  The
> *TEST.nc files are 'physical' aggregations that have great performance
> as 'Point Datasets'.  The issue is when someone tries to download a
> large part (or all) of the gridded data because the all of the data is
> read into memory before writing.  The main user right now is David
> Maidment / CUAHSI (our test group for these datasets).  If it is not
> possible to configure this setting, we may employ a filter to redirect
> these requests to a friendly 'operation not supported' page.

can you look in the logs and send me a sample of the requests he is making, esp 
the big ones?
> 
> We can't wait for 4.0!  We are hoping the streaming netcdf output helps
> for this scenario.  If you need any early testers please let us know.

ok, thanks

> 
> With regards to the IOSPs we discussed at GO-ESSP, we are working on
> getting a standard test environment together so we can evaluate the
> performance for some standard use cases.  After we get that worked out a
> little, we should have something good to pass your way.

great

> 
> Another unrelated question:  We have an simple IOSP that decodes lat/lon
> gridded data (actually RIDGE gifs).  This is mainly a proof of concept /
> easy first IOSP for a developer.  What is needed to have this recognized
> as a GridDatatype, so we can use the subset service?  We can get OPeNDAP
> to work, but that is not at the CDM level.  Do we need to add a Dataset
> impl?  I've always been a little confused by this...  The data is online
> at: 
> http://www.ncdc.noaa.gov/thredds/catalog/radar/scratch/sansari/catalog.html?dataset=radar/scratch/sansari/RidgeMosaic_Conus_20080723_0038_N0Ronly.gif

are these really gif files?

anyway, lat/lon is pretty damn easy. just add some CF metadata, which it looks 
like you already have. then a "CoordSysBuilder" object adds the coordinate 
systems, and then they are grids. For difficult cases, one has to write a 
CoordSysBuilder subclass, but that shouldnt be needed here. also can see

http://www.unidata.ucar.edu/software/netcdf-java/tutorial/

especially

http://www.unidata.ucar.edu/software/netcdf-java/tutorial/CoordSysBuilder.html

i looked at the file, you should change the rgb attributes in the following way:

  <variable name="rgb">
    <attribute name="coordinates" value="time lat lon" />    
    <attribute name="units" value="" />
    <remove name="_CoordinateAxes" type="attribute" />
    <remove name="_CoordinateSystems" type="attribute" />
  </variable>

with these changes, it works fine as a CF grid.

you also dont have strides implemented correctly yet, but you may already know 
that.

im seeing another bug in the dods server that i will investigate.

----------

BTW, i thing we talked briefly about if NCDC was interested in hosting the NSSL 
radar mosaic dataset. We want to convert to netcdf4 and put on a TDS, but 
neither Unidata or NSSL has compute/server power at the moment.

see: http://www.nssl.noaa.gov/projects/q2/

im pretty sure they are keen on someone doing this, but i might need to ping 
them about it if you guys are in fact interested.