Re: [thredds] Best way to handle missing grids in an aggregation.

Hi John,

I inserted some responsed below.

John Caron wrote:
Hi Ken:

Ken Tanaka wrote:
Hi,

I'm aggregating some gridded data with an ncml entry in our THREDDS catalog.xml file. The entry looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<catalog name="NGDC THREDDS Server Default Catalog"
xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0";
       xmlns:xlink="http://www.w3.org/1999/xlink";
       version="1.0.2">

 <service name="aggServices" serviceType="compound" base="">
<service name="dapService2" serviceType="OpenDAP" base="/thredds/dodsC/" />
   <service name="wcsService2" serviceType="WCS" base="/thredds/wcs/" />
   <service name="wms2" serviceType="WMS" base="/thredds/wms/" />
 </service>

 <dataset name="USTEC STEC Aggregation" ID="STEC-Agg"
          urlPath="stec-aggregation">
   <metadata inherited="true">
     <serviceName>aggServices</serviceName>
     <dataType>Grid</dataType>
   </metadata>

<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
     <attribute name="Comment" value="Aggregation for USTEC STEC" />
<aggregation dimName="Time" type="joinExisting" recheckEvery="15 min">
       <scan location="/nfs/ustec_netcdf/stec/latest7days"
             suffix=".nc" />
     </aggregation>
   </netcdf>

 </dataset>
</catalog>

Inside the netCDF files are grids from satellites:
time 1:  satellite_01, satellite_02, satellite_03
time 2:  satellite_01, satellite_02, satellite_03
time 3:  satellite_01, satellite_03
time 4:  satellite_03, satellite_02, satellite_01
and so on.

Sometimes data from a satellite is not present, as satellite_02 is missing at time 3 at in the example above. This appears as an all white image in the time sequence when viewing the satellite_02 result with IDV. What is the best way to handle this. Is there a missing grid flag, or should there be an empty image inserted, so that there is always a satellite_02 grid?
Can you try setting timeUnitsChange = "true" on the aggregation element and see if the problem persists? thanks.
I made this change, so the new line looks like this:

<aggregation dimName="Time" timeUnitsChange="true" type="joinExisting" recheckEvery="15 min">

I then saved the catalog.xml and used the "Reinitialize" link on the thredds/admin/debug administration page, but the still get multiple errors from IDV as shown below (in a previous message).

We are also getting an error from IDV

java.io.IOException: Inconsistent array length read: 1165128303 != 1914731274
  at ucar.nc2.dt.grid.GeoGrid.readDataSlice(GeoGrid.java:616)
  at ucar.nc2.dt.grid.GeoGrid.readDataSlice(GeoGrid.java:524)
  at ucar.nc2.dt.grid.GeoGrid.readVolumeData(GeoGrid.java:445)
at ucar.visad.data.GeoGridFlatField.readData(GeoGridFlatField.java:220) at ucar.visad.data.CachedFlatField.getMyValues(CachedFlatField.java:479) at ucar.visad.data.CachedFlatField.getMyValues(CachedFlatField.java:467) at ucar.visad.data.CachedFlatField.unpackFloats(CachedFlatField.java:614)
  at visad.FlatField.getFloats(FlatField.java:1567)
at visad.ShadowFunctionOrSetType.doTransform(ShadowFunctionOrSetType.java:1404) at visad.java3d.ShadowFunctionOrSetTypeJ3D.doTransform(ShadowFunctionOrSetTypeJ3D.java:195) at visad.java3d.ShadowFunctionOrSetTypeJ3D.recurseRange(ShadowFunctionOrSetTypeJ3D.java:1209) at visad.java3d.ShadowFunctionOrSetTypeJ3D$1.run(ShadowFunctionOrSetTypeJ3D.java:181)
  at visad.util.ThreadManager$1.run(ThreadManager.java:292)
  at visad.util.ThreadPool$ThreadMinnow.run(ThreadPool.java:98)
this is the error on the client, the only thing it tells you is that there was an error on the server. Go to the sever logs and see what error is there.
There are errors in threddsServlet.log like this one:
=================================
2009-10-16T11:02:12.451 -0600 [ 252504351][ 2269] INFO - thredds.server.opendap.OpendapServlet - doGet(): Remote host: 140.172.184.245 - Request: "GET /thredds/dodsC/stec-aggregation.dods?satellite_25%5B3:1:3%5D%5B0:1:50%5D%5B0:1:100%5D HTTP/1.0" 2009-10-16T11:02:12.493 -0600 [ 252504393][ 2269] ERROR - thredds.server.opendap.OpendapServlet - path= /stec-aggregation.dods
java.lang.NullPointerException
at ucar.nc2.ncml.AggregationOuterDimension$DatasetOuterDimension.read(AggregationOuterDimension.java:785) at ucar.nc2.ncml.AggregationOuterDimension.read(AggregationOuterDimension.java:427)
   at ucar.nc2.dataset.VariableDS._read(VariableDS.java:498)
   at ucar.nc2.Variable.read(Variable.java:778)
   at ucar.nc2.Variable.read(Variable.java:752)
   at thredds.server.opendap.NcSDArray.read(NcSDArray.java:114)
   at thredds.server.opendap.NcSDGrid.read(NcSDGrid.java:72)
   at opendap.dap.Server.SDGrid.serialize(SDGrid.java:516)
   at opendap.dap.Server.CEEvaluator.send(CEEvaluator.java:275)
at thredds.server.opendap.OpendapServlet.doGetDAP2Data(OpendapServlet.java:530)
   at thredds.server.opendap.OpendapServlet.doGet(OpendapServlet.java:220)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
   at thredds.servlet.filter.CookieFilter.doFilter(CookieFilter.java:54)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at thredds.servlet.filter.RequestQueryFilter.doFilter(RequestQueryFilter.java:121) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at thredds.servlet.filter.RequestPathFilter.doFilter(RequestPathFilter.java:105) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
   at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
   at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
   at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704) at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685)
   at java.lang.Thread.run(Thread.java:619)
2009-10-16T11:02:12.493 -0600 [ 252504393][ 2269] INFO - thredds.server.opendap.OpendapServlet - Request Completed - 500 - -1 - 42
=================================


Do you think this error is due to the missing satellite_02 grids?

One more question: Also, as can be seen at time 4, the satellite grids are not always in the same order in the netCDF file. Is this a problem? I can have my colleague regenerate the data with grids in a consistent order if needed.
Are the satellite_03, satellite_02, satellite_01 variable name? if so, order doesnt matter.
These are variable names containing grid data. It looked like the order didn't matter--thanks for confirming this.

If I have answers to the above, I can probably figure out the following on my own. If all satellites must be present, maybe we need to start a new aggregation if a new satellite is added to the set or one is retired from service? Maybe we should be splitting out each satellite into their own netCDF files?

Thanks,
Ken

_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/


--
= Enterprise Data Services Division ===============
| CIRES, National Geophysical Data Center / NOAA  |
| 303-497-6221                                    |
= Ken.Tanaka@xxxxxxxx =============================



  • 2009 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: