[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[THREDDS #FJC-813079]: truncated data download



Hi Fan:

Hard to debug this. It appears from TDS that the data was sent truncated from 
the opendap request.

It would be a good idea for us to make a single request for this data. However, 
this is single threaded so the hyrax server is only getting one request from us 
at a time. So it shouldnt be getting overwhelmed.

You should upgrade to 4.5.5 (or wait for a stable release soon for 4.6.1) and 
see if it fixes. Also I would report a possible bug to Hyrax.

John



> I attach threddsServlet.log here.
> 
> I think I had a glimpse of the error.  In the Hyrax bes.log, I can see a 
> series of requests from TDS like these:
> 
> [GMT Mon Mar 16 11:17:16 2015 id: 27928] 27928 from ip 127.0.0.1, port 53431 
> [set container in catalog values 
> catalogContainer,/ncml/aggregation/TRMM_3A25/TRMM_3A25_Aggregation.ncml;] 
> received
> [GMT Mon Mar 16 11:17:16 2015 id: 27928] 27928 from ip 127.0.0.1, port 53431 
> [define d1 in default as catalogContainer with 
> catalogContainer.constraint="bbPixNum1%5b0:1:0%5d%5b62:1:62%5d%5b13:1:13%5d";]
>  received
> [GMT Mon Mar 16 11:17:16 2015 id: 27928] 27928 from ip 127.0.0.1, port 53431 
> [get dods for d1 return as dap2;] received
> ...
> [GMT Mon Mar 16 11:17:52 2015 id: 20069] 20069 from ip 127.0.0.1, port 53415 
> [set container in catalog values 
> catalogContainer,/ncml/aggregation/TRMM_3A25/TRMM_3A25_Aggregation.ncml;] 
> received
> [GMT Mon Mar 16 11:17:52 2015 id: 20069] 20069 from ip 127.0.0.1, port 53415 
> [define d1 in default as catalogContainer with 
> catalogContainer.constraint="bbPixNum1%5b75:1:75%5d%5b62:1:62%5d%5b13:1:13%5d";]
>  received
> [GMT Mon Mar 16 11:17:52 2015 id: 20069] 20069 from ip 127.0.0.1, port 53415 
> [get dods for d1 return as dap2;] received
> 
> Since I was requesting a full time series, the time index of variable 
> 'bbPixNum1" increments from [0:1:0] up, in this case the request stopped at 
> [75:1:75], and what I saw in TDS was that it returned the truncated 76 lines 
> of data.  It is unclear to me from the log if TDS stopped requesting, or 
> Hyrax BES simply ignored any further requests from TDS.
> 
> Since sometimes the server can be overwhelmed, I wonder why it is not sent as 
> one single request? I tested in Hyrax such a request and the data was fully 
> returned:
> 
> [GMT Mon Mar 16 11:12:59 2015 id: 20069] 20069 from ip 127.0.0.1, port 53415 
> [set container in catalog values 
> catalogContainer,/ncml/aggregation/TRMM_3A25/TRMM_3A25_Aggregation.ncml;] 
> received
> [GMT Mon Mar 16 11:12:59 2015 id: 20069] 20069 from ip 127.0.0.1, port 53415 
> [define d1 in default as catalogContainer with 
> catalogContainer.constraint="bbPixNum1[0:1:202][62:1:62][13:1:13]";] received
> [GMT Mon Mar 16 11:12:59 2015 id: 20069] 20069 from ip 127.0.0.1, port 53415 
> [get dods for d1 return as dap2;] received
> 
> Is this doable, I mean unless we know exactly what happened between the 
> communication of these two servers?
> 
> -Fan
> 
> ________________________________________
> From: Unidata THREDDS Support address@hidden
> Sent: Saturday, March 14, 2015 1:51 PM
> To: Fang, Fan (GSFC-610.2)[ADNET SYSTEMS INC]
> Cc: address@hidden; Fang, Fan (GSFC-610.2)[ADNET SYSTEMS INC]
> Subject: [THREDDS #FJC-813079]: truncated data download
> 
> Hi Fan:
> 
> Can you see if theres an error message in:
> 
> 1) {tomcat}/content/thredds/logs/threddsServlet.log
> 
> (note that will roll over every hour, so generate the error then look).
> 
> 2) on the Hyrax server when the problem happens?
> 
> thanks
> 
> > Hi Ryan,
> >
> > We lately upgraded the server to the current stable 4.3 version (Version 
> > 4.3.23 - 20140826.1617):
> >
> > http://disc2.nascom.nasa.gov/thredds/catalog.html
> >
> > The trouble we reported remains.
> >
> > -Fan
> >
> > ________________________________________
> > From: Fang, Fan (GSFC-610.2)[ADNET SYSTEMS INC]
> > Sent: Thursday, March 05, 2015 10:19 PM
> > To: address@hidden
> > Subject: RE: [THREDDS #FJC-813079]: truncated data download
> >
> > Hi Ryan,
> >
> > The config catalog is a "datasetScan" of a list of NcML files, including 
> > the one for 3A25.  I am not sure it would be very informative:
> >
> > <service name="nodl" base="" serviceType="compound">
> > <service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" />
> > <service name="wcs" serviceType="WCS" base="/thredds/wcs/" />
> > <service name="wms" serviceType="WMS" base="/thredds/wms/" />
> > <service name="ncss" serviceType="NetcdfSubset" base="/thredds/ncss/grid/" 
> > />
> > <service name="ncml" serviceType="NCML" base="/thredds/ncml/"/>
> > <service name="uddc" serviceType="UDDC" base="/thredds/uddc/"/>
> > <service name="iso" serviceType="ISO" base="/thredds/iso/"/>
> > </service>
> >
> > <datasetScan name="TRMM_aggregation" ID="TRMM_aggregation" 
> > path="aggregation" location="/ftp/data/tds/ncml/aggregation">
> > <metadata inherited="true">
> > <serviceName>nodl</serviceName>
> > <dataType>Grid</dataType>
> > </metadata>
> > <filter>
> > <include wildcard="*.ncml"/>
> > </filter>
> > </datasetScan>
> >
> > The NcML file is simply reading via DAP the aggregation NcML data 
> > configured in Hyrax.  It has two lines only:
> >
> > <?xml version="1.0"?>
> > <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2"; 
> > location="http://disc2.gesdisc.eosdis.nasa.gov/opendap/ncml/aggregation/TRMM_3A25/TRMM_3A25_Aggregation.ncml"/>
> >
> > The aggregation is done in Hyrax via file "TRMM_3A25_Aggregation.ncml".  
> > You can access the Hyrax aggregation based on the URL, or here:
> >
> > http://disc2.nascom.nasa.gov/opendap/ncml/aggregation/TRMM_3A25/TRMM_3A25_Aggregation.ncml.html
> >
> > Let me know if you need anything else.  Thanks.
> >
> > -Fan
> >
> > ________________________________________
> > From: Unidata THREDDS Support address@hidden
> > Sent: Thursday, March 05, 2015 4:34 PM
> > To: Fang, Fan (GSFC-610.2)[ADNET SYSTEMS INC]
> > Cc: address@hidden; Fang, Fan (GSFC-610.2)[ADNET SYSTEMS INC]
> > Subject: [THREDDS #FJC-813079]: truncated data download
> >
> > Fan,
> >
> > While you're at it, can you also send the relevant NCML file?
> >
> > Thanks,
> >
> > Ryan
> >
> > > Fan,
> > >
> > > Well there's definitely an exception there--looks like it's accessing a 
> > > remote dataset over DODS. Can you send me the configuration catalog for 
> > > this aggregation?
> > >
> > > Thanks,
> > >
> > > Ryan
> > >
> > > > Hi Ryan,
> > > >
> > > > I just sent the second URL in my original report again and sorted out 
> > > > the logs you wanted (attached).  I got the following truncated response.
> > > >
> > > > This particular aggregation is for TRMM_3A25 monthly data, 203 time 
> > > > steps from year 1998 to late 2014.
> > > >
> > > > -Fan
> > > >
> > > > date,lat[unit="degrees_north"],lon[unit="degrees_east"],bbPixNum1[unit=""]
> > > > 190097-12-16T12:00:00Z,26.0,132.0,2699.0
> > > > 190098-01-16T12:00:00Z,26.0,132.0,4256.0
> > > > 190098-02-15T00:00:00Z,26.0,132.0,6364.0
> > > > 190098-03-16T12:00:00Z,26.0,132.0,4634.0
> > > > 190098-04-16T00:00:00Z,26.0,132.0,6216.0
> > > > 190098-05-16T12:00:00Z,26.0,132.0,10090.0
> > > > 190098-06-16T00:00:00Z,26.0,132.0,10004.0
> > > > 190098-07-16T12:00:00Z,26.0,132.0,3976.0
> > > > 190098-08-16T12:00:00Z,26.0,132.0,343.0
> > > > 190098-09-16T00:00:00Z,26.0,132.0,3445.0
> > > > 190098-10-16T12:00:00Z,26.0,132.0,7464.0
> > > > 190098-11-16T00:00:00Z,26.0,132.0,8957.0
> > > > 190098-12-16T12:00:00Z,26.0,132.0,4054.0
> > > > 190099-01-16T12:00:00Z,26.0,132.0,1033.0
> > > > 190099-02-15T00:00:00Z,26.0,132.0,235.0
> > > > 190099-03-16T12:00:00Z,26.0,132.0,12310.0
> > > > 190099-04-16T00:00:00Z,26.0,132.0,8380.0
> > > > 190099-05-16T12:00:00Z,26.0,132.0,10625.0
> > > > 190099-06-16T00:00:00Z,26.0,132.0,8768.0
> > > > 190099-07-16T12:00:00Z,
> >
> > Ticket Details
> > ===================
> > Ticket ID: FJC-813079
> > Department: Support THREDDS
> > Priority: Normal
> > Status: Closed
> >
> >
> >
> 
> 
> Ticket Details
> ===================
> Ticket ID: FJC-813079
> Department: Support THREDDS
> Priority: High
> Status: Open
> 
> 
> 


Ticket Details
===================
Ticket ID: FJC-813079
Department: Support THREDDS
Priority: High
Status: Open


NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.