[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDV #KOO-280301]: Reading data via URLs in IDV: setting Content-Length and Accept-Ranges headers



Hi Dave,

> It sounds (from an email that you sent after the one above) that you
> are seeing the same behavior on Unidata's server as I see on mine.
> Below is a sample of the type of error I'm getting anyway. It occurs
> when I try to access NOAA MADIS UrbaNet point data in NetCDF format: 
> http://funnel.sfsu.edu/data/LDAD/urbanet/200806010_1200.nc
> 
> Note that the error occurs AFTER I added the entry below to the
> httpd.conf file and restarted the Web server:
> 
> <Directory "/opt/local_htdocs">
> Options FollowSymLinks Indexes
> AllowOverride None
> Order allow,deny
> Allow from all
> Header set Accept-Ranges bytes
> </Directory>
> 
> (There's no Content-Length specification; I see from a later email
> that you're wondering whether that matters or not, too.)
> 
> In contrast, I don't have any problem reading and plotting the same
> data as a local file (from the same server) (see attached JPEG).
> 
> -- Dave

Well, I'm back to thinking that Apache by default will do byte range 
subsetting. Below is what I see if I telnet to your web server and construct a 
byte range request by hand. The request is just three lines of text:

GET /data/LDAD/urbanet/20080610_1200.nc HTTP/1.1
HOST: funnel.sfsu.edu
Range: bytes=0-2000

followed by two blank lines and indicates it only wants the first 2001 bytes of 
the file. The result is a 206 Partial Content which is what we would expect and 
it looks like the response breaks in the middle (I snipped out a bunch of 
stuff).

When I try the URL (http://funnel.sfsu.edu/data/LDAD/urbanet/20080610_1200.nc) 
in the netCDF-Java 4.0 ToolsUI, it opens it just fine. It also opens fine in 
the netCDF-Java 2.2 ToolsUI (which should be the same underlying netCDF-Java 
library the IDV is using).

However, for the dataset URL 
http://www.unidata.ucar.edu/software/netcdf-java/testdata/20010721.nc, I can 
telnet to the unidata web server and can get a byte range but it fails in 
netCDF-Java ToolsUI with the exception shown below.

So, I think something other than byte range problems is going on here.

John, any ideas?

Ethan



---------------------------------------------------------------------------
zero.unidata.ucar.edu: telnet funnel.sfsu.edu 80
Trying 130.212.21.19...
Connected to funnel.sfsu.edu.
Escape character is '^]'.
GET /data/LDAD/urbanet/20080610_1200.nc HTTP/1.1
HOST: funnel.sfsu.edu
Range: bytes=0-2000

HTTP/1.1 206 Partial Content
Date: Thu, 12 Jun 2008 20:11:03 GMT
Server: Apache/2.2.8 (Unix) mod_ssl/2.2.8 OpenSSL/0.9.7l DAV/2 PHP/5.2.5
Last-Modified: Tue, 10 Jun 2008 14:19:39 GMT
ETag: "5e9620-96c41c-44f509e5214c0"
Accept-Ranges: bytes
Content-Length: 2001
MS-Author-Via: DAV
Content-Range: bytes 0-2000/9880604
Content-Type: application/x-netcdf

CDF
... [snip] ...
Conventions MADIS surface observations, v1.0title3MADIS - Meteorological 
Surface - Integrated Mesonet      
startTime2008-06-10T12:00:00ZendTime2008-06-10T12:59:59ZstationIdVariable   
stationIdstationDescriptionVariable
                                                                                
                           
stationNamestatioMADIS_MESONETnaming_authoritnoaa.govthredds_data_typeStationhistory.revision:
 1.1; date: 2007/01/19; author: barth
                                                                                
                                  creator_nameMichael F. Barth
address@hidden
...[snip]...
includes quality-controlled reports from many observing netw
Connection to funnel.sfsu.edu closed by foreign host.
zero.unidata.ucar.edu: 

---------------------------------------------------------------------------

Exception in thread "AWT-EventQueue-0" opendap.dap.parser.TokenMgrError: 
Lexical error at line 1, column 3.  Encountered: "<" (60), after : ""
        at 
opendap.dap.parser.DDSParserTokenManager.getNextToken(DDSParserTokenManager.java:1085)
        at opendap.dap.parser.DDSParser.jj_ntk(DDSParser.java:824)
        at opendap.dap.parser.DDSParser.Dataset(DDSParser.java:93)
        at opendap.dap.DDS.parse(DDS.java:1063)
        at opendap.dap.DConnect2$DDSCommand.process(DConnect2.java:452)
        at opendap.dap.DConnect2.openConnection(DConnect2.java:267)
        at opendap.dap.DConnect2.getDDS(DConnect2.java:444)
        at opendap.dap.DConnect2.getDDS(DConnect2.java:424)
        at ucar.nc2.dods.DODSNetcdfFile.<init>(DODSNetcdfFile.java:178)
        at ucar.nc2.dataset.NetcdfDataset.openDODS(NetcdfDataset.java:378)
        at ucar.nc2.dataset.NetcdfDataset.access$000(NetcdfDataset.java:78)
        at ucar.nc2.dataset.NetcdfDataset$2.open(NetcdfDataset.java:368)
        at ucar.nc2.NetcdfFileCache.acquire(NetcdfFileCache.java:188)
        at ucar.nc2.dataset.NetcdfDataset.acquireDODS(NetcdfDataset.java:366)
        at ucar.nc2.dataset.NetcdfDataset.acquireFile(NetcdfDataset.java:257)
        at ucar.nc2.ui.ToolsUI.openFile(ToolsUI.java:760)
        at ucar.nc2.ui.ToolsUI.access$3600(ToolsUI.java:67)




Ticket Details
===================
Ticket ID: KOO-280301
Department: Support THREDDS
Priority: Normal
Status: Open


NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.