Re: Use of http proxy?

Bummer. Found the real exception and the source - DODS libraries are using the commons http client, which requires explicit proxy setting at an application level (ie proxies must be a supported feature of DAP). But even when skirting DNS availability using hosts file, we go from this,

java.net.UnknownHostException: www.marine.csiro.au
       at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:177)
       at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
       at java.net.Socket.connect(Socket.java:519)
       at java.net.Socket.connect(Socket.java:469)
       at java.net.Socket.<init>(Socket.java:366)
       at java.net.Socket.<init>(Socket.java:239)
at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:79) at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:121) at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:706) at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1321) at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:386) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324)
       at opendap.dap.DConnect2.openConnection(DConnect2.java:218)

to this,

C:\Program Files\IDV_2.2>jre\bin\java -Dhttp.proxyHost=proxyhost -Dhttp.proxyPor
t=3128 -Xmx512m -jar idv.jar
25/05/2007 14:52:15 org.apache.commons.httpclient.HttpMethodDirector executeWith
Retry
INFO: I/O exception (java.net.ConnectException) caught when processing request:
Connection timed out: connect
25/05/2007 14:52:15 org.apache.commons.httpclient.HttpMethodDirector executeWith
Retry
INFO: Retrying request
25/05/2007 14:53:00 org.apache.commons.httpclient.HttpMethodDirector executeWith
Retry
INFO: I/O exception (java.net.ConnectException) caught when processing request:
Connection timed out: connect

Looking at the source below, it would indeed seem DAP does not support proxies. This is hard to believe - it's more effective than any EULA for curtailing commercial use. I'll take my grumbles to the DAP group I guess... :(

opendap.dap.DConnect2.java:

private void openConnection(String urlString, Command command) throws IOException, DAP2Exception, ParseException {

   initHttpClient();
   GetMethod method = new GetMethod(urlString);
   method.setFollowRedirects(true);

   if (acceptCompress)
method.setRequestHeader(new Header("Accept-Encoding", "deflate,gzip"));

   // enable sessions
   if (allowSessions)
     method.setRequestHeader(new Header("X-Accept-Session", "true"));

   InputStream is = null;
   try {
     // Execute the method.

     int statusCode = _httpClient.executeMethod(method);
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^





Nick Bower wrote:
IDV seems to be having trouble with our http proxy (squid). I've set the following global parameters in runIDV.bat in the usual way;

-Dhttp.proxyHost=proxyhost -Dhttp.proxyPort=3128

however IDV fails to retrieve any data in the Data Chooser. See traceback below. Could it have something to do with IDV insisting DODS URI's begin with dods:// perhaps? What is the recommended solution for using a http proxy?

By the way, I can navigate the server in the "Display" / web browsing tab. It's only the Data Chooser that fails.

Thanks, Nick

An error has occurred:
There was an error loading the data:
Error creating data source:dods.grid with: http://www.marine.csiro.au/dods/nph-dods/dods-data/bl/SPINUP4/temp/ Grid data source failed making data set: dods://www.marine.csiro.au/dods/nph-dods/dods-data/bl/SPINUP4/temp/ Cant open dods://www.marine.csiro.au/dods/nph-dods/dods-data/bl/SPINUP4/temp/ or as DODS dods://www.marine.csiro.au/dods/nph-dods/dods-data/bl/SPINUP4/temp/
www.marine.csiro.au

java.io.FileNotFoundException: Cant open dods://www.marine.csiro.au/dods/nph-dods/dods-data/bl/SPINUP4/temp/ or as DODS dods://www.marine.csiro.au/dods/nph-dods/dods-data/bl/SPINUP4/temp/
www.marine.csiro.au
   at ucar.nc2.dataset.NetcdfDataset.openDODS(NetcdfDataset.java:380)
   at ucar.nc2.dataset.NetcdfDataset.openFile(NetcdfDataset.java:316)
   at ucar.nc2.dataset.NetcdfDataset.openDataset(NetcdfDataset.java:175)
at ucar.nc2.dataset.NetcdfDatasetCache.acquire(NetcdfDatasetCache.java:203) at ucar.nc2.dataset.NetcdfDatasetCache.acquire(NetcdfDatasetCache.java:183) at ucar.nc2.dataset.NetcdfDatasetCache.acquire(NetcdfDatasetCache.java:171)
   at ucar.nc2.dt.grid.GridDataset.open(GridDataset.java:71)
at ucar.unidata.data.grid.GeoGridDataSource.doMakeDataSet(GeoGridDataSource.java:726) at ucar.unidata.data.grid.GeoGridDataSource.getDataset(GeoGridDataSource.java:743) at ucar.unidata.data.grid.GeoGridDataSource.initAfterCreation(GeoGridDataSource.java:335) at ucar.unidata.data.DataManager.createDataSource(DataManager.java:1186) at ucar.unidata.idv.IntegratedDataViewer.createDataSource(IntegratedDataViewer.java:1698) at ucar.unidata.idv.IntegratedDataViewer.makeDataSource(IntegratedDataViewer.java:1617) at ucar.unidata.idv.IntegratedDataViewer.makeDataSource(IntegratedDataViewer.java:1595) at ucar.unidata.idv.IntegratedDataViewer.handleAction(IntegratedDataViewer.java:1446)
   at ucar.unidata.idv.DefaultIdv.handleAction(DefaultIdv.java:112)
at ucar.unidata.idv.IntegratedDataViewer.handleAction(IntegratedDataViewer.java:1378) at ucar.unidata.idv.chooser.UrlChooser.loadURLInner(UrlChooser.java:250)
   at ucar.unidata.idv.chooser.UrlChooser.loadURL(UrlChooser.java:222)
at ucar.unidata.idv.chooser.UrlChooser.doLoadInThread(UrlChooser.java:269)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:585)
   at ucar.unidata.util.Misc$2.run(Misc.java:919)
   at ucar.unidata.util.Misc$3.run(Misc.java:947)


And another example from your own servers...

java.io.FileNotFoundException: Cant open dods://motherlode.ucar.edu:8080/thredds/dodsC/modelsNc/NCEP/NAM/CONUS_80km/NAM_CONUS_80km_20070524_1200.nc or as DODS dods://motherlode.ucar.edu:8080/thredds/dodsC/modelsNc/NCEP/NAM/CONUS_80km/NAM_CONUS_80km_20070524_1200.nc
motherlode.ucar.edu
   at ucar.nc2.dataset.NetcdfDataset.openDODS(NetcdfDataset.java:380)
   at ucar.nc2.dataset.NetcdfDataset.openFile(NetcdfDataset.java:316)
   at ucar.nc2.dataset.NetcdfDataset.openDataset(NetcdfDataset.java:175)
at ucar.nc2.dataset.NetcdfDatasetCache.acquire(NetcdfDatasetCache.java:203) at ucar.nc2.dataset.NetcdfDatasetCache.acquire(NetcdfDatasetCache.java:183) at ucar.nc2.dataset.NetcdfDatasetCache.acquire(NetcdfDatasetCache.java:171)
   at ucar.nc2.dt.grid.GridDataset.open(GridDataset.java:71)
at ucar.unidata.data.grid.GeoGridDataSource.doMakeDataSet(GeoGridDataSource.java:726) at ucar.unidata.data.grid.GeoGridDataSource.getDataset(GeoGridDataSource.java:743) at ucar.unidata.data.grid.GeoGridDataSource.initAfterCreation(GeoGridDataSource.java:335) at ucar.unidata.data.DataManager.createDataSource(DataManager.java:1186) at ucar.unidata.idv.IntegratedDataViewer.createDataSource(IntegratedDataViewer.java:1698) at ucar.unidata.idv.IntegratedDataViewer.makeDataSource(IntegratedDataViewer.java:1617) at ucar.unidata.idv.IntegratedDataViewer.makeDataSource(IntegratedDataViewer.java:1595) at ucar.unidata.idv.IntegratedDataViewer.makeDataSource(IntegratedDataViewer.java:1576) at ucar.unidata.idv.chooser.IdvChooser.makeDataSource(IdvChooser.java:610) at ucar.unidata.idv.chooser.ThreddsHandler.processNodes(ThreddsHandler.java:842) at ucar.unidata.idv.chooser.ThreddsHandler.processNode(ThreddsHandler.java:782)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:585)
   at ucar.unidata.util.Misc$2.run(Misc.java:921)
   at ucar.unidata.util.Misc$3.run(Misc.java:947)