[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDFJava #OSY-357654]: delays in opening remote NCML file



So I did some simple experiments.
When that URL is opened, some coordinate variables are preloaded.
It appears that the time to do this preload read is about 8.5 seconds.
This seems rather long to me. My speculation is that the server
is responsible for most of this time. I suspect that code is somehow
associated with ncml processing on the server.


> I guess the question is is what is the griddataset.open call doing? There
> shouldn’t be any data reading going on and I assume it’s just hitting the
> .das url?
> 
> Jeff
> 
> 
> address@hidden> wrote:
> 
> > My speculation is that this just a very big file and
> > the cost is transmission + read time + time to convert to
> > CDM on the client. Does that seem reasonable,
> > or am I missing something.
> >
> > > I get ~7 seconds in the browser. Same with CURL
> > >
> > > -Jeff
> > >
> > >
> > address@hidden>
> > > wrote:
> > >
> > > > A few seconds to return this, for me
> > > >
> > > > Error {
> > > >     code = 403;
> > > >     message = "Request too big=58988.094912 Mbytes, max=500.0";
> > > > };
> > > >
> > > >
> > > >
> > > >
> > > > address@hidden> wrote:
> > > >
> > > > How long does the browser take if you access the .dods e.g.
> > > > https://na01.safelinks.protection.outlook.com/?url=
> > > > https%3A%2F%2Fdataserver.nccs.nasa.gov%2Fthredds%2FdodsC%
> > > > 2Fbypass%2FCREATE-IP%2Freanalysis%2FCFSR%2F6hr%
> > > > 2Fatmos%2Fpsl.ncml.dods&data=02%7C01%7Cbmapes%40rsmas.miami.edu%
> > > > 7Cb669f01390d34fe8a83108d53a871659%7C2a144b72f23942d48c0e6f0f17c4
> > > > 8e33%7C0%7C0%7C636479274232120145&sdata=TJcc4icYvzL6%
> > > > 2Fb59dRDbNTtWQdx14zuCUJBoQciV42Q%3D&reserved=0
> > > >
> > > >
> > > > I'm tracking a performance problem down for Brian Mapes (cc'ed). He
> > has an
> > > > ncml file that aggregates a number of ncml files and it takes a very
> > very
> > > > long time to open the file.
> > > >
> > > > Just looking at one of the NCML links that is being accessed:
> > > >
> > > > https://na01.safelinks.protection.outlook.com/?url=
> > > > https%3A%2F%2Fdataserver.nccs.nasa.gov%2Fthredds%2FdodsC%
> > > >
> > 2Fbypass%2FCREATE-IP%2Freanalysis%2FCFSR%2F6hr%2Fatmos%2Fpsl.ncml&data=02%
> > > > 7C01%7Cbmapes%40rsmas.miami.edu%7Cb669f01390d34fe8a83108d53a871659%
> > > > 7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%7C636479274232120145&sdata=
> > > > 5EBQlAWULuqQldFMKTm%2FJGYOTIKIvSAdVfL%2Fip6pDRA%3D&reserved=0
> > > >
> > > > In the browser it is sub-second to access the .das of this file.
> > However,
> > > > running CURL or the IDV's IOUtil.readcontents it takes ~10 seconds to
> > > > fetch.  Running ucar.nc2.dt.grid.GridDataset.open it takes ~ 80
> > seconds to
> > > > open the link
> > > >
> > > > Any ideas?
> > > >
> > > > -Jeff
> > > >
> > > >
> > > >
> > > > =Dennis Heimbigner
> > > >  Unidata
> > > >
> > > >
> > > > Ticket Details
> > > > ===================
> > > > Ticket ID: OSY-357654
> > > > Department: Support netCDF Java
> > > > Priority: Normal
> > > > Status: Open
> > > > ===================
> > > > NOTE: All email exchanges with Unidata User Support are recorded in the
> > > > Unidata inquiry tracking system and then made publicly available
> > through
> > > > the web.  If you do not want to have your interactions made available
> > in
> > > > this way, you must let us know in each email you send to us.
> > > >
> > > >
> > > >
> > > > *********************************************
> > > > Brian Mapes, Professor
> > > > Department of Atmospheric Sciences
> > > > Meteorology and Physical Oceanography Program
> > > > RSMAS, University of Miami
> > > > 4600 Rickenbacker Causeway
> > https://maps.google.com/?q=4600+Rickenbacker+Causeway+%0D+Miami,+FL+33149&entry=gmail&source=g
> > >
> > > > Miami, FL 33149-1098
> > > >
> > > > phone: (305) 421-4275
> > > > fax: (305) 421-4696
> > > > email: address@hidden
> > > > Web: http://www.rsmas.miami.edu/users/bmapes/
> > > > **********************************************
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> >
> > =Dennis Heimbigner
> >   Unidata
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: OSY-357654
> > Department: Support netCDF Java
> > Priority: Normal
> > Status: Open
> > ===================
> > NOTE: All email exchanges with Unidata User Support are recorded in the
> > Unidata inquiry tracking system and then made publicly available through
> > the web.  If you do not want to have your interactions made available in
> > this way, you must let us know in each email you send to us.
> >
> >
> >
> 
> 

=Dennis Heimbigner
  Unidata


Ticket Details
===================
Ticket ID: OSY-357654
Department: Support netCDF Java
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.