[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[THREDDS #ZIZ-818863]: Thredds inflation of data



Ok, this proposed fix has been inserted into our
master branch (see https://github.com/Unidata/thredds/pull/976)
now we just need to get you a snapshot to test.

> Yes, I think that will work well.
> 
> Thanks
> 
> Martyn
> 
> Martyn Hunt  
> Technical Lead, Mainframe
> Met Office  FitzRoy Road  Exeter  Devon  EX1 3PB  United Kingdom
> Tel: +44 (0)1392 884897  
> Email: address@hidden  Website: www.metoffice.gov.uk
> 
> -----Original Message-----
> From: Unidata THREDDS Support [mailto:address@hidden]
> Sent: 13 December 2017 21:39
> To: Hunt, Martyn <address@hidden>
> Cc: address@hidden; Hunt, Martyn <address@hidden>
> Subject: [THREDDS #ZIZ-818863]: Thredds inflation of data
> 
> What I was planning to do was set the output buffer size to be the same size 
> as 8*chunk size being inflated (which is as large or larger than the deflated 
> size).
> This means that the chunk size for the input file will also determine the 
> output buffer size.
> Does this seem usable?
> 
> 
> > Yes, as long as there is way for us to set the buffersize like:
> >
> > -Dunidata.h5iosp.inflate.outputbuffersize=<integer>
> >
> > Martyn Hunt
> > Technical Lead, Mainframe
> > Met Office  FitzRoy Road  Exeter  Devon  EX1 3PB  United Kingdom
> > Tel: +44 (0)1392 884897
> > Email: address@hidden  Website: www.metoffice.gov.uk
> >
> > -----Original Message-----
> > From: Unidata THREDDS Support
> > [mailto:address@hidden]
> > Sent: 12 December 2017 19:01
> > To: Hunt, Martyn <address@hidden>
> > Cc: address@hidden; Hunt, Martyn
> > <address@hidden>
> > Subject: [THREDDS #ZIZ-818863]: Thredds inflation of data
> >
> > Sorry, I was not clear. The simplest fix is to change the call (in 
> > H5tiledLayoutBB) IO.copy to IO.copyB.
> > The copyB function allows H5tiledLayoutBB to pass in a buffer size.
> > Is this an acceptable fix?
> >
> > > It's the value in IO.java that gets passed as:
> > >
> > > byte[] buffer = new byte[default_file_buffersize];
> > >
> > > Martyn Hunt
> > > Technical Lead, Mainframe
> > > Met Office  FitzRoy Road  Exeter  Devon  EX1 3PB  United Kingdom
> > > Tel: +44 (0)1392 884897
> > > Email: address@hidden  Website: www.metoffice.gov.uk
> > >
> > > -----Original Message-----
> > > From: Unidata THREDDS Support
> > > [mailto:address@hidden]
> > > Sent: 11 December 2017 17:29
> > > To: Hunt, Martyn <address@hidden>
> > > Cc: address@hidden; Hunt, Martyn
> > > <address@hidden>
> > > Subject: [THREDDS #ZIZ-818863]: Thredds inflation of data
> > >
> > > I have a question.
> > > Is the issue the default_file_buffersize in IO.java in general, or is it 
> > > specifically the IO.copy() call in H5tiledLayoutBB?
> > >
> > > >
> > > > Thanks for this work, this is now sending much bigger chunks across to 
> > > > the hardware engine.  However, I think it's now exposed that the output 
> > > > buffer size (set by default_file_buffersize=9200 in IO.java) is also 
> > > > much too small.  Could you make a change here as well, maybe to set it 
> > > > to something like 8*input buffer, rather than a static 9200 bytes?  It 
> > > > looks like you use that 8x value in H5tiledLayoutBB.java to setup the 
> > > > outputStream, so it's consistent maths at least.  Or just make it a 
> > > > tuneable as you have done for the inflate buffersize?
> > > >
> > > > Thanks
> > > >
> > > > Martyn
> > > >
> > > > Martyn Hunt
> > > > Technical Lead, Mainframe
> > > > Met Office  FitzRoy Road  Exeter  Devon  EX1 3PB  United Kingdom
> > > > Tel: +44 (0)1392 884897
> > > > Email: address@hidden  Website: www.metoffice.gov.uk
> > > >
> > > > -----Original Message-----
> > > > From: Unidata THREDDS Support
> > > > [mailto:address@hidden]
> > > > Sent: 22 November 2017 10:09
> > > > To: Hunt, Martyn <address@hidden>
> > > > Cc: address@hidden; Hunt, Martyn
> > > > <address@hidden>
> > > > Subject: [THREDDS #ZIZ-818863]: Thredds inflation of data
> > > >
> > > > Hi Martyn,
> > > >
> > > > Here is a recent SNAPSHOT build of toolsUI that includes the fix:
> > > > https://artifacts.unidata.ucar.edu/repository/unidata-snapshots/ed
> > > > u/ uc
> > > > ar/toolsUI/4.6.11-SNAPSHOT/toolsUI-4.6.11-20171119.194235-20.jar
> > > >
> > > > Alternatively, here is the TDS build from the same commit:
> > > > https://artifacts.unidata.ucar.edu/repository/unidata-snapshots/ed
> > > > u/ uc ar/tds/4.6.11-SNAPSHOT/tds-4.6.11-20171119.194513-36.war
> > > > Make sure to rename it to "thredds.war" before deploying it in Tomcat.
> > > >
> > > > Cheers,
> > > > Christian
> > > >
> > > > > Sorry, I'm being thick here, what can I do with/how do I test this 
> > > > > jar?  It's not ToolsUI or Thredds...
> > > > >
> > > > > Thanks
> > > > > Martyn
> > > >
> > > > Ticket Details
> > > > ===================
> > > > Ticket ID: ZIZ-818863
> > > > Department: Support THREDDS
> > > > Priority: Normal
> > > > Status: Closed
> > > > ===================
> > > > NOTE: All email exchanges with Unidata User Support are recorded in the 
> > > > Unidata inquiry tracking system and then made publicly available 
> > > > through the web.  If you do not want to have your interactions made 
> > > > available in this way, you must let us know in each email you send to 
> > > > us.
> > > >
> > > >
> > > >
> > >
> > > =Dennis Heimbigner
> > > Unidata
> > >
> > >
> > > Ticket Details
> > > ===================
> > > Ticket ID: ZIZ-818863
> > > Department: Support THREDDS
> > > Priority: Normal
> > > Status: Open
> > > ===================
> > > NOTE: All email exchanges with Unidata User Support are recorded in the 
> > > Unidata inquiry tracking system and then made publicly available through 
> > > the web.  If you do not want to have your interactions made available in 
> > > this way, you must let us know in each email you send to us.
> > >
> > >
> > >
> >
> > =Dennis Heimbigner
> > Unidata
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: ZIZ-818863
> > Department: Support THREDDS
> > Priority: Normal
> > Status: Open
> > ===================
> > NOTE: All email exchanges with Unidata User Support are recorded in the 
> > Unidata inquiry tracking system and then made publicly available through 
> > the web.  If you do not want to have your interactions made available in 
> > this way, you must let us know in each email you send to us.
> >
> >
> >
> 
> =Dennis Heimbigner
> Unidata
> 
> 
> Ticket Details
> ===================
> Ticket ID: ZIZ-818863
> Department: Support THREDDS
> Priority: Normal
> Status: Open
> ===================
> NOTE: All email exchanges with Unidata User Support are recorded in the 
> Unidata inquiry tracking system and then made publicly available through the 
> web.  If you do not want to have your interactions made available in this 
> way, you must let us know in each email you send to us.
> 
> 
> 

=Dennis Heimbigner
  Unidata


Ticket Details
===================
Ticket ID: ZIZ-818863
Department: Support THREDDS
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.