[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: NetCDF subsetting service - PatmosX data



On 2/27/2013 3:04 AM, Tim Smyth wrote:

Dear Dr. Caron,

 

I am currently investigating the possibilities of using the AVHRR visible channel record to put together a proposal for investigating how a species of highly reflective marine phytoplankton have shifted in their global distribution over the past 30 years.  Mike Foster and Andrew Heidinger kindly pointed me towards the AVHRR Pathfinder Atmospheres- Extended (PATMOS-x) Reflectance data set available from the catalog:

 

http://data.ncdc.noaa.gov/thredds/patmosxCatalog.html

 

It seems with some testing and code writing on my part, that the dataset is ideal (I require Channel 1 and 2 reflectances as well as the attendant geometries to do other corrections).

 

However, each daily global file at 0.1 degree resolution (which is required to look at the features of interest) and the parameters required results in a netCDF file of approximately 155 Mb.  Each of these takes about a minute to put together the request on the website and then a further 8 minutes to download.

 

Putting all of these numbers together, for the 35 year time series, results in:

 

2 Tb of data to download

90 days of download time(!)

 

I was wondering whether or not there are some alternative options that we could investigate to:

(1)    Speed up the data subsetting and;

(2)    Transfer the data across.

 

Please?

 

I would be very grateful if you could help me in this matter please?

 

Yours sincerely,

 

Tim Smyth

 

Dr Tim Smyth
Acting Head of Science: Cycling in the Sunlit Ocean

Plymouth Marine Laboratory
Prospect Place
Plymouth
Devon
PL1 3DH, UK

Tel: +44 1752 633466
Fax: +44 1752 633101

 

Latest news: www.pml.ac.uk and @PlymouthMarine

Plymouth Marine Laboratory (PML) is a company limited by guarantee registered in England & Wales, company number 4178503. Registered Charity No. 1091222. Registered Office: Prospect Place, The Hoe, Plymouth  PL1 3DH, UK. 

This message is private and confidential. If you have received this message in error, please notify the sender and remove it from your system. You are reminded that e-mail communications are not secure and may contain viruses; PML accepts no liability for any loss or damage which may be caused by viruses.

Hi Tim:

We have made a lot of improvements to NCSS in latest version of TDS 4.3. Can you contact NCDC and urge them to upgrade? Then repeat your timings and we can go from there.

8 minutes to download 155 Mb may be that their server is overloaded. One can throw money at this, but that may not be in the cards.

You could also post to thredds or other lists to see if this data is available anywhere else.

John.