[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #TSI-527912]: nccopy advice - rechunking very large files



thanks dan, i can use the one that russ generated and save 3 hours of 
download time.

On 10/11/2012 7:56 AM, Dan Swank wrote:
> New Client Reply: nccopy advice - rechunking very large files
>
> John,
> All files may be accessed here:
> http://nomads.ncdc.noaa.gov/data/offlinestage/custom/narr-physaggs/
> Same directory Russ got it from.
> narr-TMP-850mb*
> are our first test case for which  all variants are in place.
>
>            -Dan
>
>
> On Wed, Oct 10, 2012 at 6:37 PM, Unidata THREDDS Support
> <address@hidden> wrote:
>> Hi Mike:
>>
>> If you can make one of those file available on ftp or something, ill be able 
>> to profile the ncss. thanks!
>>
>> John
>>
>>> My use of "points" in my e-mail description might be misleading.  I am
>>> really meaning timeseries for individual grid cells from a gridded dataset.
>>>
>>> Thanks,
>>>
>>> Mike
>>>
>>> --
>>> D. Michael Grogan
>>> -----------------------------------------
>>> STG, Inc. - Government ContractorData Access & Applications Branch
>>> Climate Services and Monitoring Division
>>> National Climatic Data address@hidden
>>> Phone: (828) 271-4680
>>>
>>>
>>>
>>> On Wed, Oct 10, 2012 at 5:23 PM, Mike Grogan <address@hidden> wrote:
>>>
>>>> THREDDS Support,
>>>>
>>>> We have been experimenting with different chunking schemes to support fast
>>>> timeseries access for points within a large (~34 year) physical
>>>> aggregation.  Please reference Dan Swank's recent discussions with Russ Rew
>>>> under ticket [netCDF #TSI-527912] in netCDF e-mail support.
>>>>
>>>> Dan has created a .nc4 version of this physical aggregation with chunk
>>>> sizes time/98128,x/8,y/6.
>>>>
>>>> When we use the OPeNDAP interface in TDS, we are able to retrieve the
>>>> entire ~ 34-year timeseries for a single x/y coordinate point in < 1/10th
>>>> of a second when hitting TDS from localhost.
>>>>
>>>> However, when we use the NetCDF Subset Service (NCSS) in TDS to retrieve
>>>> even just a one-month timeseries for a single lat/lon coordinate it takes
>>>> 40 seconds or more.
>>>>
>>>> What is going on under the hood here with NCSS that would cause this
>>>> discrepancy?  I know the nearest x/y has to be found in coordinate space
>>>> for the specified lat/lon.  Is this calculation happening for each timestep
>>>> for some reason?  Something else?
>>>>
>>>> We are using the tds-4.3.13-20120807.191518-1.war that Marcos built for
>>>> us a couple of months back.
>>>>
>>>> Sample URLs [unfortunately not public] are:
>>>>
>>>>
>>>> http://SERVER_NAME:8080/thredds/dodsC/NARRphysagg33/narr-TMP-850mb_221_yyyymmdd_hh00_000.grb.grb2.nc4.ts.d0.ascii?TMP_850mb[0:1:98127][40][240]
>>>>
>>>>
>>>> http://SERVER_NAME:8080/thredds/ncss/grid/NARRphysagg33/narr-TMP-850mb_221_yyyymmdd_hh00_000.grb.grb2.nc4.ts.d0?var=TMP_850mb&latitude=45&longitude=-83&time_start=1979-01-01T00%3A00%3A00.000Z&time_end=1979-01-31T21%3A00%3A00.000Z&vertCoord=&accept=csv
>>>>
>>>> Ignore the .grb and .grb2 parts of the filenames.  These ARE NC4 files.
>>>>   The grb and grb2 are just in there showing our original conversion path 
>>>> in
>>>> our experiments.
>>>>
>>>> Our d0 files are no compression.
>>>>
>>>> Thanks,
>>>>
>>>> Mike Grogan
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> D. Michael Grogan
>>>> -----------------------------------------
>>>> STG, Inc. - Government ContractorData Access & Applications Branch
>>>> Climate Services and Monitoring Division
>>>> National Climatic Data address@hidden
>>>> Phone: (828) 271-4680
>>>>
>>>>
>>>
>>>
>>
>>
>> Ticket Details
>> ===================
>> Ticket ID: TSI-527912
>> Department: Support THREDDS
>> Priority: Normal
>> Status: Open
>>
>
>
>



Ticket Details
===================
Ticket ID: TSI-527912
Department: Support THREDDS
Priority: Normal
Status: Open


NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.