Unidata - To provide the data services, tools, and cyberinfrastructure leadership that advance Earth system science, enhance educational opportunities, and broaden participation. Unidata
         
  advanced  
 

many little files versus one big file

I have about 1700 files totaling about 2 Mbytes of data.  If I load
and process them individually, it takes about 7 seconds.  If I
concatenate them together on the unlimited dimension to form one big
file, it takes about 35 seconds to load and process them.  Is this
expected?  Wouldn't there be less overhead with just one file?  I got
the exact same results in both cases so I don't think did anything
terribly wrong.

Thanks,
Charles




 
 
  Contact Us     Site Map     Search     Terms and Conditions     Privacy Policy     Participation Policy
 
National Science Foundation (NSF) UCAR Community Programs   Unidata is a member of the UCAR Community Programs, is managed by the University Corporation for Atmospheric Research, and is sponsored by the National Science Foundation.
P.O. Box 3000     Boulder, CO 80307-3000 USA     Tel: 303-497-8643     Fax: 303-497-8690