[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDFJava #BNA-191717]: chunking in Java

Hi Jeff,

How chunking and compression affect file size and read/write performance is a 
complex issue. I'm going to pass this along to our chunking expert (Russ Rew) 
who, I believe, is back in the office on Monday and should be able to provide 
you with some better advise than I can give.

In the mean time, here's an email he wrote in response to a conversation on the 
effect of chunking on performance that might be useful:


Sorry I don't have a better answer for you.


Jeff Johnson wrote:
> Ethan-
> I made the changes you suggested with the following result:
> 10000 records, 8 bytes / record = 80000 bytes raw data
> original program (NetCDF4, no chunking): 537880 bytes (6.7x)
> file size with chunk size of 2000 = 457852 bytes (5.7x)
> So a little better, but still not good. I then tried different chunk sizes
> of 10000, 5000, 200, and even 1, which I would've thought would give me the
> original size, but all gave the same resulting file size of 457852.
> Finally, I tried writing more records to see if it's just a symptom of a
> small data set. With 1M records:
> 8MB raw data, chunk size = 2000
> 45.4MB file (5.7x)
> This is starting to seem like a lost cause given our small data records.
> I'm wondering if you have information I could use to go back to the archive
> group and try to convince them to use NetCDF3 instead.
> jeff

Ticket Details
Ticket ID: BNA-191717
Department: Support netCDF
Priority: Normal
Status: Open

NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.