netcdf unlimited dimension in versions 3 & 4 with fortran90

[Ken Knapp <Ken.Knapp@xxxxxxxx>]

Hi-
I'm kind of new to writing unlimited dimensions in netcdf and ran into 
some odd behavior when attempting to write unlimited dimension with 
netCDF4 and Fortran90. In short, I find two problems (for details of the 
problems see below examples and code):
1) writing unlimited dimensions in netCDF4 significantly increases file 
size over same data written with version 3.6. I realize I could turn on 
deflate ... but why would netcdf4-unlimited be 10x bigger than the 
ver.3.6 or ver.4??

2) writing large amounts of unlimited data seems impossible? Is there a 
performance issue? I realize my code is likely not optimized. Maybe I'm 
using the wrong netCDF flags? When attempting to write 1 million points 
with an unlimited dimension, ver3 file (not shown) completes quickly, 
ver.4 file never completed.

Thanks for any insight.
-Ken

p.s. I have Lahey-Fortran v6.20d and am using netcdf-4.0-snapshot2007052502


I've attached code used to produce 3 files: ver.3.6 unlimited-dimension, 
ver.4 fixed-dimension data and ver.4 unlimited-dimension data

First, I run with NMAX=10,000
> haze:~/for/tests $ time ./test_unlimited ; lss Test_*
>
> real    0m3.040s
> user    0m2.640s
> sys     0m0.370s
>  48K Test_Limited_v4.nc   40K Test_Unlimited_v3.nc  420K 
> Test_Unlimited_v4.nc
Since it is storing 10K int*4's, the file size should be ~40K. But the 
ver.4 unlimited-dimension file is 10x too big?

OK, so I then ran it with NMAX=100,000
> haze:~/for/tests $ time ./test_unlimited ; lss Test_*
>
> real    0m30.478s
> user    0m27.220s
> sys     0m3.240s
> 404K Test_Limited_v4.nc  396K Test_Unlimited_v3.nc  4.0M 
> Test_Unlimited_v4.nc
Same issue: ver.4 w/ unlimited is 10x too big? And now it completed in 
10x3sec=30sec. not bad since number of writes is 10x before.

Now the kicker, NMAX=1,000,000 ... it never finishes on my system (after 
20 minutes ... where it should conceivably finish after 
3secx100=300sec=5 minutes). It completes the first 2 files, but it never 
completes writing the Test_Unlimited_v4.nc file. Here's the output after 
Cntrl-C'ing the job:
> haze:~/for/tests $ time ./test_unlimited ; lss Test_*
> real    20m22.557s
> user    4m19.840s
> sys     0m53.840s
> 3.9M Test_Limited_v4.nc  3.9M Test_Unlimited_v3.nc   40M 
> Test_Unlimited_v4.nc
It never completed the write to Test_Unlimited_v4.nc, but the file sizes 
are roughly the right size. In fact, the memory goes through the roof 
and begins page swapping. Why?




> program unlimited
>   use netcdf
>   implicit none
>   include 'netcdf4.inc'
>   integer :: ncid, dim, vid, vid2, i
>   integer :: nmax=1000000
>
>   !NetCDF3 file
>   call check( nf90_create('Test_Unlimited_v3.nc',NF90_CLOBBER,ncid) )
>   call check( nf90_def_dim(ncid,"cell",NF90_UNLIMITED,dim) )
>   call check( nf90_def_var(ncid,"temp",NF90_INT,dim,vid) )
>   call check(nf90_enddef(ncid))
>   do i=1,nmax
>      call check( nf90_put_var(ncid,vid,i,start=(/ i /) ) )
>   end do
>   call check( nf90_close(ncid) )
>
>   call check( nf90_create('Test_Limited_v4.nc',NF_NETCDF4,ncid) )
>   call check( nf90_def_dim(ncid,"cell",nmax,dim) )
>   call check( nf90_def_var(ncid,"temp",NF90_INT,dim,vid) )
>   call check(nf90_enddef(ncid))
>   do i=1,nmax
>      call check( nf90_put_var(ncid,vid,i,start=(/i /) ) )
>   end do
>   call check( nf90_close(ncid) )
>
>   call check( nf90_create('Test_Unlimited_v4.nc',NF_NETCDF4,ncid) )
>   call check( nf90_def_dim(ncid,"cell",NF90_UNLIMITED,dim) )
>   call check( nf90_def_var(ncid,"temp",NF90_INT,dim,vid) )
>   call check(nf90_enddef(ncid))
>   do i=1,nmax
>      call check( nf90_put_var(ncid,vid,i,start=(/i /) ) )
>   end do
>   call check( nf90_close(ncid) )
>
> contains
>   subroutine check(status)
>     integer, intent ( in) :: status
>     if(status /= nf90_noerr) then
>       print *, trim(nf90_strerror(status))
>       stop "Stopped"
>     end if
>   end subroutine check  
> end program unlimited


--
Ken Knapp
Remote Sensing and Applications Division
National Climatic Data Center
151 Patton Ave
Asheville, NC 28801
828-271-4339 (voice) 828-271-4328 (fax)

===============================================================================
To unsubscribe netcdfgroup, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
===============================================================================