netcdf unlimited dimension in versions 3 & 4 with fortran90

Hi-
I'm kind of new to writing unlimited dimensions in netcdf and ran into some odd behavior when attempting to write unlimited dimension with netCDF4 and Fortran90. In short, I find two problems (for details of the problems see below examples and code): 1) writing unlimited dimensions in netCDF4 significantly increases file size over same data written with version 3.6. I realize I could turn on deflate ... but why would netcdf4-unlimited be 10x bigger than the ver.3.6 or ver.4??

2) writing large amounts of unlimited data seems impossible? Is there a performance issue? I realize my code is likely not optimized. Maybe I'm using the wrong netCDF flags? When attempting to write 1 million points with an unlimited dimension, ver3 file (not shown) completes quickly, ver.4 file never completed.

Thanks for any insight.
-Ken

p.s. I have Lahey-Fortran v6.20d and am using netcdf-4.0-snapshot2007052502


I've attached code used to produce 3 files: ver.3.6 unlimited-dimension, ver.4 fixed-dimension data and ver.4 unlimited-dimension data

First, I run with NMAX=10,000
haze:~/for/tests $ time ./test_unlimited ; lss Test_*

real    0m3.040s
user    0m2.640s
sys     0m0.370s
48K Test_Limited_v4.nc 40K Test_Unlimited_v3.nc 420K Test_Unlimited_v4.nc
Since it is storing 10K int*4's, the file size should be ~40K. But the ver.4 unlimited-dimension file is 10x too big?

OK, so I then ran it with NMAX=100,000
haze:~/for/tests $ time ./test_unlimited ; lss Test_*

real    0m30.478s
user    0m27.220s
sys     0m3.240s
404K Test_Limited_v4.nc 396K Test_Unlimited_v3.nc 4.0M Test_Unlimited_v4.nc
Same issue: ver.4 w/ unlimited is 10x too big? And now it completed in 10x3sec=30sec. not bad since number of writes is 10x before.

Now the kicker, NMAX=1,000,000 ... it never finishes on my system (after 20 minutes ... where it should conceivably finish after 3secx100=300sec=5 minutes). It completes the first 2 files, but it never completes writing the Test_Unlimited_v4.nc file. Here's the output after Cntrl-C'ing the job:
haze:~/for/tests $ time ./test_unlimited ; lss Test_*
real    20m22.557s
user    4m19.840s
sys     0m53.840s
3.9M Test_Limited_v4.nc 3.9M Test_Unlimited_v3.nc 40M Test_Unlimited_v4.nc
It never completed the write to Test_Unlimited_v4.nc, but the file sizes are roughly the right size. In fact, the memory goes through the roof and begins page swapping. Why?




program unlimited
  use netcdf
  implicit none
  include 'netcdf4.inc'
  integer :: ncid, dim, vid, vid2, i
  integer :: nmax=1000000

  !NetCDF3 file
  call check( nf90_create('Test_Unlimited_v3.nc',NF90_CLOBBER,ncid) )
  call check( nf90_def_dim(ncid,"cell",NF90_UNLIMITED,dim) )
  call check( nf90_def_var(ncid,"temp",NF90_INT,dim,vid) )
  call check(nf90_enddef(ncid))
  do i=1,nmax
     call check( nf90_put_var(ncid,vid,i,start=(/ i /) ) )
  end do
  call check( nf90_close(ncid) )

  call check( nf90_create('Test_Limited_v4.nc',NF_NETCDF4,ncid) )
  call check( nf90_def_dim(ncid,"cell",nmax,dim) )
  call check( nf90_def_var(ncid,"temp",NF90_INT,dim,vid) )
  call check(nf90_enddef(ncid))
  do i=1,nmax
     call check( nf90_put_var(ncid,vid,i,start=(/i /) ) )
  end do
  call check( nf90_close(ncid) )

  call check( nf90_create('Test_Unlimited_v4.nc',NF_NETCDF4,ncid) )
  call check( nf90_def_dim(ncid,"cell",NF90_UNLIMITED,dim) )
  call check( nf90_def_var(ncid,"temp",NF90_INT,dim,vid) )
  call check(nf90_enddef(ncid))
  do i=1,nmax
     call check( nf90_put_var(ncid,vid,i,start=(/i /) ) )
  end do
  call check( nf90_close(ncid) )

contains
  subroutine check(status)
    integer, intent ( in) :: status
    if(status /= nf90_noerr) then
      print *, trim(nf90_strerror(status))
      stop "Stopped"
    end if
end subroutine check end program unlimited


--
Ken Knapp
Remote Sensing and Applications Division
National Climatic Data Center
151 Patton Ave
Asheville, NC 28801
828-271-4339 (voice) 828-271-4328 (fax)

==============================================================================
To unsubscribe netcdfgroup, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
==============================================================================