[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #MAN-367636]: netCDF-4 file grows enormous when dimension set to unlimited

Hello Margaret,

In regards to the first question, I wonder if the issue is the variables along 
this dimension have default fill values set; if so , this may explain the 
explosion in file size.  Would it be possible to get a copy of the netCDF file 
to play around with? Also, what version of netCDF are you working with?   

In regards to your second question; it sounds like there is either a bug in 
nccopy that makes it think you are using or perhaps the file is using some 
small part of the enhanced model?  That is just a guess; if you can provide the 
original 192k file, I will figure out why nccopy is giving this error.

Thanks in advance, have a great day,

-Ward

> Hello,
> 
> We have a 192K netCDF-4 file with no unlimited dimensions. One of these
> dimensions is report_number, which has dimension 1. When this dimension is
> changed to "UNLIMITED // (currently 1)" the file expands to 213M, so over
> 1000 times as large as the original. Do you know what could be causing
> this? Is there any way to avoid it aside from compressing the file?
> 
> Second question: I contacted Unidata a while ago, and Russ told me that we
> likely had netCDF-4 files that didn't actually use any of the enhanced
> features. The data producer has confirmed that this is the case. I was
> interested in seeing if using the netCDF-4 classic model would reduce the
> file size, because other people here at NOAA have found netCDF-4 classic
> sizes to be on par with netCDF-3 (i.e., much less than the netCDF-4
> enhanced model). However, when I used the nccopy utility Russ recommended,
> I got the following error:
> 
> "Attempting netcdf-4 operation on strict nc3 netcdf-4 file"
> 
> Is there some way to change the "strict nc3" flag, since this is really a
> netCDF-4 file?
> 
> Thanks for any information.
> 
> Meg Tilton
> 
> 
> 
> 
> 
> address@hidden> wrote:
> 
> > Meg,
> >
> > > Strange! That means our files are the netCDF-4 enhanced version, so I'm
> > > surprised anyone could get ncdump -x to work on them. I guess it will
> > > remain an unsolved mystery.
> >
> > A possible explanation for the mystery is that your netCDF-4 files
> > really don't use any features of the enhanced model, but aren't
> > marked as netCDF-4 classic model files.  A netCDF-4 classic model
> > file is just a netCDF-4 file with a special scalar integer attribute
> > named "_nc3_strict" in the root group, which is tested to enforce
> > never adding any features of the enhanced netCCDF-4 data model to
> > the file, so that it will always be readable using the netCDF-3 API.
> >
> > Back in version 4.1.1, I don't think ncdump tested the file type, it
> > just printed whatever it could see through the API and displayed it
> > as NcML when the "-x" flag was used. But the ncdump code was never
> > modified to present the NcML representations for any of the netCDF-4
> > enhanced model features, partly because those representations were
> > still under development when netCDF 4.1.1 was released.
> >
> > If your current netCDF files are really netCDF-4 files that don't
> > use any enhanced data model features, then you could mark them as
> > netCDF-4 classic model files using the "nccopy" utility:
> >
> >   nccopy -k "netCDF-4 classic model" foo4.nc foo4c.nc
> >
> > to convert a netCDF-4 file to a netCDF-4 classic model file. That
> > would add the extra attribute (invisible through the netCDF API).
> > You could also do the same thing through the HDF5 API, which would
> > permit adding the attribute and overwriting files, which nccopy
> > doesn't permit.
> >
> > --Russ
> >
> > > address@hidden> wrote:
> > > >
> > > > Meg,
> > > >
> > > > > Thanks for your responses to my email.
> > > > >
> > > > > When I ran the ncdump -k on one of our netCDF4 files, the response
> > was
> > > > just
> > > > > "netCDF-4." Does this mean it's the enhanced model, and it would say
> > > > > "classic" otherwise? Or is it the other way around?
> > > >
> > > > It's the other way around.  The outputs from ncdump -k are one of these
> > > > four strings:
> > > >
> > > >   classic
> > > >   64-bit offset
> > > >   netCDF-4
> > > >   netCDF-4 classic model
> > > >
> > > > The "-x" option works to specify NcML output for all but the third of
> > > > those format variants, "netCDF-4".
> > > >
> > > > > I will pass on the information about the java netCDF library to the
> > NOAA
> > > > > people who run our CLASS archive. They were the ones who were
> > originally
> > > > > asking about this, and they need to develop code to extract the NcML
> > from
> > > > > netCDF4 files. So this may be very useful for them.
> > > >
> > > > --Russ
> > > >
> > > > > address@hidden> wrote:
> > > > > >
> > > > > >
> > > > > > Margaret Tilton - NOAA Affiliate,
> > > > > >
> > > > > > Your Ticket has been received, and a Unidata staff member will
> > review
> > > > it
> > > > > > and reply accordingly. Listed below are details of this new Ticket.
> > > > Please
> > > > > > make sure the Ticket ID remains in the Subject: line on all
> > > > correspondence
> > > > > > related to this Ticket.
> > > > > >
> > > > > >     Ticket ID: NRE-269426
> > > > > >     Subject: ncdump -x with netCDF4 files
> > > > > >     Department: Support netCDF
> > > > > >     Priority: Normal
> > > > > >     Status: Open
> > > > > >
> > > > > >
> > > > > >
> > > > > > The NetCDF libraries are developed at the Unidata Program Center,
> > > > > > in Boulder, Colorado, funded primarily by the National Science
> > > > Foundation.
> > > > > >
> > > > > > All support requests are handled by the development team. No
> > dedicated
> > > > > > support staff are funded at this time. For this reason we cannot
> > > > guarantee
> > > > > > response times, nor that we can resolve every support issue,
> > although
> > > > we
> > > > > > do our best to respond within 72 hours.
> > > > > >
> > > > > > It is in the nature of support requests that the same question is
> > asked
> > > > > > many
> > > > > > times. We urge you to search the support archives for material
> > > > relating to
> > > > > > your support request:
> > > > > >
> > > > > > http://www.unidata.ucar.edu/search.jsp?support&netcdf
> > > > > >
> > > > > > If you are having trouble building netCDF, please take a look at
> > the
> > > > > > "Building NetCDF" page:
> > > > > >
> > > > > > http://www.unidata.ucar.edu/software/netcdf/docs/building.html
> > > > > >
> > > > > > or the (unfortunately somewhat out-of-date) NetCDF Build
> > Troubleshooter
> > > > > > page:
> > > > > >
> > > > > > http://www.unidata.ucar.edu/software/netcdf/docs/troubleshoot.html
> > > > > >
> > > > > > Windows users should see the FAQ list:
> > > > > >
> > > > > >
> > > >
> > http://www.unidata.ucar.edu/software/netcdf/docs/faq.html#windows_netcdf4_2
> > > > > >
> > > > > > Complete documentation (including a tutorial, and sample programs
> > in C,
> > > > > > Fortran,
> > > > > > Java, and other programming languages) can be found on the netCDF
> > > > > > Documentation page:
> > > > > >
> > > > > > http://www.unidata.ucar.edu/software/netcdf/docs/
> > > > > > http://www.unidata.ucar.edu/software/netcdf/examples/programs/
> > > > > >
> > > > > > If you resolve your issue through one of these methods, please
> > send a
> > > > > > reply to
> > > > > > this email, letting us know that you no longer need support. This
> > will
> > > > help
> > > > > > us spend more time on netCDF development.
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > > Unidata User Support
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > Margaret Tilton
> > > > > Cooperative Institute for Research in Environmental Sciences (CIRES)
> > at
> > > > > University of Colorado at Boulder and
> > > > > NOAA National Geophysical Data Center, Solar and Terrestrial Physics
> > > > > Division
> > > > > 325 Broadway, E/GC2
> > > > > Boulder, Colorado 80305
> > > > > 303-497-6223
> > > > >
> > > > >
> > > > Russ Rew                                         UCAR Unidata Program
> > > > address@hidden                      http://www.unidata.ucar.edu
> > > >
> > > >
> > > >
> > > > Ticket Details
> > > > ===================
> > > > Ticket ID: NRE-269426
> > > > Department: Support netCDF
> > > > Priority: Normal
> > > > Status: Closed
> > > >
> > > >
> > >
> > > --
> > > Margaret Tilton
> > > Cooperative Institute for Research in Environmental Sciences (CIRES) at
> > > University of Colorado at Boulder and
> > > NOAA National Geophysical Data Center, Solar and Terrestrial Physics
> > > Division
> > > 325 Broadway, E/GC2
> > > Boulder, Colorado 80305
> > > 303-497-6223
> > >
> > >
> > Russ Rew                                         UCAR Unidata Program
> > address@hidden                      http://www.unidata.ucar.edu
> >
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: NRE-269426
> > Department: Support netCDF
> > Priority: Normal
> > Status: Closed
> >
> >
> 
> 
> --
> Margaret Tilton
> Cooperative Institute for Research in Environmental Sciences (CIRES)
> at the University
> of Colorado and NOAA National Centers for Environmental Information (NCEI)
> 325 Broadway, E/GC2
> Boulder, Colorado 80305
> 303-497-6223
> 
> 


Ticket Details
===================
Ticket ID: MAN-367636
Department: Support netCDF
Priority: Normal
Status: Closed