[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #ONQ-750372]: NetCDF NC_CHAR file double the size of ASCII file

Hi Tim,

I'm assuming that you got the answer you needed from the netCDF postings about
the problem.  Please let us know if you have more questions ...

--Russ

> Dear Russ, thanks for your reply.
> 
> The result of the command you suggested is below:
> 
> netcdf plate {
> dimensions:
> Record_Lines = 4 ;
> Line_Symbols = 87 ;
> Record_Number = UNLIMITED ; // (3 currently)
> variables:
> char Record(Record_Number, Record_Lines, Line_Symbols) ;
> 
> // global attributes:
> :_Format = "classic" ;
> }
> 
> 
> 
> Please let me know how best to structure my file for best performance.
> 
> Cheers,
> 
> Tim.
> ______________________________________________
> Tim Stitt PhD
> User Support Manager (CRC)
> Research Assistant Professor (Computer Science & Engineering)
> Room 108, Center for Research Computing, University of Notre Dame, IN 46556
> Email: address@hidden
> 
> 
> 
> 
> 
> On 5/19/14, 5:22 PM, "Unidata netCDF Support"
> <address@hidden> wrote:
> 
> >Hi Tim,
> >
> >> I¹ve been trying to convert a large (40GB) ASCII text file (composed of
> >>multiple records of 4 line ASCII strings about 90 characters long) into
> >>NetCDF format. My plan was to rewrite the original serial code to use
> >>parallel NetCDF to have many MPI processes concurrently read records and
> >>process them in parallel.
> >>
> >> I was able to write some code to convert the ASCII records into
> >>[unlimited][4][90] NetCDF NC_CHAR arrays, which I was able to read
> >>concurrently via parallel NetCDF routines. My question is related to the
> >>size of the converted NetCDF file.
> >>
> >> I notice that the converted NetCDF file is always double the size of
> >>the ASCII file whereas I was hoping for it be to much reduced. I was
> >>therefore wondering if this is expected or is more due to my bad
> >>representation in NetCDF of the ASCII records? I am using
> >>nc_put_vara_text() to write my records. Maybe I need to introduce
> >>compression that I¹m not doing already?
> >
> >Doubling in size is not expected, and I need more information to explain
> >what's happening.  Could you send the output of running
> >
> >  ncdump -sh YOURFILE
> >
> >where YOURFILE is the netCDF file you've written.  That will provide
> >information that may make it clear what's happening, for example
> >inefficient chunk shapes if you've created a netCDF-4 classic model
> >file.  The size of the output from the above command shouldn't be
> >very large as it will only contain the file schema (metadata) and no
> >actual data.
> >
> >Thanks!
> >
> >--Russ
> >
> >Russ Rew                                         UCAR Unidata Program
> >address@hidden                      http://www.unidata.ucar.edu
> >
> >
> >
> >Ticket Details
> >===================
> >Ticket ID: ONQ-750372
> >Department: Support netCDF
> >Priority: Normal
> >Status: Closed
> >
> 
> 

Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: ONQ-750372
Department: Support netCDF
Priority: Normal
Status: Closed