[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Support #CUV-251255]: Nccopy extremly slow / hangs



Mark,

> Thanks for the reply. Ok, so based on your reply, I'm doing some stuff wrong. 
> My machine has 8GB of RAM, whilst the file itself is 18GB uncompressed. My 
> attempts with the command given resulted in the machine overloading and 
> killing nccopy. ...

I've done a little more testing on my machine with 24 GB of memory using nccopy 
with
smaller use of memory, so can offer somewhat better advice about rechunking 
your data 
to improve access times.

> After a holiday and a break from this work, I was finally able to have a look 
> at it again. Unfortunately, the fix doesn't seem to work for me :-( It is 
> still the same problem as previously - the copy starts out fine, but gets 
> progressively slower and slower, and ultimately "hangs". Here is the command 
> that I am using:
>
> ./nccopy -u -k3 -d1 -m 2G -h 18G -e 10001 -c time/1698,longitude/6,latitude/7 
> combined_snapshots.nc temporal_read_optimised.nc
>
> > I am wondering whether I am setting the -h and -e options correctly? How 
> > should these be set? I'm not sure I understand the difference between them.

I've had a chance to try the same size output chunks, and think the
slow performance you were seeing was due to using too small a value
for the -e option, the number of elements in the chunk cache.  Using
nccopy as follows (first with lots of memory), on my simulated data 
it takes about 16.5 minutes:

  $ /usr/bin/time nccopy -k3 -d1 -m 2G -h 18G -e 61501 -c 
time/1698,longitude/6,latitude/7 in3.nc out.nc
  685.06user 35.42system 16:24.88elapsed 73%CPU (0avgtext+0avgdata 
19547208maxresident)k
  34244584inputs+16136200outputs (55major+4886962minor)pagefaults 0swaps

It turns out you can tradeoff memory for time in a couple of ways.
First, you can just specify a smaller chunk cache (say half the size,
9GB) and it will take somewhat longer but still not hang, as long as
the number of elements in the chunk cache remains adequate:

  $ ./clear_cache.sh;/usr/bin/time nccopy -k3 -d1 -m 1G -h 18G -e 65000 -c 
time/1698,longitude/6,latitude/7 in.nc out.nc
  693.02user 32.50system 16:57.19elapsed 71%CPU (0avgtext+0avgdata 
18570612maxresident)k
  34288536inputs+16155560outputs (739major+4724291minor)pagefaults 0swaps

  $ ./clear_cache.sh;/usr/bin/time nccopy -k3 -d1 -m 1G -h 9G -e 65000 -c 
time/1698,longitude/6,latitude/7 in.nc out.nc
  757.87user 32.74system 18:28.55elapsed 71%CPU (0avgtext+0avgdata 
18570384maxresident)k
  34753296inputs+16152264outputs (8543major+4722970minor)pagefaults 0swaps

  $ ./clear_cache.sh;/usr/bin/time nccopy -k3 -d1 -m 1G -h 4.5G -e 65000 -c 
time/1698,longitude/6,latitude/7 in.nc out.nc
  777.19user 32.96system 18:08.22elapsed 74%CPU (0avgtext+0avgdata 
18570384maxresident)k
  34271496inputs+16141928outputs (189major+4643325minor)pagefaults 0swaps
I can't explain why the last example with only 4.5GB of chunk cache was faster
than the example that used a 9 GB chunk cache, unless it had to do with machine 
load from other processes.

(Note: I'm clearing the disk cache in memory between timings to make
 sure later runs aren't just using cached disk pages in memory to get
 faster times by avoiding I/O.  The shell script clear_cache.sh is:
 #!/bin/bash -x
 # Clear the disk caches.
 sync
 sudo sh -c "echo 3 > /proc/sys/vm/drop_caches"
 :End Note)

Second, you could specify the output chunks such that instead of
accessing all the times for a location with a single chunk, you use 2
or more chunks along the time dimension.  Maybe accessing 2 or 3
chunks from the file would provide acceptable client response.  Less
memory is required in this case, because you don't have to hold all of
the file in memory at once to rechunk it.  For example, here's
rechunking with 2 and then 3 chunks in the time dimension , only using
4GB of memory for chunk cache, still specifying -e 65000.  It uses
less memory, but takes more time to write more compressed chunks, and
the resulting compression may not be as good:

Using 849x6x7 chunks:
  $ ./clear_cache.sh; /usr/bin/time nccopy -k3 -d1 -m 1G -h 4G -e 65000 -c 
time/849,longitude/6,latitude/7 in.nc out.nc
  876.84user 37.56system 19:58.29elapsed 76%CPU (0avgtext+0avgdata 
10330084maxresident)k
  34254768inputs+16163344outputs (65major+6940953minor)pagefaults 0swaps

Using 566x6x7 chunks:
  $ ./clear_cache.sh; /usr/bin/time nccopy -k3 -d1 -m 1G -h 4G -e 65000 -c 
time/566,longitude/6,latitude/7 in.nc out.nc
  1146.78user 27.09system 24:08.43elapsed 81%CPU (0avgtext+0avgdata 
7446824maxresident)k
  34255448inputs+16189832outputs (68major+1861851minor)pagefaults 0swaps

> ... In the end, I have decided to make the change in two steps, the first to 
> remove the record dimension, and the second to rechunk it. The first command 
> works fine, but it is the rechunking that is being problematic. Here is the 
> command:
> 
> ./nccopy -m 1G -h 1G -e 10001 -c time/1698,longitude/24,latitude/25 
> intermediate.nc temporal_read_optimised.nc
> 
> Some strangeness occurs with this though, in that it ends up eating 7.5GB of 
> RAM (I would have thought it should use 2GB at maximum) and essentially kills 
> the machine. Changing -e to 1000 at least lets it run ok  (but with memory 
> useage of 4.8GB (!!!??) but it is very slow to produce output  - after 
> several hours less than 10% of the data has been written to disk.

I can't explain that behavior.  It doesn't match what I would expect from the 
timings
I tried.

> I would be interested to try to do the rechunking with your file, to see how 
> the results compare - It sounds like your machine has an awful lot more RAM 
> than mine, which might explain the difference. Is there any easy for you to 
> get me the file, or maybe send a script to generate a comparable one?

Yes, I'm attaching the program I used to generate the netCDF-3 input file I'm 
using.
It was created by running "ncgen -c" on your original CDL to produce a C 
program that
would generate the same metadata, modifying the main variable to be 4-byte ints 
because
I thought I could predict int compression better than floats.  (I don't think 
the fact
that the big variable is ints is relevant to timing differences.)

My timings showed it didn't make much difference whether it used a record 
dimension 
or was a netCDF-4 compressed file.  Also apparently your data compresses much 
better
than this simulated data, which only gets about 3-to-1 compression.

--Russ

> Hi Mark,
> 
> > After a holiday and a break from this work, I was finally able to have a 
> > look at it again. Unfortunately, the fix doesn't seem to work for me :-( It 
> > is still the same problem as previously - the copy starts out fine, but 
> > gets progressively slower and slower, and ultimately "hangs". Here is the 
> > command that I am using:
> >
> > ./nccopy -u -k3 -d1 -m 2G -h 18G -e 10001 -c 
> > time/1698,longitude/6,latitude/7 combined_snapshots.nc 
> > temporal_read_optimised.nc
> >
> > I am wondering whether I am setting the -h and -e options correctly? How 
> > should these be set? I'm not sure I understand the difference between them.
> 
> It looks to me like you are setting -h correctly.  Since you are using 
> smaller sizes for the longitude and latitude dimensions than I
> used in my tests (6 and 7 versus 24 and 25), you will have about 14.3 times 
> as many chunks as I used (I was aiming at each chunk
> being about 4 MB), so you could set the number of elements in the chunk cache 
> higher (61446 instead of 4301).  I had used 10001
> elements in the chunk cache to be generously larger than 4301, but I think 
> it's not too critical as long as the number of elements
> in the chunk cache is larger than the number of cache elements you need.  
> Since you are compressing the data and reordering it in
> a way that requires *all* the chunks in memory at once, you need to use at 
> least "-e 61446", and to be generous should probably
> use something like "-e 61500".  The HDF5 documentation recommends that the 
> number of elements in the chunk cache should be
> prime, but I don't see the necessity for that and haven't noticed any 
> difference whether it's prime or composite.  With the current
> setting of "-e 1001", chunks that are only partly written will have to be 
> ejected from the cache to make room for new chunks, and
> this will lead to lots of unnecessary recompressing of chunks that are 
> ejected before writing them to disk, as well as uncompressing
> partially written chunks when reading them into the chunk cache.
> 
> You also need to make sure that your computer has enough memory to hold the 
> chunk cache in memory.  You've specified a 2GB
> input buffer and 18GB of chunk cache memory, so you should have at least 20GB 
> of memory for nccopy to run,  keeping
> the data in the chunk cache uncompressed while reordering it.  You might get 
> by with a smaller input buffer, say 11MB (one time of
> 1617*1596*4 bytes) and a somewhat smaller chunk cache, "-h 17.53G", if you're 
> close to the maximum.
> 
> > The combined_snapshots.nc file is 630MB - a dump of the header is given 
> > below:
> 
> My tests have been with simulated data of the same size as you're using, but 
> my simulated data may compress better than yours.
> If you could possibly make your actual combined_snapshots.nc file available 
> somewhere for me to test nccopy on the actual data,
> I could make sure I can reproduce something like the 15 minute times I'm 
> seeing for the copy and rechunking.  It may be your
> use of 1698x7x6 chunks requires more time than the larger 1698x25x24 chunks I 
> was writing, so I could try that as well.
> 
> > Any ideas?
> 
> I really can't explain what looks like the O(n**2) behavior you seem to be 
> seeing in writing the output, unless it's something in
> the HDF5 layer involving a performance bug in the B-trees that index the 
> chunks.  You can't really judge the progress in writing
> the output file by the size of the output, as none of the chunks are complete 
> until the end of the copy.  So the output file should
> stay fairly small until all of the chunks are flushed to disk (while being 
> compressed) at the end of the rechunking.
> 
> Also the -h and -e options to nccopy have only been minimally tested, and 
> there could still be bugs ...
> 
> --Russ
> 
> > [mpayne@oleander compiler]$ ncdump combined_snapshots.nc -h -c
> > netcdf combined_snapshots {
> > dimensions:
> > latitude = 1617 ;
> > longitude = 1596 ;
> > time = UNLIMITED ; // (1698 currently)
> > variables:
> > float chl_oc5(time, latitude, longitude) ;
> > chl_oc5:_FillValue = 0.f ;
> > chl_oc5:long_name = "Chlorophyll-a concentration in sea water using the OC5 
> > algorithm" ;
> > chl_oc5:standard_name = "mass_concentration_of_chlorophyll_a_in_sea_water" ;
> > chl_oc5:grid_mapping = "mercator" ;
> > chl_oc5:units = "milligram m-3" ;
> > chl_oc5:missing_value = 0.f ;
> > chl_oc5:units_nonstandard = "mg m^-3" ;
> > float latitude(latitude) ;
> > latitude:_FillValue = -999.f ;
> > latitude:standard_name = "latitude" ;
> > latitude:long_name = "latitude" ;
> > latitude:valid_min = -90. ;
> > latitude:units = "degrees_north" ;
> > latitude:valid_max = 90. ;
> > latitude:axis = "Y" ;
> > float longitude(longitude) ;
> > longitude:_FillValue = -999.f ;
> > longitude:standard_name = "longitude" ;
> > longitude:long_name = "longitude" ;
> > longitude:valid_min = -180. ;
> > longitude:units = "degrees_east" ;
> > longitude:valid_max = 180. ;
> > longitude:axis = "X" ;
> > int mercator ;
> > mercator:false_easting = 0L ;
> > mercator:standard_parallel = 0L ;
> > mercator:grid_mapping_name = "mercator" ;
> > mercator:false_northing = 0L ;
> > mercator:longitude_of_projection_origin = 0L ;
> > double time(time) ;
> > time:_FillValue = -1. ;
> > time:time_origin = "1970-01-01 00:00:00" ;
> > time:valid_min = 0. ;
> > time:long_name = "time" ;
> > time:standard_name = "time" ;
> > time:units = "seconds since 1970-01-01 00:00:00" ;
> > time:calendar = "gregorian" ;
> > time:axis = "T" ;
> >
> > // global attributes:
> > :site_name = "UK Shelf Seas" ;
> > :citation = "If you use this data towards any publication, please 
> > acknowledge this using: \'The authors thank the NERC Earth Observation Data 
> > Acquisition and Analysis Service (NEODAAS) for supplying data for this 
> > study\' and then email NEODAAS (address@hidden) with the details. The 
> > service relies on users\' publications as one measure of success." ;
> > :creation_date = "Thu Jun 02 10:51:37 2011" ;
> > :easternmost_longitude = 13. ;
> > :creator_url = "http://rsg.pml.ac.uk"; ;
> > :references = "See NEODAAS webpages at http://www.neodaas.ac.uk/ or RSG 
> > pages at http://rsg.pml.ac.uk/"; ;
> > :Metadata_Conventions = "Unidata Dataset Discovery v1.0" ;
> > :keywords = "satellite,observation,ocean" ;
> > :summary = "This data is Level-3 satellite observation data (Level 3 
> > meaning raw observations processedto geophysical quantities, and placed 
> > onto a regular grid)." ;
> > :id = 
> > "M2010001.1235.uk.postproc_products.MYO.01jan101235.v1.20111530951.data.nc" 
> > ;
> > :naming_authority = "uk.ac.pml" ;
> > :geospatial_lat_max = 62.999108 ;
> > :title = "Level-3 satellite data from Moderate Resolution Imaging 
> > Spectroradiometer sensor" ;
> > :source = "Moderate Resolution Imaging Spectroradiometer" ;
> > :northernmost_latitude = 62.999108 ;
> > :creator_name = "Plymouth Marine Laboratory Remote Sensing Group" ;
> > :processing_level = "Level-3 (NASA EOS Conventions)" ;
> > :creator_email = "address@hidden" ;
> > :netcdf_library_version = "4.0.1 of Sep  3 2010 11:27:29 $" ;
> > :date_issued = "Thu Jun 02 10:51:37 2011" ;
> > :geospatial_lat_min = 47. ;
> > :date_created = "Thu Jun 02 10:51:37 2011" ;
> > :institution = "Plymouth Marine Laboratory Remote Sensing Group" ;
> > :geospatial_lon_max = 13. ;
> > :geospatial_lon_min = -15. ;
> > :contact1 = "email: address@hidden" ;
> > :license = "If you use this data towards any publication, please 
> > acknowledge this using: \'The authors thank the NERC Earth Observation Data 
> > Acquisition and Analysis Service (NEODAAS) for supplying data for this 
> > study\' and then email NEODAAS (address@hidden) with the details. The 
> > service relies on users\' publications as one measure of success." ;
> > :Conventions = "CF-1.4" ;
> > :project = "NEODAAS (NERC Earth Observation Data Acquisition and Analysis 
> > Service)" ;
> > :cdm_data_type = "Grid" ;
> > :RSG_sensor = "MODIS" ;
> > :westernmost_longitude = -15. ;
> > :RSG_areacode = "uk" ;
> > :southernmost_latitude = 47. ;
> > :netcdf_file_type = "NETCDF4_CLASSIC" ;
> > :history = "Created during RSG Standard Mapping (Mapper) [SGE Job Number: 
> > 2577153]" ;
> > :NCO = "4.0.7" ;
> > }
> > [mpayne@oleander compiler]$
> >
> >
> 
> Russ Rew                                         UCAR Unidata Program
> address@hidden                      http://www.unidata.ucar.edu
> 
> 
> 
> Ticket Details
> ===================
> Ticket ID: CUV-251255
> Department: Support netCDF
> Priority: Normal
> Status: Closed
> 
> 

Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: CUV-251255
Department: Support netCDF
Priority: Normal
Status: Closed
#include <stdio.h>
#include <stdlib.h>
#include <netcdf.h>

#define FILENAME "in.nc"
#define LAT_LEN 1617
#define LON_LEN 1596
#define TIME_LEN 1698
/* #define TIME_LEN 1698 */
/* #define TIME_LEN 10 */
/* #define LAT_LEN 160 */
/* #define LON_LEN 150 */

void
check_err(const int stat, const int line, const char *file) {
    if (stat != NC_NOERR) {
        (void)fprintf(stderr,"line %d of %s: %s\n", line, file, 
nc_strerror(stat));
        fflush(stderr);
        exit(1);
    }
}

int
main(int argc, char *argv[]) {/* create in.nc for testing */

    int  stat;  /* return status */
    int  ncid;  /* netCDF id */

    /* dimension ids */
    int latitude_dim;
    int longitude_dim;
    int time_dim;

    /* dimension lengths */
    size_t latitude_len = LAT_LEN;
    size_t longitude_len = LON_LEN;
    size_t time_len = TIME_LEN; /* changed from NC_UNLIMITED so time coordinate 
not chunked */

    /* variable ids */
    int chl_oc5_id;
    int latitude_id;
    int longitude_id;
    int mercator_id;
    int time_id;

    /* rank (number of dimensions) for each variable */
#   define RANK_chl_oc5 3
#   define RANK_latitude 1
#   define RANK_longitude 1
#   define RANK_mercator 0
#   define RANK_time 1

    /* variable shapes */
    int chl_oc5_dims[RANK_chl_oc5];
    int latitude_dims[RANK_latitude];
    int longitude_dims[RANK_longitude];
    int time_dims[RANK_time];

    /* enter define mode */
    stat = nc_create(FILENAME, NC_CLOBBER, &ncid);
    check_err(stat,__LINE__,__FILE__);

    /* define dimensions */
    stat = nc_def_dim(ncid, "latitude", latitude_len, &latitude_dim);
    check_err(stat,__LINE__,__FILE__);
    stat = nc_def_dim(ncid, "longitude", longitude_len, &longitude_dim);
    check_err(stat,__LINE__,__FILE__);
    stat = nc_def_dim(ncid, "time", time_len, &time_dim);
    check_err(stat,__LINE__,__FILE__);

    /* define variables */

    latitude_dims[0] = latitude_dim;
    stat = nc_def_var(ncid, "latitude", NC_FLOAT, RANK_latitude, latitude_dims, 
&latitude_id);
    check_err(stat,__LINE__,__FILE__);

    longitude_dims[0] = longitude_dim;
    stat = nc_def_var(ncid, "longitude", NC_FLOAT, RANK_longitude, 
longitude_dims, &longitude_id);
    check_err(stat,__LINE__,__FILE__);

    stat = nc_def_var(ncid, "mercator", NC_INT, RANK_mercator, 0, &mercator_id);
    check_err(stat,__LINE__,__FILE__);

    time_dims[0] = time_dim;
    stat = nc_def_var(ncid, "time", NC_DOUBLE, RANK_time, time_dims, &time_id);
    check_err(stat,__LINE__,__FILE__);

    chl_oc5_dims[0] = time_dim;
    chl_oc5_dims[1] = latitude_dim;
    chl_oc5_dims[2] = longitude_dim;
    stat = nc_def_var(ncid, "chl_oc5", NC_INT, RANK_chl_oc5, chl_oc5_dims, 
&chl_oc5_id);
    check_err(stat,__LINE__,__FILE__);

    /* assign global attributes */
    { /* site_name */
    stat = nc_put_att_text(ncid, NC_GLOBAL, "site_name", 13, "UK Shelf Seas");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* citation */
    stat = nc_put_att_text(ncid, NC_GLOBAL, "citation", 336, "If you use this 
data towards any publication, please acknowledge this using: 'The authors thank 
the NERC Earth Observation Data Acquisition and Analysis Service (NEODAAS) for 
supplying data for this study' and then email NEODAAS (address@hidden) with the 
details. The service relies on users' publications as one measure of success.");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* creation_date */
    stat = nc_put_att_text(ncid, NC_GLOBAL, "creation_date", 24, "Thu Jun 02 
10:51:37 2011");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* easternmost_longitude */
    static const double easternmost_longitude_att[1] = {13} ;
    stat = nc_put_att_double(ncid, NC_GLOBAL, "easternmost_longitude", 
NC_DOUBLE, 1, easternmost_longitude_att);
    check_err(stat,__LINE__,__FILE__);
    }
    { /* creator_url */
    stat = nc_put_att_text(ncid, NC_GLOBAL, "creator_url", 20, 
"http://rsg.pml.ac.uk";);
    check_err(stat,__LINE__,__FILE__);
    }
    { /* references */
    stat = nc_put_att_text(ncid, NC_GLOBAL, "references", 87, "See NEODAAS 
webpages at http://www.neodaas.ac.uk/ or RSG pages at http://rsg.pml.ac.uk/";);
    check_err(stat,__LINE__,__FILE__);
    }
    { /* Metadata_Conventions */
    stat = nc_put_att_text(ncid, NC_GLOBAL, "Metadata_Conventions", 30, 
"Unidata Dataset Discovery v1.0");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* keywords */
    stat = nc_put_att_text(ncid, NC_GLOBAL, "keywords", 27, 
"satellite,observation,ocean");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* summary */
    stat = nc_put_att_text(ncid, NC_GLOBAL, "summary", 150, "This data is 
Level-3 satellite observation data (Level 3 meaning raw observations 
processedto geophysical quantities, and placed onto a regular grid).");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* id */
    stat = nc_put_att_text(ncid, NC_GLOBAL, "id", 73, 
"M2010001.1235.uk.postproc_products.MYO.01jan101235.v1.20111530951.data.nc");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* naming_authority */
    stat = nc_put_att_text(ncid, NC_GLOBAL, "naming_authority", 9, "uk.ac.pml");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* geospatial_lat_max */
    static const double geospatial_lat_max_att[1] = {62.999108} ;
    stat = nc_put_att_double(ncid, NC_GLOBAL, "geospatial_lat_max", NC_DOUBLE, 
1, geospatial_lat_max_att);
    check_err(stat,__LINE__,__FILE__);
    }
    { /* title */
    stat = nc_put_att_text(ncid, NC_GLOBAL, "title", 80, "Level-3 satellite 
data from Moderate Resolution Imaging Spectroradiometer sensor");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* source */
    stat = nc_put_att_text(ncid, NC_GLOBAL, "source", 45, "Moderate Resolution 
Imaging Spectroradiometer");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* northernmost_latitude */
    static const double northernmost_latitude_att[1] = {62.999108} ;
    stat = nc_put_att_double(ncid, NC_GLOBAL, "northernmost_latitude", 
NC_DOUBLE, 1, northernmost_latitude_att);
    check_err(stat,__LINE__,__FILE__);
    }
    { /* creator_name */
    stat = nc_put_att_text(ncid, NC_GLOBAL, "creator_name", 47, "Plymouth 
Marine Laboratory Remote Sensing Group");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* processing_level */
    stat = nc_put_att_text(ncid, NC_GLOBAL, "processing_level", 30, "Level-3 
(NASA EOS Conventions)");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* creator_email */
    stat = nc_put_att_text(ncid, NC_GLOBAL, "creator_email", 17, 
"address@hidden");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* netcdf_library_version */
    stat = nc_put_att_text(ncid, NC_GLOBAL, "netcdf_library_version", 31, 
"4.0.1 of Sep  3 2010 11:27:29 $");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* date_issued */
    stat = nc_put_att_text(ncid, NC_GLOBAL, "date_issued", 24, "Thu Jun 02 
10:51:37 2011");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* geospatial_lat_min */
    static const double geospatial_lat_min_att[1] = {47} ;
    stat = nc_put_att_double(ncid, NC_GLOBAL, "geospatial_lat_min", NC_DOUBLE, 
1, geospatial_lat_min_att);
    check_err(stat,__LINE__,__FILE__);
    }
    { /* date_created */
    stat = nc_put_att_text(ncid, NC_GLOBAL, "date_created", 24, "Thu Jun 02 
10:51:37 2011");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* institution */
    stat = nc_put_att_text(ncid, NC_GLOBAL, "institution", 47, "Plymouth Marine 
Laboratory Remote Sensing Group");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* geospatial_lon_max */
    static const double geospatial_lon_max_att[1] = {13} ;
    stat = nc_put_att_double(ncid, NC_GLOBAL, "geospatial_lon_max", NC_DOUBLE, 
1, geospatial_lon_max_att);
    check_err(stat,__LINE__,__FILE__);
    }
    { /* geospatial_lon_min */
    static const double geospatial_lon_min_att[1] = {-15} ;
    stat = nc_put_att_double(ncid, NC_GLOBAL, "geospatial_lon_min", NC_DOUBLE, 
1, geospatial_lon_min_att);
    check_err(stat,__LINE__,__FILE__);
    }
    { /* contact1 */
    stat = nc_put_att_text(ncid, NC_GLOBAL, "contact1", 24, "email: 
address@hidden");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* license */
    stat = nc_put_att_text(ncid, NC_GLOBAL, "license", 336, "If you use this 
data towards any publication, please acknowledge this using: 'The authors thank 
the NERC Earth Observation Data Acquisition and Analysis Service (NEODAAS) for 
supplying data for this study' and then email NEODAAS (address@hidden) with the 
details. The service relies on users' publications as one measure of success.");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* Conventions */
    stat = nc_put_att_text(ncid, NC_GLOBAL, "Conventions", 6, "CF-1.4");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* project */
    stat = nc_put_att_text(ncid, NC_GLOBAL, "project", 70, "NEODAAS (NERC Earth 
Observation Data Acquisition and Analysis Service)");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* cdm_data_type */
    stat = nc_put_att_text(ncid, NC_GLOBAL, "cdm_data_type", 4, "Grid");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* RSG_sensor */
    stat = nc_put_att_text(ncid, NC_GLOBAL, "RSG_sensor", 5, "MODIS");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* westernmost_longitude */
    static const double westernmost_longitude_att[1] = {-15} ;
    stat = nc_put_att_double(ncid, NC_GLOBAL, "westernmost_longitude", 
NC_DOUBLE, 1, westernmost_longitude_att);
    check_err(stat,__LINE__,__FILE__);
    }
    { /* RSG_areacode */
    stat = nc_put_att_text(ncid, NC_GLOBAL, "RSG_areacode", 2, "uk");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* southernmost_latitude */
    static const double southernmost_latitude_att[1] = {47} ;
    stat = nc_put_att_double(ncid, NC_GLOBAL, "southernmost_latitude", 
NC_DOUBLE, 1, southernmost_latitude_att);
    check_err(stat,__LINE__,__FILE__);
    }
    { /* netcdf_file_type */
    stat = nc_put_att_text(ncid, NC_GLOBAL, "netcdf_file_type", 15, 
"NETCDF4_CLASSIC");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* history */
    stat = nc_put_att_text(ncid, NC_GLOBAL, "history", 70, "Created during RSG 
Standard Mapping (Mapper) [SGE Job Number: 2577153]");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* NCO */
    stat = nc_put_att_text(ncid, NC_GLOBAL, "NCO", 5, "4.0.7");
    check_err(stat,__LINE__,__FILE__);
    }


    /* assign per-variable attributes */
    { /* _FillValue */
    static const int chl_oc5_FillValue_att[1] = {0} ;
    stat = nc_put_att_int(ncid, chl_oc5_id, "_FillValue", NC_INT, 1, 
chl_oc5_FillValue_att);
    check_err(stat,__LINE__,__FILE__);
    }
    { /* long_name */
    stat = nc_put_att_text(ncid, chl_oc5_id, "long_name", 64, "Chlorophyll-a 
concentration in sea water using the OC5 algorithm");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* standard_name */
    stat = nc_put_att_text(ncid, chl_oc5_id, "standard_name", 48, 
"mass_concentration_of_chlorophyll_a_in_sea_water");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* grid_mapping */
    stat = nc_put_att_text(ncid, chl_oc5_id, "grid_mapping", 8, "mercator");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* units */
    stat = nc_put_att_text(ncid, chl_oc5_id, "units", 13, "milligram m-3");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* missing_value */
    static const float chl_oc5_missing_value_att[1] = {0} ;
    stat = nc_put_att_float(ncid, chl_oc5_id, "missing_value", NC_FLOAT, 1, 
chl_oc5_missing_value_att);
    check_err(stat,__LINE__,__FILE__);
    }
    { /* units_nonstandard */
    stat = nc_put_att_text(ncid, chl_oc5_id, "units_nonstandard", 7, "mg m^-3");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* _FillValue */
    static const float latitude_FillValue_att[1] = {-999} ;
    stat = nc_put_att_float(ncid, latitude_id, "_FillValue", NC_FLOAT, 1, 
latitude_FillValue_att);
    check_err(stat,__LINE__,__FILE__);
    }
    { /* standard_name */
    stat = nc_put_att_text(ncid, latitude_id, "standard_name", 8, "latitude");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* long_name */
    stat = nc_put_att_text(ncid, latitude_id, "long_name", 8, "latitude");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* valid_min */
    static const double latitude_valid_min_att[1] = {-90} ;
    stat = nc_put_att_double(ncid, latitude_id, "valid_min", NC_DOUBLE, 1, 
latitude_valid_min_att);
    check_err(stat,__LINE__,__FILE__);
    }
    { /* units */
    stat = nc_put_att_text(ncid, latitude_id, "units", 13, "degrees_north");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* valid_max */
    static const double latitude_valid_max_att[1] = {90} ;
    stat = nc_put_att_double(ncid, latitude_id, "valid_max", NC_DOUBLE, 1, 
latitude_valid_max_att);
    check_err(stat,__LINE__,__FILE__);
    }
    { /* axis */
    stat = nc_put_att_text(ncid, latitude_id, "axis", 1, "Y");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* _FillValue */
    static const float longitude_FillValue_att[1] = {-999} ;
    stat = nc_put_att_float(ncid, longitude_id, "_FillValue", NC_FLOAT, 1, 
longitude_FillValue_att);
    check_err(stat,__LINE__,__FILE__);
    }
    { /* standard_name */
    stat = nc_put_att_text(ncid, longitude_id, "standard_name", 9, "longitude");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* long_name */
    stat = nc_put_att_text(ncid, longitude_id, "long_name", 9, "longitude");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* valid_min */
    static const double longitude_valid_min_att[1] = {-180} ;
    stat = nc_put_att_double(ncid, longitude_id, "valid_min", NC_DOUBLE, 1, 
longitude_valid_min_att);
    check_err(stat,__LINE__,__FILE__);
    }
    { /* units */
    stat = nc_put_att_text(ncid, longitude_id, "units", 12, "degrees_east");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* valid_max */
    static const double longitude_valid_max_att[1] = {180} ;
    stat = nc_put_att_double(ncid, longitude_id, "valid_max", NC_DOUBLE, 1, 
longitude_valid_max_att);
    check_err(stat,__LINE__,__FILE__);
    }
    { /* axis */
    stat = nc_put_att_text(ncid, longitude_id, "axis", 1, "X");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* false_easting */
    static const int mercator_false_easting_att[1] = {0} ;
    stat = nc_put_att_int(ncid, mercator_id, "false_easting", NC_INT, 1, 
mercator_false_easting_att);
    check_err(stat,__LINE__,__FILE__);
    }
    { /* standard_parallel */
    static const int mercator_standard_parallel_att[1] = {0} ;
    stat = nc_put_att_int(ncid, mercator_id, "standard_parallel", NC_INT, 1, 
mercator_standard_parallel_att);
    check_err(stat,__LINE__,__FILE__);
    }
    { /* grid_mapping_name */
    stat = nc_put_att_text(ncid, mercator_id, "grid_mapping_name", 8, 
"mercator");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* false_northing */
    static const int mercator_false_northing_att[1] = {0} ;
    stat = nc_put_att_int(ncid, mercator_id, "false_northing", NC_INT, 1, 
mercator_false_northing_att);
    check_err(stat,__LINE__,__FILE__);
    }
    { /* longitude_of_projection_origin */
    static const int mercator_longitude_of_projection_origin_att[1] = {0} ;
    stat = nc_put_att_int(ncid, mercator_id, "longitude_of_projection_origin", 
NC_INT, 1, mercator_longitude_of_projection_origin_att);
    check_err(stat,__LINE__,__FILE__);
    }
    { /* _FillValue */
    static const double time_FillValue_att[1] = {-1} ;
    stat = nc_put_att_double(ncid, time_id, "_FillValue", NC_DOUBLE, 1, 
time_FillValue_att);
    check_err(stat,__LINE__,__FILE__);
    }
    { /* time_origin */
    stat = nc_put_att_text(ncid, time_id, "time_origin", 19, "1970-01-01 
00:00:00");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* valid_min */
    static const double time_valid_min_att[1] = {0} ;
    stat = nc_put_att_double(ncid, time_id, "valid_min", NC_DOUBLE, 1, 
time_valid_min_att);
    check_err(stat,__LINE__,__FILE__);
    }
    { /* long_name */
    stat = nc_put_att_text(ncid, time_id, "long_name", 4, "time");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* standard_name */
    stat = nc_put_att_text(ncid, time_id, "standard_name", 4, "time");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* units */
    stat = nc_put_att_text(ncid, time_id, "units", 33, "seconds since 
1970-01-01 00:00:00");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* calendar */
    stat = nc_put_att_text(ncid, time_id, "calendar", 9, "gregorian");
    check_err(stat,__LINE__,__FILE__);
    }
    { /* axis */
    stat = nc_put_att_text(ncid, time_id, "axis", 1, "T");
    check_err(stat,__LINE__,__FILE__);
    }


    /* leave define mode */
    stat = nc_enddef (ncid);
    check_err(stat,__LINE__,__FILE__);

    /* assign variable data */
    {
        float latitude_data[LAT_LEN] ;
        size_t latitude_startset[1] = {0} ;
        size_t latitude_countset[1] = {LAT_LEN} ;
        int lat;
        double start = 63.0;
        double end = 47.0;
        double delta = (start - end) / (LAT_LEN - 1);
        for (lat=0; lat < LAT_LEN; lat++) {
            latitude_data[lat] = start + lat * delta;
        }
        stat = nc_put_vara(ncid, latitude_id, latitude_startset, 
latitude_countset, latitude_data);
        check_err(stat,__LINE__,__FILE__);
    }
    
    {
        float longitude_data[LON_LEN];
        size_t longitude_startset[1] = {0} ;
        size_t longitude_countset[1] = {LON_LEN} ;
        int lon;
        double start = -15.0;
        double end = 13.0;
        double delta = (start - end) / (LAT_LEN - 1);
        for (lon=0; lon < LON_LEN; lon++) {
            longitude_data[lon] = start + lon * delta;
        }
        stat = nc_put_vara(ncid, longitude_id, longitude_startset, 
longitude_countset, longitude_data);
        check_err(stat,__LINE__,__FILE__);
    }

    {
    double time_data[TIME_LEN] = {1262349300} ;
    size_t time_startset[1] = {0} ;
    size_t time_countset[1] = {TIME_LEN} ;
    double start = 1262349300;
    double delta = 300;
    int time;
    for (time  = 0; time < TIME_LEN; time++) {
        time_data[time] = start + time * delta;
    }
    stat = nc_put_vara(ncid, time_id, time_startset, time_countset, time_data);
    check_err(stat,__LINE__,__FILE__);
    }

    {
        /* double chl_oc5_data[LAT_LEN][LON_LEN] ; */
        size_t chl_oc5_startset[3] = {0, 0, 0} ;
        size_t chl_oc5_countset[3] = {1, LAT_LEN, LON_LEN} ;
        int lat, lon, time;
        int *chl_oc5_data = malloc(sizeof(int) * LAT_LEN * LON_LEN);
        int val = 0;
        
        for(time = 0; time < TIME_LEN; time++) {
            size_t ll = 0;
            chl_oc5_startset[0] = time;
            for(lat = 0; lat < LAT_LEN; lat++)
                for(lon = 0; lon < LON_LEN; lon++)
                    chl_oc5_data[ll++] = val++;
            stat = nc_put_vara_int(ncid, chl_oc5_id, chl_oc5_startset, 
chl_oc5_countset, chl_oc5_data);
            check_err(stat,__LINE__,__FILE__);
        }
    }

    stat = nc_close(ncid);
    check_err(stat,__LINE__,__FILE__);
    return 0;
}