Re: [netcdfgroup] How convert csv files to a netcdf file

  • To: Dave Allured - NOAA Affiliate <dave.allured@xxxxxxxx>, nazaniti6 <nazitkk1375@xxxxxxxxx>
  • Subject: Re: [netcdfgroup] How convert csv files to a netcdf file
  • From: Joe Lee <hyoklee@xxxxxxxxxxxx>
  • Date: Tue, 15 Dec 2020 21:56:19 +0000
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=hdfgroup.org; dmarc=pass action=none header.from=hdfgroup.org; dkim=pass header.d=hdfgroup.org; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=TDEWNAnp7a/YoGqrftkZ8Wpe4fkl1u405JA2EVZ2hc8=; b=m/EE0J61TLymEq0IDObJN3e3TbCGAQkPv8QKyMbmtyOop3q2twFa7zee00lEpjTXWVM+6xJb43XzuXy/hkyr1mxWmrWwcbBM2VnhkdAUUl1oauF1U6lVYhp1ErNl5Ka30QBycwhQvfpTaFD/wSr+J8B9HZ1PfxSjWTogGRVRdOKTdowPko++lHb8IDcM5hF7rSgQFp6ay1S++Up1+bMj9qbvwEwMpkHJLe31ScYEZXs+FYUWCGlhAtOa58iG1LV63Pl8IzE9S1BLhfote3Knbcd/zxVdXKzEdfT3u3NGYWGVDLsJBmhNpwarMxFnrDm0It+9Fal7/wNG2Lgw7iVzEA==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kShQ8b24gv/nBloykvYnBsV5RgOB2Ukr5xRPZ3Q2a8cvMqzuhh6LloMN3qV/rBPxXD/LmjeyqvwXAlVAuXr+0N4wTC8IO52OoVe9xg8TvGX8ZzrfPuf8hun6CUCuaPudg1BStiv0LDtJdVAnKPb1EwH8itFAVlXDn+w2F7YK3f5nN8RMn18KscrcKfZK8Sy4SYqTvdpEEPcW1tY1MJXGXWXRwFOnz8gIrYCNkHlvskemdUS+cLFGUmUbhsvAL+fmN0xno9wrBb1yuFn9tZeMLwLfC+ppkhH3kuS/qGGzCXB5WcarQ1RkYfjPOG/fKu5Plrg/18tfsrQQt9wYPPTfWg==
  • Authentication-results: noaa.gov; dkim=none (message not signed) header.d=none; noaa.gov; dmarc=none action=none header.from=hdfgroup.org;
You can easily create a template netCDF-4/HDF5 file using GUI called HDF 
Product Designer
and generate CDL [1] or Python code [2] that you can add data from CSV.

[1] https://hpd.readthedocs.io/en/v1.6/usage.html#id8
[2] https://hpd.readthedocs.io/en/v1.6/usage.html#python-source-code



From: netcdfgroup <netcdfgroup-bounces@xxxxxxxxxxxxxxxx> On Behalf Of Dave 
Allured - NOAA Affiliate via netcdfgroup
Sent: Tuesday, December 15, 2020 3:38 PM
To: nazaniti6 <nazitkk1375@xxxxxxxxx>
Cc: netcdfgroup@xxxxxxxxxxxxxxxx
Subject: Re: [netcdfgroup] How convert csv files to a netcdf file

I agree with Gus and Chris, a higher level language such as NCL or Python would 
be easiest if you are familiar and have access.  I would also be interested to 
know if Rosetta works.

If you have trouble with these, here is an alternative method that needs only 
an existing Netcdf file with the desired array structure and coordinates, plus 
Netcdf command line tools and a text editor.  There is a similar method that 
does not even need a previous Netcdf file.  Ask if needed.

* ncdump -c other-file.nc<http://other-file.nc> > cdl.txt
* Split cdl.txt into header and suffix (suffix is just one line).
* Add new line "tmp = " (or var name of your choice) at the end of the header.
* Change the header to match your chosen var name, if needed.
* Adjust other metadata as desired.
* Ensure correct delimiters.
* cat header.txt input.csv suffix.txt > concat.txt
* ncgen -o new.nc<http://new.nc> concat.txt

Some debugging will be expected.  Use ncdump to get familiar with what a 
complete CDL text file should look like.  You should learn this regardless of 
which method you use.  Good luck.


On Tue, Dec 15, 2020 at 1:30 PM Gus Correa 
<gus@xxxxxxxxxxxxxxxxx<mailto:gus@xxxxxxxxxxxxxxxxx>> wrote:
Hi

This is probably much easier to do using a higher level generic tool such as 
NCL,
than writing a file-specific program in C, Fortran, Python, whatever, using the 
NetCDF API.

NCL documentation includes examples on how to read CSV files:
https://www.ncl.ucar.edu/Applications/read_csv.shtml
and how to write NetCDF files:
https://www.ncl.ucar.edu/Applications/method_1.shtml
https://www.ncl.ucar.edu/Applications/method_2.shtml

For atmosphere/ocean/climate/weather data there hardly anything better than NCL:
https://www.ncl.ucar.edu/index.shtml

If you run into problems, their mailing list is active and very helpful:
https://www.ncl.ucar.edu/Support/email_lists.shtml

I hope this helps,
Gus Correa

On Tue, Dec 15, 2020 at 12:58 PM nazaniti6 
<nazitkk1375@xxxxxxxxx<mailto:nazitkk1375@xxxxxxxxx>> wrote:
Hello,
I have six CSV files which I have sent in this link
(https://drive.google.com/drive/folders/1GQtyY1mI1YrK8GFP9SEo7dZfFHexKro3?usp=sharing).
If you open them you will understand that each of these files has 720 rows
and 360 columns which indicates longitude and latitude respectively. I
should tell you that the first point of latitude is -89.75, the first point
of longitude is -179.75, and the resolution of these data is 0.5 degrees. I
want to convert them to a NetCDF file which has the following properties:
Dimensions:
longitude = 720
latitude = 360
time = 6
Variables:
longitude
Size: 720x1
Dimensions: longitude
Datatype: single
Attributes:
long_name = 'longitude'
units = 'degrees_east'
latitude
Size: 360x1
Dimensions: latitude
Datatype: single
Attributes:
long_name = 'latitude'
units = 'degrees_north'
time
Size: 6x1
Dimensions: time
Datatype: single
Attributes:
long_name = 'time'
units = 'days since 1900-1-1'
calendar = 'gregorian'
tmp
Size: 720x360x6
Dimensions: longitude, latitude,time
Datatype: single
Attributes:
long_name = 'monthly mean temperature'
units = 'degrees Celsius'
correlation_decay_distance = 1200
_FillValue = 9.969209968386869e+36
missing_value = 9.969209968386869e+36
Could you please help me?
  • 2020 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: