[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #NUC-119714]: struggling with writing data to nc file



Hi Laurens,

> Hi Russ, this last change has indeed resulted in that the emission field is 
> now written away to the output file! The field is not yet properly written 
> (looks like a mix up of lon, lat, time) but the nc file is produced and I am 
> sure that I can solve this last minor issue. Anyhow, thanks for all this help 
> that has been for me a very good tutorial on using netcdf. So far, I have 
> been basically using scripts from others only making some minor 
> modifications. Now, with all your instructions/explanations I should know 
> much more about potential issues and how to solve these more structural 
> features of using netcdf also in F77. The manuals have been useful but 
> apparently do not always suffice to overcome some of these problems.

I'm glad to hear about progress solving this problem.

Another tool that may help in dealing with the F77 interface in the netCDF-3 
version you are using is "ncgen",
which should be available with your netCDF installation.  If you have a small 
CDL file, such as output from
ncdump, you can use ncgen to generate F77 source code that would create the 
corresponding netCDF file:

  ncgen -f foo.cdl > foo.f

Sometimes this is the easiest way to get the dimension order right when dealing 
with confusing issues 
of C/CDL "row-major" order versus Fortran "column-major" order.  Looking at the 
generated Fortran
code when there's at least a few rows of data in the CDL file helps with the 
declarations and Fortran
dimensions.

An example of a small CDL example file you can use to see this generated F77 
code is 

  http://www.unidata.ucar.edu/netcdf/examples/programs/pres_temp_4D.cdl

and you can compare the ncgen-generated F77 code using "ncgen -f" with a better 
commented
version here:

  http://www.unidata.ucar.edu/netcdf/examples/programs/pres_temp_4D_wr.f

Sometimes small working examples are better than reference documentation  for 
communicating
how the intended way to use a software package ...

> I guess you are in Boulder? I should then definitely once on one of my trips 
> to Boulder (I work together with INSTAAR/NCAR colleagues) treat you on a good 
> beer in the New Belgium brewing company...

I'm always up for beer :-) ...

--Russ

> Laurens
> =======================================================================
> Dr. ir. Laurens Ganzeveld
> Assistant professor
> Department of Environmental Sciences, Chair group Earth System Sciences
> Wageningen University and ResearchÂCentre
> Lumen, room D.001, Droevendaalsesteeg 4, 6708 PB, Wageningen, Netherlands
> Phone: (+31) 317-486651 / 613066079, Fax: (+31) 317-419000,
> Being in in Colombia: +57 310 2979313
> E-mail: address@hidden
> Â
> Affiliate of Universidad de Los Andes, Depart de BiologÃa, Bogota, Colombia
> Affiliate of Max Planck Institute for Chemistry, Depart. of Atmospheric 
> Chemistry, Mainz, Germany,
> =======================================================================
> 
> 
> -----Original Message-----
> From: Unidata netCDF Support [mailto:address@hidden
> Sent: woensdag 16 oktober 2013 22:42
> To: Ganzeveld, Laurens
> Cc: address@hidden
> Subject: [netCDF #NUC-119714]: struggling with writing data to nc file
> 
> Laurens,
> 
> I think I see the problem, which I should have spotted earlier.
> 
> In subroutine write_nc_file, where you have
> 
> cnt2d   = (/ 180,  360,              1 /)
> 
> it should be
> 
> cnt2d   = (/ 360,  180,              1 /)
> 
> because the order in which you defined the shape of the netCDF "emis"
> variable was (lon, lat, time), the Fortran order corresponding to the 
> reversed CDL/C order of the variable declaration
> 
> float emis(time, lat, lon) ;
> 
> With the order you specified, you should have gotten a run-time error message 
> something like "Start+count exceeds dimension bound" corresponding to error 
> number NF_EEDGE (-57).
> 
> Please let me know if that doesn't fix the problem ...
> 
> --Russ
> 
> > Hi Russ, thanks for extensive feedback; I went through all your
> > comments/suggestions and got your points also on reading in the file.
> > I checked it out and realized that the code was apparently then 12
> > times reading in the whole 4-D field (ilat, ilon, nlev, nmonths) and
> > have simplified the code getting the call to the reading outside the
> > timeloop. I also got your point on a problem assigning the output
> > emission fields and have modified this to be sure that the ilon and
> > ilat loop indices were properly used. Also using the start1d and cnt1d
> > code works fine and the script now properly writes the time array to
> > the new nc file. However, you said that it seems that then the actual
> > writing of the 2-D emission field inside this timeloop of 1,12 months
> > was properly coded but that's exactly where it still goes wrong. I
> > went again through all your previous emails, the manual that you
> > suggested for the netcdf3.6.2 version including the use of the
> > functions nf_put_vara_real etc but don't
> get this last part solved (a frustrating experience now that I am so close 
> but really want to completely get this since I will need to use this netcdf 
> reading and writing code for many more applications). I know that I am asking 
> a lot but have attached once more again the latest version including all 
> these modifications; if you happen to have another glimpse at it to see if 
> you see a remaining potential explanation for this problem I would really 
> appreciate it.
> >
> > Cheers, Laurens
> > ======================================================================
> > =
> > Dr. ir. Laurens Ganzeveld
> > Assistant professor
> > Department of Environmental Sciences, Chair group Earth System
> > Sciences Wageningen University and ResearchÂCentre Lumen, room D.001,
> > Droevendaalsesteeg 4, 6708 PB, Wageningen, Netherlands
> > Phone: (+31) 317-486651 / 613066079, Fax: (+31) 317-419000, Being in
> > in Colombia: +57 310 2979313
> > E-mail: address@hidden
> > Â
> > Affiliate of Universidad de Los Andes, Depart de BiologÃa, Bogota,
> > Colombia Affiliate of Max Planck Institute for Chemistry, Depart. of
> > Atmospheric Chemistry, Mainz, Germany,
> > ======================================================================
> > =
> >
> >
> > -----Original Message-----
> > From: Unidata netCDF Support [mailto:address@hidden
> > Sent: dinsdag 15 oktober 2013 18:45
> > To: Ganzeveld, Laurens
> > Cc: address@hidden
> > Subject: [netCDF #NUC-119714]: struggling with writing data to nc file
> >
> > > Below is the header information of this input file that I read with
> > > the script. But the reading seems to work all fine. The problems are
> > > involved in then simply writing out the read data which I want to do
> > > to be sure that I get the data for the proper location. I will also
> > > check for these other locations to get the information on the
> > > netcdf-3 F77 functions.
> >
> > First, even though the reading "seems to work all fine", it's probably
> > not doing what you think it is.  In a loop with ii=1,12, you have
> >
> > CALL read_nc_file(ii,ncid,emisid,emis)
> >
> > but the code is just reading the entire 4D co_flux variable, including
> > data from all 12 times, each time through the loop, with the
> > statement:
> >
> > CALL nf(nf_get_var_real(ncid,emisid,emis)) ! emis.
> >
> > in the read_nc_file subroutine.  If you really intended to read data from 
> > one time with each call, making use of the value of the "ii"
> > parameter, you would instead have something like
> >
> > start = (/ ii, 1, 1, 1 /)
> > count = (/ 1, 6, 180, 360 /)
> > CALL nf(nf_get_vara_real(ncid,emisid,start,count, & emis(ii, 1, 1,
> > 1))) ! emis.
> >
> > and each call would read in only the data for the ii-th time and store that 
> > 3D slab of co_flux data in the right place in the 4D emis variable.
> >
> > Second, I'll assume maybe you already knew all the above and really just 
> > want to figure out what's wrong with the writing.
> >
> > In your output_netcdf subroutine, the loop for setting the emis_output 
> > array, declared emis_output(ilat,ilon), looks wrong:
> >
> > FORALL (I=1:nlon, J=1:ngl)
> > emis_output(I,J) = emis(ii,I,J)
> > END FORALL
> >
> > If the declaration is right, this should instead be
> >
> > FORALL (I=1:ilon, J=1:ilat)
> > emis_output(I,J) = emis(ii,I,J)
> > END FORALL
> >
> > unless you pass nlon and ngl as arguments to the subroutine.
> >
> > In the write_nc_file subroutine, you first write the 1-dimensional
> > variable time using
> >
> > start1d = (/ 1,           timestep /)
> > cnt1d   = (/ 12,                 1 /)
> >
> > status=nf_put_vara_int(ncid, timid, start1d, cnt1d,
> > &     timestep)
> >
> > Since you're only writing one value, you could instead use
> > nf_put_var1_int with fewer arguments, but to use nf_put_vara this
> > should instead be
> >
> > start1d = (/ timestep /)
> > cnt1d   = (/ 1 /)
> >
> > status=nf_put_vara_int(ncid, timid, start1d, cnt1d,
> > &     timestep)
> > ! instead could have used
> > ! status=nf_put_var1_int(ncid, timid, (/ timestep /),
> > timestep)
> >
> > For writing the 2D emis data into the appropriate time slab of the 3D 
> > netCDF variable identified by emisid, it looks to me like you've got it 
> > right!
> >
> > start2d = (/ 1,    1,         timestep /)
> > cnt2d   = (/ 180,  360,              1 /)
> >
> > ! write 2d data: emissions
> > CALL nf(nf_put_vara_double(ncid, emisid, start2d, cnt2d, emis)) ! emis.
> >
> > Just as an aside, you don't need to pass netCDF constants such as
> > nf_global, nf_float, etc. to a subroutine.  Instead, just use the
> > statement
> >
> > include 'netcdf.inc'   ! for Fortran 77
> > or
> > use 'netcdf"           ! for Fortran 90
> >
> > in the subroutine declarations, and all those parameter constants are 
> > available from the appropriate include file or module file installed with 
> > netCDF Fortran.
> >
> > --Russ
> >
> > > -----Original Message-----
> > > From: Unidata netCDF Support
> > > [mailto:address@hidden
> > > Sent: maandag 14 oktober 2013 19:09
> > > To: Ganzeveld, Laurens
> > > Cc: address@hidden
> > > Subject: [netCDF #NUC-119714]: struggling with writing data to nc
> > > file
> > >
> > > Laurens,
> > >
> > > In order to understand what your code is intended to do, we will need for 
> > > you to send the output from running the following command:
> > >
> > > ncdump -h /data/ganzevl/inputdata/emissions/EDGARV3.2/co_2000.nc
> > >
> > > I suspect there might be a confusion of Fortran array order (first index 
> > > varying fastest) with C and CDL array order (last index varying fastest).
> > >
> > > Also, since you are only using the netCDF-3 F77 functions, it might
> > > be easier and less confusing for you to look at the older
> > > documentation for that version of the F77 library, which is here (in
> > > HTML, or PDF, or
> > > ASCII):
> > >
> > > http://www.unidata.ucar.edu/software/netcdf/old_docs/docs_3_6_3/netc
> > > df
> > > -f77.html
> > > http://www.unidata.ucar.edu/software/netcdf/old_docs/docs_3_6_3/netc
> > > df
> > > -f77.pdf
> > > http://www.unidata.ucar.edu/software/netcdf/old_docs/docs_3_6_3/netc
> > > df
> > > -f77.txt
> > >
> > > That's the documentation I'll refer to in discussing how to use
> > > nf_get_vara_real
> > > from:
> > >
> > > http://www.unidata.ucar.edu/netcdf/old_docs/docs_3_6_3/netcdf-f77.ht
> > > ml
> > > #NF_005fGET_005fVARA_005f-type
> > >
> > > for example.
> > >
> > > --Russ
> > >
> > > > Dear Unidata employees,
> > > >
> > > > I hope that you are actually working at the moment and not
> > > > affected by the government shutdown; I am already struggling now
> > > > for some weeks with a simple script to read and write data to
> > > > netcdf in a F77 model code. I am able to read-in and then write
> > > > some manipulated data in F90 version but the F77 version gives
> > > > problems with the writing; I have copied below the F90 version of
> > > > the write_nc routine I used and that works with my installation of
> > > > the netcdf software
> > > > (3.6.2) on my
> > > > Suse11.2 linux system. I have also copied a version of the F77
> > > > subroutine that doesn't work; I am basically looking for the F77
> > > > equivalent of some of the calls of the various nc functions where
> > > > the help page has helped a lot but not to solve this remaining problem.
> > > > Your feedback would be highly appreciated,
> > > >
> > > > Thanks, Laurens Ganzeveld
> > > >
> > > > F90:
> > > >
> > > > The subroutine is called in a timeloop over 12 months writing out
> > > > the global emission field (emis3D), at a resolution of 1x1 degree
> > > > for each month to a new netcdf file:
> > > >
> > > > SUBROUTINE write_nc_file
> > > >
> > > > ! write timestep
> > > > CALL nf(nf90_inquire_dimension(ncid, tid, len=timestep)) timestep
> > > > = timestep + 1
> > > >
> > > > print *,'write_nc: timestepping ',timestep read (*,*)
> > > >
> > > > CALL nf(nf90_put_var(ncid, tid, time, (/timestep/) ))
> > > >
> > > > ! syntax: nf90_put_var(ncid, varid, values, start, cnt) ! start:
> > > > start in netcdf variable
> > > > ! cnt:    number of netcdf variable points
> > > > ! values: starting point of the fortran variable
> > > > start1d = (/ 1,               timestep /)
> > > > start2d = (/ 1,    1,         timestep /)
> > > > start3d = (/ 1,    1,   1,    timestep /)
> > > >
> > > > CALL nf(nf90_put_var(ncid, emis3Did,      emis3D,   start3d, cnt3d)) ! 
> > > > emis.
> > > >
> > > > END SUBROUTINE write_nc_file
> > > >
> > > >
> > > > F77:
> > > > Ncid, tid, emisid are all properly defined as well as the
> > > > parameter emis
> > > >
> > > > SUBROUTINE write_nc_file(ii,time,ncid,tid,emisid,emis)
> > > >
> > > > integer :: ii,timestep,status,ncid,tid,dimtid,timelen,nrecs,
> > > > &             emisid,start2d(3),cnt2d(3),j,k
> > > > real    :: time,emis(180,360)
> > > >
> > > > character*(20) recnam
> > > >
> > > > ! write timestep
> > > > timestep = ii
> > > > print *,'timestep',timestep,time
> > > >
> > > > CALL nf(nf_put_var_double(ncid, tid, time, (/timestep/) )) read
> > > > (*,*)
> > > >
> > > > ! syntax: nf90_put_var(ncid, varid, values, start, cnt) ! start:
> > > > start in netcdf variable
> > > > ! cnt:    number of netcdf variable points
> > > > ! values: starting point of the fortran variable
> > > > start2d = (/ 1,    1,         timestep /)
> > > > cnt2d   = (/ 180,  360,              1 /)
> > > >
> > > > ! write 2d data
> > > >
> > > > CALL nf(nf_put_var_real(ncid, emisid, emis, start2d, cnt2d)) ! emis .
> > > >
> > > > END SUBROUTINE write_nc_file
> > > >
> > > > ==================================================================
> > > > ==
> > > > ==
> > > > =
> > > > Dr. ir. Laurens Ganzeveld
> > > > Assistant professor
> > > > Department of Environmental Sciences, Chair group Earth System
> > > > Sciences Wageningen University and Research Centre Lumen, room
> > > > D.001, Droevendaalsesteeg 4, 6708 PB, Wageningen, Netherlands
> > > > Phone: (+31) 317-486651 / 613066079, Fax: (+31) 317-419000, Being
> > > > in in Colombia: +57 310 2979313
> > > > E-mail: address@hidden<mailto:address@hidden>
> > > >
> > > > Affiliate of Universidad de Los Andes, Depart de BiologÃa, Bogota,
> > > > Colombia Affiliate of Max Planck Institute for Chemistry, Depart.
> > > > of Atmospheric Chemistry, Mainz, Germany,
> > > > ==================================================================
> > > > ==
> > > > ==
> > > > =
> > > >
> > > >
> > > >
> > > Russ Rew                                         UCAR Unidata Program
> > > address@hidden                      http://www.unidata.ucar.edu
> > >
> > >
> > >
> > > Ticket Details
> > > ===================
> > > Ticket ID: NUC-119714
> > > Department: Support netCDF
> > > Priority: Normal
> > > Status: Closed
> > >
> > >
> > >
> >
> > Russ Rew                                         UCAR Unidata Program
> > address@hidden                      http://www.unidata.ucar.edu
> >
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: NUC-119714
> > Department: Support netCDF
> > Priority: Normal
> > Status: Closed
> >
> >
> >
> Russ Rew                                         UCAR Unidata Program
> address@hidden                      http://www.unidata.ucar.edu
> 
> 
> 
> Ticket Details
> ===================
> Ticket ID: NUC-119714
> Department: Support netCDF
> Priority: Normal
> Status: Closed
> 
> 
> 
Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: NUC-119714
Department: Support netCDF
Priority: Normal
Status: Closed


NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.