Re: [netcdf-java] How to create a structure in netcdf4

  • To: Christian Ward-Garrison <cwardgar@xxxxxxxx>
  • Subject: Re: [netcdf-java] How to create a structure in netcdf4
  • From: José Évora <jose.evora@xxxxxxxx>
  • Date: Wed, 16 Dec 2015 11:58:50 +0000
The screenshot is from Hdfview. Thanks for all, no problem if the
<structure_name>_t cannot be customised yet. It was just to know it.

Best regards,

José Évora Gómez
Instituto Universitario SIANI
Universidad de Las Palmas de GC
@j_evora

2015-12-11 22:23 GMT+00:00 Christian Ward-Garrison <cwardgar@xxxxxxxx>:

> Hi José,
>
> I'm not sure what program that image came from, but when I ncdump the
> NetCDF-4 file created after incorporating your changes, I get:
>
> $ ncdump -h foo.nc4
> netcdf foo {
> types:
>   compound CHANGELOG_t {
>     char Date1(30) ;
>     char Date2(53) ;
>   }; // CHANGELOG_t
> dimensions:
> City = 1 ;
> CHANGELOG = 1 ;
> variables:
> string City(City) ;
> CHANGELOG_t CHANGELOG(CHANGELOG) ;
> }
>
> "CHANGELOG_t" is the name of the compound type. Currently, there is no way
> in NetCDF-Java to configure that name; it'll always be
> "<structure_name>_t".
>
> Cheers,
> Christian
>
> On Fri, Dec 11, 2015 at 3:06 AM, José Évora <jose.evora@xxxxxxxx> wrote:
>
>> Hi Christian!
>>
>> Thank you very much. It worked perfectly and now the netcdf file is much
>> more clean. I just did one change which is calling the dimension
>> "changelog" and the structured variable "changelog" too:
>>
>> structure = (Structure) ncFile.addVariable(null, "CHANGELOG", STRUCTURE, 
>> singletonList(ncFile.addDimension(null, "CHANGELOG", 1, true, false, 
>> false)));
>>
>> As result I have next file structure:
>>
>> [image: Imágenes integradas 2]
>>
>> Is there anyway to have CHANGE-LOG-ENTRY instead of CHANGELOG_t in the
>> yellow marked entry?
>>
>> Thanks in advance, you're helping me alot
>>
>>
>> José Évora Gómez
>> Instituto Universitario SIANI
>> Universidad de Las Palmas de GC
>> @j_evora
>>
>> 2015-12-08 23:57 GMT+00:00 Christian Ward-Garrison <cwardgar@xxxxxxxx>:
>>
>>> Hi José,
>>>
>>> Once again, sorry for the delay. I've modified my last example to
>>> incorporate the changes you requested [1]. A few things to note:
>>>
>>> 1. The "Other" dimension is now shared instead of private.
>>> NetcdfFileWriter.addDimension creates a dimension and adds it to a
>>> Group. When Dimensions are contained by Groups, they should *always* be
>>> shared. In fact, NetcdfFileWriter.addDimension shouldn't even have the
>>> isShared parameter, as it is confusing and error-prone. In v5.0.0, it
>>> won't [2]. Thanks for bringing this to my attention!
>>>
>>> 2. In the first example, the Date1 and Date2 variables use *anonymous
>>> dimensions*. Anonymous dimensions are both private (i.e. contained by a
>>> Variable, not a Group) and nameless. NetcdfFileWriter does not provide
>>> a direct way to create them, so you'll have to do a little more work, as
>>> I've shown.
>>>
>>> 3. In the first example, it looks like the "CHANGELOG=2" snippet may
>>> indicate a private, *named* dimension. If that's the case, create it
>>> the same way you did the anonymous dimensions but just – you know – give it
>>> a name.
>>>
>>> Cheers,
>>> Christian
>>>
>>> [1] https://gist.github.com/cwardgar/35427c2e245ec0157416
>>> [2] https://github.com/Unidata/thredds/pull/326
>>>
>>> On Tue, Nov 24, 2015 at 1:23 AM, José Évora <jose.evora@xxxxxxxx> wrote:
>>>
>>>> Hi Christian,
>>>>
>>>> Thank you very much, it worked perfectly!. However, I'm trying to
>>>> replicate the output as expected for a netcdf reader of a specific kind of
>>>> data. When I read a netcdf that is accepted by the specific reader using
>>>> the netcdf java library, this is what I obtain (I've have adapted the
>>>> content to our toy case, but it can be extrapolated to the real case):
>>>>
>>>> netcdf res/in original.nc {
>>>>>   dimensions:
>>>>
>>>>     City = 1;
>>>>
>>>>     Other = UNLIMITED;   // (2 currently)
>>>>>     ...
>>>>>   variables:
>>>>
>>>>     String City(City=1);
>>>>
>>>>
>>>>
>>>>     Structure {
>>>>>       char Date1(30);
>>>>>       char Date2(53);
>>>>>     } CHANGELOG(CHANGELOG=2);
>>>>>       :_ChunkSizes = 1; // int
>>>>> }
>>>>
>>>>
>>>> With the code you provided me, I obtain:
>>>>
>>>>  netcdf res/mockSimulation2.nc {
>>>>>   dimensions:
>>>>>     City = 1;
>>>>>     Other-Entry_Dim0 = 1;
>>>>>     Other = 1;
>>>>>     strlen = 30;
>>>>>   variables:
>>>>>     String City(City=1);
>>>>>     Structure {
>>>>>       char Date1(30);
>>>>>       char Date2(30);
>>>>>     } Other-Entry(Other-Entry_Dim0=1);
>>>>>
>>>>> }
>>>>
>>>>
>>>> I only did one change to what you sent me at line 26:
>>>>
>>>> Structure otherVar = (Structure) ncFile.addVariable(root,
>>>>> "Other-Entry", STRUCTURE, singletonList(other));
>>>>
>>>>
>>>> I don't know how I can:
>>>>
>>>> 1) Avoid having a dimension strlen
>>>> 2) Avoid having Other-Entry_Dim0
>>>>
>>>> Thanks in advance!
>>>>
>>>> José Évora Gómez
>>>> Instituto Universitario SIANI
>>>> Universidad de Las Palmas de GC
>>>> @j_evora
>>>>
>>>> 2015-11-23 6:59 GMT+00:00 Christian Ward-Garrison <cwardgar@xxxxxxxx>:
>>>>
>>>>> Hi José,
>>>>>
>>>>> Sorry for the late response—hopefully this is still useful to you.
>>>>>
>>>>> Writing structures to NetCDF-4 is a recent addition to our library and
>>>>> hasn't been tested as comprehensively as we'd like, especially since most
>>>>> of our users that are writing NetCDF-4 are only utilizing the classic data
>>>>> model (i.e. no structures). Your example uncovered multiple bugs 
>>>>> [1][2][3],
>>>>> so thank you for that!
>>>>>
>>>>> That said, it is still possible to make your example work, although
>>>>> the admittedly limited documentation makes it tricky to get right. I've
>>>>> corrected your NetCdfCreatorLite2 class and posted it here [4]. Notice 
>>>>> that
>>>>> the "Other" dimension is no longer unlimited and that I'm recalculating 
>>>>> the
>>>>> structure's size after adding members to it.
>>>>>
>>>>> Also, for future reference, the latest versions of our library
>>>>> artifacts are not available on Maven Central. Instead, grab them from our
>>>>> organization's repository, as described here [5].
>>>>>
>>>>> Cheers,
>>>>> Christian
>>>>>
>>>>> [1] https://github.com/Unidata/thredds/issues/296
>>>>> [2] https://github.com/Unidata/thredds/issues/298
>>>>> [3] https://github.com/Unidata/thredds/issues/299
>>>>> [4] https://gist.github.com/cwardgar/7aa866886468e97c8eea
>>>>> [5]
>>>>> https://www.unidata.ucar.edu/software/thredds/current/netcdf-java/reference/BuildDependencies.html
>>>>>
>>>>> On Thu, Nov 5, 2015 at 8:01 AM, José Évora <jose.evora@xxxxxxxx>
>>>>> wrote:
>>>>>
>>>>>> The dependency is being taken from Maven central. However, I have
>>>>>> tried this jar instead of maven:
>>>>>>
>>>>>> ftp://ftp.unidata.ucar.edu/pub/netcdf-java/v4.6/netcdfAll-4.6.jar
>>>>>>
>>>>>> And I get exactly the same problems in both java files.
>>>>>>
>>>>>> José Évora Gómez
>>>>>> Instituto Universitario SIANI
>>>>>> Universidad de Las Palmas de GC
>>>>>> @j_evora
>>>>>>
>>>>>> 2015-11-05 14:09 GMT+00:00 John Caron <jcaron1129@xxxxxxxxx>:
>>>>>>
>>>>>>> what repo is
>>>>>>>
>>>>>>> <dependency>
>>>>>>> <groupId>edu.ucar</groupId>
>>>>>>> <artifactId>netcdf4</artifactId>
>>>>>>> <version>LATEST</version>
>>>>>>> </dependency>
>>>>>>>
>>>>>>> coming from?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Nov 5, 2015 at 12:13 AM, José Évora <jose.evora@xxxxxxxx>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> When maven resolves the code quoted, this is the library that
>>>>>>>> downloads:
>>>>>>>>
>>>>>>>> Maven: edu.ucar:netcdf4:4.5.5
>>>>>>>>
>>>>>>>>
>>>>>>>> Version seems to be 4.5.5 for the netcdf java library. For the
>>>>>>>> netcdf C libraries, I downloaded and installed: netCDF 4.3.3.1.
>>>>>>>> Specifically, this file:
>>>>>>>>
>>>>>>>>
>>>>>>>> http://www.unidata.ucar.edu/downloads/netcdf/ftp/netCDF4.3.3.1-NC4-DAP-64.exe
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> José Évora Gómez
>>>>>>>> Instituto Universitario SIANI
>>>>>>>> Universidad de Las Palmas de GC
>>>>>>>> @j_evora
>>>>>>>>
>>>>>>>> 2015-11-04 23:33 GMT+00:00 John Caron <jcaron1129@xxxxxxxxx>:
>>>>>>>>
>>>>>>>>> Hi Jose, what version of netcdf-java and netcdf C libraries are
>>>>>>>>> you using? John
>>>>>>>>>
>>>>>>>>> On Wed, Nov 4, 2015 at 12:54 AM, José Évora <jose.evora@xxxxxxxx>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi everyone,
>>>>>>>>>>
>>>>>>>>>> I'm trying to create a structure (also know compounded type). In
>>>>>>>>>> this email I attach two different ways I've tried to do it and their
>>>>>>>>>> respective error message. This is the dependency I'm using
>>>>>>>>>>
>>>>>>>>>> <dependency>
>>>>>>>>>>> <groupId>edu.ucar</groupId>
>>>>>>>>>>> <artifactId>netcdf4</artifactId>
>>>>>>>>>>> <version>LATEST</version>
>>>>>>>>>>> </dependency>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *1) NetCdfCreatorLite.java*
>>>>>>>>>>
>>>>>>>>>> The execution throws a NullPointerException at
>>>>>>>>>>
>>>>>>>>>> Structure otherVar = ncFile.addStructure(root,
>>>>>>>>>>> root.getParentStructure(), "Other", singletonList(other));
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Exception in thread "main" java.lang.NullPointerException
>>>>>>>>>>> at
>>>>>>>>>>> ucar.nc2.NetcdfFileWriter.addStructure(NetcdfFileWriter.java:584)
>>>>>>>>>>> at org.mock.NetCdfCreatorLite.main(NetCdfCreatorLite.java:27)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *2) NetCdfCreatorLite2.java*
>>>>>>>>>>
>>>>>>>>>> According to the java doc, the method addstructure shouldn't be
>>>>>>>>>> used. Then, in this second version, I uses a addVariable method to 
>>>>>>>>>> create
>>>>>>>>>> the structure:
>>>>>>>>>>
>>>>>>>>>> Structure otherVar = (Structure) ncFile.addVariable(root,
>>>>>>>>>>> "Other", STRUCTURE, singletonList(other));
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> The exeception given this time is:
>>>>>>>>>>
>>>>>>>>>> Exception in thread "main" java.io.IOException: nc_put_vars: -49:
>>>>>>>>>>> NetCDF: Variable not found grpid=65536 objid=1
>>>>>>>>>>> at
>>>>>>>>>>> ucar.nc2.jni.netcdf.Nc4Iosp.writeCompoundData(Nc4Iosp.java:2895)
>>>>>>>>>>> at ucar.nc2.jni.netcdf.Nc4Iosp.writeData(Nc4Iosp.java:2839)
>>>>>>>>>>> at ucar.nc2.jni.netcdf.Nc4Iosp.writeData(Nc4Iosp.java:2721)
>>>>>>>>>>> at ucar.nc2.NetcdfFileWriter.write(NetcdfFileWriter.java:934)
>>>>>>>>>>> at ucar.nc2.NetcdfFileWriter.write(NetcdfFileWriter.java:918)
>>>>>>>>>>> at org.mock.NetCdfCreatorLite2.main(NetCdfCreatorLite2.java:46)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> At line:
>>>>>>>>>>
>>>>>>>>>> ncFile.write(otherVar, otherStructure);
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> If I comment the creation of structures, the application executes
>>>>>>>>>> perfectly. Even more, by only adding the addVariable of a structure 
>>>>>>>>>> line, I
>>>>>>>>>> get this exception, even though I only write the cityVar at line 50.
>>>>>>>>>>
>>>>>>>>>> What is wrong? Can you please provide me with an example of use
>>>>>>>>>> of the structures?.
>>>>>>>>>>
>>>>>>>>>> Best regards, thank you very much,
>>>>>>>>>>
>>>>>>>>>> José Évora Gómez
>>>>>>>>>> Instituto Universitario SIANI
>>>>>>>>>> Universidad de Las Palmas de GC
>>>>>>>>>> @j_evora
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> netcdf-java mailing list
>>>>>>>>>> netcdf-java@xxxxxxxxxxxxxxxx
>>>>>>>>>> For list information or to unsubscribe, visit:
>>>>>>>>>> http://www.unidata.ucar.edu/mailing_lists/
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> netcdf-java mailing list
>>>>>> netcdf-java@xxxxxxxxxxxxxxxx
>>>>>> For list information or to unsubscribe, visit:
>>>>>> http://www.unidata.ucar.edu/mailing_lists/
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

PNG image

  • 2015 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdf-java archives: