Hi Pedro,
Is there a way to represent compound HDF5 objects in your scheme? I
see STAR_INT8, STAR_FLOAT, etc., but no STAR_COMPOUND.
In any event, FYI, I have implemented a similar thing but in a
narrower domain. I needed to map VOTable [1] (an xml format with a
defined data model for astronomy) to HDF5. Along the way, I also
mapped it (with varying levels of fidelity) to JSON, CSV, HTML, FITS,
and plain old ascii. This makes my work very table focused, with a
particular eye towards what relational databases like to output.
The code is on github [2]. It is mainly meant to be used as a
library, but the code also builds a simple program that converts files
between formats. In case you are interested, I am attaching the same
file in plain text, JSON, and HDF5.
Cheers,
Walter Landry
[1] http://www.ivoa.net/documents/VOTable/
[2] https://github.com/Caltech-IPAC/tablator
Pedro Vicente <pedro.vicente@xxxxxxxxxxxxxxxxxx> wrote:
> Hi Charlie !
>
>
> So, I am doing that exact same thing.
>
> I wrote
>
> 1) The specification to convert netCDF/HDF5 to "a" JSON format (note
> the "a" here)
>
> 2) I wrote the parsing of the JSON format using an open source C JSON
> library (janson).
>
> http://www.digip.org/jansson/
>
> 3) left to do: the actual C code of reading/writing netCDF/HDF5 to
> JSON and vice-versa (the straightforward part).
>
> The "a" above means that JSON is not really a format in the sense of
> netCDF but really a format that allows to define formats , for a lack
> of a better
> explanation.
>
> This means that anyone that writes this tool has to write code that
> write in a particular JSON representation , only valid for that tool.
>
> Like you , I searched and there was not a good one, so I wrote one.
>
> The first criteria was that it had to be obvious for anyone looking at
> the JSON text file, that that was indeed a netCDF/HDF5 file: hierarchy
> clearly show, metadata and data
> clearly shown
>
> My first look was HDF5-JSON
>
> http://hdf5-json.readthedocs.io/en/latest/
>
> but the format seemed like a mess to look at
>
> example
>
> http://hdf5-json.readthedocs.io/en/latest/examples/nullspace_dset.html
>
>
> and the reader is written in Python
>
> @John Readey
>
> (why Phyton? HDF5 developer tools should be all about writing in
> C/C++)
>
>
>
> The specification is here
>
> http://www.space-research.org/
>
> Click on menu
> "Code blog",
> then
> "netCDF/HDF5 to JSON and vice-versa"
>
>
> In the process I learned all about JSON and it is a neat format to
> represent data .
>
> In particular, it allows nested structures and arrays, which suits
> perfectly for netCDF
>
> here are two nested groups
>
> {
> "group_name1":
> {
> "group_name2": "group"
> }
> }
>
> a dataset
>
> {
> "dset1" : ["dataset", "STAR_INT32", 2, [3, 4], [1, 2, 3, 4, 5, 6, 7,
> 8, 9, 10, 11, 12]]
> }
>
>
>
> This is still under development,
>
> I would like to make this some kind of "official" netCDF/HDF5 JSON
> format for the community, so I encourage anyone to read the
> specification
>
> direct link
>
> http://www.space-research.org/blog/star_json.html
>
> If you see any flaw in the design or antything in the design that you
> would like to have change please let me know now
>
> At the moment it only (intentionally) uses common generic features of
> both netCDF and HDF5, which are the numeric atomic types and strings.
>
> Enjoy
>
>
> ----------------------
> Pedro Vicente
> pedro.vicente@xxxxxxxxxxxxxxxxxx
> http://www.space-research.org/
>
>
>
>
> ----- Original Message -----
> From: "Charlie Zender" <zender@xxxxxxx>
> To: "netCDF Mail List" <netcdfgroup@xxxxxxxxxxxxxxxx>
> Sent: Thursday, October 13, 2016 11:10 PM
> Subject: [netcdfgroup] How to dump netCDF to JSON?
>
>
>> Hello netCDFers,
>>
>> A project I am working on wants to convert netCDF files to JSON.
>> The requirements are to dump an arbitrary netCDF-extended file
>> (with groups but without funky vlen/compound types) to JSON.
>> The first few solutions that we googled (ncdump-json, netcdf2json.py)
>> do not satisfy these requirements. What is the most robust and easy
>> command-line tool (not web-service) that dumps netCDF to JSON?
>> Ideally it would be somewhat configurable like ncdump -h/-x or
>> ncks --cdl/--xml.
>>
>> Charlie
>> --
>> Charlie Zender, Earth System Sci. & Computer Sci.
>> University of California, Irvine 949-891-2429 )'(
>>
>> _______________________________________________
>> NOTE: All exchanges posted to Unidata maintained email lists are
>> recorded in the Unidata inquiry tracking system and made publicly
>> available through the web. Users who post to any of the lists we
>> maintain are reminded to remove any personal information that they
>> do not want to be made public.
>>
>>
>> netcdfgroup mailing list
>> netcdfgroup@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@xxxxxxxxxxxxxxxxxx
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
> Twitter: https://twitter.com/hdf5
| object | ra | dec | htm20
| htm7 | htm3 | shtm20 | shtm7 | shtm3 |
flags | SSO |
| char | double | real | ulong
| uint | ushort | long | int | short |
byte | bool |
118289arstratraetratratsrastratsrastrats 359.88703 50.832570
16446744073709551616 3294967296 12000 8223372036854775808 1147483648
12000 122 0
113368 344.41273 -29.622250
8446744073709551616 294967296 43002 -7223372036854775808 -2047483648
13002 0xf2 true
113368 344.41273 -29.622250
8446744073709551616 294967296 43002 -7223372036854775808 -2047483648
-23002 211 False
113368 344.41273 -29.622250
8446744073709551616 294967296 43002 -7223372036854775808 -2047483648
-31002 211 1
{
"VOTABLE":
{
"<xmlattr>":
{
"version": "1.3",
"xmlns:xsi": "http:\/\/www.w3.org\/2001\/XMLSchema-instance",
"xmlns": "http:\/\/www.ivoa.net\/xml\/VOTable\/v1.3",
"xmlns:stc": "http:\/\/www.ivoa.net\/xml\/STC\/v1.30"
},
"RESOURCE":
{
"TABLE":
{
"FIELD":
{
"<xmlattr>":
{
"name": "object",
"datatype": "char",
"arraysize": "*"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "ra",
"datatype": "double"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "dec",
"datatype": "double"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "htm20",
"datatype": "ulong"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "htm7",
"datatype": "uint"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "htm3",
"datatype": "ushort"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "shtm20",
"datatype": "long"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "shtm7",
"datatype": "int"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "shtm3",
"datatype": "short"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "flags",
"datatype": "unsignedByte"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "SSO",
"datatype": "boolean"
}
},
"DATA":
{
"TABLEDATA":
[
[
"118289arstratraetratratsrastratsrastrats",
"359.88702999999998",
"50.832569999999997",
"16446744073709551616",
"3294967296",
"12000",
"8223372036854775808",
"1147483648",
"12000",
"0x7a",
"0"
],
[
"113368",
"344.41273000000001",
"-29.622250000000001",
"8446744073709551616",
"294967296",
"43002",
"-7223372036854775808",
"-2047483648",
"13002",
"0xf2",
"1"
],
[
"113368",
"344.41273000000001",
"-29.622250000000001",
"8446744073709551616",
"294967296",
"43002",
"-7223372036854775808",
"-2047483648",
"-23002",
"0xd3",
"0"
],
[
"113368",
"344.41273000000001",
"-29.622250000000001",
"8446744073709551616",
"294967296",
"43002",
"-7223372036854775808",
"-2047483648",
"-31002",
"0xd3",
"1"
]
]
}
}
}
}
}
�HDF
