[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDV #LPW-471935]: Plotting .hdf files in IDV



Alexa,

Great! I am glad it worked.

I was contemplating writing a blog entry for our Unidata Developer Blog on this
experience as it makes a good case study. Would it be OK if I used your
data/ncml for this purpose? It is no problem if you would prefer not to.

Best,

Unidata IDV Support

> Dear IDV Support (and Sean),
> 
> This works perfectly. I realized the file naming scheme was actually actually 
> giving year + day of the year, so I adjusted the dateFormatMark using the 
> syntax in this example 
> http://docs.opendap.org/index.php/Dynamic_Aggregation_Tutorial and now the 
> dates/times are being read accurately.
> 
> As a side note, I noticed that Projectionsâ>Enable Adaptive Resolution is 
> on by default in the IDV, which automatically subsamples the data. But once 
> that function was unchecked and the data reloaded, everything looks good. Big 
> thanks to the both of you.
> 
> All the best
> 
> Alexa
> 
> 
> > On Apr 1, 2016, at 12:06 PM, Unidata IDV Support <address@hidden> wrote:
> >
> > Hi Alexa,
> >
> > What I was describing below had nothing to do with the problem. Please see 
> > the
> > attached NCML. This file should correctly aggregate the variables over
> > the time dimension, and also it will extract the times from the file name, 
> > so
> > you don't have to manually describe them.
> >
> > Let us know how it goes.
> >
> > Best,
> >
> > Unidata IDV Support
> >
> > PS Thanks to my colleague Sean Arms for the NCML lessons :-)
> >
> >> Sure thing, Iâll have a play and keep you posted. All the best, Alexa
> >>
> >>
> >>> On Mar 31, 2016, at 4:21 PM, Unidata IDV Support <address@hidden> wrote:
> >>>
> >>> Alexa,
> >>>
> >>> I think I know what the problem is, but don't have time to try before I 
> >>> go home.
> >>>
> >>> I think the problem is that we have only changed one variable when we 
> >>> should
> >>> have changed them all, at least all those that are listed under the Field
> >>> Selector.  The IDV may be getting confused on this.
> >>>
> >>> Do you want to give that a shot and let me know what happens?
> >>>
> >>> Best,
> >>>
> >>> Unidata IDV Support
> >>>
> >>>> Alexa,
> >>>>
> >>>> Yes, thanks for the clarification. (I think the ordering of our messages 
> >>>> got
> >>>> jumbled there.)
> >>>>
> >>>> I see what you are saying, and when I load each individual file 
> >>>> separately I can
> >>>> see the data are definitely not the same so something is amiss. We'll 
> >>>> resume
> >>>> tomorrow.
> >>>>
> >>>> Best,
> >>>>
> >>>> Unidata IDV Support
> >>>>
> >>>>> Hi there â thank you for for your reply. Using your syntax below, I 
> >>>>> tried aggregating three files (edited script attached) â different 
> >>>>> files this time. Again, the files are read into IDV with the correct 
> >>>>> time stamps, and I can toggle between the times, but the strange thing 
> >>>>> is the plot does not change with time. Itâs like itâs only plotting 
> >>>>> one time. -Alexa
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>> On Mar 31, 2016, at 3:46 PM, Unidata IDV Support <address@hidden> 
> >>>>>> wrote:
> >>>>>>
> >>>>>> Alexa,
> >>>>>>
> >>>>>> Great! Glad we are making progress and also that you are hanging in 
> >>>>>> there. This
> >>>>>> stuff is not so easy.
> >>>>>>
> >>>>>> Replace the current values element with something like this:
> >>>>>>
> >>>>>> <values>0 3 11</values>
> >>>>>>
> >>>>>> But if that is too onerous because you have many file, we will have to 
> >>>>>> go via
> >>>>>> the time extraction route. Let me know...
> >>>>>>
> >>>>>> Best,
> >>>>>>
> >>>>>> Unidata IDV Support
> >>>>>>
> >>>>>>> Hello â this looks great, and I appreciate your help with this. 
> >>>>>>> Your ncml script worked for the two test files and the times look 
> >>>>>>> right. However, since we have some missing satellite images, not all 
> >>>>>>> the files are spaced exactly 10 mins apart. Is there a way to specify 
> >>>>>>> time gaps between individual files begin aggregated? Otherwise, 
> >>>>>>> perhaps extracting the time from the file name would be good route. 
> >>>>>>> -Alexa
> >>>>>>>
> >>>>>>>
> >>>>>>>> On Mar 31, 2016, at 3:29 PM, Unidata IDV Support <address@hidden> 
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Hi again,
> >>>>>>>>
> >>>>>>>> The embedded NcML is a bit hard to read so attached, you will find
> >>>>>>>> that same NcML file.
> >>>>>>>>
> >>>>>>>> Best,
> >>>>>>>>
> >>>>>>>> Unidata IDV Support
> >>>>>>>>
> >>>>>>>>> Hi Alexa,
> >>>>>>>>>
> >>>>>>>>> It is a little tricky but you can fix these problems with a bit of
> >>>>>>>>> NcML. To get us going, let's look at the following NcML which can be
> >>>>>>>>> read into the IDV and will allow you to view the
> >>>>>>>>> `channel_14_brightness_temperature' over time. This file should be
> >>>>>>>>> located next to your `hdf' files. Please read the comments in the 
> >>>>>>>>> NcML
> >>>>>>>>> for more information about what is going on here.
> >>>>>>>>>
> >>>>>>>>> ,----
> >>>>>>>>> | <?xml version="1.0" encoding="UTF-8"?>
> >>>>>>>>> | <netcdf 
> >>>>>>>>> xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
> >>>>>>>>> |   <!-- Create a time coordinate variable that will be used for 
> >>>>>>>>> the aggregation. -->
> >>>>>>>>> |   <variable name="time" type="int" shape="time" >
> >>>>>>>>> |     <!-- Let's add some units. Alexa: please make sure these 
> >>>>>>>>> units are correct. -->
> >>>>>>>>> |     <attribute name="units" value="minutes since 2014-04-04 
> >>>>>>>>> 19:09"/>
> >>>>>>>>> |     <!-- Specify this variable as a time coordinate -->
> >>>>>>>>> |     <attribute name="_CoordinateAxisType" value="Time" />
> >>>>>>>>> |     <!-- Increment each file listed below by 10 minutes. Alexa, 
> >>>>>>>>> please make sure this -->
> >>>>>>>>> |     <!-- assumption about the times is correct. -->
> >>>>>>>>> |     <values start="0" increment="10" />
> >>>>>>>>> |   </variable>
> >>>>>>>>> |   <!--   Specify the coordinates of time lat lon for this one 
> >>>>>>>>> variable. Please use -->
> >>>>>>>>> |   <!-- this as an example for other variables you are interersted 
> >>>>>>>>> in. In other -->
> >>>>>>>>> |   <!-- words, you can copy and paste this variable NcML element 
> >>>>>>>>> for your other -->
> >>>>>>>>> |   <!-- variables. -->
> >>>>>>>>> |   <variable name="channel_14_brightness_temperature" shape="lines 
> >>>>>>>>> elements" type="short">
> >>>>>>>>> |     <attribute name="coordinates" value="time pixel_latitude 
> >>>>>>>>> pixel_longitude" />
> >>>>>>>>> |   </variable>
> >>>>>>>>> |   <!-- Now you are ready to do your aggregation, which is the 
> >>>>>>>>> easy part at this point -->
> >>>>>>>>> |   <aggregation dimName="time" type="joinNew">
> >>>>>>>>> |     <variableAgg name="T"/>
> >>>>>>>>> |     <netcdf location="geocatL1.MTSAT-1R.2014044.190900.hdf"/>
> >>>>>>>>> |     <netcdf location="geocatL1.MTSAT-1R.2014044.191900.hdf"/>
> >>>>>>>>> |   </aggregation>
> >>>>>>>>> | </netcdf>
> >>>>>>>>> `----
> >>>>>>>>>
> >>>>>>>>> In summary, this NcML should aggregate over time the two of HDF 
> >>>>>>>>> files
> >>>>>>>>> you provided. You will see only one variable that is aggregated over
> >>>>>>>>> time; `channel_14_brightness_temperature'.  I'll leave it up to you 
> >>>>>>>>> to
> >>>>>>>>> do the others though do not hesitate to ask for help.
> >>>>>>>>>
> >>>>>>>>> Also, if interested, there are also fancier tricks we can do here 
> >>>>>>>>> such
> >>>>>>>>> as NcML can be smart enough to extract the times from the file 
> >>>>>>>>> names.
> >>>>>>>>>
> >>>>>>>>> Best,
> >>>>>>>>>
> >>>>>>>>> Unidata IDV Support
> >>>>>>>>>
> >>>>>>>>>> Ok sounds promising. Here is the link to another file, which 
> >>>>>>>>>> should differ only in time â 
> >>>>>>>>>> https://drive.google.com/file/d/0B1jVlP9ZfAZCRDVyczU3N2NMbFU/view?usp=sharing
> >>>>>>>>>>    -Alexa
> >>>>>>>>>>
> >>>>>>>>>>> On Mar 30, 2016, at 4:49 PM, Unidata IDV Support <address@hidden> 
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Alexa,
> >>>>>>>>>>>
> >>>>>>>>>>> Indeed we may be able to patch things up with a bit of ncml. I 
> >>>>>>>>>>> only found one
> >>>>>>>>>>> file at the drive.google location. Would it be possible to see 
> >>>>>>>>>>> more so that we
> >>>>>>>>>>> have something to aggregate?
> >>>>>>>>>>>
> >>>>>>>>>>> Best,
> >>>>>>>>>>>
> >>>>>>>>>>> Unidata IDV Support
> >>>>>>>>>>>
> >>>>>>>>>>>> Hello IDV Support,
> >>>>>>>>>>>>
> >>>>>>>>>>>> I am working on reading in a number of HDF files using 
> >>>>>>>>>>>> âaggregate grids by timeâ or similar. I am able to read the 
> >>>>>>>>>>>> files in individually without a problem, but it looks like the 
> >>>>>>>>>>>> time variable is not recognized. The error message is "Grid data 
> >>>>>>>>>>>> source failed making data setâ.Dimension time does not 
> >>>>>>>>>>>> existâ. Any ideas for how to specify the time variable? Could 
> >>>>>>>>>>>> this be achieved with an .ncml script?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Example file is located here: 
> >>>>>>>>>>>> https://drive.google.com/file/d/0B1jVlP9ZfAZCRFhkRVM2aVVRVGc/view?usp=sharing
> >>>>>>>>>>>>
> >>>>>>>>>>>> All the best
> >>>>>>>>>>>>
> >>>>>>>>>>>> Alexa
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Ticket Details
> >>>>>>>>>>> ===================
> >>>>>>>>>>> Ticket ID: LPW-471935
> >>>>>>>>>>> Department: Support IDV
> >>>>>>>>>>> Priority: Normal
> >>>>>>>>>>> Status: Closed
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Ticket Details
> >>>>>>>> ===================
> >>>>>>>> Ticket ID: LPW-471935
> >>>>>>>> Department: Support IDV
> >>>>>>>> Priority: Normal
> >>>>>>>> Status: Closed<alexa.ncml>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Ticket Details
> >>>>>> ===================
> >>>>>> Ticket ID: LPW-471935
> >>>>>> Department: Support IDV
> >>>>>> Priority: Normal
> >>>>>> Status: Closed
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>> Ticket Details
> >>> ===================
> >>> Ticket ID: LPW-471935
> >>> Department: Support IDV
> >>> Priority: Normal
> >>> Status: Open
> >>>
> >>
> >>
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: LPW-471935
> > Department: Support IDV
> > Priority: Normal
> > Status: Closed<alexa.ncml>
> 
> 


Ticket Details
===================
Ticket ID: LPW-471935
Department: Support IDV
Priority: Normal
Status: Closed


NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.