[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDV #DGO-203144]: Point data questions



Hi Hans-

> Thanks for your reply and for the explanations.
> This is not really a big problem for me, though it would probably be useful 
> if the point data import feature would be be a bit more "robust" sometime in 
> the future. And perhaps a data error handling scheme that would support the 
> average user in locating errors in the data. Something to add to the to do 
> list for a rainy day maybe.

I just checked in a couple of other changes that might address your
painting issue.  I thought they were already in the nightly build.  We
are moving toward 2.7b1 tomorrow and the fixes will be in there.

Hopefully, the recent fixes go a ways toward better handling of bad data.  
I'll put it on the list, though. Since we get 300 days of sunshine here 
in Boulder, there aren't many rainy days to work on these. ;-)   

Don

> -----Original Message-----
> From: Unidata IDV Support [mailto:address@hidden]
> Sent: Monday, May 11, 2009 5:19 PM
> To: address@hidden
> Cc: address@hidden; address@hidden
> Subject: [IDV #DGO-203144]: Point data questions
> 
> Hi Hans-
> 
> > Have done a quick test on builds from May 8 and 9.
> >
> > First issue: The Point Data windows is still not being re-drawn (filled in) 
> > after having applied the saved preferences. I have to resize the window to 
> > get that to happen.
> 
> Okay, I'm having issues recreating this on my machine.  I'll test on
> on others.  I did fix another bug where the skip lines weren't being
> done.  Before I had to apply the preference twice.
> 
> > Second issue: The original csv file still generates the same error.
> 
> Still having trouble getting this to fail consistently, but will  keep
> plugging away on it.
> 
> > But the good news is that other csv files, that didn't work before, now 
> > works!
> 
> At least there's some progress!
> 
> > However, a file downloaded today ("metars.current090509.csv") comes up with 
> > a new error (see below, "samples values may not be missing", is that a bad 
> > thing i.e. something NOT missing?).
> 
> It's coming from the low level VisAD code where you can't have a missing value
> (NaN) for data when trying to create the set of times.  I meant to fix that in
> the last round of changes.
> 
> The problem is that one ob:
> 
> VOTV 090740Z 31006KT 6000 SCT018 FEW025CB(W,N,E) BKN100 31/24 Q1008 
> NOSIG,VOTV,2
> 009-05-09T07:40:00Z,8.47,76.95,31.0,24.0,310,6,,3.73,29.763779,,,,,,TRUE,,,,,SCT
> ,1800,BKN,10000,,,,,MVFR,,,,,,,,,,,,METAR,8.0
> 
> has commas in the raw observation (W,N,E) which throws off the parser.  We 
> don't
> have any way around that, so I just skip the ob now.
> 
> > And I sometimes still have the problem with different behavior (different 
> > errors) the first and the second time I read a specific csv file. The file 
> > that caused the error below ("samples values may not be missing") actually 
> > generated another error the first time I read it (first time error was the 
> > one I reported before, i.e. "An error occurred reading column number:6" (as 
> > per my previous email)
> 
> Again, I'll keep trying to reproduce this, but it's been tough to replicate.  
> I know
> it is an issue and I think it's a timing issue way down in the low level code 
> where it
> tries to create the variables.
> 
> The error below will be fixed in the next release:
> 
> > visad.SetException: Gridded1DDoubleSet: samples values may not be missing
> > at visad.Gridded1DDoubleSet.<init>(Gridded1DDoubleSet.java:120)
> > at visad.Gridded1DDoubleSet.<init>(Gridded1DDoubleSet.java:89)
> > at visad.DateTime.makeTimeSet(DateTime.java:537)
> > at 
> > ucar.unidata.data.point.PointObFactory.makeTimeSequenceOfPointObs(PointObFactory.java:252)
> > at 
> > ucar.unidata.data.point.PointObFactory.makeTimeSequenceOfPointObs(PointObFactory.java:178)
> > at 
> > ucar.unidata.data.point.PointObFactory.makeTimeSequenceOfPointObs(PointObFactory.java:155)
> > at 
> > ucar.unidata.data.point.PointDataSource.getDataInner(PointDataSource.java:1121)
> > at 
> > ucar.unidata.data.point.TextPointDataSource.getDataInner(TextPointDataSource.java:1667)
> > at ucar.unidata.data.DataSourceImpl.getData(DataSourceImpl.java:2122)
> > at ucar.unidata.data.DirectDataChoice.getData(DirectDataChoice.java:332)
> > at ucar.unidata.data.DataChoice.getData(DataChoice.java:634)
> > at ucar.unidata.data.DataInstance.getData(DataInstance.java:239)
> > at ucar.unidata.data.DataInstance.getData(DataInstance.java:203)
> > at ucar.unidata.data.grid.GridDataInstance.init(GridDataInstance.java:204)
> > at ucar.unidata.data.grid.GridDataInstance.<init>(GridDataInstance.java:161)
> > at ucar.unidata.data.grid.GridDataInstance.<init>(GridDataInstance.java:142)
> > at 
> > ucar.unidata.idv.control.PlanViewControl.doMakeDataInstance(PlanViewControl.java:1274)
> > at 
> > ucar.unidata.idv.control.DisplayControlImpl.initializeDataInstance(DisplayControlImpl.java:2968)
> > at 
> > ucar.unidata.idv.control.DisplayControlImpl.setData(DisplayControlImpl.java:2950)
> > at 
> > ucar.unidata.idv.control.PlanViewControl.setData(PlanViewControl.java:559)
> > at 
> > ucar.unidata.idv.control.ContourPlanViewControl.setData(ContourPlanViewControl.java:112)
> > at ucar.unidata.idv.control.PlanViewControl.init(PlanViewControl.java:425)
> > at 
> > ucar.unidata.idv.control.DisplayControlImpl.init(DisplayControlImpl.java:1269)
> > at 
> > ucar.unidata.idv.control.DisplayControlImpl.init(DisplayControlImpl.java:978)
> > at 
> > ucar.unidata.idv.ControlDescriptor.initControl(ControlDescriptor.java:979)
> > at ucar.unidata.idv.ControlDescriptor$1.run(ControlDescriptor.java:904)
> > at ucar.unidata.util.Misc$3.run(Misc.java:1060)
> >
> >
> > I realize I was supposed to provide you with sample files and headers 
> > before, but haven't. Sorry about that. Doing it now. The attached csv file 
> > is the one I described in the previous paragraph. The simple header file is 
> > also attached, as is the original file I tested ("metars.current.csv", i.e. 
> > the generating only the "An error occurred reading column number:6").
> 
> Thanks.  Of course, there is no way to import the header, so we're going to 
> add that
> capability today.
> 
> > So, maybe all this is just bad data symptoms? Is there a way to find out 
> > without having to read the files manually, checking every value?
> 
> Some yes, some no.  There will be an error printed out when we can't parse a 
> observation
> correctly but we'll just ignore the observation.
> 
> > And, yes, GRIB issue seems OK.
> 
> At least we got one thing right. ;-)  Thanks for your patience and reporting 
> these
> issues.
> 
> Don
> 
> > -----Original Message-----
> > From: Unidata IDV Support [mailto:address@hidden]
> > Sent: Thursday, May 07, 2009 9:06 PM
> > To: address@hidden
> > Cc: address@hidden
> > Subject: [IDV #DGO-203144]: Point data questions
> >
> > Hi Hans-
> >
> > I thought I had replied last week, but just found my reply session sitting 
> > in
> > one of my browser windows.
> >
> > > > Just tried today's (Apr 28) build. The data looks much better now!
> > > > However, I do get some errors trying to read, and plot, the data.
> > >
> > > Sorry about this.  Fix one thing, break another. ;-)
> >
> > Hopefully we've made more fixes this time.  Please try the latest nightly
> > build.
> >
> > > > Firstly, in the previous version I defined names and units for the 
> > > > fields in the csv file and saved it under its own name as a preference. 
> > > > In the previous version (I think), when selecting the saved preference 
> > > > the field names popped up in the point data form. That does not happen 
> > > > with the new version.
> > >
> > > If you resize the window slightly, you should see the fields displayed. 
> > > We're
> > > working on a fix for this.  At this time, you'll need to scroll down to 
> > > the
> > > first observation before applying the old preference.
> >
> > Please try this with the latest build and see if the problem still
> > exists.
> >
> > > > Secondly, the first time I try to read the file (i.e. clicking OK after 
> > > > having applied the saved preference def that does not show), I get an 
> > > > error message. Trying a second time, however, the data is usually read 
> > > > without error messages. Could perhaps have something to do with some 
> > > > data missing in the file which results in an uninitialized field in IDV 
> > > > the first time the file is read (just my wild guess)?
> > >
> > > There was a change made to the underlying VisAD units package that is 
> > > causing
> > > this.  We are trying to track down the problem and hope to have a fix 
> > > soon.
> >
> > We put in some fixes for the units, but I'm not sure the issue below
> > is completely resolved.  I haven't been able to reproduce it today with
> > the latest build.  However, I was in that same situation yesterday and
> > then it popped up again.
> >
> > > > The error message shown is as follows:
> > > > " There was an error loading the data:
> > > > Error creating data source:file.pointtext with: 
> > > > E:\WorkingData\Data\Observations\metars.current.csv
> > > > Making grid parameters
> > > > An error occurred reading column number:6"
> > > > Detailed:
> > > > " java.lang.IllegalArgumentException: An error occurred reading column 
> > > > number:6
> > > > at visad.data.text.TextAdapter.readit(TextAdapter.java:990)
> > > > at visad.data.text.TextAdapter.<init>(TextAdapter.java:210)
> > > > at 
> > > > ucar.unidata.data.point.TextPointDataSource.makeObs(TextPointDataSource.java:386)
> > > > at 
> > > > ucar.unidata.data.point.TextPointDataSource.makeObs(TextPointDataSource.java:316)
> > > > at 
> > > > ucar.unidata.data.point.TextPointDataSource.getSample(TextPointDataSource.java:1578)
> > > > at 
> > > > ucar.unidata.data.point.PointDataSource.doMakeDataChoices(PointDataSource.java:900)
> > > > at 
> > > > ucar.unidata.data.point.TextPointDataSource.doMakeDataChoices(TextPointDataSource.java:1586)
> > > > at 
> > > > ucar.unidata.data.DataSourceImpl.getDataChoices(DataSourceImpl.java:1729)
> > > > at ucar.unidata.data.DataManager.createDataSource(DataManager.java:1484)
> > > > at 
> > > > ucar.unidata.idv.IntegratedDataViewer.createDataSource(IntegratedDataViewer.java:1954)
> > > > at 
> > > > ucar.unidata.idv.IntegratedDataViewer.makeDataSource(IntegratedDataViewer.java:1871)
> > > > at 
> > > > ucar.unidata.idv.IntegratedDataViewer.makeDataSource(IntegratedDataViewer.java:1805)
> > > > at 
> > > > ucar.unidata.idv.IntegratedDataViewer.makeDataSource(IntegratedDataViewer.java:1778)
> > > > at 
> > > > ucar.unidata.idv.chooser.IdvChooser.makeDataSource(IdvChooser.java:839)
> > > > at 
> > > > ucar.unidata.idv.chooser.FileChooser.selectFilesInner(FileChooser.java:646)
> > > > at 
> > > > ucar.unidata.idv.chooser.FileChooser.selectFiles(FileChooser.java:515)
> > > > at 
> > > > ucar.unidata.idv.chooser.FileChooser.doLoadInThread(FileChooser.java:537)
> > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > > at 
> > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > > > at 
> > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > > at ucar.unidata.util.Misc$2.run(Misc.java:1032)
> > > > at ucar.unidata.util.Misc$3.run(Misc.java:1060)
> > > > "
> > > >
> > > > Thirdly:
> > > > Plotting the data usually generates an error with no plot being shown. 
> > > > Trying with a much smaller subset of the file works though. The 
> > > > original file contains about 3800 lines of METAR data, each consisting 
> > > > of about 44 data fields. The file size of that file is about 700 kB. Is 
> > > > there a size restriction in IDV that might be the cause of this error 
> > > > perhaps?
> > >
> > > There shouldn't be.  I wonder if there is bad data causing a problem.  
> > > Could
> > > you send me two things:
> > >
> > > - the file you are trying to use
> > > - your preference for reading the file.  You can export this by reading in
> > > the data and under the Preferences button, select Export Header.
> > >
> > > Also, which field are you trying to contour?
> >
> > The issue here is that the objective analysis was breaking when there was 
> > only
> > one station.  There are two fixes to this:
> >
> > - I put in a check to have at least 3 stations before trying to do the 
> > analysis
> > - Since the times in the observations are not grouped, you will get lots of 
> > times,
> > each with as few as one or more, it would be better to group the 
> > observations
> > into manageable chunks of time.  Bring up the data source properties and 
> > set the
> > bin size and the round to times.  I tried a bin size of 1 hour and a round 
> > to of
> > on the hour.
> >
> > The latter avoids trying to do an objective analysis on too few stations, 
> > but
> > my fix will prevent that as well.
> >
> > The latest nightly build should also have the fix for your grib problem.
> >
> > Don Murray
> >
> > Ticket Details
> > ===================
> > Ticket ID: DGO-203144
> > Department: Support IDV
> > Priority: Normal
> > Status: Open
> >
> >
> >
> 
> 
> Ticket Details
> ===================
> Ticket ID: DGO-203144
> Department: Support IDV
> Priority: Normal
> Status: Open
> 
> 


Ticket Details
===================
Ticket ID: DGO-203144
Department: Support IDV
Priority: Normal
Status: Open