[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDV #AWX-208976]: Dataset subsetting gets lost, perhaps because for reasons related to this projection related NullPointer error?



Hi Brian,

Both Yuan and I were able to load your bundle off of your RAMADDA server, change
the Time Driver to use Sept. 16th, -1hr relative start time, and a three hour 
interval
via the animation widget, without any projection null pointer issue, and 
without any
issues with spatial subsetting. The only issue was slow data transfer times 
(which 
the IDV cannot control). Here is a link to the .xidv I made for 9 time steps:

http://motherlode.ucar.edu/repository/entry/show/RAMADDA/IDV+Community+Resources/Bundles/Sean/Bob+monsoon+depression+dissected+9time.xidv?entryid=e0ee895e-03f8-49c6-ac36-a3827a44dcde

> Can you clarify a little more what to watch and how and what consequences to 
> expect?
> Isn't there lot of fixed memory cost up front, which grows only modestly with 
> added data

The main issue is the number on the left (memory in use). The middle number is 
the 'high water
mark', or the maximum used so far in the session. As this gets close the right 
number (total memory
allocated to the Java Virtual Machine), the garbage collector will kick in and 
you will get a performance
hit. It could be possible that something gets accidentally garbage collected, 
which could have
unpredictable consequences (and would be sporadic depending on what gets 
clobbered).
However, this really should not be an issue. Now, if the left number hits the 
right number, then
we are in trouble - you've blown heap memory and you *must* restart the IDV 
(there's no
coming back from that one).

> Sean, happily the error is instant.
> 
> When I load this bundle, and then change the time driver even to just add one 
> more time level,
> the NullPointer exception appears instantly, and those zany vector-font white 
> error messages in the viewport appear.
> 
> http://weather.rsmas.miami.edu:8080/repository/entry/show/RSMAS-UM+Repository+for+atm-ocean+data+and+its+science/The+Mapes+IDV+collection/IDV+Bundles/Accessing+remote+datasets/1.+Templates.+You+adjust+time+driver./Past+weather+archives/Indian+Ocean/BoB+monsoon+depression+dissected+1time?entryid=30e31f43-3ab5-4cd1-b650-99f4bcf040a5
> 
> 
> Here is a snapshot of the memory counter as it is dying.
> No red flags, right?
> [cid:36b33ac2-f95d-402e-8b0b-1e0bbcee07ae@amazon.rsmas.miami.edu]
> 
> 
> 
> What happens for you?
> 
> Brian
> 
> 
> > The most reliable crash seems to be to load this template,
> and then set the time driver Start Time and Interval to 12h or 1d before the 
> End Time
> (rather than a 3h spacing and interval which has worked OK for me).
> 
> http://weather.rsmas.miami.edu:8080/repository/entry/get/BoB%20monsoon%20depression%20dissected%201time.xidv?entryid=30e31f43-3ab5-4cd1-b650-99f4bcf040a5
> 
> 
> On Oct 21, 2013, at 8:29 PM,  wrote:
> 
> I had some luck, it worked for a while.
> I tried setting the time driver to 2 adjacent (3h spaced) times, then 8, then 
> 9, and it never crashed.
> Now I just published a 9-time .zidv bundle, it is only 144MB in size. 
> Workable. Memory not near cieling.
> 
> 
> The only weird thing I see is that the dtdttot cross section has no data at 
> the last time in the time driver range.
> All other fields have data at all times, except that one cross section. Weird.
> So I thought I would go remake that display, and see if its 9th time works in 
> that case.
> At that moment, I noticed that all the subsetting is gone when I go to Field 
> Selector.
> All the subsetting is gone from all the datasets, I see in their Properties 
> menus.
> Did it happen during publishing the .zidv? I think so.
> 
> Now when I jump the Time Driver to the 2007-08-06 case (1 time level), global 
> data come up. Subsetting is lost.
> 
> 
> ----------------
> May I again suggest trying to perform my scientific workflow?
> Load the template. Make 9-frame animations for the two dates I indicated.
> 2007-08-06, 2008-09-16. Publish them as .zidv.
> If you can do that without errors, great, and thanks, and maybe it's my 
> machine.
> I suspect you won't be able to.
> 
> At that stage you'd be right there when an error happens, with tools for 
> understanding
> the errors, unlike me. That's the debug moment. After another day, I still 
> can't get you there
> absolutely reliably in one step that I have checked twice is indeed a bug.
> But I bet you a beer you can't do the work I need to do by Thursday without
> encountering the bug.
> 
> Appreciating any help.
> Brian
> 
> 

Brian,
      I spent some times on this bundle, and I am 100 percent sure there is no 
bug.
      This is a very big bundle, and you need a lot memory if you want to do a 
9 time steps loop. In my MacBook with 8G memory, I can load the template and 
change the time driver to get 4 time steps. If I try 5 time steps, I got error 
message. If you kill my InteliJ (the Java IDE), I can do 5.

      Checking this bundle, there are 6 3D volume displays (include cross 
section) and this requires many memory. If you want, you can run the activity 
monitor in your Mac to check the memory usage. 

      In the IDV, we use 80% of the total memory of your system instead of the 
free memory. This may cause some confusion. Sometime, you already max out the 
memory before you reach the third number.

      The solution is either to add more memory to your Mac, or to remove a few 
3D displays. 


Yuan



> 
> On Oct 21, 2013, at 4:19 PM, Brian Mapes wrote:
> 
> Sean, happily the error is instant.
> 
> When I load this bundle, and then change the time driver even to just add one 
> more time level,
> the NullPointer exception appears instantly, and those zany vector-font white 
> error messages in the viewport appear.
> 
> http://weather.rsmas.miami.edu:8080/repository/entry/show/RSMAS-UM+Repository+for+atm-ocean+data+and+its+science/The+Mapes+IDV+collection/IDV+Bundles/Accessing+remote+datasets/1.+Templates.+You+adjust+time+driver./Past+weather+archives/Indian+Ocean/BoB+monsoon+depression+dissected+1time?entryid=30e31f43-3ab5-4cd1-b650-99f4bcf040a5
> 
> 
> Here is a snapshot of the memory counter as it is dying.
> No red flags, right?
> <PastedGraphic-1.png>
> 
> 
> 
> What happens for you?
> 
> Brian
> 
> 
> Brian Mapes
> address@hidden<mailto:address@hidden> or address@hidden<mailto:address@hidden>
> 
> 
> 
> 
> Brian Mapes
> address@hidden<mailto:address@hidden> or address@hidden<mailto:address@hidden>
> 
> 
> 
> 
> 


Ticket Details
===================
Ticket ID: AWX-208976
Department: Support IDV
Priority: Normal
Status: Open