[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDV #AFS-891091]: Using Large WRF Gridded Ouput Files in IDV

> Yuan,
> 
> I sent the WRF data file as an attachment to the Support Request Form.
> 
> Jim
> 

Is this a HWRF output file? There is no unit for the vertical coordinate 
variable, the IDV can not
locate the vertical position correctly. If this is a HWRF output, you need to 
run a script to
post process it.

Yuan
> -----Original Message-----
> From: Unidata IDV Support [mailto:address@hidden]
> Sent: Monday, March 19, 2018 10:24 AM
> To: address@hidden
> Cc: address@hidden; address@hidden; address@hidden
> Subject: [IDV #AFS-891091]: Using Large WRF Gridded Ouput Files in IDV
> 
> > Hi,
> >
> >
> >
> > I have been using VAPOR successfully for several years now to study
> > Hurricane Sandy WRF output from NetCDF files of size 892 MB for each
> > time step, but I now need to do more intensive analyses and create
> > publication-quality plots on even larger files and I hope IDV will
> > become my go-to package for these purposes. I have been using VAPOR
> > very efficiently on a Windows 8.1 Intel quad-core i7 machine with 16 GB RAM.
> >
> 
> Hi Jim,
> 
> This is a long support question. I will answer each part under the question. 
> You need to allocate as much as possible: Edit > Preference > System, ..... 
> to reset the memory.
> >
> >
> > With IDV so far I have been experimenting with my existing WRF files
> > and I already came up against a number of limitations which I hope you
> > can help me resolve so I can indeed eventually become a daily user of
> > IDV on even larger WRF files:
> >
> 
> The larger WRF files we are working on regularly are the HRRR operation 
> outputs.
> >
> >
> > .         Vertical scaling: Some of my files are unable to scale vertical
> > coordinates properly. My experimenting tells me that the 4 WRF
> > variables P, PB, PH, PHB are required, but please confirm whether this
> > indeed is the minimum set to scale vertical coordinates. I ask because
> > VAPOR did not require the P and PB variables to scale vertically.
> 
> The vertical coordinate is sigma, you don't need to do anything. We can deal 
> with the original output but might be slow if we need to destaggeering.
> 
> >
> > .         Slow computation of derived fields: Computing speed from WRF U and
> > V variables for a 1000x1000x30 grid took about 15 minutes and
> > computing relative vorticity took > 90 minutes before I terminated the
> > process. IDV User Guide Section 4.5 recommends using ARWPost or
> > WRFPost "to destagger the grids and compute the standard
> > meteorological variables on pressure or height coordinates" and states
> > that "IDV can read the GRIB (1 & 2), GEMPAK and Vis5D output from these 
> > post-process routines."
> >
> > o   Which format among GRIB (1 & 2), GEMPAK and Vis5D will provide the most
> > efficient usage for IDV?
> 
> Girb 2 is my choice.
> >
> > o   Do I decide between computing variables on pressure or height
> > coordinates based solely on my particular planned analyses?
> 
> I don't understand this, the IDV can deal with the coordinate transform 
> between pressure and height automatically.
> >
> > o   I assume that IDV will be able to input files if I compute variables on
> > both pressure and height coordinates (at the expense of doubling file
> > size) but please confirm.
> 
> Yes.
> >
> > .         List of standard meteorological variables: What is the complete
> > set of standard meteorological variables that IDV recognizes and which
> > I should consider for computation on pressure or height coordinates?
> 
> You can always add the unrecognized variable to IDV alias list, so the IDV 
> will recognize it next time.
> >
> > .         List of derived fields: I pored through the 519-page IDV User
> > Guide and searched elsewhere, but nowhere could I find a complete list
> > of the different built-in Derived parameters [e.g., Potential
> > Vorticity (from theta & flowvectors) and Horizontal Divergence (from U
> > & V)] that show up in the Field Selector panels and which depend on
> > the set of WRF variables contained in my particular WRF data file. I
> > need to know the complete list and on which WRF variables they depend,
> > to evaluate which WRF variables to include in future WRF data files.
> > And since each raw WRF time step file is
> > 224 GB in size, it is essential that I extract only those WRF
> > variables that I absolutely need to keep the file sizes manageable.
> 
> There are too many derived fields, we don't have an official list, it all 
> depends on the original variables the model generated.
> >
> > .         File size limits: Is there a maximum WRF file size or a size
> > guideline for reading into IDV? For example, will IDV handle 10 GB WRF
> > NetCDF files, presumably after conversion by a recommended WRF
> > post-processor? Is there a limit on the number of WRF variables that
> > IDV will read? I presume that any of these limits would be system
> > dependent so if you know limits for IDV on Microsoft Windows please let me 
> > know.
> >
> 
> No limit, the IDV only read in the metadata at the beginning, only when you 
> want to create the display it goes to get the real data part.
> >
> >
> > I am asking all these questions upfront since their answers bear
> > heavily on how I proceed with my research and how I design my files
> > for import into IDV, especially because of their large sizes.
> >
> 
> I will contact you sometime this week to chat over the phone, it will be 
> easier to understand your concerns and problem.
> >
> >
> > Thanks much in advance for your help - with it I might even become a
> > power user of IDV at some point!
> >
> >
> >
> > Jim
> >
> >
> >
> > James A. Schiavone
> >
> > Independent Scientist*
> >
> > Bridgewater, NJ
> >
> > Phone: 908-658-3158
> >
> >
> >
> > * Collaborating with Rutgers University, Cray and NCAR
> >
> >
> >
> >
> >
> 
> 
> Ticket Details
> ===================
> Ticket ID: AFS-891091
> Department: Support IDV
> Priority: Normal
> Status: Open
> ===================
> NOTE: All email exchanges with Unidata User Support are recorded in the 
> Unidata inquiry tracking system and then made publicly available through the 
> web.  If you do not want to have your interactions made available in this 
> way, you must let us know in each email you send to us.
> 
> 
> 
> 


Ticket Details
===================
Ticket ID: AFS-891091
Department: Support IDV
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.