[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDV #AFS-891091]: Using Large WRF Gridded Ouput Files in IDV

> Hi,
> 
> 
> 
> I have been using VAPOR successfully for several years now to study
> Hurricane Sandy WRF output from NetCDF files of size 892 MB for each time
> step, but I now need to do more intensive analyses and create
> publication-quality plots on even larger files and I hope IDV will become my
> go-to package for these purposes. I have been using VAPOR very efficiently
> on a Windows 8.1 Intel quad-core i7 machine with 16 GB RAM.
> 

Hi Jim,

This is a long support question. I will answer each part under the question. 
You need
to allocate as much as possible: Edit > Preference > System, ..... to reset the 
memory.
> 
> 
> With IDV so far I have been experimenting with my existing WRF files and I
> already came up against a number of limitations which I hope you can help me
> resolve so I can indeed eventually become a daily user of IDV on even larger
> WRF files:
> 

The larger WRF files we are working on regularly are the HRRR operation 
outputs. 
> 
> 
> .         Vertical scaling: Some of my files are unable to scale vertical
> coordinates properly. My experimenting tells me that the 4 WRF variables P,
> PB, PH, PHB are required, but please confirm whether this indeed is the
> minimum set to scale vertical coordinates. I ask because VAPOR did not
> require the P and PB variables to scale vertically.

The vertical coordinate is sigma, you don't need to do anything. We can deal 
with
the original output but might be slow if we need to destaggeering.

> 
> .         Slow computation of derived fields: Computing speed from WRF U and
> V variables for a 1000x1000x30 grid took about 15 minutes and computing
> relative vorticity took > 90 minutes before I terminated the process. IDV
> User Guide Section 4.5 recommends using ARWPost or WRFPost "to destagger the
> grids and compute the standard meteorological variables on pressure or
> height coordinates" and states that "IDV can read the GRIB (1 & 2), GEMPAK
> and Vis5D output from these post-process routines."
> 
> o   Which format among GRIB (1 & 2), GEMPAK and Vis5D will provide the most
> efficient usage for IDV?

Girb 2 is my choice.
> 
> o   Do I decide between computing variables on pressure or height
> coordinates based solely on my particular planned analyses?

I don't understand this, the IDV can deal with the coordinate transform between
pressure and height automatically.
> 
> o   I assume that IDV will be able to input files if I compute variables on
> both pressure and height coordinates (at the expense of doubling file size)
> but please confirm.

Yes.
> 
> .         List of standard meteorological variables: What is the complete
> set of standard meteorological variables that IDV recognizes and which I
> should consider for computation on pressure or height coordinates?

You can always add the unrecognized variable to IDV alias list, so the IDV will
recognize it next time.
> 
> .         List of derived fields: I pored through the 519-page IDV User
> Guide and searched elsewhere, but nowhere could I find a complete list of
> the different built-in Derived parameters [e.g., Potential Vorticity (from
> theta & flowvectors) and Horizontal Divergence (from U & V)] that show up in
> the Field Selector panels and which depend on the set of WRF variables
> contained in my particular WRF data file. I need to know the complete list
> and on which WRF variables they depend, to evaluate which WRF variables to
> include in future WRF data files. And since each raw WRF time step file is
> 224 GB in size, it is essential that I extract only those WRF variables that
> I absolutely need to keep the file sizes manageable.

There are too many derived fields, we don't have an official list, it all 
depends on
the original variables the model generated.
> 
> .         File size limits: Is there a maximum WRF file size or a size
> guideline for reading into IDV? For example, will IDV handle 10 GB WRF
> NetCDF files, presumably after conversion by a recommended WRF
> post-processor? Is there a limit on the number of WRF variables that IDV
> will read? I presume that any of these limits would be system dependent so
> if you know limits for IDV on Microsoft Windows please let me know.
> 

No limit, the IDV only read in the metadata at the beginning, only when you
want to create the display it goes to get the real data part.
> 
> 
> I am asking all these questions upfront since their answers bear heavily on
> how I proceed with my research and how I design my files for import into
> IDV, especially because of their large sizes.
> 

I will contact you sometime this week to chat over the phone, it will be easier 
to
understand your concerns and problem.
> 
> 
> Thanks much in advance for your help - with it I might even become a power
> user of IDV at some point!
> 
> 
> 
> Jim
> 
> 
> 
> James A. Schiavone
> 
> Independent Scientist*
> 
> Bridgewater, NJ
> 
> Phone: 908-658-3158
> 
> 
> 
> * Collaborating with Rutgers University, Cray and NCAR
> 
> 
> 
> 
> 


Ticket Details
===================
Ticket ID: AFS-891091
Department: Support IDV
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.