[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDV #AFS-891091]: Using Large WRF Gridded Ouput Files in IDV

> Hi Yuan,
> 
> I am aiming to use the NCEP Unified Post Processor (UPP) to convert our WRF 
> model output to GRIB2 format for ingest into IDV 5.4. I plan to implement it 
> on a Linux Ubuntu 16.04 Intel Core i7 machine on which I previously 
> successfully built and test-ran WRF 3.6 using downloads and instructions from 
> http://www2.mmm.ucar.edu/wrf/users/.
> 
> The only caveat I see in the UPP Version 3 User's Guide is this note about 
> GRIB2:
> 
> "Note: This section [GRIB2 control file] describes the control file for 
> outputting GRIB2 format. Disclaimer: This feature is still in preliminary 
> stages and not fully tested; use with caution. Updates will be provided as 
> they become available."

I think this is fine, you can do the conversion. I wonder if it is possible to 
cut the size of the dataset before the conversion, you don't need
that kind of high resolution data. For the hurricane, the 3km resolution should 
be good enough.


Yuan
> 
> Does this caveat represent a possible pitfall for my plan as proposed above?
> 
> If so, can you recommend an alternative WRF-to-GRIB2 conversion tool which 
> would work with my machine and libraries that I already installed with WRF 
> 3.6?
> 
> Thanks!
> 
> Jim
> 
> -----Original Message-----
> From: Jim Schiavone [mailto:address@hidden]
> Sent: Friday, March 30, 2018 1:39 PM
> To: address@hidden
> Cc: address@hidden; address@hidden; address@hidden
> Subject: RE: [IDV #AFS-891091]: Using Large WRF Gridded Ouput Files in IDV
> 
> Hi Yuan,
> 
> I am happy to report that just within the past hour I managed to successfully 
> display variables from one of my Sandy time step files with what appears to 
> be proper vertical scaling. I still need to convert to GRIB2 because 
> computation is very slow, but at least I know how to modify the files now. As 
> it turns out, I only needed to append the ZNU and ZNW variables to the rest 
> of my data. I also edited the system preferences on my machine to Memory = 
> 12000 MB and Disk Cache Size = 8000 MB and it improved performance but not to 
> an acceptable level yet.
> 
> My next steps are:
> 
> 1. Select and implement a WRF post-processing tool to convert files to GRIB2
> 
> 2. Convert my trial time step file to GRIB2
> 
> 3. Test its performance with IDV
> 
> 4. Process all of my time step files
> 
> 5. Purchase a new computer with 64 GB RAM and 1 TB SSD.
> 
> Let me know if you perceive any glitches or constraints in my plan.
> 
> Thanks!
> 
> Jim
> 
> 
> -----Original Message-----
> From: Unidata IDV Support [mailto:address@hidden]
> Sent: Friday, March 30, 2018 1:14 PM
> To: address@hidden
> Cc: address@hidden; address@hidden; address@hidden; address@hidden; 
> address@hidden
> Subject: [IDV #AFS-891091]: Using Large WRF Gridded Ouput Files in IDV
> 
> > Hi Yuan,
> >
> > Per Pete's email attached, he has the WRF initial conditions file for our 
> > Sandy simulation which contains the ZNU and ZNW variable data. Since (1) 
> > our grid domain is stationary, (2) ZNU and ZNW are one-dimensional, and (3) 
> > ZNU and ZNW were specified in the initial conditions file, I assume that 
> > the vertical coordinates as specified in the ZNU and ZNW vectors are 
> > invariant among all XLAT and XLONG coordinates and among all of our time 
> > steps, but please have your WRF expert confirm my assumption.
> 
> Hi Jim,
> 
> Sorry for the delay replay, I just came back from a trip.
> 
> Yes, you use the ZNU and ZNW from init file.
> >
> > If my assumption above is correct, I should be able to merge our ZNU and 
> > ZNW variables and their data with our other WRF data for each time step 
> > using an NCO tool and then have IDV recognize ZNU or ZNW as vertical 
> > coordinates for our Sandy WRF output. Please also confirm if this latter 
> > assumption is correct.
> >
> 
> But you still need to covert them to GIRB2 format at the end, I am not sure 
> this is a good idea.
> 
> 
> Yuan
> > Thanks!
> >
> > Jim
> >
> > -----Original Message-----
> > From: Jim Schiavone [mailto:address@hidden]
> > Sent: Thursday, March 22, 2018 4:45 PM
> > To: address@hidden
> > Subject: RE: [IDV #AFS-891091]: Using Large WRF Gridded Ouput Files in
> > IDV
> >
> > Hi Yuan,
> >
> > I had the same thoughts about the complexity of modifying the existing data 
> > for sigma coordinates and the inability to do significant scientific 
> > analyses with the WRF output that I have. I will give this some thought 
> > while I am traveling the next 4 days and will get back to you next week.
> >
> > Thanks!
> >
> > Jim
> >
> > -----Original Message-----
> > From: Unidata IDV Support [mailto:address@hidden]
> > Sent: Thursday, March 22, 2018 4:16 PM
> > To: address@hidden
> > Cc: address@hidden; address@hidden; address@hidden;
> > address@hidden
> > Subject: [IDV #AFS-891091]: Using Large WRF Gridded Ouput Files in IDV
> >
> > > Hi Yuan,
> > >
> > > Per Attachment 1, Pete double-checked the original Sandy WRF auxhist file 
> > > for ZNU and ZNW variables and did not find either of them. Attachment 2 
> > > is summary output of the original Sandy WRF file. Attachment 3 is similar 
> > > summary output of a Katrina WRF file (an NCAR example file) which does 
> > > contain the ZNU and ZNW variables for comparison.
> >
> > Jim,
> >
> > It is clear this is not original WRF output, without vertical coordinate 
> > variable you can not do real scientific analysis on these datasets.
> > >
> > > I presume we now need to return to my questions Q1, Q2 and Q3 about 
> > > adding the ZNU and ZNW variables to my existing files.
> > >
> > > But because you since informed me that ZNU and ZNW are one-dimensional 
> > > variables (which I also can confirm in the Katrina file), this suggests 
> > > that the sigma coordinate system is fixed across the horizontal domain 
> > > and that I, therefore, also would need to specify an appropriate set of 
> > > fixed sigma vertical coordinates and then interpolate all of my 3D WRF 
> > > variables onto the specified sigma vertical coordinate set. Please 
> > > confirm if that is required, although it might be more processing than I 
> > > would want to pursue. Maybe there are existing tools that can help me 
> > > with this task?
> >
> > The sigma coordinate is very complicated, I will not recommend modifying 
> > the dataset at this stage. There are many sandy simulation output 
> > available, it would be easier to look around for the better one. I have 
> > some sandy data available online, the problem is that it has very limit set 
> > of variable.
> >
> > http://motherlode.ucar.edu/repository/entry/show?entryid=a149968d-3e94
> > -4ca4-b532-61c84c00a9b0&output=thredds.catalog
> >
> > Yuan
> > >
> > > Thanks!
> > >
> > > Jim
> > >
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: AFS-891091
> > Department: Support IDV
> > Priority: Normal
> > Status: Open
> > ===================
> > NOTE: All email exchanges with Unidata User Support are recorded in the 
> > Unidata inquiry tracking system and then made publicly available through 
> > the web.  If you do not want to have your interactions made available in 
> > this way, you must let us know in each email you send to us.
> >
> >
> >
> >
> >
> 
> 
> Ticket Details
> ===================
> Ticket ID: AFS-891091
> Department: Support IDV
> Priority: Normal
> Status: Open
> ===================
> NOTE: All email exchanges with Unidata User Support are recorded in the 
> Unidata inquiry tracking system and then made publicly available through the 
> web.  If you do not want to have your interactions made available in this 
> way, you must let us know in each email you send to us.
> 
> 
> 
> 
> 


Ticket Details
===================
Ticket ID: AFS-891091
Department: Support IDV
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.