[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [conduit] No GFS data on CONDUIT since FV3 update

Great observations Pete!

Details for all the changes to the GFS v15 are in the SCN here:
https://www.weather.gov/media/notification/scn19-40gfs_v15_1.pdf

If you'd like to sign up to receive these notifications you can find instructions here:
https://www.weather.gov/notification/

Thanks,
Holly

On Wed, Jun 12, 2019 at 6:14 PM Pete Pokrandt <address@hidden> wrote:
All,

Adding the / between the YYYYMMDD and HH on the GFS pqact lines DOES fix the issue. Data is coming in correctly now, and should be populating thredds.aos.wisc.edu as we speak. 

I was also able to rescue the misnamed 12 UTC runs, so if anyone wants to pull that down to fill in a hole, the 1 deg, 0.5 deg, and 0.25 deg analysis and forecasts are up there on thredds.aos.wisc.edu too.

** WARNING** One thing - keep an eye on your disk space. The new times available (instead of 3h out to 240 and 12h out to 384, they now appear to be 3h out all the way to 384) and new variables available (even the 1deg GRIB2 files have increased in size from ~21 Mb/forecast hour to ~29 Mb/forecast hour)   The peak hourly CONDUIT volume increased dramaticlly starting with the 12 UTC run from ~20 Gb/h to nearly 33 Gb/h

If you push the edge on how much data you can keep before your disk fills up, and you are pretty general about what you save in your pqact files, this will put you over the edge. You'll have to consider not keeping as much data as before.

Here's an example of what the additional variables are in the new GFS files (this is from the 60h forecast of the 18 UTC run today, compared to the 06 UTC run - from wgrib.

This variable is no longer in the files:

kpds=189,1,0:60hr fcst:winds are N/S:"Land cover (nearest neighbor) [sea=0,land=1]

These variables are new:

kpds=11,100,0:60hr fcst:winds are N/S:"Temp. [K]
kpds=11,100,15:60hr fcst:winds are N/S:"Temp. [K]
kpds=11,100,40:60hr fcst:winds are N/S:"Temp. [K]
kpds=140,1,0:60hr fcst:winds are N/S:"Categorical rain [yes=1;no=0]
kpds=141,1,0:60hr fcst:winds are N/S:"Categorical freezing rain [yes=1;no=0]
kpds=142,1,0:60hr fcst:winds are N/S:"Categorical ice pellets [yes=1;no=0]
kpds=143,1,0:60hr fcst:winds are N/S:"Categorical snow [yes=1;no=0]
kpds=153,100,50:60hr fcst:winds are N/S:"Cloud water [kg/kg]
kpds=153,109,1:60hr fcst:winds are N/S:"Cloud water [kg/kg]
kpds=154,100,0:60hr fcst:winds are N/S:"Ozone mixing ratio [kg/kg]
kpds=154,100,1000:60hr fcst:winds are N/S:"Ozone mixing ratio [kg/kg]
kpds=154,100,15:60hr fcst:winds are N/S:"Ozone mixing ratio [kg/kg]
kpds=154,100,40:60hr fcst:winds are N/S:"Ozone mixing ratio [kg/kg]
kpds=154,100,500:60hr fcst:winds are N/S:"Ozone mixing ratio [kg/kg]
kpds=154,100,700:60hr fcst:winds are N/S:"Ozone mixing ratio [kg/kg]
kpds=154,100,850:60hr fcst:winds are N/S:"Ozone mixing ratio [kg/kg]
kpds=170,100,1000:60hr fcst:winds are N/S:"Rain water mixing ratio [kg/kg]
kpds=170,100,100:60hr fcst:winds are N/S:"Rain water mixing ratio [kg/kg]
kpds=170,100,150:60hr fcst:winds are N/S:"Rain water mixing ratio [kg/kg]
kpds=170,100,200:60hr fcst:winds are N/S:"Rain water mixing ratio [kg/kg]
kpds=170,100,250:60hr fcst:winds are N/S:"Rain water mixing ratio [kg/kg]
kpds=170,100,300:60hr fcst:winds are N/S:"Rain water mixing ratio [kg/kg]
kpds=170,100,350:60hr fcst:winds are N/S:"Rain water mixing ratio [kg/kg]
kpds=170,100,400:60hr fcst:winds are N/S:"Rain water mixing ratio [kg/kg]
kpds=170,100,450:60hr fcst:winds are N/S:"Rain water mixing ratio [kg/kg]
kpds=170,100,500:60hr fcst:winds are N/S:"Rain water mixing ratio [kg/kg]
kpds=170,100,50:60hr fcst:winds are N/S:"Rain water mixing ratio [kg/kg]
kpds=170,100,550:60hr fcst:winds are N/S:"Rain water mixing ratio [kg/kg]
kpds=170,100,600:60hr fcst:winds are N/S:"Rain water mixing ratio [kg/kg]
kpds=170,100,650:60hr fcst:winds are N/S:"Rain water mixing ratio [kg/kg]
kpds=170,100,700:60hr fcst:winds are N/S:"Rain water mixing ratio [kg/kg]
kpds=170,100,750:60hr fcst:winds are N/S:"Rain water mixing ratio [kg/kg]
kpds=170,100,800:60hr fcst:winds are N/S:"Rain water mixing ratio [kg/kg]
kpds=170,100,850:60hr fcst:winds are N/S:"Rain water mixing ratio [kg/kg]
kpds=170,100,900:60hr fcst:winds are N/S:"Rain water mixing ratio [kg/kg]
kpds=170,100,925:60hr fcst:winds are N/S:"Rain water mixing ratio [kg/kg]
kpds=170,100,950:60hr fcst:winds are N/S:"Rain water mixing ratio [kg/kg]
kpds=170,100,975:60hr fcst:winds are N/S:"Rain water mixing ratio [kg/kg]
kpds=170,109,1:60hr fcst:winds are N/S:"Rain water mixing ratio [kg/kg]
kpds=171,100,1000:60hr fcst:winds are N/S:"Snow mixing ratio [kg/kg]
kpds=171,100,100:60hr fcst:winds are N/S:"Snow mixing ratio [kg/kg]
kpds=171,100,150:60hr fcst:winds are N/S:"Snow mixing ratio [kg/kg]
kpds=171,100,200:60hr fcst:winds are N/S:"Snow mixing ratio [kg/kg]
kpds=171,100,250:60hr fcst:winds are N/S:"Snow mixing ratio [kg/kg]
kpds=171,100,300:60hr fcst:winds are N/S:"Snow mixing ratio [kg/kg]
kpds=171,100,350:60hr fcst:winds are N/S:"Snow mixing ratio [kg/kg]
kpds=171,100,400:60hr fcst:winds are N/S:"Snow mixing ratio [kg/kg]
kpds=171,100,450:60hr fcst:winds are N/S:"Snow mixing ratio [kg/kg]
kpds=171,100,500:60hr fcst:winds are N/S:"Snow mixing ratio [kg/kg]
kpds=171,100,50:60hr fcst:winds are N/S:"Snow mixing ratio [kg/kg]
kpds=171,100,550:60hr fcst:winds are N/S:"Snow mixing ratio [kg/kg]
kpds=171,100,600:60hr fcst:winds are N/S:"Snow mixing ratio [kg/kg]
kpds=171,100,650:60hr fcst:winds are N/S:"Snow mixing ratio [kg/kg]
kpds=171,100,700:60hr fcst:winds are N/S:"Snow mixing ratio [kg/kg]
kpds=171,100,750:60hr fcst:winds are N/S:"Snow mixing ratio [kg/kg]
kpds=171,100,800:60hr fcst:winds are N/S:"Snow mixing ratio [kg/kg]
kpds=171,100,850:60hr fcst:winds are N/S:"Snow mixing ratio [kg/kg]
kpds=171,100,900:60hr fcst:winds are N/S:"Snow mixing ratio [kg/kg]
kpds=171,100,925:60hr fcst:winds are N/S:"Snow mixing ratio [kg/kg]
kpds=171,100,950:60hr fcst:winds are N/S:"Snow mixing ratio [kg/kg]
kpds=171,100,975:60hr fcst:winds are N/S:"Snow mixing ratio [kg/kg]
kpds=171,109,1:60hr fcst:winds are N/S:"Snow mixing ratio [kg/kg]
kpds=178,100,1000:60hr fcst:winds are N/S:"Ice mixing ratio [kg/kg]
kpds=178,100,100:60hr fcst:winds are N/S:"Ice mixing ratio [kg/kg]
kpds=178,100,150:60hr fcst:winds are N/S:"Ice mixing ratio [kg/kg]
kpds=178,100,200:60hr fcst:winds are N/S:"Ice mixing ratio [kg/kg]
kpds=178,100,250:60hr fcst:winds are N/S:"Ice mixing ratio [kg/kg]
kpds=178,100,300:60hr fcst:winds are N/S:"Ice mixing ratio [kg/kg]
kpds=178,100,350:60hr fcst:winds are N/S:"Ice mixing ratio [kg/kg]
kpds=178,100,400:60hr fcst:winds are N/S:"Ice mixing ratio [kg/kg]
kpds=178,100,450:60hr fcst:winds are N/S:"Ice mixing ratio [kg/kg]
kpds=178,100,500:60hr fcst:winds are N/S:"Ice mixing ratio [kg/kg]
kpds=178,100,50:60hr fcst:winds are N/S:"Ice mixing ratio [kg/kg]
kpds=178,100,550:60hr fcst:winds are N/S:"Ice mixing ratio [kg/kg]
kpds=178,100,600:60hr fcst:winds are N/S:"Ice mixing ratio [kg/kg]
kpds=178,100,650:60hr fcst:winds are N/S:"Ice mixing ratio [kg/kg]
kpds=178,100,700:60hr fcst:winds are N/S:"Ice mixing ratio [kg/kg]
kpds=178,100,750:60hr fcst:winds are N/S:"Ice mixing ratio [kg/kg]
kpds=178,100,800:60hr fcst:winds are N/S:"Ice mixing ratio [kg/kg]
kpds=178,100,850:60hr fcst:winds are N/S:"Ice mixing ratio [kg/kg]
kpds=178,100,900:60hr fcst:winds are N/S:"Ice mixing ratio [kg/kg]
kpds=178,100,925:60hr fcst:winds are N/S:"Ice mixing ratio [kg/kg]
kpds=178,100,950:60hr fcst:winds are N/S:"Ice mixing ratio [kg/kg]
kpds=178,100,975:60hr fcst:winds are N/S:"Ice mixing ratio [kg/kg]
kpds=178,109,1:60hr fcst:winds are N/S:"Ice mixing ratio [kg/kg]
kpds=179,100,1000:60hr fcst:winds are N/S:"Graupel mixing ratio [kg/kg]
kpds=179,100,100:60hr fcst:winds are N/S:"Graupel mixing ratio [kg/kg]
kpds=179,100,150:60hr fcst:winds are N/S:"Graupel mixing ratio [kg/kg]
kpds=179,100,200:60hr fcst:winds are N/S:"Graupel mixing ratio [kg/kg]
kpds=179,100,250:60hr fcst:winds are N/S:"Graupel mixing ratio [kg/kg]
kpds=179,100,300:60hr fcst:winds are N/S:"Graupel mixing ratio [kg/kg]
kpds=179,100,350:60hr fcst:winds are N/S:"Graupel mixing ratio [kg/kg]
kpds=179,100,400:60hr fcst:winds are N/S:"Graupel mixing ratio [kg/kg]
kpds=179,100,450:60hr fcst:winds are N/S:"Graupel mixing ratio [kg/kg]
kpds=179,100,500:60hr fcst:winds are N/S:"Graupel mixing ratio [kg/kg]
kpds=179,100,50:60hr fcst:winds are N/S:"Graupel mixing ratio [kg/kg]
kpds=179,100,550:60hr fcst:winds are N/S:"Graupel mixing ratio [kg/kg]
kpds=179,100,600:60hr fcst:winds are N/S:"Graupel mixing ratio [kg/kg]
kpds=179,100,650:60hr fcst:winds are N/S:"Graupel mixing ratio [kg/kg]
kpds=179,100,700:60hr fcst:winds are N/S:"Graupel mixing ratio [kg/kg]
kpds=179,100,750:60hr fcst:winds are N/S:"Graupel mixing ratio [kg/kg]
kpds=179,100,800:60hr fcst:winds are N/S:"Graupel mixing ratio [kg/kg]
kpds=179,100,850:60hr fcst:winds are N/S:"Graupel mixing ratio [kg/kg]
kpds=179,100,900:60hr fcst:winds are N/S:"Graupel mixing ratio [kg/kg]
kpds=179,100,925:60hr fcst:winds are N/S:"Graupel mixing ratio [kg/kg]
kpds=179,100,950:60hr fcst:winds are N/S:"Graupel mixing ratio [kg/kg]
kpds=179,100,975:60hr fcst:winds are N/S:"Graupel mixing ratio [kg/kg]
kpds=179,109,1:60hr fcst:winds are N/S:"Graupel mixing ratio [kg/kg]
kpds=212,200,0:60hr fcst:winds are N/S:"Maximum/Composite radar reflectivity [dbZ]
kpds=255,1,0:60hr fcst:winds are N/S:"undefined
kpds=33,105,20:60hr fcst:winds are N/S:"u wind [m/s]
kpds=33,105,30:60hr fcst:winds are N/S:"u wind [m/s]
kpds=33,105,40:60hr fcst:winds are N/S:"u wind [m/s]
kpds=33,105,50:60hr fcst:winds are N/S:"u wind [m/s]
kpds=34,105,20:60hr fcst:winds are N/S:"v wind [m/s]
kpds=34,105,30:60hr fcst:winds are N/S:"v wind [m/s]
kpds=34,105,40:60hr fcst:winds are N/S:"v wind [m/s]
kpds=34,105,50:60hr fcst:winds are N/S:"v wind [m/s]
kpds=40,100,1000:60hr fcst:winds are N/S:"Geometric vertical velocity [m/s]
kpds=40,100,100:60hr fcst:winds are N/S:"Geometric vertical velocity [m/s]
kpds=40,100,150:60hr fcst:winds are N/S:"Geometric vertical velocity [m/s]
kpds=40,100,200:60hr fcst:winds are N/S:"Geometric vertical velocity [m/s]
kpds=40,100,250:60hr fcst:winds are N/S:"Geometric vertical velocity [m/s]
kpds=40,100,300:60hr fcst:winds are N/S:"Geometric vertical velocity [m/s]
kpds=40,100,350:60hr fcst:winds are N/S:"Geometric vertical velocity [m/s]
kpds=40,100,400:60hr fcst:winds are N/S:"Geometric vertical velocity [m/s]
kpds=40,100,450:60hr fcst:winds are N/S:"Geometric vertical velocity [m/s]
kpds=40,100,500:60hr fcst:winds are N/S:"Geometric vertical velocity [m/s]
kpds=40,100,550:60hr fcst:winds are N/S:"Geometric vertical velocity [m/s]
kpds=40,100,600:60hr fcst:winds are N/S:"Geometric vertical velocity [m/s]
kpds=40,100,650:60hr fcst:winds are N/S:"Geometric vertical velocity [m/s]
kpds=40,100,700:60hr fcst:winds are N/S:"Geometric vertical velocity [m/s]
kpds=40,100,750:60hr fcst:winds are N/S:"Geometric vertical velocity [m/s]
kpds=40,100,800:60hr fcst:winds are N/S:"Geometric vertical velocity [m/s]
kpds=40,100,850:60hr fcst:winds are N/S:"Geometric vertical velocity [m/s]
kpds=40,100,900:60hr fcst:winds are N/S:"Geometric vertical velocity [m/s]
kpds=40,100,925:60hr fcst:winds are N/S:"Geometric vertical velocity [m/s]
kpds=40,100,950:60hr fcst:winds are N/S:"Geometric vertical velocity [m/s]
kpds=40,100,975:60hr fcst:winds are N/S:"Geometric vertical velocity [m/s]
kpds=41,100,0:60hr fcst:winds are N/S:"Absolute vorticity [/s]
kpds=41,100,15:60hr fcst:winds are N/S:"Absolute vorticity [/s]
kpds=41,100,40:60hr fcst:winds are N/S:"Absolute vorticity [/s]
kpds=59,1,0:60hr fcst:winds are N/S:"Precipitation rate [kg/m^2/s]
kpds=61,1,0:0-60hr acc:winds are N/S:"Total precipitation [kg/m^2]
kpds=63,1,0:0-60hr acc:winds are N/S:"Convective precipitation [kg/m^2]
kpds=7,100,0:60hr fcst:winds are N/S:"Geopotential height [gpm]
kpds=7,100,15:60hr fcst:winds are N/S:"Geopotential height [gpm]
kpds=7,100,40:60hr fcst:winds are N/S:"Geopotential height [gpm]
kpds=71,100,1000:60hr fcst:winds are N/S:"Total cloud cover [%]
kpds=71,100,100:60hr fcst:winds are N/S:"Total cloud cover [%]
kpds=71,100,150:60hr fcst:winds are N/S:"Total cloud cover [%]
kpds=71,100,200:60hr fcst:winds are N/S:"Total cloud cover [%]
kpds=71,100,250:60hr fcst:winds are N/S:"Total cloud cover [%]
kpds=71,100,300:60hr fcst:winds are N/S:"Total cloud cover [%]
kpds=71,100,350:60hr fcst:winds are N/S:"Total cloud cover [%]
kpds=71,100,400:60hr fcst:winds are N/S:"Total cloud cover [%]
kpds=71,100,450:60hr fcst:winds are N/S:"Total cloud cover [%]
kpds=71,100,500:60hr fcst:winds are N/S:"Total cloud cover [%]
kpds=71,100,50:60hr fcst:winds are N/S:"Total cloud cover [%]
kpds=71,100,550:60hr fcst:winds are N/S:"Total cloud cover [%]
kpds=71,100,600:60hr fcst:winds are N/S:"Total cloud cover [%]
kpds=71,100,650:60hr fcst:winds are N/S:"Total cloud cover [%]
kpds=71,100,700:60hr fcst:winds are N/S:"Total cloud cover [%]
kpds=71,100,750:60hr fcst:winds are N/S:"Total cloud cover [%]
kpds=71,100,800:60hr fcst:winds are N/S:"Total cloud cover [%]
kpds=71,100,850:60hr fcst:winds are N/S:"Total cloud cover [%]
kpds=71,100,900:60hr fcst:winds are N/S:"Total cloud cover [%]
kpds=71,100,925:60hr fcst:winds are N/S:"Total cloud cover [%]
kpds=71,100,950:60hr fcst:winds are N/S:"Total cloud cover [%]
kpds=71,100,975:60hr fcst:winds are N/S:"Total cloud cover [%]


Just FYI.
Pete


--
Pete Pokrandt - Systems Programmer
UW-Madison Dept of Atmospheric and Oceanic Sciences
608-262-3086  - address@hidden


From: address@hidden <address@hidden> on behalf of Pete Pokrandt <address@hidden>
Sent: Wednesday, June 12, 2019 1:30 PM
To: Mullenax, Robert R. (WFF-820.0)[ORBITAL SCIENCES CORPORATION]; address@hidden; address@hidden; address@hidden
Subject: Re: [conduit] No GFS data on CONDUIT since FV3 update
 
Great info, thanks Robert!

I'll use that info to update my pqact ingest lines and hopefully that will fix it. Not sure what we can do about the latencies going up - more more data again, I guess..

Pete


--
Pete Pokrandt - Systems Programmer
UW-Madison Dept of Atmospheric and Oceanic Sciences
608-262-3086  - address@hidden


From: Mullenax, Robert R. (WFF-820.0)[ORBITAL SCIENCES CORPORATION] <address@hidden>
Sent: Wednesday, June 12, 2019 1:25 PM
To: Pete Pokrandt; address@hidden; address@hidden; address@hidden
Subject: RE: No GFS data on CONDUIT since FV3 update
 

They changed the directory structure on nomads/ftpprd of the GFS with todays upgrade. I had to update my grib filter scripts. Instead of the directory being gfs.YYYYMMDDHH, it is now gfs.YYYYMMDD/HH (where HH is the hour of the forecast run). I imagine that has something to do with it.

 

Regards,

Robert Mullenax

 

 

From: address@hidden <address@hidden> On Behalf Of Pete Pokrandt
Sent: Wednesday, June 12, 2019 1:21 PM
To: address@hidden; address@hidden; address@hidden
Subject: Re: [ldm-users] No GFS data on CONDUIT since FV3 update

 

So this is interesting. In a local nfs server directory where the 0.25 gfs data out to 87h is saved, one file per forecast hour named something like

 

gblav0p25.19061206_F024

 

for the 12 UTC run, there was a directory created there named

 

gblav0p25.190612/

 

and in there were files named

 

1_F000

1_F003

1_F006

..

 

So it must have come through and keyed in on something, but the file naming scheme/structure must be different.

 

Here's the existing pqact line that created the above:

 

 

# 0.25 deg GFS analysis [huge]

CONDUIT ^data/nccf/com/gfs/prod/gfs.20(..)(..)(..)(..)/.*pgrb2.0p25.(anl)

        FILE    /data/grib2/gblav0p25.\1\2\3\4_F\5

#

# GFS Global 0.25 degree forecast out to 99h only

#

CONDUIT ^data/nccf/com/gfs/prod/gfs\.20(..)(..)(..)(..).*pgrb2\.0p25\.f(0[0-8].)

        FILE

        /data/grib2/gblav0p25.\1\2\3\4_F\5

 

 

So.. I do have the 12 UTC 0.25 deg GFS run every 3h out to 87h if anyone wants it for posterity. On that machine I don't save anything beyond.

 

Actually I just looked, and on our thredds server there is a similar folder/file saved - and it should have all of the data, out to 384h.

 

FWIW.

Pete

 

 

--
Pete Pokrandt - Systems Programmer
UW-Madison Dept of Atmospheric and Oceanic Sciences
608-262-3086  - address@hidden

 

From: address@hidden <address@hidden> on behalf of Pete Pokrandt <address@hidden>
Sent: Wednesday, June 12, 2019 1:08 PM
To: address@hidden; address@hidden; address@hidden
Subject: [conduit] No GFS data on CONDUIT since FV3 update

 

All,

 

I don't know if it wasn't transmitted on CONDUIT or if it was and the pqact info for it just changed, but we didn't save any GFS data from today's 12 UTC run - since it was updated to the FV3 version. Based on the volume of CONDUIT traffic this morning, the peak of the 12 UTC run increased from ~20 Gb/h to almost 35 Gb/h so something definitely changed.

 

 

 

Looks like the Unidata thredds server is also missing the 12 UTC data.

 

I'm capturing the output from ldmadmin watch -f conduit running on our primary ingest server, so hopefully I'll be able to see what the data looks like coming through.

 

I'll let you know what I find - or if anyone else has any good info, please pass it along.

 

Thanks,

Pete

 

 

--
Pete Pokrandt - Systems Programmer
UW-Madison Dept of Atmospheric and Oceanic Sciences
608-262-3086  - address@hidden

_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web.  Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.


conduit mailing list
address@hidden
For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/ 
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web.  Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.


conduit mailing list
address@hidden
For list information or to unsubscribe, visit: 
http://www.unidata.ucar.edu/mailing_lists/