Re: [conduit] Missing grids from some 0.5 gfs files on CONDUIT

Carissa,

Looks like the 12 UTC run came in complete this morning, with no short files. I'll keep an eye on it over the next several runs and let you know if I notice any more incomplete files.

Thanks!

Pete


On 04/24/2015 10:10 AM, Carissa Klemmer - NOAA Federal wrote:
Pete,

We put a fix in the just the GFS directories in time for the 12z cycle today. Could you let us know if you have any issues after that comes out? If it is successful then we will apply the fix to all directories.

Thanks,

Carissa Klemmer
NCEP Central Operations
Production Management Branch Dataflow Team
301-683-3835

On Thu, Apr 23, 2015 at 6:35 PM, Pete Pokrandt <poker@xxxxxxxxxxxx <mailto:poker@xxxxxxxxxxxx>> wrote:

    I'm assuming the problem hasn't been fixed yet, but just for info,
    so far the same thing has happened with the 18 UTC 23 run for

    F021, F030, F036 and F087 (data is currently in through F204)

    Pete


    On 04/23/2015 03:07 PM, Carissa Klemmer - NOAA Federal wrote:
    Pete,

    Apologies for the issues. We have just determined that this is a
    disk read/write issue which is why you see it as random. We have
    engaged our IT folks to resolve this issue. We will let you know
    as soon as the problem is fixed.

    Carissa Klemmer
    NCEP Central Operations
    Production Management Branch Dataflow Team
    301-683-3835 <tel:301-683-3835>

    On Thu, Apr 23, 2015 at 3:29 PM, Pete Pokrandt
    <poker@xxxxxxxxxxxx <mailto:poker@xxxxxxxxxxxx>> wrote:

        All,

        We have noticed a strange problem with the grids coming in
        for the 0.5 degree GFS files. The problem started with the 12
        UTC run of the 2015/04/22 run, and several files (though not
        always the same ones) have been missing grids in each run
        since. The complete files contain 366 grids, the incomplete
        files always contain 327 and are always missing exactly the
        following grids:

        5WAVH 5-Wave Geopotential Height [gpm]:500 mb
        ABSV Absolute Vorticity [1/s]:100 mb
        ABSV Absolute Vorticity [1/s]:975 mb
        CLWMR Cloud Mixing Ratio [kg/kg]:150 mb
        CLWMR Cloud Mixing Ratio [kg/kg]:550 mb
        CLWMR Cloud Mixing Ratio [kg/kg]:800 mb
        CSNOW Categorical Snow [-]:surface:
        DLWRF Downward Long-Wave Rad. Flux [W/m^2]:surface
        HGT Geopotential Height [gpm]:1000 mb
        HGT Geopotential Height [gpm]:300 mb
        HGT Geopotential Height [gpm]:50 mb
        HGT Geopotential Height [gpm]:highest tropospheric freezing level
        HGT Geopotential Height [gpm]:PV=2e-06 (Km^2/kg/s) surface
        HGT Geopotential Height [gpm]:tropopause
        HPBL Planetary Boundary Layer Height [m]:surface
        O3MR Ozone Mixing Ratio [kg/kg]:200 mb
        PRES Pressure [Pa]:low cloud top level:
        RH Relative Humidity [%]:400 mb
        TMAX Maximum Temperature [K]:2 m above ground:114-120 hour
        max fcst:
        TMP Temperature [K]:10 mb
        TMP Temperature [K]:350 mb
        TMP Temperature [K]:650 mb
        TMP Temperature [K]:900 mb
        TMP Temperature [K]:max wind
        TSOIL Soil Temperature    Validation to deprecate [K]:1-2 m
        below ground
        UGRD U-Component of Wind [m/s]:1829 m above mean sea level
        UGRD U-Component of Wind [m/s]:70 mb
        VGRD V-Component of Wind [m/s]:20 mb
        VGRD V-Component of Wind [m/s]:500 mb
        VGRD V-Component of Wind [m/s]:750 mb
        VGRD V-Component of Wind [m/s]:950 mb
        V-GWD Meridional Flux of Gravity Wave Stress [N/m^2]:surface:
        VVEL Vertical Velocity (Pressure) [Pa/s]:0.995 sigma level
        VVEL Vertical Velocity (Pressure) [Pa/s]:450 mb
        VVEL Vertical Velocity (Pressure) [Pa/s]:700 mb
        VVEL Vertical Velocity (Pressure) [Pa/s]:925 mb


        I don't think it's a latency issue for us, because our
        latencies for conduit have been < 30 seconds or so for at
        least the past 3 days.

        One curious thing, the files are complete on
        motherlode.ucar.edu <http://motherlode.ucar.edu>, but the
        files that we have as short, have a later modification time
        than the rest. For example, for the 12 UTC 23 run, we were
        missing grids for the F021, F102, F123 and F252 times. Here's
        a subset of the times on motherlode that shows these files:

        gfs.t12z.pgrb2.0p50.f018  
<http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f018>
                 23-Apr-2015 09:35   67M
        gfs.t12z.pgrb2.0p50.f021  
<http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f021>
                 23-Apr-2015 09:49   67M
        gfs.t12z.pgrb2.0p50.f024  
<http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f024>
                 23-Apr-2015 09:37   66M

        gfs.t12z.pgrb2.0p50.f099  
<http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f099>
                 23-Apr-2015 10:00   66M
        gfs.t12z.pgrb2.0p50.f102  
<http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f102>
                 23-Apr-2015 10:42   67M
        gfs.t12z.pgrb2.0p50.f105  
<http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f105>
                 23-Apr-2015 10:03   66M
        gfs.t12z.pgrb2.0p50.f108  
<http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f108>
                 23-Apr-2015 10:03   66M
        gfs.t12z.pgrb2.0p50.f111  
<http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f111>
                 23-Apr-2015 10:04   66M
        gfs.t12z.pgrb2.0p50.f114  
<http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f114>
                 23-Apr-2015 10:05   66M
        gfs.t12z.pgrb2.0p50.f117  
<http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f117>
                 23-Apr-2015 10:06   66M
        gfs.t12z.pgrb2.0p50.f120  
<http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f120>
                 23-Apr-2015 10:07   66M
        gfs.t12z.pgrb2.0p50.f123  
<http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f123>
                 23-Apr-2015 10:51   66M
        gfs.t12z.pgrb2.0p50.f126  
<http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f126>
                 23-Apr-2015 10:09   67M
        gfs.t12z.pgrb2.0p50.f129  
<http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f129>
                 23-Apr-2015 10:10   66M

        gfs.t12z.pgrb2.0p50.f237  
<http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f237>
                 23-Apr-2015 10:43   66M
        gfs.t12z.pgrb2.0p50.f240  
<http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f240>
                 23-Apr-2015 10:44   67M
        gfs.t12z.pgrb2.0p50.f252  
<http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f252>
                 23-Apr-2015 11:32   62M
        gfs.t12z.pgrb2.0p50.f264  
<http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f264>
                 23-Apr-2015 10:49   62M
        gfs.t12z.pgrb2.0p50.f276  
<http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f276>
                 23-Apr-2015 10:50   63M


        We are feeding redundantly from conduit.ncep.noaa.gov
        <http://conduit.ncep.noaa.gov> and ncepldm4.woc.noaa.gov
        <http://ncepldm4.woc.noaa.gov>

        Ideas?

        Here is a (I think) complete list of the times that had short
        files since 12UTC 4/22. I'm guessing all of these files are
        complete on motherlode, but have a later modification time
        than the rest of the run.

        2015042212 <tel:2015042212>_F081
        2015042212 <tel:2015042212>_F112
        2015042212 <tel:2015042212>_F141
        2015042212 <tel:2015042212>_F156
        2015042212 <tel:2015042212>_F159
        2015042212 <tel:2015042212>_F195
        2015042212 <tel:2015042212>_F228

        2015042218 <tel:2015042218>_F063
        2015042218 <tel:2015042218>_F081
        2015042218 <tel:2015042218>_F198

        2015042300 <tel:2015042300>_F006
        2015042300 <tel:2015042300>_F018
        2015042300 <tel:2015042300>_F021
        2015042300 <tel:2015042300>_F087
        2015042300 <tel:2015042300>_F114
        2015042300 <tel:2015042300>_F120
        2015042300 <tel:2015042300>_F138

        2015042306 <tel:2015042306>_F006
        2015042306 <tel:2015042306>_F069

        2015042312 <tel:2015042312>_F021
        2015042312 <tel:2015042312>_F102
        2015042312 <tel:2015042312>_F123
        2015042312 <tel:2015042312>_F252



        Pete






-- Pete Pokrandt - Systems Programmer
        UW-Madison Dept of Atmospheric and Oceanic Sciences
        608-262-3086  <tel:608-262-3086>   -poker@xxxxxxxxxxxx  
<mailto:poker@xxxxxxxxxxxx>


        _______________________________________________
        conduit mailing list
        conduit@xxxxxxxxxxxxxxxx <mailto:conduit@xxxxxxxxxxxxxxxx>
        For list information or to unsubscribe, visit:
        http://www.unidata.ucar.edu/mailing_lists/




-- Pete Pokrandt - Systems Programmer
    UW-Madison Dept of Atmospheric and Oceanic Sciences
    608-262-3086  <tel:608-262-3086>   -poker@xxxxxxxxxxxx  
<mailto:poker@xxxxxxxxxxxx>




--
Pete Pokrandt - Systems Programmer
UW-Madison Dept of Atmospheric and Oceanic Sciences
608-262-3086  - poker@xxxxxxxxxxxx