[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #BOA-542231]: Problem writing 0.5 deg GFS to disk

Hi Gerry,

re:
> And a happy Sunday to you guys!

Sorry, I took the day off yesterday... hunkered in at home where the
temperatures hit a high of about 9F for a half hour or so before heading
back towards 0F.  Overnight it was below zero (-1 to -2) from 11 pm
until and including now (8 am).  A bit chilly!

re:
> I've talked to Tom about this late last week, but we didn't quite get
> something right... I need to snag the gfs grid 004 (0.5 deg) via conduit
> as an element of my WRF workflow, and what I'm getting isn't right.
> I've included a verbose ls listing from the CONDUIT-derived files, and
> one after I've replaced 'em with files downloaded vis FTP for
> comparison.  I've also included the pqact.conduit.conf file I'm trying
> to use.

A quick look at the sizes of files created by the CONDUIT actions shows
that the double filing problem we worked on is gone.

Question:

- what specifically is the problem you are seeing?

If it is file sizes that are smaller than what you could FTP from
the NCEP servers, please be aware that the NCEP files can have
the same field at the same time/height/etc. replicated in the
file.  This will not happen with the data delivered by LDM since
the duplicate products will be detected and rejected at the
injection source.  The status/manifest file sent along with the
CONDUIT data will show which fields were successfully injected
into the feed, and which were eliminated by duplicate product
detection.

> I suppose I could automate checking for the existence of the files on
> the NCEP server, and automate processing at that point, but ldm and
> pqact are a bit more elegant for workflows.
> 
> Any ideas?

Not until I understand what the specific problem is...

> I suspect I've FUBAR'd something with the regex, but I've
> tried to compulse it and it works using the regex tester that comes with
> the distro.

Nothing is jumping out at me at this point.

- have you made a detailed listing of each GRIB message in the forecast
  files and compared the fields seen?

  I would use wgrib2 for this.

- if yes, is there any pattern to the difference(s)?

> ### CONDUIT GFS data... 20070825 dlc
> ### This gets all the available GFS 0.5x0.5 degree data
> ### was on radar2, copied to ernie, then moved to dc-ldm2 on 20091026 dlc
> ###
> #Grid #004, 0.5x0.5 degree lat/lon grid, gridid = 0p5deg (Hours F000-F180), 
> opnx=opnt, GRIB_version=2
> # File all GFS Grid #004 to disk (note includes new NCEP file NAMES)
> ###
> ## sample
> ##CONDUIT 015 data/nccf/com/gfs/prod/gfs.2009092412/gfs.t12z.pgrb2f174

> CONDUIT               (gfs\.(....)(..)(..)(..))/(.*)(pgrb2f[0-9][0-9]) .
> FILE  data/conduit/gfsgrid004/\2/\3/\4/gfs.\2\3\4\5/\6\7
> 
> CONDUIT               (gfs\.(....)(..)(..)(..))/(.*)(pgrb2f1[012][0-9])
> FILE  data/conduit/gfsgrid004/\2/\3/\4/gfs.\2\3\4\5/\6\7
> 
> ###
> # CONDUIT RUC grids (currently this grid is a duplicate of grib2)
> # Grid #252 (20km) grib1:     prod/ruc2a.*#252

> # Grid #252 (20km) grib2:     prod/ruc2a.*grib2
> CONDUIT               (ruc2a\.(....)(..)..)\/(.*)(pgrb20f.*)[ ].*
> FILE  data/conduit/ruc2aGrid252/\2/\3/\1/\4\5
> 
> 
> total 2117144
> -rw-rw-r--  1 ldm ldm 42613214 Dec  6 03:21 gfs.t06z.pgrb2f00
> -rw-rw-r--  1 ldm ldm 48385167 Dec  6 03:22 gfs.t06z.pgrb2f03
> -rw-rw-r--  1 ldm ldm 47927617 Dec  6 03:24 gfs.t06z.pgrb2f06
> -rw-rw-r--  1 ldm ldm 47994156 Dec  6 03:25 gfs.t06z.pgrb2f09
> -rw-rw-r--  1 ldm ldm 49734511 Dec  6 03:58 gfs.t06z.pgrb2f102
> -rw-rw-r--  1 ldm ldm 49720360 Dec  6 03:59 gfs.t06z.pgrb2f105
> -rw-rw-r--  1 ldm ldm 49615322 Dec  6 04:00 gfs.t06z.pgrb2f108
> -rw-rw-r--  1 ldm ldm 49633955 Dec  6 04:01 gfs.t06z.pgrb2f111
> -rw-rw-r--  1 ldm ldm 49362470 Dec  6 04:02 gfs.t06z.pgrb2f114
> -rw-rw-r--  1 ldm ldm 49161569 Dec  6 04:04 gfs.t06z.pgrb2f117
> -rw-rw-r--  1 ldm ldm 49053176 Dec  6 03:27 gfs.t06z.pgrb2f12
> -rw-rw-r--  1 ldm ldm 49269103 Dec  6 04:04 gfs.t06z.pgrb2f120
> -rw-rw-r--  1 ldm ldm 49051036 Dec  6 04:05 gfs.t06z.pgrb2f123
> -rw-rw-r--  1 ldm ldm 49331476 Dec  6 04:06 gfs.t06z.pgrb2f126
> -rw-rw-r--  1 ldm ldm 50262901 Dec  6 04:08 gfs.t06z.pgrb2f129
> -rw-rw-r--  1 ldm ldm 49358941 Dec  6 03:27 gfs.t06z.pgrb2f15
> -rw-rw-r--  1 ldm ldm 49141058 Dec  6 03:29 gfs.t06z.pgrb2f18
> -rw-rw-r--  1 ldm ldm 48997914 Dec  6 03:29 gfs.t06z.pgrb2f21
> -rw-rw-r--  1 ldm ldm 49131718 Dec  6 03:31 gfs.t06z.pgrb2f24
> -rw-rw-r--  1 ldm ldm 49325617 Dec  6 03:32 gfs.t06z.pgrb2f27
> -rw-rw-r--  1 ldm ldm 49330131 Dec  6 03:33 gfs.t06z.pgrb2f30
> -rw-rw-r--  1 ldm ldm 49243723 Dec  6 03:34 gfs.t06z.pgrb2f33
> -rw-rw-r--  1 ldm ldm 49194879 Dec  6 03:35 gfs.t06z.pgrb2f36
> -rw-rw-r--  1 ldm ldm 49404721 Dec  6 03:36 gfs.t06z.pgrb2f39
> -rw-rw-r--  1 ldm ldm 49190932 Dec  6 03:37 gfs.t06z.pgrb2f42
> -rw-rw-r--  1 ldm ldm 49032415 Dec  6 03:38 gfs.t06z.pgrb2f45
> -rw-rw-r--  1 ldm ldm 49056011 Dec  6 03:39 gfs.t06z.pgrb2f48
> -rw-rw-r--  1 ldm ldm 49308216 Dec  6 03:40 gfs.t06z.pgrb2f51
> -rw-rw-r--  1 ldm ldm 49506478 Dec  6 03:42 gfs.t06z.pgrb2f54
> -rw-rw-r--  1 ldm ldm 49817406 Dec  6 03:42 gfs.t06z.pgrb2f57
> -rw-rw-r--  1 ldm ldm 49565262 Dec  6 03:43 gfs.t06z.pgrb2f60
> -rw-rw-r--  1 ldm ldm 49519922 Dec  6 03:45 gfs.t06z.pgrb2f63
> -rw-rw-r--  1 ldm ldm 49630050 Dec  6 03:45 gfs.t06z.pgrb2f66
> -rw-rw-r--  1 ldm ldm 49547521 Dec  6 03:46 gfs.t06z.pgrb2f69
> -rw-rw-r--  1 ldm ldm 49221995 Dec  6 03:48 gfs.t06z.pgrb2f72
> -rw-rw-r--  1 ldm ldm 49939472 Dec  6 03:49 gfs.t06z.pgrb2f75
> -rw-rw-r--  1 ldm ldm 50155290 Dec  6 03:49 gfs.t06z.pgrb2f78
> -rw-rw-r--  1 ldm ldm 50571730 Dec  6 03:51 gfs.t06z.pgrb2f81
> -rw-rw-r--  1 ldm ldm 50462265 Dec  6 03:51 gfs.t06z.pgrb2f84
> -rw-rw-r--  1 ldm ldm 50160271 Dec  6 03:53 gfs.t06z.pgrb2f87
> -rw-rw-r--  1 ldm ldm 49799725 Dec  6 03:54 gfs.t06z.pgrb2f90
> -rw-rw-r--  1 ldm ldm 49398707 Dec  6 03:55 gfs.t06z.pgrb2f93
> -rw-rw-r--  1 ldm ldm 49618158 Dec  6 03:56 gfs.t06z.pgrb2f96
> -rw-rw-r--  1 ldm ldm 50118644 Dec  6 03:57 gfs.t06z.pgrb2f99
> 
> total 2114620
> -rw-rw-r--  1 ldm ldm 42577274 Dec  6 07:52 gfs.t06z.pgrb2f00
> -rw-rw-r--  1 ldm ldm 48344106 Dec  6 07:52 gfs.t06z.pgrb2f03
> -rw-rw-r--  1 ldm ldm 47886556 Dec  6 07:53 gfs.t06z.pgrb2f06
> -rw-rw-r--  1 ldm ldm 47912034 Dec  6 07:53 gfs.t06z.pgrb2f09
> -rw-rw-r--  1 ldm ldm 49693154 Dec  6 07:48 gfs.t06z.pgrb2f102
> -rw-rw-r--  1 ldm ldm 49679003 Dec  6 07:48 gfs.t06z.pgrb2f105
> -rw-rw-r--  1 ldm ldm 49532608 Dec  6 07:48 gfs.t06z.pgrb2f108
> -rw-rw-r--  1 ldm ldm 49592598 Dec  6 07:49 gfs.t06z.pgrb2f111
> -rw-rw-r--  1 ldm ldm 49279756 Dec  6 07:49 gfs.t06z.pgrb2f114
> -rw-rw-r--  1 ldm ldm 49078855 Dec  6 07:49 gfs.t06z.pgrb2f117
> -rw-rw-r--  1 ldm ldm 49012115 Dec  6 07:53 gfs.t06z.pgrb2f12
> -rw-rw-r--  1 ldm ldm 49186389 Dec  6 07:49 gfs.t06z.pgrb2f120
> -rw-rw-r--  1 ldm ldm 49009679 Dec  6 07:49 gfs.t06z.pgrb2f123
> -rw-rw-r--  1 ldm ldm 49290119 Dec  6 07:50 gfs.t06z.pgrb2f126
> -rw-rw-r--  1 ldm ldm 50221544 Dec  6 07:50 gfs.t06z.pgrb2f129
> -rw-rw-r--  1 ldm ldm 49276819 Dec  6 07:53 gfs.t06z.pgrb2f15
> -rw-rw-r--  1 ldm ldm 49058936 Dec  6 07:53 gfs.t06z.pgrb2f18
> -rw-rw-r--  1 ldm ldm 48915792 Dec  6 07:53 gfs.t06z.pgrb2f21
> -rw-rw-r--  1 ldm ldm 49049596 Dec  6 07:54 gfs.t06z.pgrb2f24
> -rw-rw-r--  1 ldm ldm 49243495 Dec  6 07:54 gfs.t06z.pgrb2f27
> -rw-rw-r--  1 ldm ldm 49248009 Dec  6 07:54 gfs.t06z.pgrb2f30
> -rw-rw-r--  1 ldm ldm 49161601 Dec  6 07:54 gfs.t06z.pgrb2f33
> -rw-rw-r--  1 ldm ldm 49153818 Dec  6 07:54 gfs.t06z.pgrb2f36
> -rw-rw-r--  1 ldm ldm 49363660 Dec  6 07:55 gfs.t06z.pgrb2f39
> -rw-rw-r--  1 ldm ldm 49149871 Dec  6 07:55 gfs.t06z.pgrb2f42
> -rw-rw-r--  1 ldm ldm 48991354 Dec  6 07:55 gfs.t06z.pgrb2f45
> -rw-rw-r--  1 ldm ldm 49014950 Dec  6 07:55 gfs.t06z.pgrb2f48
> -rw-rw-r--  1 ldm ldm 49267155 Dec  6 07:55 gfs.t06z.pgrb2f51
> -rw-rw-r--  1 ldm ldm 49465417 Dec  6 07:55 gfs.t06z.pgrb2f54
> -rw-rw-r--  1 ldm ldm 49776345 Dec  6 07:56 gfs.t06z.pgrb2f57
> -rw-rw-r--  1 ldm ldm 49524201 Dec  6 07:56 gfs.t06z.pgrb2f60
> -rw-rw-r--  1 ldm ldm 49437800 Dec  6 07:56 gfs.t06z.pgrb2f63
> -rw-rw-r--  1 ldm ldm 49547928 Dec  6 07:56 gfs.t06z.pgrb2f66
> -rw-rw-r--  1 ldm ldm 49465399 Dec  6 07:56 gfs.t06z.pgrb2f69
> -rw-rw-r--  1 ldm ldm 49139873 Dec  6 07:57 gfs.t06z.pgrb2f72
> -rw-rw-r--  1 ldm ldm 49857350 Dec  6 07:57 gfs.t06z.pgrb2f75
> -rw-rw-r--  1 ldm ldm 50073168 Dec  6 07:57 gfs.t06z.pgrb2f78
> -rw-rw-r--  1 ldm ldm 50489608 Dec  6 07:57 gfs.t06z.pgrb2f81
> -rw-rw-r--  1 ldm ldm 50421204 Dec  6 07:57 gfs.t06z.pgrb2f84
> -rw-rw-r--  1 ldm ldm 50119210 Dec  6 07:57 gfs.t06z.pgrb2f87
> -rw-rw-r--  1 ldm ldm 49758664 Dec  6 07:58 gfs.t06z.pgrb2f90
> -rw-rw-r--  1 ldm ldm 49357646 Dec  6 07:58 gfs.t06z.pgrb2f93
> -rw-rw-r--  1 ldm ldm 49577097 Dec  6 07:58 gfs.t06z.pgrb2f96
> -rw-rw-r--  1 ldm ldm 50077583 Dec  6 07:58 gfs.t06z.pgrb2f99

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: BOA-542231
Department: Support LDM
Priority: Normal
Status: Open