[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20051129: some AREA file not being updated (cont.)

>From: Myron Kowalski <address@hidden>
>Organization: Moravian
>Keywords: 200511281523.jASFNF7s008982 ldm-mcidas McIDAS

Hi Myron,

>All the data we needed for the class was there this morning.

Excellent!

>Everything but 30 some files were upgraded, so I didn't
>touch anyting until after the class.

OK.  I just jumped onto batboat and took a look at AREA files in
/mcidasdata/mcidas that are not being updated:

-rw-rw-rw-   1 ldm      unidata   607856 Aug  8 12:18 AREA0064
-rw-rw-rw-   1 ldm      unidata   607856 Aug  8 12:18 AREA0164
-rw-rw-rw-   1 ldm      unidata   608096 Aug  8 11:04 AREA0063
-rw-rw-rw-   1 ldm      unidata   607856 Aug  8 11:04 AREA0163
-rw-rw-rw-   1 ldm      unidata   607856 Aug  8 09:37 AREA0062
-rw-rw-rw-   1 ldm      unidata   607856 Aug  8 09:36 AREA0162
-rw-rw-rw-   1 ldm      unidata   607856 Aug  8 08:37 AREA0061
-rw-rw-rw-   1 ldm      unidata   607856 Aug  8 08:36 AREA0161
-rw-rw-rw-   1 ldm      unidata   607856 Aug  8 07:37 AREA0060
-rw-rw-rw-   1 ldm      unidata   607856 Aug  8 07:36 AREA0160
-rw-rw-rw-   1 ldm      unidata   607856 Aug  8 07:04 AREA0069
-rw-rw-rw-   1 ldm      unidata   607856 Aug  8 07:03 AREA0169
-rw-rw-rw-   1 ldm      unidata   607856 Aug  8 05:37 AREA0068
-rw-rw-rw-   1 ldm      unidata   607856 Aug  8 05:36 AREA0168
-rw-rw-rw-   1 ldm      unidata   607856 Aug  8 04:37 AREA0067
-rw-rw-rw-   1 ldm      unidata   608096 Aug  8 04:36 AREA0167
-rw-rw-rw-   1 ldm      unidata   607856 Aug  8 03:37 AREA0066
-rw-rw-rw-   1 ldm      unidata   607856 Aug  8 03:36 AREA0166
-rw-rw-rw-   1 ldm      unidata   607856 Aug  8 02:37 AREA0065
-rw-rw-rw-   1 ldm      unidata   607856 Aug  8 02:36 AREA0165

None of these will get upgraded since the Educational Floaters have
been removed from the Unidata-Wisconsin (IDD UNIWISC (aka MCIDAS))
datastream.  That accounts for 20 of the files not being updated.  They
should be deleted from /mcidasdata/mcidas.

re:
-rw-rw-rw-   1 ldm      unidata   308528 Nov 27 13:05 AREA0199
-rw-rw-rw-   1 ldm      unidata   308528 Nov 27 10:08 AREA0198
-rw-rw-rw-   1 ldm      unidata   310576 Nov 27 07:05 AREA0197

These files will get updated as new Antarctic composite IR images
are received.

re:
-rw-r--r--   1 ldm      unidata  4506816 Nov 28 19:44 AREA0024
-rw-r--r--   1 ldm      unidata  4506096 Nov  1 11:40 AREA0020
-rw-r--r--   1 ldm      unidata  4506256 Jul 11 14:00 AREA0012

There are no products in the Unidata-Wisconsin datastream that will be
unpacked into AREA files of these names.  These files must have been
created by someone for something specific or are a result of the messed
up ROUTE.SYS file.  They should be deleted from /mcidasdata/mcidas.

re:
-rw-rw-rw-   1 ldm      unidata   897280 Jun 19  2004 AREA0279

This will be updated when enough Mollweide global IR images have been
received.  The image itself is a composite of the Mollweide global IR
images and topography.  It is created by a McIDAS post process upon
receipt of the Mollweide global IR images.

re:
-rw-r--r--   1 ldm      unidata  2439056 Nov 26 00:41 AREA-13428820
-rw-r--r--   1 ldm      unidata  4506256 Nov 25 15:30 AREA-12728432
-rw-r--r--   1 ldm      unidata  4506256 Jul 10 20:21 AREA-13363388
-rw-r--r--   1 ldm      unidata  4506256 Jun 19  2004 AREA-13164684
-rw-r--r--   1 ldm      unidata  2439056 May 30  2004 AREA-13164680
-rw-r--r--   1 ldm      unidata  2439056 May  8  2004 AREA-13164664
-rw-r--r--   1 ldm      unidata  1155888 Oct  5  2000 AREA4194304

All of these files are BAD (the names do not conform to any
McIDAS convention).  They should be deleted from /mcidasdata/mcidas.

re: *00271.IDX
-rw-rw-r--   1 ldm      unidata   353024 Sep 27  2000 MET00271.IDX
-rw-rw-r--   1 ldm      unidata  1060864 Sep 27  2000 SA00271.IDX
-rw-rw-r--   1 ldm      unidata   847872 Sep 27  2000 SM00271.IDX
-rw-rw-r--   1 ldm      unidata   421888 Sep 27  2000 ZZ00271.IDX
 ...
-rw-rw-r--   1 ldm      unidata      128 Sep 26  2000 MAT00271.IDX
-rw-rw-r--   1 ldm      unidata      192 Sep 26  2000 TON00271.IDX
-rw-rw-r--   1 ldm      unidata      128 Sep 26  2000 FWO00271.IDX

These files were created by XCD decoders back in 2000.  Since they will
never be deleted by the McIDAS scour process (mcscour.sh run from
cron), they should be deleted by hand from /mcidasdata/mcidas.

re:
-rw-rw-r--   1 ldm      unidata   432640 Oct 31  2000 SCHEMA.103100
-rw-rw-r--   1 ldm      unidata    24000 Oct 11  2000 SYSKEY.TAB.1011
-rwxrwxr--   1 ldm      unidata    16384 Oct 11  2000 ROUTE.SYS.1011
-rw-rw-r--   1 ldm      unidata    24000 Oct  5  2000 SYSKEY.TAB.org
-rw-r--r--   1 ldm      unidata  1155888 Oct  5  2000 AREA4194304
-rw-rw-r--   1 ldm      unidata    14080 Oct  5  2000 ROUTE.SYS.org
-rw-rw-r--   1 ldm      unidata     7312 Nov 15  1999 MDRDEC.IDT
-rw-rw-r--   1 ldm      unidata    73420 Aug  7  1999 PIRPDEC.IDT
-rw-rw-r--   1 mcidas   unidata    24804 Aug  7  1999 IDXALIAS.DAT

These files should be deleted from /mcidasdata/mcidas when upgrading
to the current McIDAS release.

re:
-rw-rw-r--   1 mcidas   unidata   465408 Oct 31  2000 SCHEMA

This file will be replaced by a current station database file
when upgrading to the McIDAS release.

>I upgraded to the new ldm, just now and data is certainly
>streaming  in now.

Sounds good!

>So now the ldm and the ldm-mcidas decoders are upgraded. I guess I
>still need to upgrade mcidas itself.

Sounds good.

>Thanks for your help on short notice.

No worries.

>I brought up a few things
>in mcidas and thought everything was ok until Joe came and said
>nothing he needed for his class was working. Oh, well. That the
>way schools work. Wait until there's just isn't enough time in
>the day left--then work a miracle.

:-)

One last comment (that I can make now that you are running LDM-6).
You can see real time statistics for the LDM ingestion on catwoman
in the following links:

Real time statistics - Site Index
http://www.unidata.ucar.edu/software/idd/rtstats/siteindex.php

catwoman.cs.moravian.edu
http://www.unidata.ucar.edu/software/idd/rtstats/siteindex.php?catwoman.cs.moravian.edu

IDS|DDPLUS latency
http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?IDS|DDPLUS+catwoman.cs.moravian.edu

HDS latency
http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?HDS+catwoman.cs.moravian.edu


A comparison of the HDS products latency (which is high) and the other
feeds (which are not at zero) indicates one of two potential problems:

- the feed from SUNY Brockport (vortex.esc.brockport.edu) is slow

- there is packet shaping (artificial volume limiting) being done on your
  network connection

To test to see if the problem you are experiencing is the latter
possibility, I just changed the feed of HDS data from
vortex.esc.brockport.edu to atm.geo.nsf.gov.

  If the latencies drop to zero, the problem is the feed from SUNY
  Brockport.

  If the latencies remain high, the problem is packet shaping somewhere
  on your campus.  If you intend to continue to ingest all of the HDS
  data (NOAAPORT model output from NCEP), you will need to contact your
  campus IT folks and get the packet shaping removed for port 388
  traffic.

My suspicion is that the problem is being caused by packet shaping, but
I will be more confident in my theory after a couple of hours.

One last comment:  Since you (Moravian) do not access data on an
ongoing basis, it may be advisable to stop ingesting data through the
IDD and switch to reliance on the remote access of data by McIDAS.
Another site did this several years back and has never regretted the
change.  The concept is that McIDAS can access data from any open
access ADDE server that is reachable by TCP/IP Ethernet.  There are
several such servers in the Unidata community that are open to all and
maintained with current data.  One user that was using McIDAS in much
the same way that you appear to be switched to this setup several years
ago and has been happy ever since: all he has to do now is maintain
McIDAS.  Please bounce this idea off of Joe to see what he thinks.

Cheers,

Tom
--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publicly available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.