Re: [idvdevelopers] Problem with shapefile (ESRI .shp) code in IDV

On 5/23/14, 2:31 PM, Yuan Ho wrote:
On 5/23/2014 12:23 PM, Tommy Jasmin wrote:
Folks - someone here found a problem with the shapefile control and data
source code.
I have test files and have coded what I believe is a fix, but would like
to discuss with
someone before moving ahead and creating a pull request.  Who would be
the best
person to describe the details of the issue with?

Tommy,

 You can send your description to this list.

Yuan

Thanks Yuan, here goes...

We have a Shapefile from NOAA that a researcher uses here, I'll call it "old". They updated the shapefile this year, I'll call that "new". Both are attached as zip files. (very small files, you can tell from file name which is from last
year and which is from this year).

Unzip these in an old and new directory.  Both validate fine and come up
fine in any GIS program.  In IDV, with the new file, it loads but you get no
filters or table in the Layer Controls.  Old file works fine.

It looks like ShapeFileDataSource builds up the vector polygons as a
SampledSet array.  Then, and this is the part I don't understand, the
attributes from the .dbf file are ONLY LOADED if the very first element
is an instance of MapSet:

    private void setProperties(Data shapefileData, DbaseFile dbFile) {
        if ((dbFile == null) || (shapefileData == null)
                || !(shapefileData instanceof UnionSet)) {
            return;
        }
        SampledSet[] sets = ((UnionSet) shapefileData).getSets();
        // TJJ Look here: why does first element need to be MapSet ??
        if ((sets.length == 0) || !(sets[0] instanceof MapSet)) {
            return;
        }
        ...otherwise drops through and loads the properties as usual
    }

In the old file, the first entry happens to be a MapSet and everything is
fine:

15:23:20.676 [Thread-211] WARN u.u.idv.control.ShapefileControl - log: Idx: 0 instance MapSet? true 15:23:20.676 [Thread-211] WARN u.u.idv.control.ShapefileControl - log: Idx: 1 instance MapSet? true 15:23:20.676 [Thread-211] WARN u.u.idv.control.ShapefileControl - log: Idx: 2 instance MapSet? true 15:23:20.676 [Thread-211] WARN u.u.idv.control.ShapefileControl - log: Idx: 3 instance MapSet? true 15:23:20.676 [Thread-211] WARN u.u.idv.control.ShapefileControl - log: Idx: 4 instance MapSet? false 15:23:20.676 [Thread-211] WARN u.u.idv.control.ShapefileControl - log: Idx: 5 instance MapSet? true 15:23:20.676 [Thread-211] WARN u.u.idv.control.ShapefileControl - log: Idx: 6 instance MapSet? true 15:23:20.677 [Thread-211] WARN u.u.idv.control.ShapefileControl - log: Idx: 7 instance MapSet? true 15:23:20.677 [Thread-211] WARN u.u.idv.control.ShapefileControl - log: Idx: 8 instance MapSet? true 15:23:20.677 [Thread-211] WARN u.u.idv.control.ShapefileControl - log: Idx: 9 instance MapSet? true 15:23:20.677 [Thread-211] WARN u.u.idv.control.ShapefileControl - log: Idx: 10 instance MapSet? false 15:23:20.677 [Thread-211] WARN u.u.idv.control.ShapefileControl - log: Idx: 11 instance MapSet? false 15:23:20.677 [Thread-211] WARN u.u.idv.control.ShapefileControl - log: Names obj: FID_improv 15:23:20.677 [Thread-211] WARN u.u.idv.control.ShapefileControl - log: Names obj: FID_persis 15:23:20.677 [Thread-211] WARN u.u.idv.control.ShapefileControl - log: Names obj: FID_dev 15:23:20.678 [Thread-211] WARN u.u.idv.control.ShapefileControl - log: Names obj: Fcst_Date 15:23:20.678 [Thread-211] WARN u.u.idv.control.ShapefileControl - log: Names obj: Target 15:23:20.678 [Thread-211] WARN u.u.idv.control.ShapefileControl - log: Names obj: FID_Remove 15:23:20.678 [Thread-211] WARN u.u.idv.control.ShapefileControl - log: populateTable...

But for the new file, the first entry is NOT a MapSet so we get no Filters or Table in Layer Controls:

15:25:21.521 [Thread-278] WARN u.u.idv.control.ShapefileControl - log: creating mapSets... 15:25:21.521 [Thread-278] WARN u.u.idv.control.ShapefileControl - log: Idx: 0 instance MapSet? false 15:25:21.522 [Thread-278] WARN u.u.idv.control.ShapefileControl - log: Idx: 1 instance MapSet? false 15:25:21.522 [Thread-278] WARN u.u.idv.control.ShapefileControl - log: Idx: 2 instance MapSet? false 15:25:21.522 [Thread-278] WARN u.u.idv.control.ShapefileControl - log: Idx: 3 instance MapSet? true 15:25:21.522 [Thread-278] WARN u.u.idv.control.ShapefileControl - log: Idx: 4 instance MapSet? true 15:25:21.522 [Thread-278] WARN u.u.idv.control.ShapefileControl - log: Idx: 5 instance MapSet? false 15:25:21.522 [Thread-278] WARN u.u.idv.control.ShapefileControl - log: Idx: 6 instance MapSet? true 15:25:21.522 [Thread-278] WARN u.u.idv.control.ShapefileControl - log: Idx: 7 instance MapSet? true 15:25:21.523 [Thread-278] WARN u.u.idv.control.ShapefileControl - log: Idx: 8 instance MapSet? true 15:25:21.523 [Thread-278] WARN u.u.idv.control.ShapefileControl - log: Idx: 9 instance MapSet? true 15:25:21.523 [Thread-278] WARN u.u.idv.control.ShapefileControl - log: Idx: 10 instance MapSet? true 15:25:21.523 [Thread-278] WARN u.u.idv.control.ShapefileControl - log: Idx: 11 instance MapSet? false 15:25:21.523 [Thread-278] WARN u.u.idv.control.ShapefileControl - log: Idx: 12 instance MapSet? false 15:25:21.523 [Thread-278] WARN u.u.idv.control.ShapefileControl - log: Idx: 13 instance MapSet? false

If I change ShapefileDataSource.setProperties to load properties as long as we find AT LEAST ONE
MapSet:

        boolean noMapSets = true;
        for (int idx = 0; idx < sets.length; idx++) {
System.err.println("Idx: " + idx + " instance MapSet? " + (sets[idx] instanceof MapSet));
            if (sets[idx] instanceof MapSet) {
                noMapSets = false;
                break;
            }
        }

        if (noMapSets) return;

...everything works great.

A similar change is needed in ShapefileControl (don't assume index zero is a MapSet, find the
index of the first and use that to pass properties to the Filter).

The only reason I am hesitant to commit is I am wondering what the non-MapSet objects are and whether I need to worry about them? Nothing in the code would make it seem so, but I'm hoping somebody familiar with these two classes can verify. I'm concerned there may be a further problem in the code where certain polygons are not getting processed correctly.

I looked at the Shapefiles with a utility and both, as I said, look fine with the exception of
the new file having 14 polygons instead of 12:

tommyj@ubuntu:~/SHP/4-17$ shpdump DO_Merge_Clip.shp | more
Shapefile Type: Polygon   # of Shapes: 12

File Bounds: (-6075469.880,-1110460.719,0,0)
         to  (  766161.497, 1506760.778,0,0)


tommyj@ubuntu:~/SHP/5-15$ shpdump DO_Merge_Clip.shp | more
Shapefile Type: Polygon   # of Shapes: 14

File Bounds: (-6075622.353,-1294963.859,0,0)
         to  (  824461.673, 1501564.819,0,0)


--
Tommy Jasmin
Space Science and Engineering Center
University of Wisconsin, Madison
1225 West Dayton Street, Madison, WI 53706

Attachment: sdo_polygons_20140417.zip
Description: Zip archive

Attachment: sdo_polygons_20140515.zip
Description: Zip archive

  • 2014 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the idvdevelopers archives: