Disclaimer: the opinions expressed by the blog author do not necessarily represent the views of the Unidata Program Center.
Multithreaded client access (2)
A more realistic test is to ask for successive slices of some data, and see if multithreading speeds things up. So I made a test that makes data requests lik:
Send http://motherlode.ucar.edu:8081/thredds/dodsC/fmrc/NCEP/GFS/Global_0p5deg/files/GFS_Global_0p5deg_20090115_0000.grib2.dods?
Absolute_vorticity[0][0][0:1:360][0:1:719]
Absolute_vorticity[1][0][0:1:360][0:1:719]
...
Each request is 1 Mbyte. These are grib2 files, which may be jpeg2000 encoded (im not sure in this particular file).
So each thread makes 20 requests of successive slices, and i try this with 1-14 simultaneous threads. The plot below is the speedup factor. Its pretty impressively linear, up to about 9 threads, then things get slower but then recover again. Not sure if that's meaningful - i need to run this 100 times and average.
.
Send http://motherlode.ucar.edu:8081/thredds/dodsC/fmrc/NCEP/GFS/Global_0p5deg/files/GFS_Global_0p5deg_20090115_0000.grib2.dods?
Absolute_vorticity[0][0][0:1:360][0:1:719]
Absolute_vorticity[1][0][0:1:360][0:1:719]
...
Each request is 1 Mbyte. These are grib2 files, which may be jpeg2000 encoded (im not sure in this particular file).
So each thread makes 20 requests of successive slices, and i try this with 1-14 simultaneous threads. The plot below is the speedup factor. Its pretty impressively linear, up to about 9 threads, then things get slower but then recover again. Not sure if that's meaningful - i need to run this 100 times and average.
.
Multithreaded TDS clients (1)
Jeff McWhirter was studying whether he could speed up IDV data access by multithreading on the client. He put together a test which fetched the same data over and over, with 1-10 client threads.
This is hitting motherlode TDS, which has 2 quad-core processors, running solaris x86 and ZFS. It asks for one field from an NCEP model file, stored in GRIB1 compression, so theres some amount of computation. Actual URL:
"http://motherlode.ucar.edu:8081/thredds/dodsC/model/NCEP/NAM/CONUS_80km/NAM_CONUS_80km_20090313_1200.grib1.dods?Temperature[0:1:0][0:1:18][0:1:64][0:1:92]";
19 * 65 *93 = 114855 floats = 459K bytes.
The result shows that you can't get more than 2.5x speedup in this particular case. I will try this on netcdf files, which should be only pure IO. Also interesting to see if the opendap stack causes significant overhead.
A more realistic test is to ask for successive time slices of non-cached data. These may be better opportunities for overlapping IO.
This is hitting motherlode TDS, which has 2 quad-core processors, running solaris x86 and ZFS. It asks for one field from an NCEP model file, stored in GRIB1 compression, so theres some amount of computation. Actual URL:
"http://motherlode.ucar.edu:8081/thredds/dodsC/model/NCEP/NAM/CONUS_80km/NAM_CONUS_80km_20090313_1200.grib1.dods?Temperature[0:1:0][0:1:18][0:1:64][0:1:92]";
19 * 65 *93 = 114855 floats = 459K bytes.
The result shows that you can't get more than 2.5x speedup in this particular case. I will try this on netcdf files, which should be only pure IO. Also interesting to see if the opendap stack causes significant overhead.
A more realistic test is to ask for successive time slices of non-cached data. These may be better opportunities for overlapping IO.
AWIPS II licensing update
Unidata has been told by the NWS that
I hope this is the case, and if so, its a smart move by NWS and Raytheon, and the right thing to do with taxpayer's dollars.
"...the AWIPS II software will indeed be available to Unidata for distribution to U.S. universities, through open source licensing. In separate conversations with Raytheon people, we received the same assurance as well."Supposedly we will get a letter from both Raytheon and the NWS "before the Users Committee meeting in early April" that will clarify the AWIPS-II licensing for the Unidata community.
I hope this is the case, and if so, its a smart move by NWS and Raytheon, and the right thing to do with taxpayer's dollars.