Multithreaded TDS Clients (1)

Jeff McWhirter was studying whether he could speed up IDV data access by multithreading on the client. He put together a test which fetched the same data over and over, with 1-10 client threads.

This is hitting motherlode TDS, which has 2 quad-core processors, running solaris x86 and ZFS. It asks for one field from an NCEP model file, stored in GRIB1 compression, so theres some amount of computation. Actual URL:

http://motherlode.ucar.edu:8081/thredds/dodsC/model/NCEP/NAM/CONUS_80km/NAM_CONUS_80km_20090313_1200.grib1.dods?Temperature[0:1:0][0:1:18][0:1:64][0:1:92]

19 * 65 *93  = 114855 floats = 459K bytes.

The result shows that you can't get more than 2.5x speedup in this particular case. I will try this on netcdf files, which should be only pure IO. Also interesting to see if the opendap stack causes significant overhead.

A more realistic test is to ask for successive time slices of non-cached data. These may be better opportunities for overlapping IO.

Comments:

Your yellow graph is showing a significant improvement. However, for multiuser server, I think you wouldn't notice any difference. I am comparing this with some cloning software. When it works by its own, multithreading is important, but when multitasks each with multithreading then you notice that improvement due to multithreading is not significant if noticeable at all.

Posted by Alzayed on April 17, 2009 at 02:59 PM MDT #

Post a Comment:
Comments are closed for this entry.
Unidata Developer's Blog
A weblog about software development by Unidata developers*
Unidata Developer's Blog
A weblog about software development by Unidata developers*

Welcome

FAQs

News@Unidata blog

Take a poll!

What if we had an ongoing user poll in here?

Browse By Topic
Browse by Topic
« September 2019
SunMonTueWedThuFriSat
1
3
4
5
6
7
8
10
11
12
13
14
15
17
18
19
20
21
22
23
24
25
26
27
28
29
30
     
       
Today