Re: [thredds] How to speed up gbx9 file generation

  • To: "Sean Arms" <sarms@xxxxxxxx>
  • Subject: Re: [thredds] How to speed up gbx9 file generation
  • From: 唐健 <tangjian@xxxxxxxxxx>
  • Date: Sun, 10 Jul 2016 16:59:18 +0800 (GMT+08:00)
Dear Sean,


With the multi-thread option turned on(I set -nthreads=10), it looks fine (TDM 
hasn't failed since I started it, so it does not have the httpclient issue 
maybe) and also the index generation seems speed up a little(about 20 mins 
faster I think, compare to before), but I am not quite sure about it.
Is there any way for me to know whether it runs actually on mulit-thread or not?
When I look into task manager of windows, it shows there is 41 threads for the 
JAVA run. I am no expert on this, maybe someone else have any experience on it? 
Please feel free to jump in.


Thank you for you kind help all this time.


Best wishes,
Jian

// "C:/Program Files/Java/jdk1.7.0_55/bin/java" -Xmx10g 
-Dtds.content.root.path="C:\Program Files\Apache Software Foundation\Tomcat 
7.0\content" -jar tdm-4.5.jar -nthreads=10 -tds "http://thredds.cma.gov.cn/";


-----原始邮件-----
发件人:"Sean Arms" <sarms@xxxxxxxx>
发送时间:2016-07-09 04:54:23 (星期六)
收件人: "唐健" <tangjian@xxxxxxxxxx>
抄送: thredds <thredds@xxxxxxxxxxxxxxxx>
主题: Re: Re: [thredds] How to speed up gbx9 file generation


Greetings Jian,


Ah yes, the -catalog option. This should work. I do not know for sure if 
multithreading in 4.5 will work for the TDM, as the initial threading issue was 
related to an apache httpclient upgrade (which I think happened in 4.6). The 
TDM will hang after about 5 updates, so it will fail pretty fast.


Cheers,


Sean




On Wed, Jul 6, 2016 at 6:49 PM, 唐健 <tangjian@xxxxxxxxxx> wrote:
Hi Sean,


Thanks for the quick reply.
I was trying to google yesterday and found it in application-config.xml also.
But cannot find the flag to use it beside from ant the source.


One more question, is it safe to use multi-thread? Because I find this 
https://github.com/Unidata/thredds/issues/496 .
I am using TDM 4.5 and TDS 4.5.5 right now.


Yesterday I also try to run multi TDM by using the flag like " -catalog 
nwp_cma_grapes_global.xml "
like this, acutually it works...


runTDM.bat:


start "" "C:/Program Files/Java/jdk1.7.0_55/bin/java" -Xmx1g 
-Dtds.content.root.path="C:\Program Files\Apache Software Foundation\Tomcat 
7.0\content" -jar tdm-4.5.jar -catalog nwp_cma_grapes_global.xml -tds 
"http://thredds.cma.gov.cn/";
start "" "C:/Program Files/Java/jdk1.7.0_55/bin/java" -Xmx1g 
-Dtds.content.root.path="C:\Program Files\Apache Software Foundation\Tomcat 
7.0\content" -jar tdm-4.5.jar -catalog nwp_cma_grapes_meso.xml -tds 
"http://thredds.cma.gov.cn/";
start "" "C:/Program Files/Java/jdk1.7.0_55/bin/java" -Xmx1g 
-Dtds.content.root.path="C:\Program Files\Apache Software Foundation\Tomcat 
7.0\content" -jar tdm-4.5.jar -catalog nwp_cma_grapes_ruc.xml -tds 
"http://thredds.cma.gov.cn/";
start "" "C:/Program Files/Java/jdk1.7.0_55/bin/java" -Xmx1g 
-Dtds.content.root.path="C:\Program Files\Apache Software Foundation\Tomcat 
7.0\content" -jar tdm-4.5.jar -catalog nwp_cma_t639.xml -tds 
"http://thredds.cma.gov.cn/";


I will try the multi-threads way this afternoon.
Thanks very much again!


Cheers,
Jian



-----原始邮件-----
发件人:"Sean Arms" <sarms@xxxxxxxx>
发送时间:2016-07-07 05:27:37 (星期四)
收件人: "唐健" <tangjian@xxxxxxxxxx>
抄送: thredds <thredds@xxxxxxxxxxxxxxxx>
主题: Re: [thredds] How to speed up gbx9 file generation



Greetings Jian,


The performance of the TDM will be limited to the performance of your I/O 
system. While running more than one TDM against a single TDS won't work due to 
the way the TDM finds collections that need scanned, you can run a single TDM 
in a multithreaded mode. In your TDM run script, you can set the number of 
threads to use by setting the flag


-nthreads=<number of threads>


By default, the TDM runs with one thread.


Cheers,


Sean




On Wed, Jul 6, 2016 at 12:40 AM, 唐健 <tangjian@xxxxxxxxxx> wrote:
HI everyone

We run our thredds server and TDM on a virtual workstation with 2.0GHz 
E78850*2, and 32GB memory, windows server 2008.
The script run TDM is like this
"C:/Program Files/Java/jdk1.7.0_55/bin/java" -Xmx10g 
-Dtds.content.root.path="C:\Program Files\Apache Software Foundation\Tomcat 
7.0\content" -jar tdm-4.5.jar -tds "http://thredds.cma.gov.cn/"; -cred ..
In operation it only take like 0.5GB memory.

My problem is almost all global model and meso-scale model output comes in at 
the most busy time around 3-4 pm local time.
and there are a lot of files. it takes quite long time for tdm to build indexes.
for example, when one model output is all there, tdm is busy building indexes 
for another model, and it cannot generate gbx9 and ncx2 file for this model.
Is there anything I can do about it?
Maybe run multi-TDM?
or put one TDM in charge of every single model output?

or can I use any other program to scan dirs and generate gbx9 for each file 
once it comes in ?

If anyone has any suggestion, please let me know.
Many thanks


Cheers
Jian


--
Dr. Jian TANG
Central Meteorological Office
National Meteorological Center of CMA



_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web.  Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.


thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit: 
http://www.unidata.ucar.edu/mailing_lists/




--

Dr. Jian TANG
Central Meteorological Office
National Meteorological Center of CMA

No. 46, Zhongguancun South Street, Haidian District, Beijing
P. R. China, 100081







--

Dr. Jian TANG
Central Meteorological Office
National Meteorological Center of CMA

No. 46, Zhongguancun South Street, Haidian District, Beijing
P. R. China, 100081



  • 2016 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: