<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=gb2312">
<META content="MSHTML 6.00.3790.2885" name=GENERATOR>
<STYLE>
<!--
/* Font Definitions */
@font-face
{font-family:宋体;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"\@宋体";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
text-align:justify;
text-justify:inter-ideograph;
font-size:10.5pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:Verdana;
color:windowtext;
font-weight:normal;
font-style:normal;
text-decoration:none none;}
/* Page Definitions */
@page Section1
{size:595.3pt 841.9pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;
layout-grid:15.6pt;}
div.Section1
{page:Section1;}
-->
</STYLE>
</HEAD>
<BODY>
<DIV><FONT face=Verdana color=#0000ff size=2>A question on MPI over infiniband
bandwidth test. Thanks for your help!</FONT></DIV>
<DIV><FONT face=Verdana size=2></FONT> </DIV>
<DIV align=left>
<DIV align=left><FONT face=Verdana size=2>
<HR style="WIDTH: 122px; HEIGHT: 2px" SIZE=2>
</FONT></DIV>
<DIV><FONT color=#c0c0c0><FONT face=Verdana size=2>zhangwl</FONT></DIV>
<DIV><FONT face=Verdana size=2>2007-06-05</FONT></FONT></DIV></DIV>
<DIV><FONT face=Verdana size=2>
<HR>
</FONT></DIV>
<DIV><FONT face=Verdana><FONT size=2><STRONG>发件人：</STRONG> Jiuxing
Liu</FONT></FONT></DIV>
<DIV><FONT face=Verdana><FONT size=2><STRONG>发送时间：</STRONG>
2007-06-04 22:05:10</FONT></FONT></DIV>
<DIV><FONT face=Verdana><FONT size=2><STRONG>收件人：</STRONG>
zhangwl@ncic.ac.cn</FONT></FONT></DIV>
<DIV><FONT face=Verdana><FONT size=2><STRONG>抄送：</STRONG> </FONT></FONT></DIV>
<DIV><FONT face=Verdana><FONT size=2><STRONG>主题：</STRONG> Re: Fw: Question for
help, thanks a lot!</FONT></FONT></DIV>
<DIV><FONT face=Verdana size=2></FONT> </DIV>
<DIV><FONT face=Verdana size=2><BR><FONT face=sans-serif size=2>Hi,</FONT>
<BR><BR><FONT face=sans-serif size=2>I have left Ohio State and do not maintain
MVAPICH any more.</FONT> <BR><FONT face=sans-serif size=2>Can you forward your
message to the MVAPICH mailing list (</FONT><A
href="mailto:mvapich-discuss@cse.ohio-state.edu"><FONT color=blue
size=3><U>mvapich-discuss@cse.ohio-state.edu</U></FONT></A><FONT size=3>
</FONT><FONT face=sans-serif size=2>)? I think that people there will be more
than happy to help you.</FONT> <BR><BR><FONT face=sans-serif
size=2>Thanks,</FONT> <BR><BR><FONT face=sans-serif size=2>-Jiuxing</FONT>
<BR><BR><BR><BR>
<TABLE width="100%">
<TBODY>
<TR vAlign=top>
<TD width="40%"><FONT face=sans-serif size=1><B>"zhangwl"
<zhangwl@ncic.ac.cn></B> </FONT>
<P><FONT face=sans-serif size=1>06/04/2007 07:58 AM</FONT><BR></P>
<TD width="59%">
<TABLE width="100%">
<TBODY>
<TR vAlign=top>
<TD>
<DIV align=right><FONT face=sans-serif size=1>To</FONT></DIV>
<TD><FONT face=sans-serif size=1>Jiuxing Liu/Watson/IBM@IBMUS</FONT>
<TR vAlign=top>
<TD>
<DIV align=right><FONT face=sans-serif size=1>cc</FONT></DIV>
<TD>
<TR vAlign=top>
<TD>
<DIV align=right><FONT face=sans-serif size=1>Subject</FONT></DIV>
<TD><FONT face=sans-serif size=1>Fw: Question for help, thanks a
lot!</FONT></TR></TBODY></TABLE><BR>
<TABLE>
<TBODY>
<TR vAlign=top>
<TD>
<TD></TR></TBODY></TABLE><BR></TR></TBODY></TABLE><BR><BR><BR><FONT face=Verdana
size=2>Hello, Dr. Jiuxing Liu,</FONT> <BR><FONT face=Verdana
size=2> </FONT> <BR><FONT face=Verdana size=2>When testing mpi over
infiniband performance, I find a problem that bandwidth at larger messages are
lower instead both intra and inter node. The detailed tests and results are
attached as follows.</FONT> <BR><FONT face=Verdana size=2> </FONT>
<BR><FONT face=Verdana size=2> </FONT> <BR><FONT face=Verdana size=2>My
system is: </FONT><BR><FONT face=Verdana size=2>-- 2.2GHz Dual Core AMD
Opteron(tm) Processor 275, 8GB Mem</FONT> <BR><FONT face=Verdana size=2>-- Linux
2.6.9-42.ELsmp x86_64</FONT> <BR><FONT face=Verdana size=2>-- openib-1.1 {
Detected the following HCAs: 1) mthca0 [ Mellanox PCI-X ]</FONT><FONT face=@宋体
size=2> }</FONT> <BR><FONT face=Verdana size=2> </FONT> <BR><FONT
face=Verdana size=2>1. Test inter-node bandwidth with -</FONT><FONT
face="Times New Roman" size=2>DVIADEV_RGET_SUPPORT .</FONT> <BR><FONT
face="Times New Roman" size=2>setup_ch_gen2 starts... -D_X86_64_
-DEARLY_SEND_COMPLETION -DMEMORY_SCALE -DVIADEV_RGET_SUPPORT
-DLAZY_MEM_UNREGISTER -DCH_GEN2 -D_SMP_ -D_SMP_RNDV_ -D_MLX_PCI_X_
-I/usr/local/ofed/include -O3</FONT> <BR><FONT face=Verdana size=2> </FONT>
<BR><A href="mailto:cmy@gnode28:~/zhangwl/osu_benchmarks$"><FONT
face="Times New Roman" color=blue size=2><U>$</U></FONT></A><FONT
face="Times New Roman" size=2> mpirun_rsh -rsh -np 2 inode28 inode30
./osu_bw<BR># OSU MPI Bandwidth Test (Version 2.2)<BR># Size
Bandwidth (MB/s)<BR>1
0.243180<BR>2
0.507795<BR>4 1.008787<BR>8
2.030054<BR>16
4.008455<BR>32
8.113140<BR>64
16.160978<BR>128
33.764735<BR>256 67.708075<BR>512
161.522157<BR>1024
335.222506<BR>2048
491.421716<BR>4096
568.259955<BR>8192
606.043232<BR>16384 662.063392<BR>32768
738.589843<BR>65536
783.586601<BR>131072
807.462616<BR>262144
820.750931<BR>524288
685.880335<BR>1048576 660.237959<BR>2097152
659.233480<BR>4194304
659.946110</FONT> <BR><FONT face=Verdana size=2>2. Test inter-node bandwidth
with -</FONT><FONT face="Times New Roman" size=2>DVIADEV_RPUT_SUPPORT .</FONT>
<BR><FONT face="Times New Roman" size=2>setup_ch_gen2 starts... -D_X86_64_
-DEARLY_SEND_COMPLETION -DMEMORY_SCALE -DVIADEV_RPUT_SUPPORT
-DLAZY_MEM_UNREGISTER -DCH_GEN2 -D_SMP_ -D_SMP_RNDV_ -D_MLX_PCI_X_
-I/usr/local/ofed/include -O3</FONT> <BR><FONT face=Verdana size=2> </FONT>
<BR><A href="mailto:cmy@gnode28:~/zhangwl/osu_benchmarks$"><FONT
face="Times New Roman" color=blue size=2><U>$</U></FONT></A><FONT
face="Times New Roman" size=2> mpirun_rsh -rsh -np 2 inode28 inode30
./osu_bw<BR># OSU MPI Bandwidth Test (Version 2.2)<BR># Size
Bandwidth (MB/s)<BR>1
0.248081<BR>2
0.516046<BR>4 1.034260<BR>8
2.069607<BR>16
4.110799<BR>32
8.282444<BR>64
16.593745<BR>128
34.620911<BR>256 69.113305<BR>512
163.455879<BR>1024
341.066875<BR>2048
496.503655<BR>4096
569.049428<BR>8192
606.183374<BR>16384 624.840449<BR>32768
713.280615<BR>65536
769.011487<BR>131072
800.359506<BR>262144
814.869019<BR>524288
679.025085<BR>1048576 652.137840<BR>2097152
650.207077<BR>4194304
650.629356</FONT> <BR><FONT face=Verdana size=2>3. Test intra-node bandwidth
with -</FONT><FONT face="Times New Roman" size=2>DVIADEV_RPUT_SUPPORT .</FONT>
<BR><A href="mailto:cmy@gnode28:~/zhangwl/osu_benchmarks$"><FONT
face="Times New Roman" color=blue size=2><U>$</U></FONT></A><FONT
face="Times New Roman" size=2> mpirun_rsh -rsh -np 2 inode28 inode28
./osu_bw<BR># OSU MPI Bandwidth Test (Version 2.2)<BR># Size
Bandwidth (MB/s)<BR>1
2.173175<BR>2
4.449079<BR>4 9.049134<BR>8
20.301348<BR>16
42.489627<BR>32
85.085168<BR>64
153.869271<BR>128
286.734337<BR>256 480.187573<BR>512
741.525232<BR>1024
932.896797<BR>2048
1145.834426<BR>4096
1291.731546<BR>8192
1388.989562<BR>16384
1428.285773<BR>32768 1453.529249<BR>65536
1431.307671<BR>131072
1445.227803<BR>262144
1393.404399<BR>524288
1168.315567<BR>1048576 1071.952093<BR>2097152
1072.327638<BR>4194304
1064.196619</FONT> <BR><FONT face=Verdana size=2> </FONT> <BR><FONT
face="Times New Roman" size=2>I have seen test results on your homepage
(</FONT><A
href="http://mvapich.cse.ohio-state.edu/performance/mvapich/opteron/MVAPICH-opteron-gen2-DDR.shtml"><FONT
face="Times New Roman" color=blue
size=2><U>http://mvapich.cse.ohio-state.edu/performance/mvapich/opteron/MVAPICH-opteron-gen2-DDR.shtml</U></FONT></A><FONT
face="Times New Roman" size=2>, </FONT><A
href="http://mvapich.cse.ohio-state.edu/performance/mvapich/intra_opteron.shtml"><FONT
face="Times New Roman" color=blue
size=2><U>http://mvapich.cse.ohio-state.edu/performance/mvapich/intra_opteron.shtml</U></FONT></A><FONT
face="Times New Roman" size=2>), that inter-node bandwidth results seem normal
but intra-node bandwidth results are like mine. And bandwidth results in you
paper BUILDING MULTIRAIL INFINIBAND CLUSTERS: MPI-LEVEL DESIGN AND PERFORMANCE
EVALUATION: SC2004(Fig. 9) seem that striping or binding optimization will
remove improve this problem.</FONT> <BR><FONT face="Times New Roman"
size=2> </FONT> <BR><FONT face="Times New Roman" size=2>What do you think
will be the problem source for my bandwidth tests? In order to get optimal
bandwidth value, what do you think I should modify based on default options in
original MVAPICH 0.9.8 packet? Does STRIPING or BINDING haven't been added to
mvapich0.9.8?</FONT> <BR><FONT face=Verdana size=2> </FONT> <BR><FONT
face="Times New Roman" size=2>Thanks a lot for your any reply!</FONT> <BR><FONT
face=Verdana size=2> </FONT> <BR><FONT face="Times New Roman" size=2>Best
Regards,</FONT> <BR><FONT face="Times New Roman" size=2>Wenli</FONT> <BR><FONT
face=Verdana size=2> </FONT> <BR>
<HR>
<BR><FONT face=Verdana color=#c0c0c0 size=2>zhangwl</FONT> <BR><FONT
face=Verdana color=#c0c0c0 size=2>2007-06-04</FONT> <BR><FONT face=Verdana
color=#c0c0c0 size=2> </FONT> <BR><FONT face=Verdana color=#c0c0c0
size=2>=====================================================<BR>Zhang
Wenli,<BR>NCIC, Institute of Computing Technology<BR>Chinese Academy of Sciences
</FONT>
<P><FONT face=Verdana color=#c0c0c0 size=2>NO. 6, Ke Xue Yuan South Road,
Zhongguancun, <BR>Beijing, P.R.China <BR>NCIC, P.O.Box 2704 Zip Code
100080</FONT>
<P><FONT face=Verdana color=#c0c0c0 size=2>Tel: 86-10-62601041<BR>Fax:
86-10-62527487 <BR>Email: </FONT><A href="mailto:zhangwl@ncic.ac.cn"
target=_blank><FONT face=Verdana color=blue
size=2><U>zhangwl@ncic.ac.cn</U></FONT></A>
<P></FONT></P></DIV>
<DIV><FONT face=Verdana size=2></FONT> </DIV>
<DIV>
<DIV><FONT
size=2>==============================<WBR>=======================<BR>Zhang
Wenli,<BR>NCIC, Institute of Computing Technology<BR>Chinese Academy of Sciences
</FONT></DIV>
<DIV>
<P><FONT size=2>NO. 6, Ke Xue Yuan South Road, Zhongguancun,
<SCRIPT><!--
D(["mb","\u003cbr\>Beijing, P.R.China \u003cbr\>NCIC, P.O.Box \n 2704 Zip Code 100080\u003c/p\>\n \u003cp\>Tel: 86-10-62601041\u003cbr\>Fax: 86-10-62527487 \u003cbr\>Email: \n \u003ca href\u003d\"mailto:zhangwl@ncic.ac.cn\" target\u003d\"_blank\" onclick\u003d\"return top.js.OpenExtLink(window,event,this)\"\>zhangwl@ncic.ac.cn\u003c/a\>\u003c/p\> \u003c/font\>\u003c/font\>\u003c/div\>\u003c/div\>\u003c/td\>\u003c/tr\>\u003c/tbody\>\u003c/table\>\u003c/font\>\n \u003cdiv align\u003d\"right\"\>\u003cimg height\u003d\"209\" src\u003d\"?attid\u003d0.0.1&disp\u003demb&view\u003datt&th\u003d11029fbd4afe2577\" width\u003d\"219\"\>\u003c/div\>\u003c/td\>\u003c/tr\>\u003c/tbody\>\u003c/table\>\u003c/div\>\u003c/div\>\u003c/td\>\u003c/tr\>\u003c/tbody\>\u003c/table\>\u003c/div\>\u003c/td\>\u003c/tr\>\n \u003ctr\>\n \u003ctd valign\u003d\"bottom\" align\u003d\"right\" width\u003d\"80\" height\u003d\"30\"\> \u003c/td\>\n \u003ctd valign\u003d\"bottom\" align\u003d\"right\" height\u003d\"30\"\>\u003c/td\>\u003c/tr\>\u003c/tbody\>\u003c/table\>\u003c/div\>\n",0]
);
//--></SCRIPT>
<BR>Beijing, P.R.China <BR>NCIC, P.O.Box 2704 Zip Code
100080</FONT></P>
<P><FONT size=2>Tel: 86-10-62601041<BR>Fax: 86-10-62527487 <BR>Email: </FONT><A
onclick="return top.js.OpenExtLink(window,event,this)"
href="mailto:zhangwl@ncic.ac.cn" target=_blank><FONT
size=2>zhangwl@ncic.ac.cn</FONT></A></P></DIV></DIV></BODY></HTML>