<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=iso-8859-1">
<META content="MSHTML 6.00.6000.16640" name=GENERATOR>
<STYLE></STYLE>
</HEAD>
<BODY bgColor=#ffffff>
<DIV><FONT face=Arial size=2> </FONT></DIV>
<DIV><FONT face=Arial size=2> one more to
add:</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2> My IB
hardware 8-port Flextronics SDR
switch</FONT></DIV>
<DIV><FONT face=Arial
size=2> MHES18 Mallanox
HCA cards</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial
size=2> ibchecknet shows the
following</FONT></DIV>
<DIV><FONT face=Arial size=2> </FONT></DIV>
<DIV><FONT face=Arial size=2> [root@divilab bin]# ibchecknet</FONT></DIV>
<DIV> </DIV>
<DIV><FONT face=Arial size=2># Checking Ca: nodeguid
0x0002c90200244228</FONT></DIV>
<DIV> </DIV>
<DIV><FONT face=Arial size=2># Checking Ca: nodeguid
0x0002c90200244230</FONT></DIV>
<DIV> </DIV>
<DIV><FONT face=Arial size=2># Checking Ca: nodeguid
0x0002c902002740fc</FONT></DIV>
<DIV> </DIV>
<DIV><FONT face=Arial size=2># Checking Ca: nodeguid
0x0002c902002441a4</FONT></DIV>
<DIV> </DIV>
<DIV><FONT face=Arial size=2># Checking Ca: nodeguid
0x0002c902002441c4</FONT></DIV>
<DIV> </DIV>
<DIV><FONT face=Arial size=2># Checking Ca: nodeguid
0x0002c9020024422c</FONT></DIV>
<DIV> </DIV>
<DIV><FONT face=Arial size=2># Checking Ca: nodeguid
0x0002c902002441ac</FONT></DIV>
<DIV> </DIV>
<DIV><FONT face=Arial size=2># Checking Ca: nodeguid
0x0002c9020024418c</FONT></DIV>
<DIV> </DIV>
<DIV><FONT face=Arial size=2>## Summary: 9 nodes checked, 0 bad nodes
found<BR>## 16 ports
checked, 0 bad ports
found<BR>## 0 ports have
errors beyond threshold<BR></FONT></DIV>
<BLOCKQUOTE dir=ltr
style="PADDING-RIGHT: 0px; PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #000000 2px solid; MARGIN-RIGHT: 0px">
<DIV style="FONT: 10pt arial">----- Original Message ----- </DIV>
<DIV
style="BACKGROUND: #e4e4e4; FONT: 10pt arial; font-color: black"><B>From:</B>
<A title=divi@ncat.edu href="mailto:divi@ncat.edu">Divi Venkateswarlu</A>
</DIV>
<DIV style="FONT: 10pt arial"><B>To:</B> <A
title=mvapich-discuss@cse.ohio-state.edu
href="mailto:mvapich-discuss@cse.ohio-state.edu">mvapich-discuss@cse.ohio-state.edu</A>
</DIV>
<DIV style="FONT: 10pt arial"><B>Sent:</B> Saturday, June 07, 2008 9:02
AM</DIV>
<DIV style="FONT: 10pt arial"><B>Subject:</B> How to check if MVAPICH is using
IB network but not ethernetwork?</DIV>
<DIV><BR></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2> Hello
all:</FONT></DIV>
<DIV><FONT face=Arial size=2> </FONT></DIV>
<DIV><FONT face=Arial size=2> Good
morning! </FONT></DIV>
<DIV><FONT face=Arial size=2> I set up a
64-core cluster based on ROCKS-5.0 using eight Dell PE2900 boxes.</FONT></DIV>
<DIV><FONT face=Arial size=2> All are
dual-processor QC machines.</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2> compiled
MVAPICH-1.0 (using intel compiler) with default parameters in
make.mvapich.gen2</FONT></DIV>
<DIV><FONT face=Arial size=2> IB stack is
OFED-1.2.5.5. </FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2> My MD
program (PMEMD/AMBER) is compiled with no errors with IFORT/MKL libraries
and</FONT></DIV>
<DIV><FONT face=Arial size=2> I could run
the code on all 64 cores, but the scaling from 16 to 32 to 64 is terrible. I
am inclosing</FONT></DIV>
<DIV><FONT face=Arial size=2> the
benchmarks on a test run.</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial
size=2> <FONT
face="Times New Roman" size=3> # of CPUs/cores Time
(sec) Nodes
(load-balanced) Scaling
(%)<BR>
8
82
8 100
<BR>
16
49
8
84
<BR>
32
42
8 49<BR>
64
39
8
26</FONT><BR></FONT></DIV>
<DIV><FONT face=Arial
size=2> In
contrast, on single box, I get a reasonable scaling.</FONT></DIV><FONT
face=Arial size=2><FONT face="Times New Roman" size=3><FONT face=Arial
size=2></FONT>
<DIV><FONT face=Arial size=2></FONT><BR> #
cores time
(sec)<BR>
2
284
(100%)<BR>
4
164 (87%<BR>
8
107 (65%)<BR><BR> For
some reason, I suspect, MPI traffic is not going over IB
net.</FONT><BR></DIV></FONT>
<DIV><FONT face=Arial
size=2> <FONT
face="Times New Roman" size=3>MVAPICH is built using make.mvapich.gen2 with
F77=ifort and CC=gcc<BR><BR> mpif77 -link_info
is:<BR><BR> /state/partition1/fc91052/bin/ifort
-L/usr/local/ofed/lib64 -L/usr/local/mvapich/lib <BR>
-lmpich -L/usr/local/ofed/lib64 -Wl,-rpath=/usr/local/ofed/lib64
-libverbs <BR> -libumad -lpthread -lpthread
-lrt</FONT><BR><BR></FONT></DIV>
<DIV><FONT face=Arial size=2> How can I be sure that
MPI traffic is going through IB network rather than ethernet?</FONT></DIV>
<DIV><FONT face=Arial size=2> Are there any specific
checks I should perform?</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2> Thanks a lot for your
help.</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2> Divi</FONT></DIV>
<DIV><FONT face=Arial size=2> </DIV></BLOCKQUOTE></FONT></BODY></HTML>