<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=iso-8859-1">
<META content="MSHTML 6.00.6000.16640" name=GENERATOR>
<STYLE></STYLE>
</HEAD>
<BODY bgColor=#ffffff>
<DIV><FONT face=Arial size=2><FONT face=Arial size=2></FONT></FONT> </DIV>
<DIV><FONT face=Arial size=2> Hello
all:</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2> I
have just built a 64-core cluster and the following is my
setup.</FONT></DIV><FONT face=Arial size=2></FONT><FONT face=Arial size=2><FONT
face="Times New Roman" size=3>
<DIV><BR> 8 DP quad-core machines running
ROCKS-5 and MVAPICH-1.0<BR> I am using
8-port flextronic switch (it is SDR switch) and the
cards<BR> are MHES18 (10 GB/sec HCA
cards)<BR><BR> I have the following
questions.<BR><BR> How do I know if my
computation is using IB network or ethernetwork?</DIV>
<DIV> I named each IB card
"fast1 .... fast8. </DIV>
<DIV> I created a host file with
8-copies of each of
fast1, fast2.....fast8.<BR> </DIV>
<DIV><FONT face=Arial size=2> The details of
8 nodes with IB config below (only two shown)..</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV> HOST
SUBNET IFACE
IP
NETMASK
NAME<BR>
divilab:
ibnet ib0
20.1.1.1
255.255.255.0 fast1</DIV>
<DIV><FONT face=Arial size=2>
................................................................................................................</FONT></DIV>
<DIV><FONT face=Arial
size=2> ................................................................................................................</FONT><BR> compute-0-6:
ibnet ib0
20.1.1.8
255.255.255.0
fast8<BR><BR> I do not see any scale-up from
16 to 32 to 64 processes.<BR><BR> One benchmark of
MD simulation (for one picosecond) of a protein
(FIXa) is<BR> given
below:<BR> </DIV>
<DIV> The MD code is PMEMD/MVAPICH with IFORT/MKL
compilation.<BR><BR> # of
CPUs/cores Time
(sec) Nodes
(load-balanced)<BR>
8
82
8<BR>
16
49
8<BR>
32
42
8<BR>
64
39
8<BR><BR> I am suspecting that I might have
not set up something right or SDR switch/card
limitations...<BR> definitely not happy with
poor scale-up...<BR><BR> I used all default
values of make.mvapich.gen2 (with intel fortran 9.0). </DIV>
<DIV> There seems too many options in this
script. Not sure what most of them would do, therefore, just
<BR> let the script run as
such.<BR><BR> Could somebody offer some help
on how to fix/improve the scaling?<BR><BR>
Thanks a lot...<BR> Divi<BR><BR>-----
Original Message ----- <BR>From: "Joshua Bernstein" <</FONT><A
href="mailto:jbernstein@penguincomputing.com"><FONT face="Times New Roman"
size=3>jbernstein@penguincomputing.com</FONT></A><FONT face="Times New Roman"
size=3>><BR>To: "Karthik Gopalakrishnan" <</FONT><A
href="mailto:gopalakk@cse.ohio-state.edu"><FONT face="Times New Roman"
size=3>gopalakk@cse.ohio-state.edu</FONT></A><FONT face="Times New Roman"
size=3>><BR>Cc: "Divi Venkateswarlu" <</FONT><A
href="mailto:divi@ncat.edu"><FONT face="Times New Roman"
size=3>divi@ncat.edu</FONT></A><FONT face="Times New Roman" size=3>>;
<BR><</FONT><A href="mailto:mvapich-discuss@cse.ohio-state.edu"><FONT
face="Times New Roman" size=3>mvapich-discuss@cse.ohio-state.edu</FONT></A><FONT
face="Times New Roman" size=3>><BR>Sent: Friday, June 06, 2008 4:59
PM<BR>Subject: Re: [mvapich-discuss] how to set "ulimit -l unlimited" at user
<BR>level?<BR><BR><BR>> Also,<BR>><BR>> If you are running AMBER with
using SSH, you will want to add the <BR>> ulimit -l command to your
/etc/init.d/sshd startup script on the nodes. <BR>> That way any proces
forked by SSH on the compute node will inherit that <BR>> setting and hence
allow AMBER to run.<BR>><BR>> -Joshua Bernstein<BR>> Software
Engineer<BR>> Penguin Computing<BR>><BR>> Karthik Gopalakrishnan
wrote:<BR>>> Try adding the "ulimit -c unlimited" line to /etc/profile
first.<BR>>><BR>>> Regards,<BR>>>
Karthik<BR>>><BR>>> On Tue, Jun 3, 2008 at 1:36 PM, Divi
Venkateswarlu <</FONT><A href="mailto:divi@ncat.edu"><FONT
face="Times New Roman" size=3>divi@ncat.edu</FONT></A><FONT
face="Times New Roman" size=3>>
wrote:<BR>>>>
Hello:<BR>>>><BR>>>> I am running
ROCKS-5 on two DP quad-core machines with mellanox IB <BR>>>>
HCA<BR>>>> cards.<BR>>>> I compiled
mvapich with ifort without any
problems.<BR>>>><BR>>>> I am able to
run at root level with NO problems. I could set <BR>>>> ulimit
-l<BR>>>> unlimited to increase<BR>>>>
RLIMIT_MEMLOCK size. My program (PMEMD of AMBER package) runs on
<BR>>>> all<BR>>>> 16-cores with no
hiccups.<BR>>>><BR>>>> When I try to
set ulimit -l unlimited at user level, I get the<BR>>>>
following error
message.<BR>>>><BR>>>>
-bash: ulimit: max locked memory: cannot modify limit: Operation
<BR>>>> not<BR>>>>
permitted.<BR>>>><BR>>>> Can somebody help
me how to fix this problem? I am running <BR>>>>
mvapich-1.0<BR>>>><BR>>>> Thanks a lot for
your help<BR>>>><BR>>>>
Divi<BR>>>><BR>>>><BR>>>>
_______________________________________________<BR>>>> mvapich-discuss
mailing list<BR>>>> </FONT><A
href="mailto:mvapich-discuss@cse.ohio-state.edu"><FONT face="Times New Roman"
size=3>mvapich-discuss@cse.ohio-state.edu</FONT></A><BR><FONT
face="Times New Roman" size=3>>>> </FONT><A
href="http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss"><FONT
face="Times New Roman"
size=3>http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss</FONT></A><BR><FONT
face="Times New Roman" size=3>>>><BR>>>><BR>>>
_______________________________________________<BR>>> mvapich-discuss
mailing list<BR>>> </FONT><A
href="mailto:mvapich-discuss@cse.ohio-state.edu"><FONT face="Times New Roman"
size=3>mvapich-discuss@cse.ohio-state.edu</FONT></A><BR><FONT
face="Times New Roman" size=3>>> </FONT><A
href="http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss"><FONT
face="Times New Roman"
size=3>http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss</FONT></A><BR><FONT
face="Times New Roman" size=3>> </FONT><BR></DIV></FONT></BODY></HTML>