From twcroc at wm.edu Wed Oct 1 18:28:30 2008 From: twcroc at wm.edu (Tom Crockett) Date: Wed Oct 1 18:28:40 2008 Subject: [mvapich-discuss] Bug in MVAPICH2 SVN Trunk Configure Script In-Reply-To: <48DC19F6.40901@mcs.anl.gov> References: <48DBFE9D.7090808@wm.edu> <20080925213334.GB2955@cse.ohio-state.edu> <48DC19F6.40901@mcs.anl.gov> Message-ID: <48E3F98E.9080805@wm.edu> Pavan Balaji wrote: > This was a bug in MPICH2 that was fixed in r1000 shortly after the 1.0.7 > release. But, this error should only mean that the Fortran bindings are > not compiled with the internal optimizations; so there shouldn't be any > noticeable performance impact. Pavan, As it turns out, the impact was more severe than that. When I was attempting a 32-bit build of MVAPICH2, gfortran apparently defaulted to a 64-bit compilation because it didn't get the "-m32" option that I had specified in F90FLAGS, and that caused one of the configuration tests to fail because it was trying to link a 64-bit object file against the 32-bit OFED libraries. >> We'll also take a look at this. We may still need to reset these flags >> but maybe it should still include various 'internal' flags that the user >> has specified. > > The CFLAGS will be included within mpicc; they won't show up externally. The configuration test that's blowing up isn't using mpicc, it's invoking gcc directly. Here's the relevant output from "configure": ----------------------------------------------------------------------- RUNNING CONFIGURE FOR MPI TESTS checking for make... make checking whether clock skew breaks make... no checking whether make supports include... yes checking whether make allows comments in actions... yes checking for virtual path format... VPATH checking whether make sets CFLAGS... no checking for gcc... /usr/bin/gcc checking for C compiler default output file name... configure: error: C compiler cannot create executables See `config.log' for more details. configure: error: /bin/sh './configure' failed for test/mpi ----------------------------------------------------------------------- And here's an excerpt from test/mpi/config.log which shows the specific test that blows up. It's essentially the same problem that gfortran ran into in src/binding/f90/configure: gcc doesn't get the right set of compiler flags, so it defaults to 64-bit mode and fails when it tries to link against the 32-bit OFED libraries: ----------------------------------------------------------------------- configure:2304: checking for C compiler default output file name configure:2307: /usr/bin/gcc -I/usr/local/OFED-1.3/include -I/usr/local/src/mvapich2-trunk-2008-09-09/src/mpid/ch3/include -I/usr/local/src/mvapich2-trunk-2008-09-09/src/mpid/ch3/include -I/usr/local/src/mvapich2-trunk-2008-09-09/src/mpid/common/datatype -I/usr/local/src/mvapich2-trunk-2008-09-09/src/mpid/common/datatype -I/usr/local/src/mvapich2-trunk-2008-09-09/src/mpid/ch3/channels/mrail/include -I/usr/local/src/mvapich2-trunk-2008-09-09/src/mpid/ch3/channels/mrail/include -I/usr/local/src/mvapich2-trunk-2008-09-09/src/mpid/ch3/channels/mrail/src/gen2 -I/usr/local/src/mvapich2-trunk-2008-09-09/src/mpid/ch3/channels/mrail/src/gen2 -I/usr/local/src/mvapich2-trunk-2008-09-09/src/mpid/common/locks -I/usr/local/src/mvapich2-trunk-2008-09-09/src/mpid/common/locks -L/usr/local/OFED-1.3/lib conftest.c -lpthread -lrdmacm -libverbs -libumad -lrt >&5 /usr/lib64/gcc/x86_64-suse-linux/4.1.2/../../../../x86_64-suse-linux/bin/ld: skipping incompatible /usr/local/OFED-1.3/lib/librdmacm.so when searching for -lrdmacm /usr/lib64/gcc/x86_64-suse-linux/4.1.2/../../../../x86_64-suse-linux/bin/ld: skipping incompatible /usr/local/OFED-1.3/lib/librdmacm.a when searching for -lrdmacm /usr/lib64/gcc/x86_64-suse-linux/4.1.2/../../../../x86_64-suse-linux/bin/ld: cannot find -lrdmacm collect2: ld returned 1 exit status configure:2310: $? = 1 configure: failed program was: | /* confdefs.h. */ | | #define PACKAGE_NAME "" | #define PACKAGE_TARNAME "" | #define PACKAGE_VERSION "" | #define PACKAGE_STRING "" | #define PACKAGE_BUGREPORT "" | /* end confdefs.h. */ | | int | main () | { | | ; | return 0; | } configure:2349: error: C compiler cannot create executables See `config.log' for more details. ----------------------------------------------------------------------- > What exact CFLAGS and MPICH2LIB_CFLAGS did you use? Note that these > are exclusive. CFLAGS sets flags that are used to compile > MPICH2/MVAPICH2 as well as show up in mpicc. MPICH2LIB_CFLAGS only > sets flags that are used to compile MPICH2/MVAPICH2. Here are the configuration parameters (from test/mpi/config.log): ----------------------------------------------------------------------- ./configure --prefix=/local/scr/root/mvapich2-trunk-2008-09-09-ib --prefix=/local/scr/root/mvapich2-trunk-2008-09-09-ib --with-rdma=gen2 --with-pm=mpirun --with-ib-include=/usr/local/OFED-1.3/include --with-ib-libpath=/usr/local/OFED-1.3/lib --enable-romio --with-file-system=ufs+nfs CC=/usr/bin/gcc CFLAGS=-march=k8 -m32 CXXFLAGS=-march=k8 -m32 CXX=/usr/bin/g++ F77=/usr/bin/gfortran F90FLAGS=-march=k8 -m32 F90=/usr/bin/gfortran FFLAGS=-march=k8 -m32 MPICH2LIB_CFLAGS=-O3 -march=k8 -msse -msse2 -msse3 -m32 MPICH2LIB_CXXFLAGS=-O3 -march=k8 -msse -msse2 -msse3 -m32 MPICH2LIB_F90FLAGS=-O3 -march=k8 -msse -msse2 -msse3 -m32 MPICH2LIB_FFLAGS=-O3 -march=k8 -msse -msse2 -msse3 -m32 --enable-header-caching --cache-file=/dev/null --srcdir=. ----------------------------------------------------------------------- Here's the output from "mpicc -show": ----------------------------------------------------------------------- /usr/bin/gcc -march=k8 -m32 -L/usr/local/OFED-1.3/lib -I/local/scr/root/mvapich2-trunk-2008-09-09-ib/include -L/local/scr/root/mvapich2-trunk-2008-09-09-ib/lib -lmpich -lpthread -lrdmacm -libverbs -libumad -lrt ----------------------------------------------------------------------- Does this clarify things? -Tom -- Tom Crockett College of William and Mary email: twcroc@wm.edu IT/High Performance Computing Group phone: (757) 221-2762 Savage House fax: (757) 221-2023 P.O. Box 8795 Williamsburg, VA 23187-8795 From balaji at mcs.anl.gov Wed Oct 1 21:48:12 2008 From: balaji at mcs.anl.gov (Pavan Balaji) Date: Wed Oct 1 21:48:17 2008 Subject: [mvapich-discuss] Bug in MVAPICH2 SVN Trunk Configure Script In-Reply-To: <48E3F98E.9080805@wm.edu> References: <48DBFE9D.7090808@wm.edu> <20080925213334.GB2955@cse.ohio-state.edu> <48DC19F6.40901@mcs.anl.gov> <48E3F98E.9080805@wm.edu> Message-ID: <48E4285C.3010002@mcs.anl.gov> > As it turns out, the impact was more severe than that. When I was > attempting a 32-bit build of MVAPICH2, gfortran apparently defaulted to > a 64-bit compilation because it didn't get the "-m32" option that I had > specified in F90FLAGS, and that caused one of the configuration tests to > fail because it was trying to link a 64-bit object file against the > 32-bit OFED libraries. I'm a little confused here. From your previous email I thought you had used MPICH2LIB_F90FLAGS, that was dropped in the fortran bindings. Which flags were dropped for you? F90FLAGS or MPICH2LIB_F90FLAGS? MPICH2LIB_F90FLAGS is only for optimizations. Things will break if you specify -m32 in MPICH2LIB_F90FLAGS, since it won't show up in mpif90. >> The CFLAGS will be included within mpicc; they won't show up externally. > > The configuration test that's blowing up isn't using mpicc, it's > invoking gcc directly. Here's the relevant output from "configure": Thanks. If this is happening in the tests directory, then it looks like something is broken. I'll look into it this weekend. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji From twcroc at wm.edu Thu Oct 2 10:01:39 2008 From: twcroc at wm.edu (Tom Crockett) Date: Thu Oct 2 10:01:47 2008 Subject: [mvapich-discuss] Bug in MVAPICH2 SVN Trunk Configure Script In-Reply-To: <48E4285C.3010002@mcs.anl.gov> References: <48DBFE9D.7090808@wm.edu> <20080925213334.GB2955@cse.ohio-state.edu> <48DC19F6.40901@mcs.anl.gov> <48E3F98E.9080805@wm.edu> <48E4285C.3010002@mcs.anl.gov> Message-ID: <48E4D443.40501@wm.edu> Pavan Balaji wrote: > I'm a little confused here. From your previous email I thought you had > used MPICH2LIB_F90FLAGS, that was dropped in the fortran bindings. Pavan, I'm using: F90FLAGS = "-march=k8 -m32" MPICH2LIB_F90FLAGS = "-O3 -march=k8 -msse -msse2 -msse3 -m32" But this is the line in src/binding/f90/configure that's causing the problem for a 32-bit build with gfortran: F90FLAGS="$MPICH2_INTERNAL_C90FLAGS" This is because there are no "MPICH2_INTERNAL_C90FLAGS". Presumably this should be "MPICH2_INTERNAL_F90FLAGS". -Tom -- Tom Crockett College of William and Mary email: twcroc@wm.edu IT/High Performance Computing Group phone: (757) 221-2762 Savage House fax: (757) 221-2023 P.O. Box 8795 Williamsburg, VA 23187-8795 From bfp at purdue.edu Tue Oct 7 10:30:26 2008 From: bfp at purdue.edu (Bryan Putnam) Date: Tue Oct 7 10:30:30 2008 Subject: [mvapich-discuss] 32-bit build of mvapich2-1.2rc2 on 64-bit system Message-ID: Hi All, I've been successfull in building a 64-bit version of mvapich2-1.2rc2 on a 64-bit Linux system. In this case our IB drivers libraries are in /usr/lib64. Now I'm attempting to do a 32-bit build of mvapich2 (driver libraries are in /usr/lib) on the same 64-bit system, using 32-bit compilers, etc. Are there any changes I need to make besides using the configure option --with-ib-libpath=/usr/lib I notice that the build scripts seem to check which type of system it is on (64-bit) and proceeds to attempt a 64-bit build regardless. I suppose what I'mm looking for is something like an --arch=ia32 configure option? Thanks for any help. Bryan From bfp at purdue.edu Tue Oct 7 15:09:01 2008 From: bfp at purdue.edu (Bryan Putnam) Date: Tue Oct 7 15:09:10 2008 Subject: [mvapich-discuss] Re: 32-bit build of mvapich2-1.2rc2 on 64-bit system In-Reply-To: References: Message-ID: On Tue, 7 Oct 2008, Bryan Putnam wrote: > Hi All, > > I've been successfull in building a 64-bit version of mvapich2-1.2rc2 on > a 64-bit Linux system. In this case our IB drivers libraries are in > /usr/lib64. > > Now I'm attempting to do a 32-bit build of mvapich2 (driver libraries are > in /usr/lib) on the same 64-bit system, using 32-bit compilers, etc. Are > there any changes I need to make besides using the configure option > > --with-ib-libpath=/usr/lib > > I notice that the build scripts seem to check which type of system it is > on (64-bit) and proceeds to attempt a 64-bit build regardless. I suppose > what I'mm looking for is something like an > > --arch=ia32 > > configure option? > > Thanks for any help. > > Bryan > > > Nevermind, it does appear that just specifying --with-ib-libpath=/usr/lib is sufficient. Thanks, Bryan From panda at cse.ohio-state.edu Wed Oct 8 16:29:44 2008 From: panda at cse.ohio-state.edu (Dhabaleswar Panda) Date: Wed Oct 8 16:29:52 2008 Subject: [mvapich-discuss] Bug in Allreduce for user-defined ops In-Reply-To: <1ef1de420809272213i45d98b62oa90bf05d15594e34@mail.gmail.com> Message-ID: To update the list users - This issue has been resolved. Jack has tested the solution. Patches #3060 and #3061 have been added to mvapich 1.0.1 branch and mavpich trunk (for 1.1), respectively. Jack - Thanks for your help in testing this solution. We are closing this report. Thanks, DK On Sun, 28 Sep 2008, Jack Poulson wrote: > I believe I've run into a bug in the implementation of Allreduce for > user-defined functions in MVAPICH 1.0 and 1.0.1 (0.9.8 works). > > In 0.9.8, for power-of-two processes, the user-op is called log2 times > with the correct length. In the new versions, it appears to be called > log2+2 times, where the first call to the user-op passes in a count of > zero (I found this by simply printing it from within the user-op). > I've looked through the intra_Allreduce routine in > src/coll/intra_fns_new.c, but I don't see why the user-op is called > more than log2 times for power-of-two processes. > > Should user-defined ops check to ensure the length is nonzero? I've > attached a driver and output that demonstrate the problem. The issue > causes problems in operations such as a custom pivoting operation in > an LU factorization, where an integer is tacked onto the end of a set > of doubles, and a zero length in bytes would cause the routine to > decide negative doubles are being operated on. I've been working > around the problem with a custom Allreduce implementation that uses a > reduce-to-one/bcast, but I would like to take advantage of your team's > multicore optimizations. > > Thank you, > Jack Poulson > From balaji at mcs.anl.gov Thu Oct 9 23:22:25 2008 From: balaji at mcs.anl.gov (Pavan Balaji) Date: Thu Oct 9 23:22:36 2008 Subject: [mvapich-discuss] Bug in MVAPICH2 SVN Trunk Configure Script In-Reply-To: <48E4D443.40501@wm.edu> References: <48DBFE9D.7090808@wm.edu> <20080925213334.GB2955@cse.ohio-state.edu> <48DC19F6.40901@mcs.anl.gov> <48E3F98E.9080805@wm.edu> <48E4285C.3010002@mcs.anl.gov> <48E4D443.40501@wm.edu> Message-ID: <48EECA71.2080408@mcs.anl.gov> Tom, Can you try the attached patch? -- Pavan On 10/02/2008 09:01 AM, Tom Crockett wrote: > Pavan Balaji wrote: >> I'm a little confused here. From your previous email I thought you had >> used MPICH2LIB_F90FLAGS, that was dropped in the fortran bindings. > > Pavan, > > I'm using: > > F90FLAGS = "-march=k8 -m32" > MPICH2LIB_F90FLAGS = "-O3 -march=k8 -msse -msse2 -msse3 -m32" > > But this is the line in src/binding/f90/configure that's causing the > problem for a 32-bit build with gfortran: > > F90FLAGS="$MPICH2_INTERNAL_C90FLAGS" > > This is because there are no "MPICH2_INTERNAL_C90FLAGS". Presumably > this should be "MPICH2_INTERNAL_F90FLAGS". > > -Tom > -- Pavan Balaji http://www.mcs.anl.gov/~balaji -------------- next part -------------- A non-text attachment was scrubbed... Name: mpich2-tsuite.diff Type: text/x-diff Size: 1129 bytes Desc: not available Url : http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20081009/94c0be6f/mpich2-tsuite.bin From tgetachew at hotmail.com Fri Oct 10 19:12:15 2008 From: tgetachew at hotmail.com (Tatek) Date: Fri Oct 10 19:12:26 2008 Subject: [mvapich-discuss] mvapich2 mpirun_rsh bug Message-ID: Hello All, There seems to be a bug with mvapich2 and mpirun_rsh command. Once I install mvapich2, to execute mpiun_rsh command, I have to give the full path of the mpirun_rsh or it will not execute. For example, I was trying to run mpiBench and after setting the PATH for the bin and LD_LIBRARY_PATH, I would get the following [root@t2k-0007 mpibench-0.1-3]# mpirun_rsh -np 2 -hostfile hosts5 ./mpiBench -i 100 Allreduce /usr/bin/env: mpispawn: No such file or directory Child exited abnormally! cleanupKilling remote processes.../usr/bin/env: mpispawn: No such file or directory DONE note that "which mpirun_rsh" shows the correct PATH so I had the PATH set correctly [root@t2k-0007 mpibench-0.1-3]# which mpirun_rsh /global/usr/mpi/mvapich2-1.2/pgi/bin/mpirun_rsh If I give the full PATH then no problem. [root@t2k-0007 mpibench-0.1-3]# /global/usr/mpi/mvapich2-1.2/pgi/bin/mpirun_rsh -np 2 -hostfile hosts5 ./mpiBench -i 100 Allreduce START mpiBench 0 : ddn-0268 1 : ddn-0302 Allreduce Bytes: 0 Iters: 100 Avg: 2.9612 Min: 2.9612 Max: 2.9612 Dim: 0 Ranks: 2 Allreduce Bytes: 8 Iters: 100 Avg: 6.4301 Min: 6.4087 Max: 6.4516 Dim: 0 Ranks: 2 After some investigation, one of my colleagues figured out the issue was that MVAPICH2 mpirun_rsh command gets the directory name from the name of the executable (passed to the program as the first argument) and starts mpispawn using this path. If the full path name is not supplied, the mpispawn program is started without a path and can't be found unless it happens to be in the default login path of the user. We think this is a bug in MVAPICH2's mpirun_rsh. It is reasonable to expect that if the MVAPICH2 binary directory is the PATH, issuing an mpirun_rsh command should work. We have a workaround for now by either using the full PATH or using wrapper scripts but ideally, this should work same as it did with mvapich1. Best Regards, Tatek Getachew Solutions Engineer, HPC Solutions Engineering Appro International, Inc. 446 South Abbott Ave Milpitas, CA 95035 408-941-8100 x504 (office) 408-640-9050 (cell) tgetachew@appro.com _________________________________________________________________ Get more out of the Web. Learn 10 hidden secrets of Windows Live. http://windowslive.com/connect/post/jamiethomson.spaces.live.com-Blog-cns!550F681DAD532637!5295.entry?ocid=TXT_TAGLM_WL_domore_092008 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20081010/17fa90da/attachment.html From vera_wx_cn at yahoo.com.cn Sun Oct 12 04:34:40 2008 From: vera_wx_cn at yahoo.com.cn (=?gb2312?q?=C7=BF=20=C2=ED?=) Date: Sun Oct 12 04:34:52 2008 Subject: [mvapich-discuss] (no subject) Message-ID: <828122.22047.qm@web15306.mail.cnb.yahoo.com> Hi, All friends I have a heterogeneous cluster with 32 IA64 nodes and 32 X86_64 nodes, But I don't know how to srun a job running on these 64 nodes. please guilde me to operate it. thanks advance. --------------------------------- ÑÅ»¢ÓÊÏ䣬ÄúµÄÖÕÉúÓÊÏ䣡 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20081012/e65ec67c/attachment.html From Lars.Paul.Huse at Sun.COM Mon Oct 13 03:55:10 2008 From: Lars.Paul.Huse at Sun.COM (Lars Paul Huse) Date: Mon Oct 13 03:55:44 2008 Subject: [mvapich-discuss] Number of IB QP's Message-ID: <48F2FEDE.8060300@Sun.COM> Hi all, As my source of difficult MPI questions: How many QP's will be used for a large application run with MVAPICH / MVAPICH2 1.x ? aka' given an application running all-to-all traffic of random size on

MPI processes (= #cores) behind each HCA on nodes/HCA's (i.e. a total of N*P application processes) will use QP's on rank 0 and QP's on the rest... Will Automatic path migration affect the number of QP's used ? Will using the MVAPICH2 multi-rail feature affect number of QP's used ? Other factors / features that will affect the number of QP's ? /lars paul From enda.obrien at dalco.ch Mon Oct 13 11:23:25 2008 From: enda.obrien at dalco.ch (Enda O'Brien) Date: Mon Oct 13 11:27:03 2008 Subject: [mvapich-discuss] Which VIADEV* parameters might free up a "hang" on 64 or more cores, when job runs fine up to 32 cores? Message-ID: Hello, I saw this address at the top of the mvapich.conf file on the system I'm using, so I thought I'd submit this question: What parameter(s) in the mvapich.conf file might be adjusted to "free" up a job that is "hanging" on 64 or more cores, but which runs fine on 8, 16 or 32 cores? When such a thing happens on a Quadrics cluster (as it sometimes does...), I can usually adjust (increase) LIBELAN_TPORT_BIGMSG and LIBELAN_ALLOC_SIZE to free the log-jam. That's just 2 parameters. However, there are ~100 VIADEV* parameters in mvapich.conf, and the ones I've adjusted so far haven't made any difference. The main MPI function in the application in question is MPI_Alltoall, but it uses only ~3 minutes out of 80 on 32 cores. Any tips, advice, recommendations gratefully received! Best wishes, Enda P.S. Here are the settings I've tried: VIADEV_VBUF_TOTAL_SIZE=49152 VIADEV_VBUF_POOL_SIZE=1024 VIADEV_ON_DEMAND_THRESHOLD=64 VIADEV_NUM_RDMA_BUFFER=64 VIADEV_USE_SHMEM_COLL=0 ADEV_USE_RDMA_BARRIER=1 VIADEV_SQ_SIZE_MAX=500 VIADEV_DEFAULT_QP_OUS_RD_ATOM=8 VIADEV_CQ_SIZE=100000 VIADEV_DEBUG=3 VIADEV_SRQ_MAX_SIZE=8192 VIADEV_ADAPTIVE_ENABLE_LIMIT=128 =========================== Enda O'Brien DALCO AG Switzerland Aille, Barna, Co. Galway, Ireland Tel. +353 91 591307 Mob. +353 87 7517969 =========================== From panda at cse.ohio-state.edu Mon Oct 13 11:58:02 2008 From: panda at cse.ohio-state.edu (Dhabaleswar Panda) Date: Mon Oct 13 11:58:11 2008 Subject: [mvapich-discuss] Which VIADEV* parameters might free up a "hang" on 64 or more cores, when job runs fine up to 32 cores? In-Reply-To: Message-ID: Thanks for your question. Can you provide some details re. the mvapich version, the mvapich interface (OpenFabrics-Gen2 or other), computing platform and the InfiniBand NIC you are using. This will help us to determine what is going on here and provide appropriate suggestions. Thanks, DK On Mon, 13 Oct 2008, Enda O'Brien wrote: > Hello, > > I saw this address at the top of the mvapich.conf file on the system I'm using, so I thought I'd submit this question: > > What parameter(s) in the mvapich.conf file might be adjusted to "free" up a job that is "hanging" on 64 or more cores, but which runs fine on 8, 16 or 32 cores? > > When such a thing happens on a Quadrics cluster (as it sometimes does...), I can usually adjust (increase) LIBELAN_TPORT_BIGMSG and LIBELAN_ALLOC_SIZE to free the log-jam. That's just 2 parameters. However, there are ~100 VIADEV* parameters in mvapich.conf, and the ones I've adjusted so far haven't made any difference. > > The main MPI function in the application in question is MPI_Alltoall, but it uses only ~3 minutes out of 80 on 32 cores. > > Any tips, advice, recommendations gratefully received! > > Best wishes, > Enda > > P.S. Here are the settings I've tried: > VIADEV_VBUF_TOTAL_SIZE=49152 > VIADEV_VBUF_POOL_SIZE=1024 > VIADEV_ON_DEMAND_THRESHOLD=64 > VIADEV_NUM_RDMA_BUFFER=64 > VIADEV_USE_SHMEM_COLL=0 > ADEV_USE_RDMA_BARRIER=1 > VIADEV_SQ_SIZE_MAX=500 > VIADEV_DEFAULT_QP_OUS_RD_ATOM=8 > VIADEV_CQ_SIZE=100000 > VIADEV_DEBUG=3 > VIADEV_SRQ_MAX_SIZE=8192 > VIADEV_ADAPTIVE_ENABLE_LIMIT=128 > > =========================== > Enda O'Brien > DALCO AG Switzerland > Aille, Barna, Co. Galway, Ireland > Tel. +353 91 591307 > Mob. +353 87 7517969 > =========================== > > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > From sridharj at cse.ohio-state.edu Mon Oct 13 12:08:31 2008 From: sridharj at cse.ohio-state.edu (Jaidev Sridhar) Date: Mon Oct 13 12:08:40 2008 Subject: [mvapich-discuss] mvapich2 mpirun_rsh bug In-Reply-To: References: Message-ID: <20081013160831.GB9190@kappa.cse.ohio-state.edu> Hi Tatek, On further investigation, we saw that the path handling was different between MVAPICH and MVAPICH2 as you pointed out. Thanks for this report. We've updated mpirun_rsh on MVAPICH2 to mimic the behavior on MVAPICH. The update will be available in the upcoming release of MVAPICH2 1.2. It is also available on the svn at: https://mvapich.cse.ohio-state.edu/svn/mpi/mvapich2/trunk Regards, Jaidev On Fri, Oct 10, 2008 at 11:12:15PM +0000, Tatek wrote: > > Hello All, > > There seems to be a bug with mvapich2 and mpirun_rsh command. Once I > install mvapich2, to execute mpiun_rsh command, I have to give the > full path of the mpirun_rsh or it will not execute. For example, I > was trying to run mpiBench and after setting the PATH for the bin and > LD_LIBRARY_PATH, I would get the following > > [root@t2k-0007 mpibench-0.1-3]# mpirun_rsh -np 2 -hostfile hosts5 > ./mpiBench -i 100 Allreduce > /usr/bin/env: mpispawn: No such file or directory > Child exited abnormally! > cleanupKilling remote processes.../usr/bin/env: mpispawn: No such file > or directory DONE > > note that "which mpirun_rsh" shows the correct PATH so I had the PATH > set correctly > > [root@t2k-0007 mpibench-0.1-3]# which mpirun_rsh > /global/usr/mpi/mvapich2-1.2/pgi/bin/mpirun_rsh > > If I give the full PATH then no problem. > > [root@t2k-0007 mpibench-0.1-3]# > /global/usr/mpi/mvapich2-1.2/pgi/bin/mpirun_rsh -np 2 -hostfile hosts5 > ./mpiBench -i 100 Allreduce START mpiBench 0 : ddn-0268 > 1 : ddn-0302 > Allreduce Bytes: 0 Iters: 100 Avg: 2.9612 Min: 2.9612 Max: 2.9612 Dim: > 0 Ranks: 2 > Allreduce Bytes: 8 Iters: 100 Avg: 6.4301 Min: 6.4087 Max: 6.4516 Dim: > 0 Ranks: 2 > > After some investigation, one of my colleagues figured out the issue > was that MVAPICH2 mpirun_rsh command gets the directory name from the > name of the executable (passed to the program as the first argument) > and starts mpispawn using this path. If the full path name is not > supplied, the mpispawn program is started without a path and can't be > found unless it happens to be in the default login path of the > user. We think this is a bug in MVAPICH2's mpirun_rsh. It is > reasonable to expect that if the MVAPICH2 binary directory is the > PATH, issuing an mpirun_rsh command should work. > > We have a workaround for now by either using the full PATH or using > wrapper scripts but ideally, this should work same as it did with > mvapich1. > > Best Regards, > > > Tatek Getachew > > Solutions Engineer, HPC Solutions Engineering > > Appro International, Inc. > > 446 South Abbott Ave > > Milpitas, CA 95035 > > 408-941-8100 x504 (office) > > 408-640-9050 (cell) > > [1]tgetachew@appro.com > > > > > _________________________________________________________________ > > Get more out of the Web. Learn 10 hidden secrets of Windows Live. > [2]Learn Now > > References > > 1. mailto:tgetachew@appro.com > 2. http://windowslive.com/connect/post/jamiethomson.spaces.live.com-Blog-cns!550F681DAD532637!5295.entry?ocid=TXT_TAGLM_WL_getmore_092008 > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss From enda.obrien at dalco.ch Mon Oct 13 12:32:37 2008 From: enda.obrien at dalco.ch (Enda O'Brien) Date: Mon Oct 13 14:23:39 2008 Subject: [mvapich-discuss] Which VIADEV* parameters might free up a "hang" on 64 or more cores, when job runs fine up to 32 cores? In-Reply-To: References: , Message-ID: Hello DK, Thanks for this quick reply! I have progress even since I wrote the message below - here are the details. First, I'm using mvapich-1.0.1, with OpenFabrics-Gen2. The installation lives at: /usr/local/ofed/1.3.1-LFS-1.5.2-RHEL4-67.0.22 . The output of /sbin/lspci is copied below my signature - the only revealing line to my eye is: 01:00.0 InfiniBand: Mellanox Technologies: Unknown device 634a (rev a0) I'm happy to report that the job is running now over 64 cores. I know of one key change I made, but there were at least 2, and I'm just not sure yet what was the 2nd one.. First, I saw in the mvapich user guide: "For example, if you want to disable MPI Allreduce, you can do: $ mpirun rsh -np 8 -hostfile hf VIADEV USE SHMEM ALLREDUCE=0 ./a.out " So I put my VIADEV* parameters in ~/mvapich_loc.conf and ran with: $ mpirun rsh -np 8 -hostfile hf MVAPICH_DEF_PARAMFILE=~/mvapich_loc.conf ./a.out But this generated the message: "Unrecognized argument MVAPICH_DEF_PARAMFILE=/home/Xeobrien/mvapich_loc.conf ignored." And if I left out the "=" sign, I got: "Unrecognized argument MVAPICH_DEF_PARAMFILE ignored." So I tried export MVAPICH_DEF_PARAMFILE=/home/Xeobrien/mvapich_loc.conf $ mpirun rsh -np 8 -hostfile hf ./a.out And that seemed to work. That's the first key thing to know. Now the contents of my current MVAPICH_DEF_PARAMFILE are: VIADEV_VBUF_TOTAL_SIZE=49152 VIADEV_VBUF_POOL_SIZE=1024 VIADEV_ON_DEMAND_THRESHOLD=64 VIADEV_NUM_RDMA_BUFFER=64 VIADEV_USE_SHMEM_COLL=0 ADEV_USE_RDMA_BARRIER=1 VIADEV_SQ_SIZE_MAX=500 VIADEV_DEFAULT_QP_OUS_RD_ATOM=8 VIADEV_CQ_SIZE=100000 VIADEV_DEBUG=3 VIADEV_SRQ_MAX_SIZE=8192 VIADEV_ADAPTIVE_ENABLE_LIMIT=128 Probably only 1 or 2 of these really count to free up the "hang". I'll do some tests now to find out which one(s). If/when I find this out, I'll let you know! The information could be very useful for future reference! Best wishes, Enda =========================== Enda O'Brien DALCO AG Switzerland Aille, Barna, Co. Galway, Ireland Tel. +353 91 591307 Mob. +353 87 7517969 =========================== 00:00.0 Host bridge: Intel Corporation: Unknown device 4003 (rev 20) 00:05.0 PCI bridge: Intel Corporation: Unknown device 4025 (rev 20) 00:09.0 PCI bridge: Intel Corporation: Unknown device 4029 (rev 20) 00:0f.0 System peripheral: Intel Corporation: Unknown device 402f (rev 20) 00:10.0 Host bridge: Intel Corporation: Unknown device 4030 (rev 20) 00:10.1 Host bridge: Intel Corporation: Unknown device 4030 (rev 20) 00:10.2 Host bridge: Intel Corporation: Unknown device 4030 (rev 20) 00:10.3 Host bridge: Intel Corporation: Unknown device 4030 (rev 20) 00:10.4 Host bridge: Intel Corporation: Unknown device 4030 (rev 20) 00:11.0 Host bridge: Intel Corporation: Unknown device 4031 (rev 20) 00:15.0 Host bridge: Intel Corporation: Unknown device 4035 (rev 20) 00:15.1 Host bridge: Intel Corporation: Unknown device 4035 (rev 20) 00:16.0 Host bridge: Intel Corporation: Unknown device 4036 (rev 20) 00:16.1 Host bridge: Intel Corporation: Unknown device 4036 (rev 20) 00:1d.0 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #1 (rev 09) 00:1d.1 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #2 (rev 09) 00:1d.2 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #3 (rev 09) 00:1d.3 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #4 (rev 09) 00:1d.7 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset EHCI USB2 Controller (rev 09) 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d9) 00:1f.0 ISA bridge: Intel Corporation 631xESB/632xESB/3100 Chipset LPC Interface Controller (rev 09) 00:1f.1 IDE interface: Intel Corporation 631xESB/632xESB IDE Controller (rev 09) 00:1f.2 IDE interface: Intel Corporation 631xESB/632xESB/3100 Chipset SATA Stora ge Controller IDE (rev 09) 00:1f.3 SMBus: Intel Corporation 631xESB/632xESB/3100 Chipset SMBus Controller ( rev 09) 01:00.0 InfiniBand: Mellanox Technologies: Unknown device 634a (rev a0) 02:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Upstream Port (rev 01) 02:00.3 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express to PCI-X Bridg e (rev 01) 03:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Downstream Por t E1 (rev 01) 03:01.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Downstream Por t E2 (rev 01) 03:02.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Downstream Por t E3 (rev 01) 06:00.0 Ethernet controller: Intel Corporation 631xESB/632xESB DPT LAN Controlle r Copper (rev 01) 06:00.1 Ethernet controller: Intel Corporation 631xESB/632xESB DPT LAN Controlle r Copper (rev 01) 08:0c.0 VGA compatible controller: ATI Technologies Inc ES1000 (rev 02) ________________________________________ From: Dhabaleswar Panda [panda@cse.ohio-state.edu] Sent: 13 October 2008 16:58 To: Enda O'Brien Cc: mvapich-discuss@cse.ohio-state.edu; Franklin Dallmann Subject: Re: [mvapich-discuss] Which VIADEV* parameters might free up a "hang" on 64 or more cores, when job runs fine up to 32 cores? Thanks for your question. Can you provide some details re. the mvapich version, the mvapich interface (OpenFabrics-Gen2 or other), computing platform and the InfiniBand NIC you are using. This will help us to determine what is going on here and provide appropriate suggestions. Thanks, DK On Mon, 13 Oct 2008, Enda O'Brien wrote: > Hello, > > I saw this address at the top of the mvapich.conf file on the system I'm using, so I thought I'd submit this question: > > What parameter(s) in the mvapich.conf file might be adjusted to "free" up a job that is "hanging" on 64 or more cores, but which runs fine on 8, 16 or 32 cores? > > When such a thing happens on a Quadrics cluster (as it sometimes does...), I can usually adjust (increase) LIBELAN_TPORT_BIGMSG and LIBELAN_ALLOC_SIZE to free the log-jam. That's just 2 parameters. However, there are ~100 VIADEV* parameters in mvapich.conf, and the ones I've adjusted so far haven't made any difference. > > The main MPI function in the application in question is MPI_Alltoall, but it uses only ~3 minutes out of 80 on 32 cores. > > Any tips, advice, recommendations gratefully received! > > Best wishes, > Enda > > P.S. Here are the settings I've tried: > VIADEV_VBUF_TOTAL_SIZE=49152 > VIADEV_VBUF_POOL_SIZE=1024 > VIADEV_ON_DEMAND_THRESHOLD=64 > VIADEV_NUM_RDMA_BUFFER=64 > VIADEV_USE_SHMEM_COLL=0 > ADEV_USE_RDMA_BARRIER=1 > VIADEV_SQ_SIZE_MAX=500 > VIADEV_DEFAULT_QP_OUS_RD_ATOM=8 > VIADEV_CQ_SIZE=100000 > VIADEV_DEBUG=3 > VIADEV_SRQ_MAX_SIZE=8192 > VIADEV_ADAPTIVE_ENABLE_LIMIT=128 > > =========================== > Enda O'Brien > DALCO AG Switzerland > Aille, Barna, Co. Galway, Ireland > Tel. +353 91 591307 > Mob. +353 87 7517969 > =========================== > > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > From chai.15 at osu.edu Mon Oct 13 18:13:21 2008 From: chai.15 at osu.edu (Lei Chai) Date: Mon Oct 13 18:13:23 2008 Subject: [mvapich-discuss] (no subject) In-Reply-To: <828122.22047.qm@web15306.mail.cnb.yahoo.com> References: <828122.22047.qm@web15306.mail.cnb.yahoo.com> Message-ID: <48F3C801.3040808@osu.edu> To use slurm to run mvapich/mvapich2, please refer to the user guide: http://mvapich.cse.ohio-state.edu/support/mvapich_user_guide-1.1.html#x1-180005.3 http://mvapich.cse.ohio-state.edu/support/user_guide_mvapich2-1.2rc2.html#x1-150005.2.2 We have done extensive testing on X86_64 platform, but not on IA64 platform recently. Your mileage on the heterogeneous cluster may vary. Lei Ç¿ Âí wrote: > Hi, All friends > I have a heterogeneous cluster with 32 IA64 nodes and 32 X86_64 nodes, > But I don't know how to srun a job running on these 64 nodes. > please guilde me to operate it. thanks advance. > > ------------------------------------------------------------------------ > ÑÅ»¢ÓÊÏ䣬ÄúµÄÖÕÉúÓÊÏ䣡 > ------------------------------------------------------------------------ > > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > From chai.15 at osu.edu Mon Oct 13 18:45:01 2008 From: chai.15 at osu.edu (Lei Chai) Date: Mon Oct 13 18:45:02 2008 Subject: [mvapich-discuss] Number of IB QP's In-Reply-To: <48F2FEDE.8060300@Sun.COM> References: <48F2FEDE.8060300@Sun.COM> Message-ID: <48F3CF6D.8020006@osu.edu> Hi Lars, My answers are inline: > How many QP's will be used for a large application run with MVAPICH / > MVAPICH2 1.x ? aka' given an application running all-to-all traffic of > random size on

MPI processes (= #cores) behind each HCA on > nodes/HCA's (i.e. a total of N*P application processes) will use > QP's on rank 0 and QP's on the rest... If it's all-to-all and the number of processes>64 (on-demand connection model is used), then each process will setup (N-1)*P QP's. Basically each process will connect to all the other processes except those that are on the same node, and each connection by default uses one QP. > Will Automatic path migration affect the number of QP's used ? Yes. It will setup 4 QP's per port for each connection, if the number of processes is in the range of 8-64, and 2 QP's per port for each connection otherwise. > Will using the MVAPICH2 multi-rail feature affect number of QP's used ? Yes, each rail will need to setup a QP. > Other factors / features that will affect the number of QP's ? You can use env variables to modify the number of QP's for each connection. And if you are using the XRC mode in mvapich-1.1 then you will need to setup less number of QP's. The details are discussed in the following paper: http://nowlab.cse.ohio-state.edu/publications/conf-papers/2008/koop-cluster08.pdf Hope this helps. Lei > /lars paul > > > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss From enda.obrien at dalco.ch Mon Oct 13 18:46:51 2008 From: enda.obrien at dalco.ch (Enda O'Brien) Date: Mon Oct 13 18:47:04 2008 Subject: [mvapich-discuss] Which VIADEV* parameters might free up a "hang" on 64 or more cores, when job runs fine up to 32 cores? In-Reply-To: References: , , Message-ID: Hello DK, Just for completeness here, the answer to my original question (i.e., how to avoid "hanging" on many MPI processes with MVAPICH) seems to be to set VIADEV_USE_SHMEM_COLL=0 There is a hint about this in section 7.1.4 of the MVAPICH User Guide ("application hangs/aborts in Collectives") - only the main collective in my application is MPI_Alltoall, which apparently not be specifically targetted as, e.g. Allreduce can with VIADEV_USE_SHMEM_ALLREDUCE=0. So that's it. Performance also seems to be okay whether this is on or off, though I need to check that some more. Best wishes, Enda ________________________________________ From: Enda O'Brien Sent: 13 October 2008 17:32 To: Dhabaleswar Panda Cc: mvapich-discuss@cse.ohio-state.edu; Franklin Dallmann Subject: RE: [mvapich-discuss] Which VIADEV* parameters might free up a "hang" on 64 or more cores, when job runs fine up to 32 cores? Hello DK, Thanks for this quick reply! I have progress even since I wrote the message below - here are the details. First, I'm using mvapich-1.0.1, with OpenFabrics-Gen2. The installation lives at: /usr/local/ofed/1.3.1-LFS-1.5.2-RHEL4-67.0.22 . The output of /sbin/lspci is copied below my signature - the only revealing line to my eye is: 01:00.0 InfiniBand: Mellanox Technologies: Unknown device 634a (rev a0) I'm happy to report that the job is running now over 64 cores. I know of one key change I made, but there were at least 2, and I'm just not sure yet what was the 2nd one.. First, I saw in the mvapich user guide: "For example, if you want to disable MPI Allreduce, you can do: $ mpirun rsh -np 8 -hostfile hf VIADEV USE SHMEM ALLREDUCE=0 ./a.out " So I put my VIADEV* parameters in ~/mvapich_loc.conf and ran with: $ mpirun rsh -np 8 -hostfile hf MVAPICH_DEF_PARAMFILE=~/mvapich_loc.conf ./a.out But this generated the message: "Unrecognized argument MVAPICH_DEF_PARAMFILE=/home/Xeobrien/mvapich_loc.conf ignored." And if I left out the "=" sign, I got: "Unrecognized argument MVAPICH_DEF_PARAMFILE ignored." So I tried export MVAPICH_DEF_PARAMFILE=/home/Xeobrien/mvapich_loc.conf $ mpirun rsh -np 8 -hostfile hf ./a.out And that seemed to work. That's the first key thing to know. Now the contents of my current MVAPICH_DEF_PARAMFILE are: VIADEV_VBUF_TOTAL_SIZE=49152 VIADEV_VBUF_POOL_SIZE=1024 VIADEV_ON_DEMAND_THRESHOLD=64 VIADEV_NUM_RDMA_BUFFER=64 VIADEV_USE_SHMEM_COLL=0 ADEV_USE_RDMA_BARRIER=1 VIADEV_SQ_SIZE_MAX=500 VIADEV_DEFAULT_QP_OUS_RD_ATOM=8 VIADEV_CQ_SIZE=100000 VIADEV_DEBUG=3 VIADEV_SRQ_MAX_SIZE=8192 VIADEV_ADAPTIVE_ENABLE_LIMIT=128 Probably only 1 or 2 of these really count to free up the "hang". I'll do some tests now to find out which one(s). If/when I find this out, I'll let you know! The information could be very useful for future reference! Best wishes, Enda =========================== Enda O'Brien DALCO AG Switzerland Aille, Barna, Co. Galway, Ireland Tel. +353 91 591307 Mob. +353 87 7517969 =========================== 00:00.0 Host bridge: Intel Corporation: Unknown device 4003 (rev 20) 00:05.0 PCI bridge: Intel Corporation: Unknown device 4025 (rev 20) 00:09.0 PCI bridge: Intel Corporation: Unknown device 4029 (rev 20) 00:0f.0 System peripheral: Intel Corporation: Unknown device 402f (rev 20) 00:10.0 Host bridge: Intel Corporation: Unknown device 4030 (rev 20) 00:10.1 Host bridge: Intel Corporation: Unknown device 4030 (rev 20) 00:10.2 Host bridge: Intel Corporation: Unknown device 4030 (rev 20) 00:10.3 Host bridge: Intel Corporation: Unknown device 4030 (rev 20) 00:10.4 Host bridge: Intel Corporation: Unknown device 4030 (rev 20) 00:11.0 Host bridge: Intel Corporation: Unknown device 4031 (rev 20) 00:15.0 Host bridge: Intel Corporation: Unknown device 4035 (rev 20) 00:15.1 Host bridge: Intel Corporation: Unknown device 4035 (rev 20) 00:16.0 Host bridge: Intel Corporation: Unknown device 4036 (rev 20) 00:16.1 Host bridge: Intel Corporation: Unknown device 4036 (rev 20) 00:1d.0 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #1 (rev 09) 00:1d.1 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #2 (rev 09) 00:1d.2 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #3 (rev 09) 00:1d.3 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #4 (rev 09) 00:1d.7 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset EHCI USB2 Controller (rev 09) 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d9) 00:1f.0 ISA bridge: Intel Corporation 631xESB/632xESB/3100 Chipset LPC Interface Controller (rev 09) 00:1f.1 IDE interface: Intel Corporation 631xESB/632xESB IDE Controller (rev 09) 00:1f.2 IDE interface: Intel Corporation 631xESB/632xESB/3100 Chipset SATA Stora ge Controller IDE (rev 09) 00:1f.3 SMBus: Intel Corporation 631xESB/632xESB/3100 Chipset SMBus Controller ( rev 09) 01:00.0 InfiniBand: Mellanox Technologies: Unknown device 634a (rev a0) 02:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Upstream Port (rev 01) 02:00.3 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express to PCI-X Bridg e (rev 01) 03:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Downstream Por t E1 (rev 01) 03:01.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Downstream Por t E2 (rev 01) 03:02.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Downstream Por t E3 (rev 01) 06:00.0 Ethernet controller: Intel Corporation 631xESB/632xESB DPT LAN Controlle r Copper (rev 01) 06:00.1 Ethernet controller: Intel Corporation 631xESB/632xESB DPT LAN Controlle r Copper (rev 01) 08:0c.0 VGA compatible controller: ATI Technologies Inc ES1000 (rev 02) ________________________________________ From: Dhabaleswar Panda [panda@cse.ohio-state.edu] Sent: 13 October 2008 16:58 To: Enda O'Brien Cc: mvapich-discuss@cse.ohio-state.edu; Franklin Dallmann Subject: Re: [mvapich-discuss] Which VIADEV* parameters might free up a "hang" on 64 or more cores, when job runs fine up to 32 cores? Thanks for your question. Can you provide some details re. the mvapich version, the mvapich interface (OpenFabrics-Gen2 or other), computing platform and the InfiniBand NIC you are using. This will help us to determine what is going on here and provide appropriate suggestions. Thanks, DK On Mon, 13 Oct 2008, Enda O'Brien wrote: > Hello, > > I saw this address at the top of the mvapich.conf file on the system I'm using, so I thought I'd submit this question: > > What parameter(s) in the mvapich.conf file might be adjusted to "free" up a job that is "hanging" on 64 or more cores, but which runs fine on 8, 16 or 32 cores? > > When such a thing happens on a Quadrics cluster (as it sometimes does...), I can usually adjust (increase) LIBELAN_TPORT_BIGMSG and LIBELAN_ALLOC_SIZE to free the log-jam. That's just 2 parameters. However, there are ~100 VIADEV* parameters in mvapich.conf, and the ones I've adjusted so far haven't made any difference. > > The main MPI function in the application in question is MPI_Alltoall, but it uses only ~3 minutes out of 80 on 32 cores. > > Any tips, advice, recommendations gratefully received! > > Best wishes, > Enda > > P.S. Here are the settings I've tried: > VIADEV_VBUF_TOTAL_SIZE=49152 > VIADEV_VBUF_POOL_SIZE=1024 > VIADEV_ON_DEMAND_THRESHOLD=64 > VIADEV_NUM_RDMA_BUFFER=64 > VIADEV_USE_SHMEM_COLL=0 > ADEV_USE_RDMA_BARRIER=1 > VIADEV_SQ_SIZE_MAX=500 > VIADEV_DEFAULT_QP_OUS_RD_ATOM=8 > VIADEV_CQ_SIZE=100000 > VIADEV_DEBUG=3 > VIADEV_SRQ_MAX_SIZE=8192 > VIADEV_ADAPTIVE_ENABLE_LIMIT=128 > > =========================== > Enda O'Brien > DALCO AG Switzerland > Aille, Barna, Co. Galway, Ireland > Tel. +353 91 591307 > Mob. +353 87 7517969 > =========================== > > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > From tgetachew at hotmail.com Tue Oct 14 11:23:47 2008 From: tgetachew at hotmail.com (Tatek) Date: Tue Oct 14 11:24:04 2008 Subject: [mvapich-discuss] mvapich2 mpirun_rsh bug In-Reply-To: <20081013160831.GB9190@kappa.cse.ohio-state.edu> References: <20081013160831.GB9190@kappa.cse.ohio-state.edu> Message-ID: Thanks!! Tatek> Date: Mon, 13 Oct 2008 12:08:31 -0400> From: sridharj@cse.ohio-state.edu> To: tgetachew@hotmail.com> CC: mvapich-discuss@cse.ohio-state.edu> Subject: Re: [mvapich-discuss] mvapich2 mpirun_rsh bug> > Hi Tatek,> > On further investigation, we saw that the path handling was different> between MVAPICH and MVAPICH2 as you pointed out. Thanks for this report.> We've updated mpirun_rsh on MVAPICH2 to mimic the behavior on MVAPICH.> > The update will be available in the upcoming release of MVAPICH2 1.2. It is> also available on the svn at:> https://mvapich.cse.ohio-state.edu/svn/mpi/mvapich2/trunk> > Regards,> Jaidev> > On Fri, Oct 10, 2008 at 11:12:15PM +0000, Tatek wrote:> > > > Hello All,> > > > There seems to be a bug with mvapich2 and mpirun_rsh command. Once I> > install mvapich2, to execute mpiun_rsh command, I have to give the> > full path of the mpirun_rsh or it will not execute. For example, I> > was trying to run mpiBench and after setting the PATH for the bin and> > LD_LIBRARY_PATH, I would get the following> > > > [root@t2k-0007 mpibench-0.1-3]# mpirun_rsh -np 2 -hostfile hosts5> > ./mpiBench -i 100 Allreduce> > /usr/bin/env: mpispawn: No such file or directory> > Child exited abnormally!> > cleanupKilling remote processes.../usr/bin/env: mpispawn: No such file> > or directory DONE> > > > note that "which mpirun_rsh" shows the correct PATH so I had the PATH> > set correctly> > > > [root@t2k-0007 mpibench-0.1-3]# which mpirun_rsh> > /global/usr/mpi/mvapich2-1.2/pgi/bin/mpirun_rsh> > > > If I give the full PATH then no problem.> > > > [root@t2k-0007 mpibench-0.1-3]#> > /global/usr/mpi/mvapich2-1.2/pgi/bin/mpirun_rsh -np 2 -hostfile hosts5> > ./mpiBench -i 100 Allreduce START mpiBench 0 : ddn-0268> > 1 : ddn-0302> > Allreduce Bytes: 0 Iters: 100 Avg: 2.9612 Min: 2.9612 Max: 2.9612 Dim:> > 0 Ranks: 2> > Allreduce Bytes: 8 Iters: 100 Avg: 6.4301 Min: 6.4087 Max: 6.4516 Dim:> > 0 Ranks: 2> > > > After some investigation, one of my colleagues figured out the issue> > was that MVAPICH2 mpirun_rsh command gets the directory name from the> > name of the executable (passed to the program as the first argument)> > and starts mpispawn using this path. If the full path name is not> > supplied, the mpispawn program is started without a path and can't be> > found unless it happens to be in the default login path of the> > user. We think this is a bug in MVAPICH2's mpirun_rsh. It is> > reasonable to expect that if the MVAPICH2 binary directory is the> > PATH, issuing an mpirun_rsh command should work.> > > > We have a workaround for now by either using the full PATH or using> > wrapper scripts but ideally, this should work same as it did with> > mvapich1.> > > > Best Regards,> > > > > > Tatek Getachew> > > > Solutions Engineer, HPC Solutions Engineering> > > > Appro International, Inc.> > > > 446 South Abbott Ave> > > > Milpitas, CA 95035> > > > 408-941-8100 x504 (office)> > > > 408-640-9050 (cell)> > > > [1]tgetachew@appro.com> > > > > > > > > > _________________________________________________________________> > > > Get more out of the Web. Learn 10 hidden secrets of Windows Live.> > [2]Learn Now> > > > References> > > > 1. mailto:tgetachew@appro.com> > 2. http://windowslive.com/connect/post/jamiethomson.spaces.live.com-Blog-cns!550F681DAD532637!5295.entry?ocid=TXT_TAGLM_WL_getmore_092008> > > _______________________________________________> > mvapich-discuss mailing list> > mvapich-discuss@cse.ohio-state.edu> > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss> _________________________________________________________________ Want to do more with Windows Live? Learn ?10 hidden secrets? from Jamie. http://windowslive.com/connect/post/jamiethomson.spaces.live.com-Blog-cns!550F681DAD532637!5295.entry?ocid=TXT_TAGLM_WL_domore_092008 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20081014/5e963877/attachment.html From chai.15 at osu.edu Tue Oct 14 13:45:00 2008 From: chai.15 at osu.edu (Lei Chai) Date: Tue Oct 14 13:45:02 2008 Subject: [mvapich-discuss] Number of IB QP's In-Reply-To: <48F3CF6D.8020006@osu.edu> References: <48F2FEDE.8060300@Sun.COM> <48F3CF6D.8020006@osu.edu> Message-ID: <48F4DA9C.4040907@osu.edu> Hi Lars, I need to clarify the number of QP's used for automatic path migration (APM). My answer below (4 QP's per port and 2 QP's per port) actually referred to the hot-spot avoidance feature (MV2_USE_HSAM=1) in mvapich2. APM doesn't need additional QP's. Sorry if this has caused confusion. Lei Lei Chai wrote: > Hi Lars, > > My answers are inline: > >> How many QP's will be used for a large application run with MVAPICH / >> MVAPICH2 1.x ? aka' given an application running all-to-all traffic >> of random size on

MPI processes (= #cores) behind each HCA on >> nodes/HCA's (i.e. a total of N*P application processes) will use >> QP's on rank 0 and QP's on the rest... > > If it's all-to-all and the number of processes>64 (on-demand > connection model is used), then each process will setup (N-1)*P QP's. > Basically each process will connect to all the other processes except > those that are on the same node, and each connection by default uses > one QP. > >> Will Automatic path migration affect the number of QP's used ? > > Yes. It will setup 4 QP's per port for each connection, if the number > of processes is in the range of 8-64, and 2 QP's per port for each > connection otherwise. > >> Will using the MVAPICH2 multi-rail feature affect number of QP's used ? > > Yes, each rail will need to setup a QP. > >> Other factors / features that will affect the number of QP's ? > You can use env variables to modify the number of QP's for each > connection. And if you are using the XRC mode in mvapich-1.1 then you > will need to setup less number of QP's. The details are discussed in > the following paper: > > http://nowlab.cse.ohio-state.edu/publications/conf-papers/2008/koop-cluster08.pdf > > > Hope this helps. > > Lei > > >> /lars paul >> >> >> _______________________________________________ >> mvapich-discuss mailing list >> mvapich-discuss@cse.ohio-state.edu >> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss From joseph.hargitai at nyu.edu Wed Oct 15 10:23:21 2008 From: joseph.hargitai at nyu.edu (Joseph Hargitai) Date: Wed Oct 15 10:23:31 2008 Subject: [mvapich-discuss] c++ include header file not created Message-ID: these files are created but the header for config is missing: mpi2c++_config.h - which is needed for code. This is the case also for version 1.1 as well as this one here 1.0.1 [hargitai@login-0-0 mpi2c++]$ ls comm.h group_inln.h pdatatype.h prequest_inln.h comm_inln.h intercomm.h perrhandler.h pstatus.h constants.h intercomm_inln.h pexception.h pstatus_inln.h datatype.h intracomm.h pgroup.h ptopology.h datatype_inln.h intracomm_inln.h pgroup_inln.h request.h errhandler.h mpi2c++_list.h pintercomm.h request_inln.h errhandler_inln.h mpi2c++_map.h pintracomm.h status.h exception.h mpi++.h pmpi++.h status_inln.h functions.h op.h pop.h topology.h functions_inln.h op_inln.h pop_inln.h topology_inln.h group.h pcomm.h prequest.h here is the config file: [hargitai@login-0-0 mvapich-1.0.1]$ more make.mvapich.gen2 #!/bin/bash # Most variables here can be overridden by exporting them in the environment # before running this script. Default values have been provided if the # environment variable is not already set. source ./make.mvapich.def # The target architecture. If not exported outside of this script, # it will be found automatically or prompted for if necessary. # Supported: "_IA32_", "_IA64_", "_EM64T_", "_X86_64_" # if [ -z "$ARCH" ]; then arch fi # Mandatory variables. All are checked except CXX and F90. IBHOME=${IBHOME:-/usr/include/infiniband} IBHOME_LIB=${IBHOME_LIB:-/usr/lib64} PREFIX=${PREFIX:-/home/hargitai/mv1} export CC=${CC:-/usr/local/intel/cce/10.0.023/bin/icc} export CXX=${CXX:-/usr/local/intel/cce/10.0.023/bin/icc} export F77=${F77:-/usr/local/intel/fce/10.0.023/bin/ifort} export F90=${F90:-/usr/local/intel/fce/10.0.023/bin/ifort} if [ $ARCH = "SOLARIS" ]; then die_setup "MVAPICH GEN2 is not supported on Solaris." elif [ $ARCH = "MAC_OSX" ]; then die_setup "MVAPICH GEN2 is not supported on MacOS." fi # # Compiler specific flags. If you are using # ICC on IA64 platform, please set COMPILER_FLAG # to "icc" # COMPILER_FLAG=${COMPILER_FLAG:-} if [ "$COMPILER_FLAG" == "icc" ]; then COMPILER_FLAG="-D_ICC_" else COMPILER_FLAG="" fi # Check mandatory variable settings. if [ -z "$IBHOME" ] || [ -z "$PREFIX" ] || [ -z "$CC" ] || [ -z "$F77" ]; then die_setup "Please set mandatory variables in this script." elif [ ! -d $IBHOME ]; then die_setup "IBHOME directory $IBHOME does not exist." fi # Optional variables. # # Whether to enable ROMIO support. This is necessary if building the # F90 modules. if [ -n "$F90" ]; then ROMIO="--with-romio" else ROMIO=${ROMIO:---without-romio} fi # PTMALLOC support for MVAPICH2 memory hooks. Enabling this will allow # MVAPICH2 to release memory to the Operating System (when registration # cache is enabled). Enabled by default. Disable with "no". PTMALLOC=${PTMALLOC:-} if [ "$PTMALLOC" = "no" ]; then PTMALLOC="-DDISABLE_PTMALLOC" else PTMALLOC="" fi # Set this to override automatic optimization setting (-03). OPT_FLAG=${OPT_FLAG:--O3} if [ -n "$PROCESSOR" ]; then PROCESSOR=-D${PROCESSOR} else PROCESSOR= fi export LIBS=${LIBS:--L${IBHOME_LIB} -Wl,-rpath=${IBHOME_LIB} -libverbs -libumad -lpthread} export FFLAGS=${FFLAGS:--L${IBHOME_LIB}} export CFLAGS=${CFLAGS:--D${ARCH} ${PROCESSOR} ${PTMALLOC} -DEARLY_SEND_COMPLETI ON -DMEMORY_SCALE -DVIADEV_RPUT_SUPPORT -D_SMP_ -D_SMP_RNDV_ -DCH_GEN2 -D_GNU_SO URCE ${COMPILER_FLAG} -I${IBHOME}/include $OPT_FLAG} export MPIRUN_CFLAGS="${MPIRUN_CFLAGS} -DLD_LIBRARY_PATH_MPI=\\\"${PREFIX}/lib/s hared\\\" -DMPI_PREFIX=\\\"${PREFIX}/\\\" -DPARAM_GLOBAL=\\\"${PREFIX}/etc/mvapi ch.conf\\\"" # Prelogue make distclean &>/dev/null set -o pipefail # Configure MVAPICH echo "Configuring MVAPICH..." ./configure --with-device=ch_gen2 --with-arch=LINUX -prefix=${PREFIX} \ $ROMIO --without-mpe --enable-cxx --enable-sharedlib --enable-debug --en able-f77 --enable-f90 -lib="$LIBS" 2>&1 |tee config-mine.log ret=$? test $ret = 0 || die "configuration." # Build MVAPICH echo "Building MVAPICH..." make 2>&1 |tee make-mine.log ret=$? test $ret = 0 || die "building MVAPICH." # Install MVAPICH echo "MVAPICH installation..." rm -f install-mine.log make install 2>&1 |tee install-mine.log ret=$? test $ret = 0 || die "installing MVAPICH." thanks, joseph From devesh28 at gmail.com Wed Oct 15 00:55:16 2008 From: devesh28 at gmail.com (Devesh Sharma) Date: Wed Oct 15 10:31:03 2008 Subject: [mvapich-discuss] MVAPICH2-1.0.1 crashes with TCP/IP Message-ID: <309a667c0810142155j1c1ffccbo924acb3a75949dec@mail.gmail.com> hello all, I am trying to run linpack_10.0.4 on TCP/IP of MVAPICH2-1.0.1with 80% of memory usage i.e. N=331776 we have a 16 node cluster with 64GB of physical memory in each node and having quad socket quad core configration when I am running 256 process on this cluster over ethernet I receive following error after a run of 1 hour approximately. please help me figuring out the issues. [cli_239]: aborting job: Fatal error in MPI_Send: Other MPI error, error stack: MPI_Send(190).............................: MPI_Send(buf=0x2b6cb1e040, count=1, dtype=USER, dest=16, tag=2857, comm=0x84000001) failed MPIDI_CH3_Progress_wait(212)..............: an error occurred while handling an event returned by MPIDU_Sock_Wait() MPIDI_CH3I_Progress_handle_sock_event(651): MPIDU_Socki_handle_pollhup(402)...........: connection closed by peer (set=0,sock=7) rank 239 in job 2 tf00_50378 caused collective abort of all ranks exit status of rank 239: killed by signal 9 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20081015/71055986/attachment.html From christian.guggenberger at rzg.mpg.de Wed Oct 15 11:13:47 2008 From: christian.guggenberger at rzg.mpg.de (Christian Guggenberger) Date: Wed Oct 15 11:14:00 2008 Subject: [mvapich-discuss] c++ include header file not created In-Reply-To: References: Message-ID: <20081015151347.GH4095@daltons.rzg.mpg.de> Joseph, > > these files are created but the header for config is missing: mpi2c++_config.h - which is needed for code. This is the case also for version 1.1 as well as this one here 1.0.1 > > > # Mandatory variables. All are checked except CXX and F90. > IBHOME=${IBHOME:-/usr/include/infiniband} > IBHOME_LIB=${IBHOME_LIB:-/usr/lib64} > PREFIX=${PREFIX:-/home/hargitai/mv1} > export CC=${CC:-/usr/local/intel/cce/10.0.023/bin/icc} > export CXX=${CXX:-/usr/local/intel/cce/10.0.023/bin/icc} 'icc' is not the C++ compiler - please use 'icpc' instead. hope this helps. - Christian From joseph.hargitai at nyu.edu Wed Oct 15 11:48:53 2008 From: joseph.hargitai at nyu.edu (Joseph Hargitai) Date: Wed Oct 15 11:49:05 2008 Subject: [mvapich-discuss] c++ include header file not created In-Reply-To: <20081015151347.GH4095@daltons.rzg.mpg.de> References: <20081015151347.GH4095@daltons.rzg.mpg.de> Message-ID: cool - reinstalling and icpc did the trick. j ----- Original Message ----- From: Christian Guggenberger Date: Wednesday, October 15, 2008 11:13 am Subject: Re: [mvapich-discuss] c++ include header file not created > Joseph, > > > > these files are created but the header for config is missing: > mpi2c++_config.h - which is needed for code. This is the case also for > version 1.1 as well as this one here 1.0.1 > > > > > > # Mandatory variables. All are checked except CXX and F90. > > IBHOME=${IBHOME:-/usr/include/infiniband} > > IBHOME_LIB=${IBHOME_LIB:-/usr/lib64} > > PREFIX=${PREFIX:-/home/hargitai/mv1} > > export CC=${CC:-/usr/local/intel/cce/10.0.023/bin/icc} > > export CXX=${CXX:-/usr/local/intel/cce/10.0.023/bin/icc} > > 'icc' is not the C++ compiler - please use 'icpc' instead. > > hope this helps. > > - Christian From Lars.Paul.Huse at Sun.COM Wed Oct 15 12:08:27 2008 From: Lars.Paul.Huse at Sun.COM (Lars Paul Huse) Date: Wed Oct 15 12:08:40 2008 Subject: [mvapich-discuss] Number of IB QP's In-Reply-To: <48F4DA9C.4040907@osu.edu> References: <48F2FEDE.8060300@Sun.COM> <48F3CF6D.8020006@osu.edu> <48F4DA9C.4040907@osu.edu> Message-ID: <48F6157B.1050301@Sun.COM> Hi Lei, Thanks for the reply and the pointer :-) So for a N=4097 node system with P=16 processes per node (i.e. a total of 65552 MPI processes - not completely unlike a full TACC Ranger run utilizing all cores) the number of QP's per node with or without APM will be : QPpN = (N-1)*P*P = 4k*16*16 = 1M /lars paul On 10/14/08 07:45 PM, Lei Chai wrote: > Hi Lars, > > I need to clarify the number of QP's used for automatic path migration > (APM). My answer below (4 QP's per port and 2 QP's per port) actually > referred to the hot-spot avoidance feature (MV2_USE_HSAM=1) in mvapich2. > APM doesn't need additional QP's. Sorry if this has caused confusion. > > Lei > > > Lei Chai wrote: >> Hi Lars, >> >> My answers are inline: >> >>> How many QP's will be used for a large application run with MVAPICH / >>> MVAPICH2 1.x ? aka' given an application running all-to-all traffic >>> of random size on

MPI processes (= #cores) behind each HCA on >>> nodes/HCA's (i.e. a total of N*P application processes) will use >>> QP's on rank 0 and QP's on the rest... >> >> If it's all-to-all and the number of processes>64 (on-demand >> connection model is used), then each process will setup (N-1)*P QP's. >> Basically each process will connect to all the other processes except >> those that are on the same node, and each connection by default uses >> one QP. >> >>> Will Automatic path migration affect the number of QP's used ? >> >> Yes. It will setup 4 QP's per port for each connection, if the number >> of processes is in the range of 8-64, and 2 QP's per port for each >> connection otherwise. >> >>> Will using the MVAPICH2 multi-rail feature affect number of QP's used ? >> >> Yes, each rail will need to setup a QP. >> >>> Other factors / features that will affect the number of QP's ? >> You can use env variables to modify the number of QP's for each >> connection. And if you are using the XRC mode in mvapich-1.1 then you >> will need to setup less number of QP's. The details are discussed in >> the following paper: >> >> http://nowlab.cse.ohio-state.edu/publications/conf-papers/2008/koop-cluster08.pdf >> >> >> Hope this helps. >> >> Lei >> >> >>> /lars paul >>> >>> >>> _______________________________________________ >>> mvapich-discuss mailing list >>> mvapich-discuss@cse.ohio-state.edu >>> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss >> >> _______________________________________________ >> mvapich-discuss mailing list >> mvapich-discuss@cse.ohio-state.edu >> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > From perkinjo at cse.ohio-state.edu Thu Oct 16 15:52:13 2008 From: perkinjo at cse.ohio-state.edu (Jonathan Perkins) Date: Thu Oct 16 15:53:28 2008 Subject: [mvapich-discuss] Re: [ofa-general] [PATCH] mvapich2-trunk-3073 In-Reply-To: References: Message-ID: <20081016195213.GQ3120@cse.ohio-state.edu> Mike: Thanks for the patch. I'll take a look at this and include it in our next srpm. Please note that you can mail mvapich-discuss@cse.ohio-state.edu for issues related to mvapich or mvapich2. On Thu, Oct 16, 2008 at 02:28:33PM -0500, Mike Heinz wrote: > The mpivars.sh and mpivars.csh scripts (created by the RPM spec file) do > not set the LD_LIBRARY_PATH, which means that mvapich2 programs may not > run or link unless the path is explicitly set. A similar problem was > found with mvapich earlier this year. > > [mheinz@homer SRPMS]> diff -ud mvapich2.spec.orig mvapich2.spec > --- mvapich2.spec.orig 2008-10-16 15:19:36.000000000 -0400 > +++ mvapich2.spec 2008-10-16 15:22:40.000000000 -0400 > @@ -126,6 +126,14 @@ > set path = ( %{_prefix}/bin ) > endif > > +if ("1" == "\$?LD_LIBRARY_PATH") then > + if ("\$LD_LIBRARY_PATH" !~ *%{_prefix}/lib) then > + setenv LD_LIBRARY_PATH %{_prefix}/lib:\${LD_LIBRARY_PATH} > + endif > +else > + setenv LD_LIBRARY_PATH %{_prefix}/lib > +endif > + > if (\$?MANPATH) then > if ( "\${MANPATH}" !~ *%{_prefix}/man* ) then > setenv MANPATH %{_prefix}/man:\$MANPATH > @@ -140,6 +148,10 @@ > PATH=%{_prefix}/bin:\${PATH} > fi > > +if ! echo \${LD_LIBRARY_PATH} | grep -q %{_prefix}/lib ; then > + export LD_LIBRARY_PATH=%{_prefix}/lib:\${LD_LIBRARY_PATH} > +fi > + > if ! echo \${MANPATH} | grep -q %{_prefix}/man ; then > MANPATH=%{_prefix}/man:\${MANPATH} > fi > > -- > Michael Heinz > Principal Engineer, Qlogic Corporation > King of Prussia, Pennsylvania > > _______________________________________________ > general mailing list > general@lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general -- Jonathan Perkins http://www.cse.ohio-state.edu/~perkinjo From joseph.hargitai at nyu.edu Fri Oct 17 21:39:53 2008 From: joseph.hargitai at nyu.edu (Joseph Hargitai) Date: Fri Oct 17 21:40:02 2008 Subject: [mvapich-discuss] cpu placement with mpiexec? Message-ID: Hi all: can one use/translate the mpirun_rsh -np 2 -hostfile $PBS_NODEFILE VIADEV_CPU_MAPPING=0,4 ./a.out construct to use with mpiexec when using mvapich1x versions? thanks, joseph From chai.15 at osu.edu Fri Oct 17 22:19:40 2008 From: chai.15 at osu.edu (Lei Chai) Date: Fri Oct 17 22:19:36 2008 Subject: [mvapich-discuss] cpu placement with mpiexec? In-Reply-To: References: Message-ID: <48F947BC.4060907@osu.edu> Joseph, If you are using mpiexec and mvapich2, you can use: $ mpiexec -n 4 -env MV2_CPU_MAPPING 0:1:4:5 ./a.out and mvapich2 supports mpirun_rsh too which is more scalable than mpiexec. More info about cpu mapping in mvapich2 can be found in the userguide: http://mvapich.cse.ohio-state.edu/support/user_guide_mvapich2-1.2rc2.html#x1-320006.8 Lei Joseph Hargitai wrote: > Hi all: > > can one use/translate the > > mpirun_rsh -np 2 -hostfile $PBS_NODEFILE VIADEV_CPU_MAPPING=0,4 ./a.out > > construct to use with mpiexec when using mvapich1x versions? > > > thanks, > joseph > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > From joseph.hargitai at nyu.edu Sat Oct 18 08:27:20 2008 From: joseph.hargitai at nyu.edu (Joseph Hargitai) Date: Sat Oct 18 08:27:28 2008 Subject: [mvapich-discuss] cpu placement with mpiexec? In-Reply-To: <48F947BC.4060907@osu.edu> References: <48F947BC.4060907@osu.edu> Message-ID: Correct - it does work with mpiexec and mvapich2 however does not work with the mvapich1x version with mpiexec. Passing that argument results in error. we can use the 2x version, i was more just curious. The key to us is that in our PBS environment, that jobs do terminate properly. This works well with mpiexec, but not always with mpirun_rsh. So when you say, mpirun_rsh is more scalable, what does scale refer to? And is there a way to make job termination/cleanup really solid with mpirun_rsh. thanks. j ----- Original Message ----- From: Lei Chai Date: Friday, October 17, 2008 10:19 pm Subject: Re: [mvapich-discuss] cpu placement with mpiexec? > Joseph, > > If you are using mpiexec and mvapich2, you can use: > > $ mpiexec -n 4 -env MV2_CPU_MAPPING 0:1:4:5 ./a.out > > and mvapich2 supports mpirun_rsh too which is more scalable than > mpiexec. More info about cpu mapping in mvapich2 can be found in the > userguide: > > http://mvapich.cse.ohio-state.edu/support/user_guide_mvapich2-1.2rc2.html#x1-320006.8 > > Lei > > > Joseph Hargitai wrote: > > Hi all: > > > > can one use/translate the > > > > mpirun_rsh -np 2 -hostfile $PBS_NODEFILE VIADEV_CPU_MAPPING=0,4 ./a.out > > > > construct to use with mpiexec when using mvapich1x versions? > > > > > > thanks, > > joseph > > _______________________________________________ > > mvapich-discuss mailing list > > mvapich-discuss@cse.ohio-state.edu > > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > > > From panda at cse.ohio-state.edu Sat Oct 18 09:43:31 2008 From: panda at cse.ohio-state.edu (Dhabaleswar Panda) Date: Sat Oct 18 09:43:37 2008 Subject: [mvapich-discuss] cpu placement with mpiexec? In-Reply-To: Message-ID: Joseph, > Correct - it does work with mpiexec and mvapich2 > however does not work with the mvapich1x version with mpiexec. Passing that argument results in error. > > we can use the 2x version, i was more just curious. > > The key to us is that in our PBS environment, that jobs do terminate > properly. This works well with mpiexec, but not always with > mpirun_rsh. So when you say, mpirun_rsh is more scalable, what does > scale refer to? And is there a way to make job termination/cleanup > really solid with mpirun_rsh. The new mpirun_rsh framework takes into account the modern multi-core architectures (i.e., establishing connections first to a leader in a node and then to other cores in that node in a hierarchical manner). Thus, the start-up becomes faster and can scale to larger number of nodes/cores. The job termination/cleanup issue with mpirun_rsh you are mentioning has been solved with some of the recent patches. If you download and use mvapich 1.1RC1 or the trunk version of mvapich2 1.2 (not rc2), you will get these fixes. These fixes will be part of the formal mvapich 1.1 and mvapich2 1.2 releases (which are coming soon). Thanks, DK > > thanks. > j > > ----- Original Message ----- > From: Lei Chai > Date: Friday, October 17, 2008 10:19 pm > Subject: Re: [mvapich-discuss] cpu placement with mpiexec? > > > Joseph, > > > > If you are using mpiexec and mvapich2, you can use: > > > > $ mpiexec -n 4 -env MV2_CPU_MAPPING 0:1:4:5 ./a.out > > > > and mvapich2 supports mpirun_rsh too which is more scalable than > > mpiexec. More info about cpu mapping in mvapich2 can be found in the > > userguide: > > > > http://mvapich.cse.ohio-state.edu/support/user_guide_mvapich2-1.2rc2.html#x1-320006.8 > > > > Lei > > > > > > Joseph Hargitai wrote: > > > Hi all: > > > > > > can one use/translate the > > > > > > mpirun_rsh -np 2 -hostfile $PBS_NODEFILE VIADEV_CPU_MAPPING=0,4 ./a.out > > > > > > construct to use with mpiexec when using mvapich1x versions? > > > > > > > > > thanks, > > > joseph > > > _______________________________________________ > > > mvapich-discuss mailing list > > > mvapich-discuss@cse.ohio-state.edu > > > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > > > > > > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > From joseph.hargitai at nyu.edu Sat Oct 18 09:50:40 2008 From: joseph.hargitai at nyu.edu (Joseph Hargitai) Date: Sat Oct 18 09:50:48 2008 Subject: [mvapich-discuss] cpu placement with mpiexec? In-Reply-To: References: Message-ID: > The new mpirun_rsh framework takes into account the modern multi-core > architectures (i.e., establishing connections first to a leader in a node > and then to other cores in that node in a hierarchical manner). Thus, > the > start-up becomes faster and can scale to larger number of nodes/cores. > The > job termination/cleanup issue with mpirun_rsh you are mentioning has been > solved with some of the recent patches. If you download and use mvapich > 1.1RC1 or the trunk version of mvapich2 1.2 (not rc2), you will get these > fixes. These fixes will be part of the formal mvapich 1.1 and mvapich2 > 1.2 releases (which are coming soon). will mpirun_rsh alone do the cleanup or you need rsh really to be working on the nodes, and pass the -rsh option? j From koop at cse.ohio-state.edu Sat Oct 18 10:21:26 2008 From: koop at cse.ohio-state.edu (Matthew Koop) Date: Sat Oct 18 10:21:33 2008 Subject: [mvapich-discuss] Number of IB QP's In-Reply-To: <48F6157B.1050301@Sun.COM> Message-ID: > So for a N=4097 node system with P=16 processes per node (i.e. a total > of 65552 MPI processes - not completely unlike a full TACC Ranger run > utilizing all cores) the number of QP's per node with or without APM > will be : > > QPpN = (N-1)*P*P = 4k*16*16 = 1M For RC, yes, this is the total number of QPs per node. Per process there will be (N-1)P QPs, or 64K QPs. Instead if the XRC transport, included in ConnectX, is used this number can potentially be (N-1) QPs per process, so less by a factor of P. The paper that Lei sent has some additional details. We also have a new hybrid device for MVAPICH, which allows either UD-RC or UD-XRC operation. Since UD QPs are not connection-oriented, there can be significantly less numbers of QPs needed in the system. Then for high-volume connections a RC or XRC connection can be used. Additional details: http://nowlab.cse.ohio-state.edu/publications/conf-papers/2008/koop-ipdps08.pdf Hope this helps, Matt > /lars paul > > On 10/14/08 07:45 PM, Lei Chai wrote: > > Hi Lars, > > > > I need to clarify the number of QP's used for automatic path migration > > (APM). My answer below (4 QP's per port and 2 QP's per port) actually > > referred to the hot-spot avoidance feature (MV2_USE_HSAM=1) in mvapich2. > > APM doesn't need additional QP's. Sorry if this has caused confusion. > > > > Lei > > > > > > Lei Chai wrote: > >> Hi Lars, > >> > >> My answers are inline: > >> > >>> How many QP's will be used for a large application run with MVAPICH / > >>> MVAPICH2 1.x ? aka' given an application running all-to-all traffic > >>> of random size on

MPI processes (= #cores) behind each HCA on > >>> nodes/HCA's (i.e. a total of N*P application processes) will use > >>> QP's on rank 0 and QP's on the rest... > >> > >> If it's all-to-all and the number of processes>64 (on-demand > >> connection model is used), then each process will setup (N-1)*P QP's. > >> Basically each process will connect to all the other processes except > >> those that are on the same node, and each connection by default uses > >> one QP. > >> > >>> Will Automatic path migration affect the number of QP's used ? > >> > >> Yes. It will setup 4 QP's per port for each connection, if the number > >> of processes is in the range of 8-64, and 2 QP's per port for each > >> connection otherwise. > >> > >>> Will using the MVAPICH2 multi-rail feature affect number of QP's used ? > >> > >> Yes, each rail will need to setup a QP. > >> > >>> Other factors / features that will affect the number of QP's ? > >> You can use env variables to modify the number of QP's for each > >> connection. And if you are using the XRC mode in mvapich-1.1 then you > >> will need to setup less number of QP's. The details are discussed in > >> the following paper: > >> > >> http://nowlab.cse.ohio-state.edu/publications/conf-papers/2008/koop-cluster08.pdf > >> > >> > >> Hope this helps. > >> > >> Lei > >> > >> > >>> /lars paul > >>> > >>> > >>> _______________________________________________ > >>> mvapich-discuss mailing list > >>> mvapich-discuss@cse.ohio-state.edu > >>> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > >> > >> _______________________________________________ > >> mvapich-discuss mailing list > >> mvapich-discuss@cse.ohio-state.edu > >> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > > > > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > From publications at kressworks.com Sat Oct 18 12:16:49 2008 From: publications at kressworks.com (publications) Date: Sat Oct 18 12:16:58 2008 Subject: [mvapich-discuss] 32 bit mvapich shared libraries Message-ID: <54F700260E724490BC434E8F04858AF0@inspiron9100> When installing OFED 1.3.1, the person doing the install is presented with a option: --build32 Build 32-bit libraries. Relevant for x86_64 and ppc64 platforms However, there are no 32 bit shared libraries for mvapich-1.0.1 created, even though the installation script finishes with "installation successful" I guess this implies that the OFED software will not provide the 32 bit mvapich shared libraries. If I use mvapich 1.0.1 alone, how does one get the 32 bit mvapich-1.0.1 libraries built and where will they be located? My system consists of: X86_64 quad-core Intel cpu Centos 5 64 bit OS -a variant of RedHat 5 (ROCKS) Standard gcc, g77, gfortran, etc that come as a part of Centos 5 Regards, Jim Kress From chai.15 at osu.edu Sat Oct 18 17:19:54 2008 From: chai.15 at osu.edu (Lei Chai) Date: Sat Oct 18 17:19:51 2008 Subject: [mvapich-discuss] cpu placement with mpiexec? In-Reply-To: References: Message-ID: <48FA52FA.5020802@osu.edu> mpirun_rsh should do the work and you don't have to use rsh and the -rsh option. By default ssh is used and it should work well. Lei Joseph Hargitai wrote: >> The new mpirun_rsh framework takes into account the modern multi-core >> architectures (i.e., establishing connections first to a leader in a node >> and then to other cores in that node in a hierarchical manner). Thus, >> the >> start-up becomes faster and can scale to larger number of nodes/cores. >> The >> job termination/cleanup issue with mpirun_rsh you are mentioning has been >> solved with some of the recent patches. If you download and use mvapich >> 1.1RC1 or the trunk version of mvapich2 1.2 (not rc2), you will get these >> fixes. These fixes will be part of the formal mvapich 1.1 and mvapich2 >> 1.2 releases (which are coming soon). >> > > > will mpirun_rsh alone do the cleanup or you need rsh really to be working on the nodes, and pass the -rsh option? > > j > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > From joseph.hargitai at nyu.edu Mon Oct 20 13:41:10 2008 From: joseph.hargitai at nyu.edu (Joseph Hargitai) Date: Mon Oct 20 13:41:19 2008 Subject: [mvapich-discuss] Error: read_ib_one: mixed version executables (5 and 8), no hope. - In-Reply-To: <48FA52FA.5020802@osu.edu> References: <48FA52FA.5020802@osu.edu> Message-ID: getting this error with newly compiled (there were two posts about this early this year - but this is the new mvapich-1.1rc1 version. [jh2@compute-0-1 m-test]$ /opt/mpiexec/bin/mpiexec -v -comm ib -np 2 ./new3 mpiexec: resolve_exe: using absolute path "./new3". mpiexec: concurrent_init: old master died, reusing his fifo as master. mpiexec: process_start_event: evt 2 task 0 on compute-0-1.local. mpiexec: process_start_event: evt 3 task 1 on compute-0-1.local. mpiexec: Warning: read_ib_one: protocol version 8 not known, but might still work. mpiexec: read_ib_one: version 5 startup. mpiexec: service_ib_startup: rank 0 in, 1 + 0 left. mpiexec: Error: read_ib_one: mixed version executables (5 and 8), no hope. same works with: mpirun_rsh -np 12 -hostfile $PBS_NODEFILE VIADEV_CPU_MAPPING=0,1,2,5,6,7 ./new3 libs: [jh2@compute-0-1 m-test]$ ldd new3 libmpich.so.1.0 => /share/apps/nyu/mpi/intel/mvapich-1.1rc1/lib/shared/libmpich.so.1.0 (0x0000002a95557000) libibverbs.so.1 => /usr/lib64/libibverbs.so.1 (0x0000002a95769000) libibumad.so.1 => /usr/lib64/libibumad.so.1 (0x0000002a95874000) libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x0000003acf100000) librt.so.1 => /lib64/tls/librt.so.1 (0x0000003acf900000) libm.so.6 => /lib64/tls/libm.so.6 (0x0000003acef00000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003acf700000) libc.so.6 => /lib64/tls/libc.so.6 (0x0000003acea00000) libdl.so.2 => /lib64/libdl.so.2 (0x0000003aced00000) libimf.so => /usr/local/intel/cce/10.0.023/lib/libimf.so (0x0000002a9599e000) libsvml.so => /usr/local/intel/cce/10.0.023/lib/libsvml.so (0x0000002a95d00000) libintlc.so.5 => /usr/local/intel/cce/10.0.023/lib/libintlc.so.5 (0x0000002a95e81000) libibcommon.so.1 => /usr/lib64/libibcommon.so.1 (0x0000002a95fba000) /lib64/ld-linux-x86-64.so.2 (0x0000003ace600000) compiled as: /share/apps/nyu/mpi/intel/mvapich-1.1rc1/mpicc cpi.c -o new3 env: LD_LIBRARY_PATH=/usr/local/intel/cce/10.0.023/lib/:/usr/local/intel/cce/10.0.023/lib:/usr/local/intel/fce/10.0.023/lib::/usr/local/intel/mkl/9.1.023/lib/em64t:/usr/local/intel/mkl/9.1.023/include:/usr/local/intel/mkl/9.1.023/lib_ilp64:/usr/mpi/intel/openmpi-1.2.2-1/lib64:/usr/mpi/intel/openmpi-1.2.2-1/include:/share/apps/nyu/mpi/intel/mvapich-1.1rc1/lib:/share/apps/nyu/mpi/intel/mvapich-1.1rc1/lib/shared bin PATH=/usr/local/intel/cce/10.0.023/bin:/usr/kerberos/bin:/usr/java/jdk1.5.0_10/bin:/usr/lpp/mmfs/bin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/opt/ganglia/bin:/opt/ganglia/sbin:/usr/sbin:/opt/openmpi/bin/:/opt/maui/bin:/opt/torque/bin:/opt/torque/sbin:/opt/rocks/bin:/opt/rocks/sbin:/usr/local/tecplot/bin:/home/jh2/bin:/opt/torque/bin:/share/apps/nyu/mpi/intel/mvapich-1.1rc1/bin j From joseph.hargitai at nyu.edu Mon Oct 20 15:09:52 2008 From: joseph.hargitai at nyu.edu (Joseph Hargitai) Date: Mon Oct 20 15:10:02 2008 Subject: [mvapich-discuss] Error: read_ib_one: mixed version executables (5 and 8), no hope Message-ID: Hi all: this error message was posted a while ago but did not have a resolution. /share/apps/nyu/mpi/intel/mvapich-1.1rc1/bin/mpirun_rsh -np 12 -hostfile $PBS_NODEFILE ./new-f works same with mpiexec errs: /opt/mpiexec/bin/mpiexec -comm ib -np 12 ./new-f mpiexec: Warning: read_ib_one: protocol version 8 not known, but might still work. mpiexec: Error: read_ib_one: mixed version executables (5 and 8), no hope. libraries loaded as such: ldd new-f libmpich.so.1.0 => /share/apps/nyu/mpi/intel/mvapich-1.1rc1/lib/shared/libmpich.so.1.0 (0x0000002a95557000) libibverbs.so.1 => /usr/lib64/libibverbs.so.1 (0x0000002a95769000) libibumad.so.1 => /usr/lib64/libibumad.so.1 (0x0000002a95874000) libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x0000003acf100000) librt.so.1 => /lib64/tls/librt.so.1 (0x0000003acf900000) libm.so.6 => /lib64/tls/libm.so.6 (0x0000003acef00000) libc.so.6 => /lib64/tls/libc.so.6 (0x0000003acea00000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003acf700000) libdl.so.2 => /lib64/libdl.so.2 (0x0000003aced00000) libimf.so => /usr/local/intel/fce/10.0.023/lib/libimf.so (0x0000002a9599e000) libsvml.so => /usr/local/intel/fce/10.0.023/lib/libsvml.so (0x0000002a95d00000) libintlc.so.5 => /usr/local/intel/fce/10.0.023/lib/libintlc.so.5 (0x0000002a95e81000) libibcommon.so.1 => /usr/lib64/libibcommon.so.1 (0x0000002a95fba000) /lib64/ld-linux-x86-64.so.2 (0x0000003ace600000) -- mpiexec can run version 0.9x libraries mpirun_rsh 1.1rc1 version can run with either libraries loaded mpirun_rsh 0.9 version can run only with 0.9x version of libraries. --- here is how mpirun_rsh 1.1rc was compiled: [jh2@compute-0-1 m-test]$ /share/apps/nyu/mpi/intel/mvapich-1.1rc1/bin/mpichversion MPICH Version: 1.2.7 MPICH Release date: $Date: 2005/06/22 16:33:49$ MPICH Patches applied: none MPICH configure: --with-device=ch_gen2 --with-arch=LINUX -prefix=/share/apps/nyu/mpi/intel/mvapich-1.1rc1 --with-romio --enable-cxx --enable-sharedlib --enable-debug --enable-f77 --enable-f90 --without-mpe -lib=-L/usr/lib64 -Wl,-rpath=/usr/lib64 -libverbs -libumad -lpthread MPICH Device: ch_gen2 here is the config file: #!/bin/bash # Most variables here can be overridden by exporting them in the environment # before running this script. Default values have been provided if the # environment variable is not already set. source ./make.mvapich.def # The target architecture. If not exported outside of this script, # it will be found automatically or prompted for if necessary. # Supported: "_IA32_", "_IA64_", "_EM64T_", "_X86_64_" # if [ -z "$ARCH" ]; then arch fi # Mandatory variables. All are checked except CXX and F90. IBHOME=${IBHOME:-/usr/include/infiniband} IBHOME_LIB=${IBHOME_LIB:-/usr/lib64} PREFIX=${PREFIX:-/share/apps/nyu/mpi/intel/mvapich-1.1rc1} export CC=${CC:-/usr/local/intel/cce/10.0.023/bin/icc} export CXX=${CXX:-/usr/local/intel/cce/10.0.023/bin/icpc} export F77=${F77:-/usr/local/intel/fce/10.0.023/bin/ifort} export F90=${F90:-/usr/local/intel/fce/10.0.023/bin/ifort} if [ $ARCH = "SOLARIS" ]; then die_setup "MVAPICH GEN2 is not supported on Solaris." elif [ $ARCH = "MAC_OSX" ]; then die_setup "MVAPICH GEN2 is not supported on MacOS." fi # # Compiler specific flags. If you are using # ICC on IA64 platform, please set COMPILER_FLAG # to "icc" # COMPILER_FLAG=${COMPILER_FLAG:-} if [ "$COMPILER_FLAG" == "icc" ]; then COMPILER_FLAG="-D_ICC_" else COMPILER_FLAG="" fi # Check mandatory variable settings. if [ -z "$IBHOME" ] || [ -z "$PREFIX" ] || [ -z "$CC" ] || [ -z "$F77" ]; then die_setup "Please set mandatory variables in this script." elif [ ! -d $IBHOME ]; then die_setup "IBHOME directory $IBHOME does not exist." fi # Optional variables. # # Whether to enable ROMIO support. This is necessary if building the # F90 modules. if [ -n "$F90" ]; then ROMIO="--with-romio" else ROMIO=${ROMIO:---without-romio} fi # PTMALLOC support for MVAPICH2 memory hooks. Enabling this will allow # MVAPICH2 to release memory to the Operating System (when registration # cache is enabled). Enabled by default. Disable with "no". PTMALLOC=${PTMALLOC:-} if [ "$PTMALLOC" = "no" ]; then PTMALLOC="-DDISABLE_PTMALLOC" else PTMALLOC="" fi # Set this to override automatic optimization setting (-03). OPT_FLAG=${OPT_FLAG:--O3} if [ -n "$PROCESSOR" ]; then PROCESSOR=-D${PROCESSOR} else PROCESSOR= fi export LIBS=${LIBS:--L${IBHOME_LIB} -Wl,-rpath=${IBHOME_LIB} -libverbs -libumad -lpthread} export FFLAGS=${FFLAGS:--L${IBHOME_LIB}} export CFLAGS=${CFLAGS:--D${ARCH} ${PROCESSOR} ${PTMALLOC} -DEARLY_SEND_COMPLETI ON -DMEMORY_SCALE -DVIADEV_RPUT_SUPPORT -D_SMP_ -D_SMP_RNDV_ -DCH_GEN2 -D_GNU_S OURCE ${COMPILER_FLAG} -I${IBHOME}/include $OPT_FLAG} export MPIRUN_CFLAGS="${MPIRUN_CFLAGS} -DLD_LIBRARY_PATH_MPI=\\\"${PREFIX}/lib/s hared\\\" -DMPI_PREFIX=\\\"${PREFIX}/\\\" -DPARAM_GLOBAL=\\\"${PREFIX}/etc/mvapi ch.conf\\\"" # Prelogue make distclean &>/dev/null set -o pipefail # Configure MVAPICH echo "Configuring MVAPICH..." ./configure --with-device=ch_gen2 --with-arch=LINUX -prefix=${PREFIX} \ $ROMIO --enable-cxx --enable-sharedlib --enable-debug --enable-f77 --ena ble-f90 --without-mpe -lib="$LIBS" 2>&1 |tee config-mine.log ret=$? test $ret = 0 || die "configuration." # Build MVAPICH echo "Building MVAPICH..." make 2>&1 |tee make-mine.log ret=$? test $ret = 0 || die "building MVAPICH." # Install MVAPICH echo "MVAPICH installation..." rm -f install-mine.log make install 2>&1 |tee install-mine.log ret=$? test $ret = 0 || die "installing MVAPICH." --------------- mpiexec version and config - maybe a low version? /opt/mpiexec/bin/mpiexec -version Version 0.82, configure options: '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--target=x86_64-redhat-linux-gnu' '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--localstatedir=/var' '--sharedstatedir=/usr/com' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--prefix=/opt/mpiexec' '--with-pbs=/opt/torque' '--with-default-comm=mpich-p4' 'CFLAGS=-O2 -g -pipe -m64' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'target_alias=x86_64-redhat-linux-gnu' From perkinjo at cse.ohio-state.edu Mon Oct 20 15:11:16 2008 From: perkinjo at cse.ohio-state.edu (Jonathan Perkins) Date: Mon Oct 20 15:11:25 2008 Subject: [mvapich-discuss] Error: read_ib_one: mixed version executables (5 and 8), no hope. - In-Reply-To: References: <48FA52FA.5020802@osu.edu> Message-ID: <20081020191116.GC5537@cse.ohio-state.edu> On Mon, Oct 20, 2008 at 01:41:10PM -0400, Joseph Hargitai wrote: > > > getting this error with newly compiled (there were two posts about this early this year - but this is the new mvapich-1.1rc1 version. > > > [jh2@compute-0-1 m-test]$ /opt/mpiexec/bin/mpiexec -v -comm ib -np 2 ./new3 > mpiexec: resolve_exe: using absolute path "./new3". > mpiexec: concurrent_init: old master died, reusing his fifo as master. > mpiexec: process_start_event: evt 2 task 0 on compute-0-1.local. > mpiexec: process_start_event: evt 3 task 1 on compute-0-1.local. > mpiexec: Warning: read_ib_one: protocol version 8 not known, but might still work. > mpiexec: read_ib_one: version 5 startup. > mpiexec: service_ib_startup: rank 0 in, 1 + 0 left. > mpiexec: Error: read_ib_one: mixed version executables (5 and 8), no hope. > It looks like you're using mpiexec provided from OSC. You'll need to update this with 0.84 when it is released or use their svn trunk version until then. You can find their main webpage is located at http://www.osc.edu/~pw/mpiexec/index.php. You can use 'svn co http://svn.osc.edu/repos/mpiexec/trunk mpiexec' to check out their trunk. > > same works with: > > mpirun_rsh -np 12 -hostfile $PBS_NODEFILE VIADEV_CPU_MAPPING=0,1,2,5,6,7 ./new3 -- Jonathan Perkins http://www.cse.ohio-state.edu/~perkinjo From joseph.hargitai at nyu.edu Mon Oct 20 15:36:54 2008 From: joseph.hargitai at nyu.edu (Joseph Hargitai) Date: Mon Oct 20 15:37:05 2008 Subject: [mvapich-discuss] Error: read_ib_one: mixed version executables (5 and 8), no hope. - In-Reply-To: <20081020191116.GC5537@cse.ohio-state.edu> References: <48FA52FA.5020802@osu.edu> <20081020191116.GC5537@cse.ohio-state.edu> Message-ID: Can you run multiple versions of mpiexec? Just want to make sure there isn't an issue with hooking two versions into pbs - We would like to try the trunk version of the new mpiexec. Also, same issue exists with version of mpiexec regarding mvapich2x ? thanks, j ----- Original Message ----- From: Jonathan Perkins Date: Monday, October 20, 2008 3:11 pm Subject: Re: [mvapich-discuss] Error: read_ib_one: mixed version executables (5 and 8), no hope. - > > It looks like you're using mpiexec provided from OSC. You'll need to > update this with 0.84 when it is released or use their svn trunk > version until then. > > You can find their main webpage is located at > http://www.osc.edu/~pw/mpiexec/index.php. You can use 'svn co > http://svn.osc.edu/repos/mpiexec/trunk mpiexec' to check out their > trunk. From stevejones at stanford.edu Mon Oct 20 15:39:06 2008 From: stevejones at stanford.edu (Steve Jones) Date: Mon Oct 20 15:39:17 2008 Subject: [mvapich-discuss] Error: read_ib_one: mixed version executables (5 and 8), no hope In-Reply-To: References: Message-ID: <20081020123906.hujb95r8sowgc0c0@webmail.stanford.edu> Hi Joseph. I'm assuming this is your Rocks Cluster. The version of MPIEXEC (0.82<=) you're using doesn't support MVAPICH 1.1. I'm not sure of the release date for the newer version of MPIEXEC 0.84 from OSC, though the web site states "coming soon". I do know I've built MPIEXEC from their SVN repo and it's working with MVAPICH 1.0.1, so you might try using it. Here are the basics to check out the source and build on your cluster: If you don't have svn: # yum install subversion Once you have the above installed: # cd /share/apps # svn co http://svn.osc.edu/repos/mpiexec/trunk mpiexec # cd mpiexec # ./configure --prefix=/share/apps/mpiexec --with-default-comm=mpich-ib # make ; make install You can now run it using absolute path of /share/apps/mpiexec/bin/mpiexec or add it to your logon scripts for frontend and compute nodes. You can post to the rocks list for customizations on the cluster. Steve Quoting Joseph Hargitai : > Hi all: > > this error message was posted a while ago but did not have a resolution. > > > > /share/apps/nyu/mpi/intel/mvapich-1.1rc1/bin/mpirun_rsh -np 12 > -hostfile $PBS_NODEFILE ./new-f > > works > > same with mpiexec errs: > > /opt/mpiexec/bin/mpiexec -comm ib -np 12 ./new-f > > mpiexec: Warning: read_ib_one: protocol version 8 not known, but > might still work. > mpiexec: Error: read_ib_one: mixed version executables (5 and 8), no hope. > > > libraries loaded as such: > > ldd new-f > libmpich.so.1.0 => > /share/apps/nyu/mpi/intel/mvapich-1.1rc1/lib/shared/libmpich.so.1.0 > (0x0000002a95557000) > libibverbs.so.1 => /usr/lib64/libibverbs.so.1 (0x0000002a95769000) > libibumad.so.1 => /usr/lib64/libibumad.so.1 (0x0000002a95874000) > libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x0000003acf100000) > librt.so.1 => /lib64/tls/librt.so.1 (0x0000003acf900000) > libm.so.6 => /lib64/tls/libm.so.6 (0x0000003acef00000) > libc.so.6 => /lib64/tls/libc.so.6 (0x0000003acea00000) > libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003acf700000) > libdl.so.2 => /lib64/libdl.so.2 (0x0000003aced00000) > libimf.so => /usr/local/intel/fce/10.0.023/lib/libimf.so > (0x0000002a9599e000) > libsvml.so => /usr/local/intel/fce/10.0.023/lib/libsvml.so > (0x0000002a95d00000) > libintlc.so.5 => > /usr/local/intel/fce/10.0.023/lib/libintlc.so.5 (0x0000002a95e81000) > libibcommon.so.1 => /usr/lib64/libibcommon.so.1 (0x0000002a95fba000) > /lib64/ld-linux-x86-64.so.2 (0x0000003ace600000) > > > -- > mpiexec can run version 0.9x libraries > mpirun_rsh 1.1rc1 version can run with either libraries loaded > mpirun_rsh 0.9 version can run only with 0.9x version of libraries. > > > --- > > here is how mpirun_rsh 1.1rc was compiled: > > [jh2@compute-0-1 m-test]$ > /share/apps/nyu/mpi/intel/mvapich-1.1rc1/bin/mpichversion > MPICH Version: 1.2.7 > MPICH Release date: $Date: 2005/06/22 16:33:49$ > MPICH Patches applied: none > MPICH configure: --with-device=ch_gen2 --with-arch=LINUX > -prefix=/share/apps/nyu/mpi/intel/mvapich-1.1rc1 --with-romio > --enable-cxx --enable-sharedlib --enable-debug --enable-f77 > --enable-f90 --without-mpe -lib=-L/usr/lib64 -Wl,-rpath=/usr/lib64 > -libverbs -libumad -lpthread > MPICH Device: ch_gen2 > > here is the config file: > > #!/bin/bash > > # Most variables here can be overridden by exporting them in the environment > # before running this script. Default values have been provided if the > # environment variable is not already set. > > source ./make.mvapich.def > > # The target architecture. If not exported outside of this script, > # it will be found automatically or prompted for if necessary. > # Supported: "_IA32_", "_IA64_", "_EM64T_", "_X86_64_" > # > if [ -z "$ARCH" ]; then > arch > fi > > # Mandatory variables. All are checked except CXX and F90. > IBHOME=${IBHOME:-/usr/include/infiniband} > IBHOME_LIB=${IBHOME_LIB:-/usr/lib64} > PREFIX=${PREFIX:-/share/apps/nyu/mpi/intel/mvapich-1.1rc1} > export CC=${CC:-/usr/local/intel/cce/10.0.023/bin/icc} > export CXX=${CXX:-/usr/local/intel/cce/10.0.023/bin/icpc} > export F77=${F77:-/usr/local/intel/fce/10.0.023/bin/ifort} > export F90=${F90:-/usr/local/intel/fce/10.0.023/bin/ifort} > > if [ $ARCH = "SOLARIS" ]; then > die_setup "MVAPICH GEN2 is not supported on Solaris." > elif [ $ARCH = "MAC_OSX" ]; then > die_setup "MVAPICH GEN2 is not supported on MacOS." > fi > > # > # Compiler specific flags. If you are using > # ICC on IA64 platform, please set COMPILER_FLAG > # to "icc" > # > > COMPILER_FLAG=${COMPILER_FLAG:-} > > if [ "$COMPILER_FLAG" == "icc" ]; then > COMPILER_FLAG="-D_ICC_" > else > COMPILER_FLAG="" > fi > > # Check mandatory variable settings. > if [ -z "$IBHOME" ] || [ -z "$PREFIX" ] || [ -z "$CC" ] || [ -z > "$F77" ]; then > die_setup "Please set mandatory variables in this script." > elif [ ! -d $IBHOME ]; then > die_setup "IBHOME directory $IBHOME does not exist." > fi > > # Optional variables. > # > > # Whether to enable ROMIO support. This is necessary if building the > # F90 modules. > if [ -n "$F90" ]; then > ROMIO="--with-romio" > else > ROMIO=${ROMIO:---without-romio} > fi > > # PTMALLOC support for MVAPICH2 memory hooks. Enabling this will allow > # MVAPICH2 to release memory to the Operating System (when registration > # cache is enabled). Enabled by default. Disable with "no". > PTMALLOC=${PTMALLOC:-} > > if [ "$PTMALLOC" = "no" ]; then > PTMALLOC="-DDISABLE_PTMALLOC" > else > PTMALLOC="" > fi > > # Set this to override automatic optimization setting (-03). > OPT_FLAG=${OPT_FLAG:--O3} > > if [ -n "$PROCESSOR" ]; then > PROCESSOR=-D${PROCESSOR} > else > PROCESSOR= > fi > > export LIBS=${LIBS:--L${IBHOME_LIB} -Wl,-rpath=${IBHOME_LIB} > -libverbs -libumad > -lpthread} > export FFLAGS=${FFLAGS:--L${IBHOME_LIB}} > export CFLAGS=${CFLAGS:--D${ARCH} ${PROCESSOR} ${PTMALLOC} > -DEARLY_SEND_COMPLETI > ON -DMEMORY_SCALE -DVIADEV_RPUT_SUPPORT -D_SMP_ -D_SMP_RNDV_ > -DCH_GEN2 -D_GNU_S > OURCE ${COMPILER_FLAG} -I${IBHOME}/include $OPT_FLAG} > > export MPIRUN_CFLAGS="${MPIRUN_CFLAGS} > -DLD_LIBRARY_PATH_MPI=\\\"${PREFIX}/lib/s > hared\\\" -DMPI_PREFIX=\\\"${PREFIX}/\\\" > -DPARAM_GLOBAL=\\\"${PREFIX}/etc/mvapi > ch.conf\\\"" > > # Prelogue > make distclean &>/dev/null > set -o pipefail > > # Configure MVAPICH > > echo "Configuring MVAPICH..." > > ./configure --with-device=ch_gen2 --with-arch=LINUX -prefix=${PREFIX} \ > $ROMIO --enable-cxx --enable-sharedlib --enable-debug > --enable-f77 --ena > ble-f90 --without-mpe -lib="$LIBS" 2>&1 |tee config-mine.log > ret=$? > test $ret = 0 || die "configuration." > > # Build MVAPICH > echo "Building MVAPICH..." > make 2>&1 |tee make-mine.log > ret=$? > test $ret = 0 || die "building MVAPICH." > > # Install MVAPICH > echo "MVAPICH installation..." > rm -f install-mine.log > make install 2>&1 |tee install-mine.log > ret=$? > test $ret = 0 || die "installing MVAPICH." > > --------------- > mpiexec version and config - maybe a low version? > > /opt/mpiexec/bin/mpiexec -version > Version 0.82, configure options: '--build=x86_64-redhat-linux-gnu' > '--host=x86_64-redhat-linux-gnu' '--target=x86_64-redhat-linux-gnu' > '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' > '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' > '--datadir=/usr/share' '--includedir=/usr/include' > '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' > '--localstatedir=/var' '--sharedstatedir=/usr/com' > '--mandir=/usr/share/man' '--infodir=/usr/share/info' > '--prefix=/opt/mpiexec' '--with-pbs=/opt/torque' > '--with-default-comm=mpich-p4' 'CFLAGS=-O2 -g -pipe -m64' > 'build_alias=x86_64-redhat-linux-gnu' > 'host_alias=x86_64-redhat-linux-gnu' > 'target_alias=x86_64-redhat-linux-gnu' > > > > > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > From joseph.hargitai at nyu.edu Mon Oct 20 15:52:54 2008 From: joseph.hargitai at nyu.edu (Joseph Hargitai) Date: Mon Oct 20 15:53:05 2008 Subject: [mvapich-discuss] Error: read_ib_one: mixed version executables (5 and 8), no hope In-Reply-To: <20081020123906.hujb95r8sowgc0c0@webmail.stanford.edu> References: <20081020123906.hujb95r8sowgc0c0@webmail.stanford.edu> Message-ID: Thanks, Steve: does it find pbs on its own? or need to pass flag for it? > Hi Joseph. > > I'm assuming this is your Rocks Cluster. The version of MPIEXEC > (0.82<=) you're using doesn't support MVAPICH 1.1. I'm not sure of the > > release date for the newer version of MPIEXEC 0.84 from OSC, though > the web site states "coming soon". I do know I've built MPIEXEC from > > their SVN repo and it's working with MVAPICH 1.0.1, so you might try > > using it. Here are the basics to check out the source and build on > your cluster: > > If you don't have svn: > > # yum install subversion > > Once you have the above installed: > > # cd /share/apps > # svn co http://svn.osc.edu/repos/mpiexec/trunk mpiexec > # cd mpiexec > # ./configure --prefix=/share/apps/mpiexec --with-default-comm=mpich-ib > # make ; make install > > You can now run it using absolute path of > /share/apps/mpiexec/bin/mpiexec or add it to your logon scripts for > frontend and compute nodes. You can post to the rocks list for > customizations on the cluster. > > Steve > > Quoting Joseph Hargitai : > > > Hi all: > > > > this error message was posted a while ago but did not have a resolution. > > > > > > > > /share/apps/nyu/mpi/intel/mvapich-1.1rc1/bin/mpirun_rsh -np 12 > > -hostfile $PBS_NODEFILE ./new-f > > > > works > > > > same with mpiexec errs: > > > > /opt/mpiexec/bin/mpiexec -comm ib -np 12 ./new-f > > > > mpiexec: Warning: read_ib_one: protocol version 8 not known, but > > might still work. > > mpiexec: Error: read_ib_one: mixed version executables (5 and 8), no > hope. > > > > > > libraries loaded as such: > > > > ldd new-f > > libmpich.so.1.0 => > > /share/apps/nyu/mpi/intel/mvapich-1.1rc1/lib/shared/libmpich.so.1.0 > > > (0x0000002a95557000) > > libibverbs.so.1 => /usr/lib64/libibverbs.so.1 (0x0000002a95769000) > > libibumad.so.1 => /usr/lib64/libibumad.so.1 (0x0000002a95874000) > > libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x0000003acf100000) > > librt.so.1 => /lib64/tls/librt.so.1 (0x0000003acf900000) > > libm.so.6 => /lib64/tls/libm.so.6 (0x0000003acef00000) > > libc.so.6 => /lib64/tls/libc.so.6 (0x0000003acea00000) > > libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003acf700000) > > libdl.so.2 => /lib64/libdl.so.2 (0x0000003aced00000) > > libimf.so => /usr/local/intel/fce/10.0.023/lib/libimf.so > > (0x0000002a9599e000) > > libsvml.so => /usr/local/intel/fce/10.0.023/lib/libsvml.so > > > (0x0000002a95d00000) > > libintlc.so.5 => > > /usr/local/intel/fce/10.0.023/lib/libintlc.so.5 (0x0000002a95e81000) > > libibcommon.so.1 => /usr/lib64/libibcommon.so.1 (0x0000002a95fba000) > > /lib64/ld-linux-x86-64.so.2 (0x0000003ace600000) > > > > > > -- > > mpiexec can run version 0.9x libraries > > mpirun_rsh 1.1rc1 version can run with either libraries loaded > > mpirun_rsh 0.9 version can run only with 0.9x version of libraries. > > > > > > --- > > > > here is how mpirun_rsh 1.1rc was compiled: > > > > [jh2@compute-0-1 m-test]$ > > /share/apps/nyu/mpi/intel/mvapich-1.1rc1/bin/mpichversion > > MPICH Version: 1.2.7 > > MPICH Release date: $Date: 2005/06/22 16:33:49$ > > MPICH Patches applied: none > > MPICH configure: --with-device=ch_gen2 --with-arch=LINUX > > -prefix=/share/apps/nyu/mpi/intel/mvapich-1.1rc1 --with-romio > > --enable-cxx --enable-sharedlib --enable-debug --enable-f77 > > --enable-f90 --without-mpe -lib=-L/usr/lib64 -Wl,-rpath=/usr/lib64 > > > -libverbs -libumad -lpthread > > MPICH Device: ch_gen2 > > > > here is the config file: > > > > #!/bin/bash > > > > # Most variables here can be overridden by exporting them in the environment > > # before running this script. Default values have been provided if > the > > # environment variable is not already set. > > > > source ./make.mvapich.def > > > > # The target architecture. If not exported outside of this script, > > # it will be found automatically or prompted for if necessary. > > # Supported: "_IA32_", "_IA64_", "_EM64T_", "_X86_64_" > > # > > if [ -z "$ARCH" ]; then > > arch > > fi > > > > # Mandatory variables. All are checked except CXX and F90. > > IBHOME=${IBHOME:-/usr/include/infiniband} > > IBHOME_LIB=${IBHOME_LIB:-/usr/lib64} > > PREFIX=${PREFIX:-/share/apps/nyu/mpi/intel/mvapich-1.1rc1} > > export CC=${CC:-/usr/local/intel/cce/10.0.023/bin/icc} > > export CXX=${CXX:-/usr/local/intel/cce/10.0.023/bin/icpc} > > export F77=${F77:-/usr/local/intel/fce/10.0.023/bin/ifort} > > export F90=${F90:-/usr/local/intel/fce/10.0.023/bin/ifort} > > > > if [ $ARCH = "SOLARIS" ]; then > > die_setup "MVAPICH GEN2 is not supported on Solaris." > > elif [ $ARCH = "MAC_OSX" ]; then > > die_setup "MVAPICH GEN2 is not supported on MacOS." > > fi > > > > # > > # Compiler specific flags. If you are using > > # ICC on IA64 platform, please set COMPILER_FLAG > > # to "icc" > > # > > > > COMPILER_FLAG=${COMPILER_FLAG:-} > > > > if [ "$COMPILER_FLAG" == "icc" ]; then > > COMPILER_FLAG="-D_ICC_" > > else > > COMPILER_FLAG="" > > fi > > > > # Check mandatory variable settings. > > if [ -z "$IBHOME" ] || [ -z "$PREFIX" ] || [ -z "$CC" ] || [ -z > > "$F77" ]; then > > die_setup "Please set mandatory variables in this script." > > elif [ ! -d $IBHOME ]; then > > die_setup "IBHOME directory $IBHOME does not exist." > > fi > > > > # Optional variables. > > # > > > > # Whether to enable ROMIO support. This is necessary if building the > > # F90 modules. > > if [ -n "$F90" ]; then > > ROMIO="--with-romio" > > else > > ROMIO=${ROMIO:---without-romio} > > fi > > > > # PTMALLOC support for MVAPICH2 memory hooks. Enabling this will allow > > # MVAPICH2 to release memory to the Operating System (when registration > > # cache is enabled). Enabled by default. Disable with "no". > > PTMALLOC=${PTMALLOC:-} > > > > if [ "$PTMALLOC" = "no" ]; then > > PTMALLOC="-DDISABLE_PTMALLOC" > > else > > PTMALLOC="" > > fi > > > > # Set this to override automatic optimization setting (-03). > > OPT_FLAG=${OPT_FLAG:--O3} > > > > if [ -n "$PROCESSOR" ]; then > > PROCESSOR=-D${PROCESSOR} > > else > > PROCESSOR= > > fi > > > > export LIBS=${LIBS:--L${IBHOME_LIB} -Wl,-rpath=${IBHOME_LIB} > > -libverbs -libumad > > -lpthread} > > export FFLAGS=${FFLAGS:--L${IBHOME_LIB}} > > export CFLAGS=${CFLAGS:--D${ARCH} ${PROCESSOR} ${PTMALLOC} > > -DEARLY_SEND_COMPLETI > > ON -DMEMORY_SCALE -DVIADEV_RPUT_SUPPORT -D_SMP_ -D_SMP_RNDV_ > > -DCH_GEN2 -D_GNU_S > > OURCE ${COMPILER_FLAG} -I${IBHOME}/include $OPT_FLAG} > > > > export MPIRUN_CFLAGS="${MPIRUN_CFLAGS} > > -DLD_LIBRARY_PATH_MPI=\\\"${PREFIX}/lib/s > > hared\\\" -DMPI_PREFIX=\\\"${PREFIX}/\\\" > > -DPARAM_GLOBAL=\\\"${PREFIX}/etc/mvapi > > ch.conf\\\"" > > > > # Prelogue > > make distclean &>/dev/null > > set -o pipefail > > > > # Configure MVAPICH > > > > echo "Configuring MVAPICH..." > > > > ./configure --with-device=ch_gen2 --with-arch=LINUX > -prefix=${PREFIX} \ > > $ROMIO --enable-cxx --enable-sharedlib --enable-debug > > --enable-f77 --ena > > ble-f90 --without-mpe -lib="$LIBS" 2>&1 |tee config-mine.log > > ret=$? > > test $ret = 0 || die "configuration." > > > > # Build MVAPICH > > echo "Building MVAPICH..." > > make 2>&1 |tee make-mine.log > > ret=$? > > test $ret = 0 || die "building MVAPICH." > > > > # Install MVAPICH > > echo "MVAPICH installation..." > > rm -f install-mine.log > > make install 2>&1 |tee install-mine.log > > ret=$? > > test $ret = 0 || die "installing MVAPICH." > > > > --------------- > > mpiexec version and config - maybe a low version? > > > > /opt/mpiexec/bin/mpiexec -version > > Version 0.82, configure options: '--build=x86_64-redhat-linux-gnu' > > > '--host=x86_64-redhat-linux-gnu' '--target=x86_64-redhat-linux-gnu' > > > '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' > > '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' > > '--datadir=/usr/share' '--includedir=/usr/include' > > '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' > > '--localstatedir=/var' '--sharedstatedir=/usr/com' > > '--mandir=/usr/share/man' '--infodir=/usr/share/info' > > '--prefix=/opt/mpiexec' '--with-pbs=/opt/torque' > > '--with-default-comm=mpich-p4' 'CFLAGS=-O2 -g -pipe -m64' > > 'build_alias=x86_64-redhat-linux-gnu' > > 'host_alias=x86_64-redhat-linux-gnu' > > 'target_alias=x86_64-redhat-linux-gnu' > > > > > > > > > > _______________________________________________ > > mvapich-discuss mailing list > > mvapich-discuss@cse.ohio-state.edu > > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > > > > From joseph.hargitai at nyu.edu Mon Oct 20 16:44:16 2008 From: joseph.hargitai at nyu.edu (Joseph Hargitai) Date: Mon Oct 20 16:44:26 2008 Subject: [mvapich-discuss] Error: read_ib_one: mixed version executables (5 and 8), no hope In-Reply-To: <20081020123906.hujb95r8sowgc0c0@webmail.stanford.edu> References: <20081020123906.hujb95r8sowgc0c0@webmail.stanford.edu> Message-ID: Okay - the trunk version of mpiexec 0.84 compiled per Steve's instructions does fix the problem. thanks, j ----- Original Message ----- From: Steve Jones Date: Monday, October 20, 2008 3:39 pm Subject: Re: [mvapich-discuss] Error: read_ib_one: mixed version executables (5 and 8), no hope > Hi Joseph. > > I'm assuming this is your Rocks Cluster. The version of MPIEXEC > (0.82<=) you're using doesn't support MVAPICH 1.1. I'm not sure of the > > release date for the newer version of MPIEXEC 0.84 from OSC, though > the web site states "coming soon". I do know I've built MPIEXEC from > > their SVN repo and it's working with MVAPICH 1.0.1, so you might try > > using it. Here are the basics to check out the source and build on > your cluster: > > If you don't have svn: > > # yum install subversion > > Once you have the above installed: > > # cd /share/apps > # svn co http://svn.osc.edu/repos/mpiexec/trunk mpiexec > # cd mpiexec > # ./configure --prefix=/share/apps/mpiexec --with-default-comm=mpich-ib > # make ; make install > > You can now run it using absolute path of > /share/apps/mpiexec/bin/mpiexec or add it to your logon scripts for > frontend and compute nodes. You can post to the rocks list for > customizations on the cluster. > > Steve > > Quoting Joseph Hargitai : > > > Hi all: > > > > this error message was posted a while ago but did not have a resolution. > > > > > > > > /share/apps/nyu/mpi/intel/mvapich-1.1rc1/bin/mpirun_rsh -np 12 > > -hostfile $PBS_NODEFILE ./new-f > > > > works > > > > same with mpiexec errs: > > > > /opt/mpiexec/bin/mpiexec -comm ib -np 12 ./new-f > > > > mpiexec: Warning: read_ib_one: protocol version 8 not known, but > > might still work. > > mpiexec: Error: read_ib_one: mixed version executables (5 and 8), no > hope. > > > > > > libraries loaded as such: > > > > ldd new-f > > libmpich.so.1.0 => > > /share/apps/nyu/mpi/intel/mvapich-1.1rc1/lib/shared/libmpich.so.1.0 > > > (0x0000002a95557000) > > libibverbs.so.1 => /usr/lib64/libibverbs.so.1 (0x0000002a95769000) > > libibumad.so.1 => /usr/lib64/libibumad.so.1 (0x0000002a95874000) > > libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x0000003acf100000) > > librt.so.1 => /lib64/tls/librt.so.1 (0x0000003acf900000) > > libm.so.6 => /lib64/tls/libm.so.6 (0x0000003acef00000) > > libc.so.6 => /lib64/tls/libc.so.6 (0x0000003acea00000) > > libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003acf700000) > > libdl.so.2 => /lib64/libdl.so.2 (0x0000003aced00000) > > libimf.so => /usr/local/intel/fce/10.0.023/lib/libimf.so > > (0x0000002a9599e000) > > libsvml.so => /usr/local/intel/fce/10.0.023/lib/libsvml.so > > > (0x0000002a95d00000) > > libintlc.so.5 => > > /usr/local/intel/fce/10.0.023/lib/libintlc.so.5 (0x0000002a95e81000) > > libibcommon.so.1 => /usr/lib64/libibcommon.so.1 (0x0000002a95fba000) > > /lib64/ld-linux-x86-64.so.2 (0x0000003ace600000) > > > > > > -- > > mpiexec can run version 0.9x libraries > > mpirun_rsh 1.1rc1 version can run with either libraries loaded > > mpirun_rsh 0.9 version can run only with 0.9x version of libraries. > > > > > > --- > > > > here is how mpirun_rsh 1.1rc was compiled: > > > > [jh2@compute-0-1 m-test]$ > > /share/apps/nyu/mpi/intel/mvapich-1.1rc1/bin/mpichversion > > MPICH Version: 1.2.7 > > MPICH Release date: $Date: 2005/06/22 16:33:49$ > > MPICH Patches applied: none > > MPICH configure: --with-device=ch_gen2 --with-arch=LINUX > > -prefix=/share/apps/nyu/mpi/intel/mvapich-1.1rc1 --with-romio > > --enable-cxx --enable-sharedlib --enable-debug --enable-f77 > > --enable-f90 --without-mpe -lib=-L/usr/lib64 -Wl,-rpath=/usr/lib64 > > > -libverbs -libumad -lpthread > > MPICH Device: ch_gen2 > > > > here is the config file: > > > > #!/bin/bash > > > > # Most variables here can be overridden by exporting them in the environment > > # before running this script. Default values have been provided if > the > > # environment variable is not already set. > > > > source ./make.mvapich.def > > > > # The target architecture. If not exported outside of this script, > > # it will be found automatically or prompted for if necessary. > > # Supported: "_IA32_", "_IA64_", "_EM64T_", "_X86_64_" > > # > > if [ -z "$ARCH" ]; then > > arch > > fi > > > > # Mandatory variables. All are checked except CXX and F90. > > IBHOME=${IBHOME:-/usr/include/infiniband} > > IBHOME_LIB=${IBHOME_LIB:-/usr/lib64} > > PREFIX=${PREFIX:-/share/apps/nyu/mpi/intel/mvapich-1.1rc1} > > export CC=${CC:-/usr/local/intel/cce/10.0.023/bin/icc} > > export CXX=${CXX:-/usr/local/intel/cce/10.0.023/bin/icpc} > > export F77=${F77:-/usr/local/intel/fce/10.0.023/bin/ifort} > > export F90=${F90:-/usr/local/intel/fce/10.0.023/bin/ifort} > > > > if [ $ARCH = "SOLARIS" ]; then > > die_setup "MVAPICH GEN2 is not supported on Solaris." > > elif [ $ARCH = "MAC_OSX" ]; then > > die_setup "MVAPICH GEN2 is not supported on MacOS." > > fi > > > > # > > # Compiler specific flags. If you are using > > # ICC on IA64 platform, please set COMPILER_FLAG > > # to "icc" > > # > > > > COMPILER_FLAG=${COMPILER_FLAG:-} > > > > if [ "$COMPILER_FLAG" == "icc" ]; then > > COMPILER_FLAG="-D_ICC_" > > else > > COMPILER_FLAG="" > > fi > > > > # Check mandatory variable settings. > > if [ -z "$IBHOME" ] || [ -z "$PREFIX" ] || [ -z "$CC" ] || [ -z > > "$F77" ]; then > > die_setup "Please set mandatory variables in this script." > > elif [ ! -d $IBHOME ]; then > > die_setup "IBHOME directory $IBHOME does not exist." > > fi > > > > # Optional variables. > > # > > > > # Whether to enable ROMIO support. This is necessary if building the > > # F90 modules. > > if [ -n "$F90" ]; then > > ROMIO="--with-romio" > > else > > ROMIO=${ROMIO:---without-romio} > > fi > > > > # PTMALLOC support for MVAPICH2 memory hooks. Enabling this will allow > > # MVAPICH2 to release memory to the Operating System (when registration > > # cache is enabled). Enabled by default. Disable with "no". > > PTMALLOC=${PTMALLOC:-} > > > > if [ "$PTMALLOC" = "no" ]; then > > PTMALLOC="-DDISABLE_PTMALLOC" > > else > > PTMALLOC="" > > fi > > > > # Set this to override automatic optimization setting (-03). > > OPT_FLAG=${OPT_FLAG:--O3} > > > > if [ -n "$PROCESSOR" ]; then > > PROCESSOR=-D${PROCESSOR} > > else > > PROCESSOR= > > fi > > > > export LIBS=${LIBS:--L${IBHOME_LIB} -Wl,-rpath=${IBHOME_LIB} > > -libverbs -libumad > > -lpthread} > > export FFLAGS=${FFLAGS:--L${IBHOME_LIB}} > > export CFLAGS=${CFLAGS:--D${ARCH} ${PROCESSOR} ${PTMALLOC} > > -DEARLY_SEND_COMPLETI > > ON -DMEMORY_SCALE -DVIADEV_RPUT_SUPPORT -D_SMP_ -D_SMP_RNDV_ > > -DCH_GEN2 -D_GNU_S > > OURCE ${COMPILER_FLAG} -I${IBHOME}/include $OPT_FLAG} > > > > export MPIRUN_CFLAGS="${MPIRUN_CFLAGS} > > -DLD_LIBRARY_PATH_MPI=\\\"${PREFIX}/lib/s > > hared\\\" -DMPI_PREFIX=\\\"${PREFIX}/\\\" > > -DPARAM_GLOBAL=\\\"${PREFIX}/etc/mvapi > > ch.conf\\\"" > > > > # Prelogue > > make distclean &>/dev/null > > set -o pipefail > > > > # Configure MVAPICH > > > > echo "Configuring MVAPICH..." > > > > ./configure --with-device=ch_gen2 --with-arch=LINUX > -prefix=${PREFIX} \ > > $ROMIO --enable-cxx --enable-sharedlib --enable-debug > > --enable-f77 --ena > > ble-f90 --without-mpe -lib="$LIBS" 2>&1 |tee config-mine.log > > ret=$? > > test $ret = 0 || die "configuration." > > > > # Build MVAPICH > > echo "Building MVAPICH..." > > make 2>&1 |tee make-mine.log > > ret=$? > > test $ret = 0 || die "building MVAPICH." > > > > # Install MVAPICH > > echo "MVAPICH installation..." > > rm -f install-mine.log > > make install 2>&1 |tee install-mine.log > > ret=$? > > test $ret = 0 || die "installing MVAPICH." > > > > --------------- > > mpiexec version and config - maybe a low version? > > > > /opt/mpiexec/bin/mpiexec -version > > Version 0.82, configure options: '--build=x86_64-redhat-linux-gnu' > > > '--host=x86_64-redhat-linux-gnu' '--target=x86_64-redhat-linux-gnu' > > > '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' > > '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' > > '--datadir=/usr/share' '--includedir=/usr/include' > > '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' > > '--localstatedir=/var' '--sharedstatedir=/usr/com' > > '--mandir=/usr/share/man' '--infodir=/usr/share/info' > > '--prefix=/opt/mpiexec' '--with-pbs=/opt/torque' > > '--with-default-comm=mpich-p4' 'CFLAGS=-O2 -g -pipe -m64' > > 'build_alias=x86_64-redhat-linux-gnu' > > 'host_alias=x86_64-redhat-linux-gnu' > > 'target_alias=x86_64-redhat-linux-gnu' > > > > > > > > > > _______________________________________________ > > mvapich-discuss mailing list > > mvapich-discuss@cse.ohio-state.edu > > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > > > > From David_Kewley at Dell.com Mon Oct 20 17:37:40 2008 From: David_Kewley at Dell.com (David_Kewley@Dell.com) Date: Mon Oct 20 17:37:54 2008 Subject: [mvapich-discuss] Fwd: fortran system calls In-Reply-To: References: <1f31dac10809181815u70d75ab9l520d42c084057dc@mail.gmail.com> Message-ID: David, 2.6.9-67.0.7.ELsmp is a RHEL4 kernel that, unless I'm badly mistaken, does include a backport of whatever patches are required to support IBV_FORK_SAFE=1. Give it a try. David -----Original Message----- From: mvapich-discuss-bounces@cse.ohio-state.edu [mailto:mvapich-discuss-bounces@cse.ohio-state.edu] On Behalf Of Matthew Koop Sent: Thursday, September 18, 2008 6:36 PM To: David Stuebe Cc: mvapich-discuss@cse.ohio-state.edu Subject: Re: [mvapich-discuss] Fwd: fortran system calls Hi David, This is a known problem with OFED. Your kernel is too old to support system calls and OFED at the same time. To have fork() and system call support you need to have a 2.6.16 or later kernel with OFED 1.2+ and also export the IBV_FORK_SAFE=1 environment variable. This is why it isn't having any problems on a single node since shared memory (and not IB) is being used for communication. Matt On Thu, 18 Sep 2008, David Stuebe wrote: > Hello MVAPICH > > I am helping set up and new cluster and I have run into a problem > using mvapich to compile and run a Fortran90 code which uses system > calls. The program compiles, but will not run on more than one node, > even though only one processor makes the system call. Very strange! > > All is well when run on only one node of the cluster. > > Running: > mpvapich2 1.0.2 > Intel 10.1 compiler > OFED 1.2.5.3 > Linux X86_64 2.6.9-67.0.7.ELsmp > > Cluster built by aspen systems - dual processor Quad core hardware. > > Has anyone seen anything similar - I am not sure it is worth trying to > fix, but if by posting it I save someones else some time, I will feel > warm and fuzzy inside... > > !================================================== > program mpi_test > USE MPI > implicit none > > INTEGER:: MYID,NPROCS, IERR > > WRITE(6,*)"START TEST" > CALL MPI_INIT(IERR) > WRITE(6,*)"MPI_INIT: MPI_COMM_WORLD,IERR",MPI_COMM_WORLD,IERR > > CALL MPI_COMM_RANK(MPI_COMM_WORLD,MYID,IERR) > WRITE(6,*)"MPI_COMM_RANK: MYID,IERR",MYID,IERR > CALL MPI_COMM_SIZE(MPI_COMM_WORLD,NPROCS,IERR) > WRITE(6,*)"MPI_COMM_RANK: NPROCS,IERR",NPROCS,IERR > > CALL MPI_BARRIER(MPI_COMM_WORLD,IERR) > > WRITE(6,*) "CALLED BARRIER: myid",myid,IERR > > > IF(MYID==0) THEN > > CALL SYSTEM( "uptime > up_out" ) > WRITE(6,*) "CALLED SYSTEM: myid",myid > END IF > > CALL MPI_BARRIER(MPI_COMM_WORLD,IERR) > > WRITE(6,*) "CALLED BARRIER: myid",myid,IERR > > > > CALL MPI_FINALIZE(IERR) > > > end program mpi_test > !================================================== > > RESULT FROM RUN:mpiexec -n 2 ./mpit > > START TEST > START TEST > MPI_INIT: MPI_COMM_WORLD,IERR 1140850688 0 > MPI_COMM_RANK: MYID,IERR 0 0 > MPI_COMM_RANK: NPROCS,IERR 2 0 > MPI_INIT: MPI_COMM_WORLD,IERR 1140850688 0 > MPI_COMM_RANK: MYID,IERR 1 0 > MPI_COMM_RANK: NPROCS,IERR 2 0 > CALLED BARRIER: myid 1 0 > CALLED BARRIER: myid 0 0 > CALLED SYSTEM: myid 0 > CALLED BARRIER: myid 0 0 > send desc error > [0] Abort: [] Got completion with error 4, vendor code=52, dest rank=1 > at line 513 in file ibv_channel_manager.c > rank 0 in job 50 cpr_52824 caused collective abort of all ranks > exit status of rank 0: killed by signal 9 > > > Thanks so much > > David > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > _______________________________________________ mvapich-discuss mailing list mvapich-discuss@cse.ohio-state.edu http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss From floydfan at innogrid.com Wed Oct 22 00:54:05 2008 From: floydfan at innogrid.com (Soon-Heum Ko) Date: Wed Oct 22 08:11:58 2008 Subject: [mvapich-discuss] Running code hanged using MVAPICH Message-ID: <18E5E5EECA3A4F11A18FA449F774A02A@MyLaptop> Hi, I'm working at KISTI Supercomputing Center, Korea, as a member of SUN support team. I have some trouble using MVAPICH. So, I'd like to ask you the option for installing MVAPICH. Recently, we made a supercomputer which consists of Sun blade 6048 nodes (4 quadcore Barcelona CPUs at each node) and Voltaire ISR 2012 switch for infiniband network. Admins installed MVAPICH in this system and I tested the operation of MVAPICH library. While running various codes, I found that some codes don't work with MVAPICH though they don't make any trouble with other MPI libraries. Particularly, these codes work well with MVAPICH when I use several processors in the same node; but they don't work if more than 2 nodes cooperate. In detail, code is hung at the first MPI communication command. (i.e., when I use 16 processors in the same node, it works well... If I use 16 processors in 2 nodes, it pends in the MPI communication routine...) Mysterious thing is that, in some simple codes, MPI communication routines make no trouble in inter-node communication. On the other hand, some complex codes that spend vast size memory show communication tourble even when they transfer only 4 bytes data (one integer). Just in my imagination, this happened by one of these reasons: - MVAPICH trouble : which is fixed in latest version. (Note that we currently use MVAPICH Version 1.0.1) - MVAPICH trouble : which has not been reported or not been fixed. - Installing option : our installation option is as follows. ./configure --with-device=ch_gen2 --with-arch=LINUX -prefix=/usr/local/mvapich \ --enable-shared --enable-static --enable-debug --enable-sharedlib \ --enable-cxx --enable-f77 --enable-f90 --enable-f90modules \ --with-romio --without-mpe - Bugs on users' codes : which cannot happen. Do you have any idea or comments? If you know the reason, please let me know. Thank you in advance. Best regards, Jeff Soon-Heum Ko, Ph.D, Computational Fluid Dynamics Parallel Optimization Analyst, SUN Support Team (InnoGrid) at KISTI -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20081022/d39affca/attachment.html From panda at cse.ohio-state.edu Wed Oct 22 11:20:41 2008 From: panda at cse.ohio-state.edu (Dhabaleswar Panda) Date: Wed Oct 22 11:20:54 2008 Subject: [mvapich-discuss] Running code hanged using MVAPICH In-Reply-To: <18E5E5EECA3A4F11A18FA449F774A02A@MyLaptop> Message-ID: Soon-Heum, Thanks for your note. It is very strange that your MPI jobs are not able to communicate across two nodes. Most probably, it seems to be some installation/systems problem. Many organizations with large-scale IB clusters with Barcelona-based AMD nodes (like your) are running MVAPICH 1.0.1 successfully. Are you able to run the standard OSU benchmarks (latency, bandwidth, bi-directional bandwidth, broadcast, etc.) across two nodes. You should check these first before running your MPI applications. You can refer to MVAPICH 1.0.1 user guide available from the following URL to verify that your build and installation process is fine. http://mvapich.cse.ohio-state.edu/support/mvapich_user_guide.html#x1-100004.4.1 DK On Wed, 22 Oct 2008, Soon-Heum Ko wrote: > Hi, > > > I'm working at KISTI Supercomputing Center, Korea, as a member of SUN support team. > > I have some trouble using MVAPICH. So, I'd like to ask you the option for installing MVAPICH. > > Recently, we made a supercomputer which consists of Sun blade 6048 nodes (4 quadcore Barcelona CPUs at each node) and Voltaire ISR 2012 switch for infiniband network. Admins installed MVAPICH in this system and I tested the operation of MVAPICH library. > > While running various codes, I found that some codes don't work with MVAPICH though they don't make any trouble with other MPI libraries. Particularly, these codes work well with MVAPICH when I use several processors in the same node; but they don't work if more than 2 nodes cooperate. In detail, code is hung at the first MPI communication command. (i.e., when I use 16 processors in the same node, it works well... If I use 16 processors in 2 nodes, it pends in the MPI communication routine...) > > Mysterious thing is that, in some simple codes, MPI communication routines make no trouble in inter-node communication. On the other hand, some complex codes that spend vast size memory show communication tourble even when they transfer only 4 bytes data (one integer). > > Just in my imagination, this happened by one of these reasons: > - MVAPICH trouble : which is fixed in latest version. (Note that we currently use MVAPICH Version 1.0.1) > - MVAPICH trouble : which has not been reported or not been fixed. > - Installing option : our installation option is as follows. > ./configure --with-device=ch_gen2 --with-arch=LINUX -prefix=/usr/local/mvapich \ > --enable-shared --enable-static --enable-debug --enable-sharedlib \ > --enable-cxx --enable-f77 --enable-f90 --enable-f90modules \ > --with-romio --without-mpe > - Bugs on users' codes : which cannot happen. > > Do you have any idea or comments? If you know the reason, please let me know. > > Thank you in advance. > > > Best regards, > Jeff > > > > Soon-Heum Ko, > Ph.D, Computational Fluid Dynamics > Parallel Optimization Analyst, > SUN Support Team (InnoGrid) at KISTI From floydfan at innogrid.com Wed Oct 22 22:46:12 2008 From: floydfan at innogrid.com (Soon-Heum Ko) Date: Wed Oct 22 23:41:32 2008 Subject: [mvapich-discuss] Running code hanged using MVAPICH In-Reply-To: References: Message-ID: <7AAAEF9CC0D44CDBBE5BCDA8CA14FDFB@MyLaptop> Dear DK, Thank you for your comments. Let me first re-install MVAPICH and report the result again. Jeff (Soon-Heum) ----- Original Message ----- From: "Dhabaleswar Panda" To: "Soon-Heum Ko" Cc: ; "??????" ; "??????" ; "??????" Sent: Thursday, October 23, 2008 12:20 AM Subject: Re: [mvapich-discuss] Running code hanged using MVAPICH > Soon-Heum, > > Thanks for your note. It is very strange that your MPI jobs are not able > to communicate across two nodes. Most probably, it seems to be some > installation/systems problem. Many organizations with large-scale IB > clusters with Barcelona-based AMD nodes (like your) are running MVAPICH > 1.0.1 successfully. > > Are you able to run the standard OSU benchmarks (latency, bandwidth, > bi-directional bandwidth, broadcast, etc.) across two nodes. You should > check these first before running your MPI applications. > > You can refer to MVAPICH 1.0.1 user guide available from the following URL > to verify that your build and installation process is fine. > > http://mvapich.cse.ohio-state.edu/support/mvapich_user_guide.html#x1-100004.4.1 > > DK > > On Wed, 22 Oct 2008, Soon-Heum Ko wrote: > >> Hi, >> >> >> I'm working at KISTI Supercomputing Center, Korea, as a member of SUN >> support team. >> >> I have some trouble using MVAPICH. So, I'd like to ask you the option for >> installing MVAPICH. >> >> Recently, we made a supercomputer which consists of Sun blade 6048 nodes >> (4 quadcore Barcelona CPUs at each node) and Voltaire ISR 2012 switch for >> infiniband network. Admins installed MVAPICH in this system and I tested >> the operation of MVAPICH library. >> >> While running various codes, I found that some codes don't work with >> MVAPICH though they don't make any trouble with other MPI libraries. >> Particularly, these codes work well with MVAPICH when I use several >> processors in the same node; but they don't work if more than 2 nodes >> cooperate. In detail, code is hung at the first MPI communication >> command. (i.e., when I use 16 processors in the same node, it works >> well... If I use 16 processors in 2 nodes, it pends in the MPI >> communication routine...) >> >> Mysterious thing is that, in some simple codes, MPI communication >> routines make no trouble in inter-node communication. On the other hand, >> some complex codes that spend vast size memory show communication tourble >> even when they transfer only 4 bytes data (one integer). >> >> Just in my imagination, this happened by one of these reasons: >> - MVAPICH trouble : which is fixed in latest version. (Note that we >> currently use MVAPICH Version 1.0.1) >> - MVAPICH trouble : which has not been reported or not been fixed. >> - Installing option : our installation option is as follows. >> ./configure --with-device=ch_gen2 --with-arch=LINUX -prefix=/usr/local/mvapich >> \ >> --enable-shared --enable-static --enable-debug --enable-sharedlib \ >> --enable-cxx --enable-f77 --enable-f90 --enable-f90modules \ >> --with-romio --without-mpe >> - Bugs on users' codes : which cannot happen. >> >> Do you have any idea or comments? If you know the reason, please let me >> know. >> >> Thank you in advance. >> >> >> Best regards, >> Jeff >> >> >> >> Soon-Heum Ko, >> Ph.D, Computational Fluid Dynamics >> Parallel Optimization Analyst, >> SUN Support Team (InnoGrid) at KISTI > From pasha at dev.mellanox.co.il Thu Oct 30 07:32:23 2008 From: pasha at dev.mellanox.co.il (Pavel Shamis (Pasha)) Date: Thu Oct 30 07:32:43 2008 Subject: [mvapich-discuss] 32 bit mvapich shared libraries In-Reply-To: <54F700260E724490BC434E8F04858AF0@inspiron9100> References: <54F700260E724490BC434E8F04858AF0@inspiron9100> Message-ID: <49099B47.6080902@dev.mellanox.co.il> > However, there are no 32 bit shared libraries for mvapich-1.0.1 created, > even though the installation script finishes with "installation successful" > The flag says to OFED to build 32bit version (in additional to 64bit) for libibvebs and other IB libs. It does not effect on user level applications like MPI. -Pasha From Terrence.LIAO at total.com Thu Oct 30 13:22:38 2008 From: Terrence.LIAO at total.com (Terrence.LIAO@total.com) Date: Thu Oct 30 13:22:54 2008 Subject: [mvapich-discuss] Is negative host_id number from gethostbyname() a problem? Message-ID: Dear Mvapich, I am building mvapich-1.1rc1 for infinipath HTX card with CentOS5.2 and OFED1.4 using make.mavpich.gen2. The build is successfully, but the run fails, it never passes the MPI_Init(). It seems fall into an infinite loop. I check the gethostbyname() using this code struct hostent *he; he = gethostbyname (myname); int host_id = ((struct in_addr *) he->h_addr_list[0])->s_addr; printf("host_id: %d\n", host_id); and get host_id: -486364992 Will this the root cause, and any advice what I should do to fix this problem? Thank you very much. -- Terrence -------------------------------------------------------- Terrence Liao, Ph.D. Research Computer Scientist TOTAL E&P RESEARCH & TECHNOLOGY USA, LLC 1201 Louisiana, Suite 1800, Houston, TX 77002 Tel: 713.647.3498 Fax: 713.647.3638 Email: terrence.liao@total.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20081030/1c9ba231/attachment.html From panda at cse.ohio-state.edu Thu Oct 30 15:09:07 2008 From: panda at cse.ohio-state.edu (Dhabaleswar Panda) Date: Thu Oct 30 15:09:18 2008 Subject: [mvapich-discuss] Is negative host_id number from gethostbyname() a problem? In-Reply-To: Message-ID: Hi Terrence, Thanks for your report. We are taking a look at it. It looks like you are using make.mvapich.gen2 to run your applications on systems with infinipath HTX cards. Not sure whether you are aware of a different interface in MVAPICH 1.1 which is specifically designed for infinipath cards. This design is on top of the PSM layer provided by QLogic and delivers better performance. You need to use make.mvapich.psm script for this. More details are available in mvapich 1.1 user guide from the following URL: http://mvapich.cse.ohio-state.edu/support/mvapich_user_guide-1.1.html#x1-120004.4.3 Thanks, DK On Thu, 30 Oct 2008 Terrence.LIAO@total.com wrote: > Dear Mvapich, > > I am building mvapich-1.1rc1 for infinipath HTX card with CentOS5.2 and > OFED1.4 using make.mavpich.gen2. The build is successfully, but the run > fails, it never passes the MPI_Init(). It seems fall into an infinite > loop. I check the gethostbyname() using this code > struct hostent *he; > he = gethostbyname (myname); > int host_id = ((struct in_addr *) > he->h_addr_list[0])->s_addr; > printf("host_id: %d\n", host_id); > > and get > host_id: -486364992 > > Will this the root cause, and any advice what I should do to fix this > problem? > > > Thank you very much. > > -- Terrence > -------------------------------------------------------- > Terrence Liao, Ph.D. > Research Computer Scientist > TOTAL E&P RESEARCH & TECHNOLOGY USA, LLC > 1201 Louisiana, Suite 1800, Houston, TX 77002 > Tel: 713.647.3498 Fax: 713.647.3638 > Email: terrence.liao@total.com > > > From sridharj at cse.ohio-state.edu Thu Oct 30 15:47:43 2008 From: sridharj at cse.ohio-state.edu (Jaidev Sridhar) Date: Thu Oct 30 15:45:13 2008 Subject: [mvapich-discuss] Is negative host_id number from gethostbyname() a problem? In-Reply-To: References: Message-ID: <1225396063.25946.6.camel@t13.nowlab.cis.ohio-state.edu> Hi Terrence, To add to what Dr. Panda said, the host_id -486364992 is just a 32 bit representation of the IP address 192.168.2.227. It does not appear to be the problem here. -Jaidev On Thu, 2008-10-30 at 15:09 -0400, Dhabaleswar Panda wrote: > Hi Terrence, > > Thanks for your report. We are taking a look at it. It looks like you are > using make.mvapich.gen2 to run your applications on systems with > infinipath HTX cards. Not sure whether you are aware of a different > interface in MVAPICH 1.1 which is specifically designed for infinipath > cards. This design is on top of the PSM layer provided by QLogic and > delivers better performance. You need to use make.mvapich.psm script for > this. More details are available in mvapich 1.1 user guide from the > following URL: > > http://mvapich.cse.ohio-state.edu/support/mvapich_user_guide-1.1.html#x1-120004.4.3 > > Thanks, > > DK > > > On Thu, 30 Oct 2008 Terrence.LIAO@total.com wrote: > > > Dear Mvapich, > > > > I am building mvapich-1.1rc1 for infinipath HTX card with CentOS5.2 and > > OFED1.4 using make.mavpich.gen2. The build is successfully, but the run > > fails, it never passes the MPI_Init(). It seems fall into an infinite > > loop. I check the gethostbyname() using this code > > struct hostent *he; > > he = gethostbyname (myname); > > int host_id = ((struct in_addr *) > > he->h_addr_list[0])->s_addr; > > printf("host_id: %d\n", host_id); > > > > and get > > host_id: -486364992 > > > > Will this the root cause, and any advice what I should do to fix this > > problem? > > > > > > Thank you very much. > > > > -- Terrence > > -------------------------------------------------------- > > Terrence Liao, Ph.D. > > Research Computer Scientist > > TOTAL E&P RESEARCH & TECHNOLOGY USA, LLC > > 1201 Louisiana, Suite 1800, Houston, TX 77002 > > Tel: 713.647.3498 Fax: 713.647.3638 > > Email: terrence.liao@total.com > > > > > > > > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > From Terrence.LIAO at total.com Thu Oct 30 15:54:06 2008 From: Terrence.LIAO at total.com (Terrence.LIAO@total.com) Date: Thu Oct 30 15:54:24 2008 Subject: [mvapich-discuss] Is negative host_id number from gethostbyname() a problem? In-Reply-To: Message-ID: Professor DP, Yes, we are aware of the Qlogic driver on PSM and your make.mvapich.psm . We choose GEN2 for two reasons: 1. The OFED stack seems work fine with our luster system. We have a tough time to make our lustre to talk to cluster via IB, with this OFED GEN2, we can do it. 2. Our infinipath card is old (or ancient!) about 3 years old, before Qlogic bought infinipath. I am not sure PSM will work on it. Our application (seismic apps solving wave equations) is IO intensive, we believe we can scarify the MPI performance a little and potentially gain more from IO performance with lustre on IB. Of course, if PSM works on the lustre driver too, this will be a perfect solution. Thank you very much for your many help. -- Terrence Dhabaleswar Panda 10/30/2008 02:09 PM To Terrence.LIAO@total.com cc mvapich-discuss@cse.ohio-state.edu, Brian Stevens , Subject Re: [mvapich-discuss] Is negative host_id number from gethostbyname() a problem? Hi Terrence, Thanks for your report. We are taking a look at it. It looks like you are using make.mvapich.gen2 to run your applications on systems with infinipath HTX cards. Not sure whether you are aware of a different interface in MVAPICH 1.1 which is specifically designed for infinipath cards. This design is on top of the PSM layer provided by QLogic and delivers better performance. You need to use make.mvapich.psm script for this. More details are available in mvapich 1.1 user guide from the following URL: http://mvapich.cse.ohio-state.edu/support/mvapich_user_guide-1.1.html#x1-120004.4.3 Thanks, DK On Thu, 30 Oct 2008 Terrence.LIAO@total.com wrote: > Dear Mvapich, > > I am building mvapich-1.1rc1 for infinipath HTX card with CentOS5.2 and > OFED1.4 using make.mavpich.gen2. The build is successfully, but the run > fails, it never passes the MPI_Init(). It seems fall into an infinite > loop. I check the gethostbyname() using this code > struct hostent *he; > he = gethostbyname (myname); > int host_id = ((struct in_addr *) > he->h_addr_list[0])->s_addr; > printf("host_id: %d\n", host_id); > > and get > host_id: -486364992 > > Will this the root cause, and any advice what I should do to fix this > problem? > > > Thank you very much. > > -- Terrence > -------------------------------------------------------- > Terrence Liao, Ph.D. > Research Computer Scientist > TOTAL E&P RESEARCH & TECHNOLOGY USA, LLC > 1201 Louisiana, Suite 1800, Houston, TX 77002 > Tel: 713.647.3498 Fax: 713.647.3638 > Email: terrence.liao@total.com > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20081030/85381a32/attachment.html From Terrence.LIAO at total.com Thu Oct 30 15:56:43 2008 From: Terrence.LIAO at total.com (Terrence.LIAO@total.com) Date: Thu Oct 30 15:57:03 2008 Subject: [mvapich-discuss] Is negative host_id number from gethostbyname() a problem? In-Reply-To: <1225396063.25946.6.camel@t13.nowlab.cis.ohio-state.edu> Message-ID: Jaidev, It is great to know, this is not a problem. This IP is correct. Once I get the gdb installed, I will trace down where the code is locked, and get back to you. Thank you very much. -- Terrence Jaidev Sridhar 10/30/2008 02:47 PM To mvapich-discuss cc Terrence.LIAO@total.com, Brian Stevens , John.WANG@total.com, Dhabaleswar Panda Subject Re: [mvapich-discuss] Is negative host_id number from gethostbyname() a problem? Hi Terrence, To add to what Dr. Panda said, the host_id -486364992 is just a 32 bit representation of the IP address 192.168.2.227. It does not appear to be the problem here. -Jaidev On Thu, 2008-10-30 at 15:09 -0400, Dhabaleswar Panda wrote: > Hi Terrence, > > Thanks for your report. We are taking a look at it. It looks like you are > using make.mvapich.gen2 to run your applications on systems with > infinipath HTX cards. Not sure whether you are aware of a different > interface in MVAPICH 1.1 which is specifically designed for infinipath > cards. This design is on top of the PSM layer provided by QLogic and > delivers better performance. You need to use make.mvapich.psm script for > this. More details are available in mvapich 1.1 user guide from the > following URL: > > http://mvapich.cse.ohio-state.edu/support/mvapich_user_guide-1.1.html#x1-120004.4.3 > > Thanks, > > DK > > > On Thu, 30 Oct 2008 Terrence.LIAO@total.com wrote: > > > Dear Mvapich, > > > > I am building mvapich-1.1rc1 for infinipath HTX card with CentOS5.2 and > > OFED1.4 using make.mavpich.gen2. The build is successfully, but the run > > fails, it never passes the MPI_Init(). It seems fall into an infinite > > loop. I check the gethostbyname() using this code > > struct hostent *he; > > he = gethostbyname (myname); > > int host_id = ((struct in_addr *) > > he->h_addr_list[0])->s_addr; > > printf("host_id: %d\n", host_id); > > > > and get > > host_id: -486364992 > > > > Will this the root cause, and any advice what I should do to fix this > > problem? > > > > > > Thank you very much. > > > > -- Terrence > > -------------------------------------------------------- > > Terrence Liao, Ph.D. > > Research Computer Scientist > > TOTAL E&P RESEARCH & TECHNOLOGY USA, LLC > > 1201 Louisiana, Suite 1800, Houston, TX 77002 > > Tel: 713.647.3498 Fax: 713.647.3638 > > Email: terrence.liao@total.com > > > > > > > > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20081030/8356b1d4/attachment-0001.html From panda at cse.ohio-state.edu Thu Oct 30 21:45:31 2008 From: panda at cse.ohio-state.edu (Dhabaleswar Panda) Date: Thu Oct 30 21:45:40 2008 Subject: [mvapich-discuss] Is negative host_id number from gethostbyname() a problem? In-Reply-To: Message-ID: > Yes, we are aware of the Qlogic driver on PSM and your make.mvapich.psm . Thanks. > We choose GEN2 for two reasons: > 1. The OFED stack seems work fine with our luster system. We have a tough > time to make our lustre to talk to cluster via IB, with this OFED GEN2, we > can do it. > 2. Our infinipath card is old (or ancient!) about 3 years old, before > Qlogic bought infinipath. I am not sure PSM will work on it. Thanks for letting us know your objectives. We have a similar older setup here (in addition to the newer cards and setups). PSM and MVAPICH-PSM work on these cards. > Our application (seismic apps solving wave equations) is IO intensive, we > believe we can scarify the MPI performance a little and potentially gain > more from IO performance with lustre on IB. Of course, if PSM works on the > lustre driver too, this will be a perfect solution. I am not sure whether PSM works with Lustre or not. Might be somebody from QLogic can answer this. Thanks, DK > Thank you very much for your many help. > > -- Terrence > > > > > > > Dhabaleswar Panda > 10/30/2008 02:09 PM > > To > Terrence.LIAO@total.com > cc > mvapich-discuss@cse.ohio-state.edu, Brian Stevens , > > Subject > Re: [mvapich-discuss] Is negative host_id number from gethostbyname() a > problem? > > > > > > > Hi Terrence, > > Thanks for your report. We are taking a look at it. It looks like you are > using make.mvapich.gen2 to run your applications on systems with > infinipath HTX cards. Not sure whether you are aware of a different > interface in MVAPICH 1.1 which is specifically designed for infinipath > cards. This design is on top of the PSM layer provided by QLogic and > delivers better performance. You need to use make.mvapich.psm script for > this. More details are available in mvapich 1.1 user guide from the > following URL: > > http://mvapich.cse.ohio-state.edu/support/mvapich_user_guide-1.1.html#x1-120004.4.3 > > > Thanks, > > DK > > > On Thu, 30 Oct 2008 Terrence.LIAO@total.com wrote: > > > Dear Mvapich, > > > > I am building mvapich-1.1rc1 for infinipath HTX card with CentOS5.2 and > > OFED1.4 using make.mavpich.gen2. The build is successfully, but the run > > fails, it never passes the MPI_Init(). It seems fall into an infinite > > loop. I check the gethostbyname() using this code > > struct hostent *he; > > he = gethostbyname (myname); > > int host_id = ((struct in_addr *) > > he->h_addr_list[0])->s_addr; > > printf("host_id: %d\n", host_id); > > > > and get > > host_id: -486364992 > > > > Will this the root cause, and any advice what I should do to fix this > > problem? > > > > > > Thank you very much. > > > > -- Terrence > > -------------------------------------------------------- > > Terrence Liao, Ph.D. > > Research Computer Scientist > > TOTAL E&P RESEARCH & TECHNOLOGY USA, LLC > > 1201 Louisiana, Suite 1800, Houston, TX 77002 > > Tel: 713.647.3498 Fax: 713.647.3638 > > Email: terrence.liao@total.com > > > > > > > > From Terrence.LIAO at total.com Fri Oct 31 13:25:06 2008 From: Terrence.LIAO at total.com (Terrence.LIAO@total.com) Date: Fri Oct 31 13:25:26 2008 Subject: [mvapich-discuss] Is negative host_id number from gethostbyname() a problem? In-Reply-To: Message-ID: Professor DK, We have solved the problem by reinstalling the OS and OFED. mvapich1.0 Gen2 now can run on our system. However, we saw the one-way bandwidth is about 33% (600MBps on 4xSDR IB) less than the original pathscale MPI in this system which is using pathscale driver (900MBps) . I will try the PSM as you recommended. Thank you very much. -- Terrence Dhabaleswar Panda 10/30/2008 08:45 PM To Terrence.LIAO@total.com cc Brian Stevens , , Subject Re: [mvapich-discuss] Is negative host_id number from gethostbyname() a problem? > Yes, we are aware of the Qlogic driver on PSM and your make.mvapich.psm . Thanks. > We choose GEN2 for two reasons: > 1. The OFED stack seems work fine with our luster system. We have a tough > time to make our lustre to talk to cluster via IB, with this OFED GEN2, we > can do it. > 2. Our infinipath card is old (or ancient!) about 3 years old, before > Qlogic bought infinipath. I am not sure PSM will work on it. Thanks for letting us know your objectives. We have a similar older setup here (in addition to the newer cards and setups). PSM and MVAPICH-PSM work on these cards. > Our application (seismic apps solving wave equations) is IO intensive, we > believe we can scarify the MPI performance a little and potentially gain > more from IO performance with lustre on IB. Of course, if PSM works on the > lustre driver too, this will be a perfect solution. I am not sure whether PSM works with Lustre or not. Might be somebody from QLogic can answer this. Thanks, DK > Thank you very much for your many help. > > -- Terrence > > > > > > > Dhabaleswar Panda > 10/30/2008 02:09 PM > > To > Terrence.LIAO@total.com > cc > mvapich-discuss@cse.ohio-state.edu, Brian Stevens , > > Subject > Re: [mvapich-discuss] Is negative host_id number from gethostbyname() a > problem? > > > > > > > Hi Terrence, > > Thanks for your report. We are taking a look at it. It looks like you are > using make.mvapich.gen2 to run your applications on systems with > infinipath HTX cards. Not sure whether you are aware of a different > interface in MVAPICH 1.1 which is specifically designed for infinipath > cards. This design is on top of the PSM layer provided by QLogic and > delivers better performance. You need to use make.mvapich.psm script for > this. More details are available in mvapich 1.1 user guide from the > following URL: > > http://mvapich.cse.ohio-state.edu/support/mvapich_user_guide-1.1.html#x1-120004.4.3 > > > Thanks, > > DK > > > On Thu, 30 Oct 2008 Terrence.LIAO@total.com wrote: > > > Dear Mvapich, > > > > I am building mvapich-1.1rc1 for infinipath HTX card with CentOS5.2 and > > OFED1.4 using make.mavpich.gen2. The build is successfully, but the run > > fails, it never passes the MPI_Init(). It seems fall into an infinite > > loop. I check the gethostbyname() using this code > > struct hostent *he; > > he = gethostbyname (myname); > > int host_id = ((struct in_addr *) > > he->h_addr_list[0])->s_addr; > > printf("host_id: %d\n", host_id); > > > > and get > > host_id: -486364992 > > > > Will this the root cause, and any advice what I should do to fix this > > problem? > > > > > > Thank you very much. > > > > -- Terrence > > -------------------------------------------------------- > > Terrence Liao, Ph.D. > > Research Computer Scientist > > TOTAL E&P RESEARCH & TECHNOLOGY USA, LLC > > 1201 Louisiana, Suite 1800, Houston, TX 77002 > > Tel: 713.647.3498 Fax: 713.647.3638 > > Email: terrence.liao@total.com > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20081031/134a379e/attachment.html