From wgy at altair.com.cn Sat Sep 1 01:48:57 2007 From: wgy at altair.com.cn (wgy@altair.com.cn) Date: Sat Sep 1 01:49:32 2007 Subject: [mvapich-discuss] Verify the application is really running over IB In-Reply-To: References: <8554.203.193.122.66.1188404755.squirrel@mail.altair.com.cn> <9FEC4E50-1D18-4FCB-A0CA-A95921CE9B37@cisco.com> <9432.203.193.122.66.1188406465.squirrel@mail.altair.com.cn> Message-ID: <4832.218.242.127.98.1188625737.squirrel@mail.altair.com.cn> Hello, Jeff and Dr, Panda: I get back with the following resluts and further ques: 1)latency test reslut: the latency.c I downloaded from mvapich website can be complied in the cluster with errors: [radioss@hpc-node-01 job1]$ /usr/local/topspin/mpi/mpich/bin/mpicc latency.c -o lat latency.c:66: error: syntax error before "for" latency.c:68: error: `i' undeclared here (not in a function) latency.c:68: warning: data definition has no type or storage class latency.c:69: error: syntax error before '}' token latency.c:73: error: `skip_large' undeclared here (not in a function) latency.c:73: warning: data definition has no type or storage class latency.c:74: error: syntax error before '}' token latency.c:76: warning: parameter names (without types) in function declaration latency.c:76: warning: data definition has no type or storage class latency.c:78: error: syntax error before "if" latency.c:82: error: syntax error before numeric constant latency.c:82: warning: data definition has no type or storage class latency.c:83: error: syntax error before numeric constant latency.c:84: warning: data definition has no type or storage class latency.c:86: error: initializer element is not constant latency.c:86: warning: data definition has no type or storage class latency.c:88: error: syntax error before '}' token latency.c:92: error: syntax error before numeric constant latency.c:92: warning: data definition has no type or storage class latency.c:98: error: `t_start' undeclared here (not in a function) latency.c:98: error: `loop' undeclared here (not in a function) latency.c:98: warning: data definition has no type or storage class latency.c:99: error: syntax error before string constant latency.c:99: warning: conflicting types for built-in function 'fprintf' latency.c:99: warning: data definition has no type or storage class latency.c:104: warning: data definition has no type or storage class latency.c:105: error: syntax error before "return" latency.c:107:2: warning: no newline at end of file latency.c:68: error: storage size of `r_buf' isn't know I had to use mpi_latency.c shipped with mvapich in the cluster and got the following latency test results. [radioss@hpc-node-01 job1]$ /usr/local/topspin/mpi/mpich/bin/mpirun_rsh -np 2 -hostfile appfile ./lat 10000 1 1 6.288650 [radioss@hpc-node-01 job1]$ /usr/local/topspin/mpi/mpich/bin/mpirun_rsh -np 2 -hostfile appfile ./lat 10000 4 4 6.410350 while Topspin's Host-Side Drivers User Guide for Linux Release 3.1.0 gives the following latency test figure as an example: [root@qa-bc1-blade2 root]# /usr/local/topspin/mpi/mpich/bin/mpirun_ssh -np 2 qabc1- blade2 qa-bc1-blade3 /usr/local/topspin/mpi/mpich/bin/mpi_latency 10000 1 1 6.684000 2) Jeff Squyres once asked me: >> I have 4-cores nodes here.. >> I would expect to run it as: >> /usr/local/topspin/mpi/mpich/bin/mpirun_ssh -np 2 -hostfile hosts >^^ Is that the right path? Or is it "mvapich"? Regardless, I think wherever you find mpirun_ssh under /usr/local/topspin/mpi is probably the right one. the path is right, and I am pretty sure it is mavapich because: i)rpm -qf /usr/local/topspin/mpi/mpich/bin/mpirun_ssh gives: topspin-ib-mpi-rhel4-3.2.0-118 ii)[radioss@hpc-node-01 local]$ /usr/local/topspin/mpi/mpich/bin/mpirun_rsh -v OSU MVAPICH VERSION 0.9.5-SingleRail 3)when I try to use hp mpi 2.2.5 over the IB network I got the following: [radioss@hpc-node-01 job1]$ /opt/hpmpi/bin/mpirun -stdio=i0 -cpu_bind=cyclic -VAPI -f appfile < PFTANKD01 dlopen test for MPI_ICLIB_VAPI__VAPI_MAIN could not open libs in list libmtl_common.so libmpga.so libmosal.so libvapi.so: /usr/local/topspin/lib64/libmosal.so: undefined symbol: pthread_create dlopen test for MPI_ICLIB_VAPI__VAPI_CISCO could not open libs in list libpthread.so libmosal.so libvapi.so: /usr/lib64/libpthread.so: invalid ELF header mpid: MPI BUG: VAPI requested but not available what does it probably indicate? anything is wrong with the IB configuration? RPM packages installed there: [radioss@hpc-node-01 job1]$ rpm -qa|grep topspin topspin-ib-rhel4-3.2.0-118 topspin-ib-mpi-rhel4-3.2.0-118 topspin-ib-mod-rhel4-2.6.9-42.ELsmp-3.2.0-118 4)You guys suggested me to use HP MPI (no native mvapich) and OFED IB stack if possible. now I have some questiosn hope you can have a quick comment or refer me some website link so that I can read through: i)how to verify which IB stack is used here, OFED or Cisco/Topspin IB stack? what's the advantages of OFED IB stack over Cisco/Topspin IB stack? ii)what's the advatages of HP HPI over "native mvapich"? what means by "native mvapich"? the one shipped with Cisco/Topspin? is it enough to upgrade mvapich to the latest one which is availabel on mvapich website? Thanks a lot for all of you for your kindly help! Henry, Wu. | On Aug 29, 2007, at 12:54 PM, wgy@altair.com.cn wrote: | |> Yes, I think I used mavapich shipped with Topspin, but I am not sure |> unless I know how to verify it. | | If it's in the /usr/local/topspin directory, it's the Topspin (later | Cisco) MVAPICH. | |> about latency test, I downloaded |> https://mvapich.cse.ohio-state.edu/svn/mpi/mvapich/trunk/ |> osu_benchmarks/osu_latency.c |> and will compile it to run a benchmark. Can you please tell me how |> should |> I run it? how many nodes should be used and how many cpus should be |> involved? | | You typically run it with 2 MPI processes; one on each host. It | measures the MPI network latency between those two hosts. | |> I have 4-cores nodes here.. |> I would expect to run it as: |> /usr/local/topspin/mpi/mpich/bin/mpirun_ssh -np 2 -hostfile hosts | | ^^ Is that the right path? Or is it "mvapich"? Regardless, I think | wherever you find mpirun_ssh under /usr/local/topspin/mpi is probably | the right one. | |> osu_latency.o | | Is your executable really named osu_latency.o? That's uncommon. | Regardless, run the executable that you got when you compiled | osu_latency.c with mpicc. | |> and include the following in the hosts file |> hpc-node-01 |> hpc-node-02 | | Sounds right. I'm not an MVAPICH expert, though -- so I defer to the | maintainers here on this list for the finer details... | |> Is it right? |> Thanks a lot, I am really a newbie with Infiniband.... | | If this is your own system, I do want to stress that OFED is really | the way to go with HPC InfiniBand installations these days. The | MPI's that are included are much more recent, and all new development | work is happening in the OFED arena. | | I recommend that you upgrade if you can. | | |> Henry, Wu |> |> |> | On Aug 29, 2007, at 12:25 PM, wgy@altair.com.cn wrote: |> | |> |> Hello, Jeff: |> |> The mvapich version is OSU mvapich0.95. |> |> does it mean that it is Cisco IB stack and therefor the application |> |> I run |> |> with mvapich is really running over IB network? |> | |> | The version of MVAPICH, by itself, does not mean that it is or is |> not |> | running over IB. |> | |> | What *implies* that you are running over IB is: |> | |> | - You implied that you are using the MVAPICH shipped with the |> Topspin |> | IB stack (which is not OFED). Is that correct? |> | - I *believe* that the Topspin MVAPICH did not have TCP support |> | compiled into it (Topspin was before my time, but I am pretty sure |> | that the Cisco MVAPICH shipped with the Cisco IB stack does not) |> | |> | What would *prove* that you are using IB (vs. gige) is: |> | |> | - Run a simple latency test, as Dr. Panda suggested. Your latency |> | should be single-digit microseconds (exact numbers depend on your |> | hardware -- this might be all older stuff since you mentioned |> | "Topspin", not "Cisco"; Topspin was acquired by Cisco quite a while |> | ago...). If your latency is much higher than that (e.g., 50 us), |> | you're using gige. |> | |> | |> | |> |> Thanks. |> |> |> |> Henry, Wu. |> |> | In addition to what Dr. Panda said, Cisco recommends that all HPC |> |> | customers upgrade to the OFED IB driver stack if possible (some |> |> | customers cannot upgrade for various reasons). FWIW: all new |> HPC/ |> |> MPI |> |> | work is occurring in the OFED arena. |> |> | |> |> | I bring this up because you specifically mention Topspin |> Infiniband, |> |> | which I'm *assuming* is the Cisco IB stack (not the OFED IB |> stack), |> |> | and is therefore shipping with a somewhat older version of |> MVAPICH |> |> | that was derived from the OSU MVAPICH. The Cisco MVAPICH should |> |> only |> |> | be compiled with IB support enabled; a simple latency test should |> |> | prove that you're running over IB and not ethernet. |> |> | |> |> | Much more recent versions of MPI implementations are included |> with |> |> | the OFED stack (Cisco provides binary distributions of OFED on |> |> | www.cisco.com). |> |> | |> |> | |> |> | On Aug 29, 2007, at 11:44 AM, Dhabaleswar Panda wrote: |> |> | |> |> |> |> |> |> |> |> |> On Wed, 29 Aug 2007 wgy@altair.com.cn wrote: |> |> |> |> |> |>> Hello, list: |> |> |>> It might be a silly questions but I wonder how to verify run |> with |> |> |>> mvapich |> |> |>> (come with Topspin Infiniband) over Infiniband, NOT Gigabite |> |> network. |> |> |>> Is there an option to force mvapich to use IB network otherwise |> |> |>> just exits? |> |> |> |> |> |> MVAPICH has several underlying interfaces: Gen2, uDAPL, VAPI, |> TCP/ |> |> |> IP and |> |> |> shared memory. Please take a look at the user guide |> (available from |> |> |> mvapich project page) to see the differences and capabilities of |> |> these |> |> |> interfaces. Gen2 interface (corresponding to OFED) will give you |> |> |> the best performance and scalability. If you have OFED stack |> |> |> installed, |> |> |> you should be able to configure mvapich to run over Gen2 |> interface |> |> |> (as per the instructions indicated in the user guide). During |> OFED |> |> |> installation, you can also select mvapich from the package. |> |> |> |> |> |> On your existing installation, you can also run OSU benchmarks |> |> (such |> |> |> as OSU latency). If you get latency number in the range of 2~4 |> |> |> microsec |> |> |> for short messages (say 4 bytes), it is already running over the |> |> |> native |> |> |> IB. |> |> |> |> |> |> Hope this helps. |> |> |> |> |> |> DK |> |> |> |> |> |>> Thanks for your suggestion. |> |> |>> Rdgs. |> |> |>> Henry, Wu |> |> |>> |> |> |>> _______________________________________________ |> |> |>> mvapich-discuss mailing list |> |> |>> mvapich-discuss@cse.ohio-state.edu |> |> |>> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss |> |> |>> |> |> |> |> |> |> _______________________________________________ |> |> |> mvapich-discuss mailing list |> |> |> mvapich-discuss@cse.ohio-state.edu |> |> |> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss |> |> | |> |> | |> |> | -- |> |> | Jeff Squyres |> |> | Cisco Systems |> |> | |> |> | |> | |> | |> | -- |> | Jeff Squyres |> | Cisco Systems |> | |> | | | | -- | Jeff Squyres | Cisco Systems | | From jsquyres at cisco.com Sat Sep 1 08:45:28 2007 From: jsquyres at cisco.com (Jeff Squyres) Date: Sat Sep 1 08:46:24 2007 Subject: [mvapich-discuss] Verify the application is really running over IB In-Reply-To: <4832.218.242.127.98.1188625737.squirrel@mail.altair.com.cn> References: <8554.203.193.122.66.1188404755.squirrel@mail.altair.com.cn> <9FEC4E50-1D18-4FCB-A0CA-A95921CE9B37@cisco.com> <9432.203.193.122.66.1188406465.squirrel@mail.altair.com.cn> <4832.218.242.127.98.1188625737.squirrel@mail.altair.com.cn> Message-ID: <4E85CA66-A8CE-4864-9F1D-EFB96AF524FA@cisco.com> On Sep 1, 2007, at 1:48 AM, wgy@altair.com.cn wrote: > 3)when I try to use hp mpi 2.2.5 over the IB network I got the > following: > [radioss@hpc-node-01 job1]$ /opt/hpmpi/bin/mpirun -stdio=i0 > -cpu_bind=cyclic -VAPI -f appfile < PFTANKD01 > dlopen test for MPI_ICLIB_VAPI__VAPI_MAIN could not open libs in list > libmtl_common.so libmpga.so libmosal.so libvapi.so: > /usr/local/topspin/lib64/libmosal.so: undefined symbol: pthread_create > dlopen test for MPI_ICLIB_VAPI__VAPI_CISCO could not open libs in list > libpthread.so libmosal.so libvapi.so: /usr/lib64/ > libpthread.so: > invalid ELF header > mpid: MPI BUG: VAPI requested but not available > what does it probably indicate? anything is wrong with the IB > configuration? This is a question for HP MPI support. Please contact them with your questions about HP MPI. > 4)You guys suggested me to use HP MPI (no native mvapich) and OFED IB > stack if possible. HP MPI was not recommended on this list; you must be referring to some other conversation...? Cisco QA qualifies both HP MPI and MVAPICH (and other MPI's) on our hardware. > now I have some questiosn hope you can have a quick comment or > refer me > some website link so that I can read through: > i)how to verify which IB stack is used here, OFED or Cisco/Topspin IB > stack? Your RPMs are named "topspin"; hence you are using the Cisco/Topspin IB stack. > what's the advantages of OFED IB stack over Cisco/Topspin IB stack? There are several: - The OpenFabrics stack is developed by the community of all InfiniBand vendors to provide a single set of drivers and tools that are compatible across all IB hardware - The OpenFabrics stack is open source, the kernel parts of which are included in Linus' git tree (and therefore are in main-line Linux distros) - All new work is occurring (and has been occurring for 1-2 years) in the OpenFabrics stack for HPC customers - The OpenFabrics stack has several low-level features that the Topspin IB stack does not (features that MPI implementations can take advantage of) - MPI implementations are (and have been for quite a while) focusing on the OFED stack rather than the mVAPI-based stacks. For example: I have not worked on the Topspin IB stack in about 1.5 years. Binary distributions of OFED are available from Cisco under the somewhat-confusing moniker of "Server networking and virtualization software", here: http://www.cisco.com/kobayashi/sw-center/snv.shtml You need to have a [free] www.cisco.com username/password in order to reach the download page. (if the above link doesn't work for some reason, go to www.cisco.com - > support -> download software -> server networking and virtualization software -> linux host driver software packages) If you have any questions / problems with this procedure, please contact me off-list; we're straying a bit off-topic here... > ii)what's the advatages of HP HPI over "native mvapich"? Every MPI claims to have advantages over its competitors; you should probably ask your HP MPI contact for their marketing information. Some obvious differences: - MVAPICH is open source; HP MPI is closed source - MVAPICH is supported by the community; HP MPI has paid/contract support > what means by > "native mvapich"? the one shipped with Cisco/Topspin? is it enough to > upgrade mvapich to the latest one which is availabel on mvapich > website? Cisco certifies the versions of MVAPICH that are included with OFED distributions. The version of MVAPICH included in OFED v1.2 is *much* more recent compared to the version included in the Topspin IB stack. Cisco does not certify or support other versions of MVAPICH, but the community (e.g., this mailing list) provides good support. -- Jeff Squyres Cisco Systems From perkinjo at cse.ohio-state.edu Sat Sep 1 09:26:28 2007 From: perkinjo at cse.ohio-state.edu (Jonathan Perkins) Date: Sat Sep 1 09:27:00 2007 Subject: [mvapich-discuss] process limits In-Reply-To: <46D8D8E1.8090303@hpcapplications.com> References: <46D5AE8D.8050608@hpcapplications.com> <46D70E5C.9020006@cse.ohio-state.edu> <46D8D8E1.8090303@hpcapplications.com> Message-ID: <46D96884.40107@cse.ohio-state.edu> Mark Potts wrote: > Jonathan, That patch seems to solve the non-bash shell problems. I've > asked some others to try it for themselves, and if I learn anything > new, I'll let you know. But for now, it appears that problem can be > checked off. regards, That's good to hear. Below my quoted response you'll find the solution to your problem when trying to run many processes on one node. > > Jonathan L. Perkins wrote: >> Mark Potts wrote: >>> DK, Thanks for the quick reply. >>> >>> I'm maybe jumping ahead here, but assuming the "timeout" is the >>> one that I suggested from the 10 July patch (mentioned below) I >>> have a question related to that patch. Could you/your people >>> suggest why this patch seems to fail for users running tcsh (as >>> opposed to bash)? My initial testing, which used bash, of this >>> patch found that it worked quite nicely to clean up jobs when one >>> or more of the processes killed themselves. An mpirun_rsh >>> thread detected this condition and then killed the remaining >>> remote tasks. However, we now find that users employing tcsh >>> login shell are not so lucky. The detection part of the patch >>> works successfully and the remote sshd tasks are killed >>> successfully but the remote MPI processes continue to run. Quite >>> interesting if it weren't so bad. We can be left with large >>> numbers of CPU burning tasks that are difficult to find and kill. >>> >> >> Mark: Attached you find a patch that should solve the issue with >> remote processes not being killed when using tcsh. The problem >> arose from using the signal name with the kill command. Since in >> many cases the kill command is a shell built in, there can be some >> minor differences such as SIGKILL in bash and just KILL in tcsh. >> >> After applying this patch to the MVAPICH source you can simply >> re-install using the proper make script. Let me know if you have >> any further questions regarding this. >> >> As for your original question, it seems that the problem isn't the >> timeout while waiting for the other child process to end. More >> troublesome is that the child died in the first place. We have >> reproduced this issue and are working on finding the best solution >> to resolve it. We'll keep you posted. Can you try setting the following in /etc/ssh/sshd_config: MaxStartups 20 > From the sshd_config manual page: > > MaxStartups > Specifies the maximum number of concurrent unauthenticated con- > nections to the sshd daemon. Additional connections will be > dropped until authentication succeeds or the LoginGraceTime > expires for a connection. The default is 10. We were able to narrow the problem down to the area where we spawn ssh processes and it seemed to consistently have problems whenever this limit was reached. After changing this configuration and restarting sshd the problem went away. >>> Shell experts' ideas welcome. >>> >>> regards, >>> >>> Dhabaleswar Panda wrote: >>>> Hi Mark, >>>> >>>> Thanks for providing us the details. There appears to be some >>>> `time out' with the new mpirun_rsh. We are taking a look at it >>>> and will be able to send you some solution soon. >>>> >>>> Thanks, >>>> >>>> DK >>>> >>>> On Tue, 28 Aug 2007, Mark Potts wrote: >>>> >>>>> Hi, I tried VIADEV_USE_SHMEM_COLL=0 and separately tried >>>>> VIADEV_USE_BLOCKING=1, with no change in results. During >>>>> task startup I get either "Unable to find child nnnn!", >>>>> "Child died. Timeout while waiting", and/or simply "done." >>>>> >>>>> I tried repeatedly but was never able to consistently run >>>>> more than 10 ranks (-np 10) on a single node. I, of course, >>>>> am able to run many more ranks, when I spread the targets >>>>> across more nodes. >>>>> >>>>> My experiment is to start a very simple code with multiple >>>>> processes on a single node. Specific details of my setup on >>>>> two machines. The results were the same: >>>>> >>>>> Machine Cpus per Cores per Avail Cpu MVAPICH >>>>> MVAPICH Node Cpu Nodes Type version Device >>>>> A 1 2 3 X86-64 -0.9.9-1168 >>>>> ch_gen2 B 2 4 16 X86_64 -0.9.9-1326 >>>>> ch_gen2 >>>>> >>>>> The MVAPICH code, which was obtained from ofed 1.2 >>>>> installation, has two patches as follows: (1) for >>>>> mpirun_rsh.c from Sayatan Sur of 10 Jul for MVAPICH errant >>>>> process/job cleanup. (2) for comm_free.c from Amith Rajith >>>>> Mamidla of 11 Jul for MVAPICH segmentation fault during >>>>> MPI_Finalize() in large jobs. >>>>> >>>>> Is it possible that the mpirun_rsh.c patch is prematurely >>>>> killing tasks when it determines that the processes on the >>>>> oversubscribed node are not responding fast enough? Or is >>>>> there another clean explanation? As I understand the note >>>>> from DK this morning, oversubscription should work... >>>>> regards, >>>>> >>>>> amith rajith mamidala wrote: >>>>>> Hi Mark, >>>>>> >>>>>> Can you check if you get this error by setting the >>>>>> environment variable: VIADEV_USE_SHMEM_COLL to 0 e.g. >>>>>> mpirun_rsh -np N VIADEV_USE_SHMEM_COLL=0 ./a.out >>>>>> >>>>>> -thanks, Amith >>>>>> >>>>>> On Tue, 28 Aug 2007, Mark Potts wrote: >>>>>> >>>>>>> Hi, Is there an effective or hard limit on the number of >>>>>>> MVAPICH processes that can be run on a single node? >>>>>>> >>>>>>> Given N cpus, each having M cores, on a single node, I've >>>>>>> been told that one can not run more than N*M MVAPICH >>>>>>> processes on a single node. In fact, I observe that if I >>>>>>> try to even approach this number with "-np 16" (for a >>>>>>> node with N=8 and M=4), I observe a "unable to find child >>>>>>> nnnn!" or "Child died" message. Is this a configuration >>>>>>> problem with this system or somehow an expected >>>>>>> behavior? >>>>>>> >>>>>>> >>>>>>> More pointedly, should oversubscription of cores, np > >>>>>>> N*M, on a single node work in MVAPICH? How about in >>>>>>> MVAPICH2? >>>>>>> >>>>>>> regards, -- *********************************** >>>>>>>>> Mark J. Potts, PhD >>>>>>>>> >>>>>>>>> HPC Applications Inc. phone: 410-992-8360 Bus >>>>>>>>> 410-313-9318 Home 443-418-4375 Cell email: >>>>>>>>> potts@hpcapplications.com potts@excray.com >>>>>>> *********************************** >>>>>>> _______________________________________________ >>>>>>> mvapich-discuss mailing list >>>>>>> mvapich-discuss@cse.ohio-state.edu >>>>>>> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss >>>>>>> >>>>>>> >>>>>>> >>>>> -- *********************************** >>>>>>> Mark J. Potts, PhD >>>>>>> >>>>>>> HPC Applications Inc. phone: 410-992-8360 Bus >>>>>>> 410-313-9318 Home 443-418-4375 Cell email: >>>>>>> potts@hpcapplications.com potts@excray.com >>>>> *********************************** >>>>> _______________________________________________ >>>>> mvapich-discuss mailing list >>>>> mvapich-discuss@cse.ohio-state.edu >>>>> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss >>>>> >>>>> >>>>> >>> >> >> > -- Jonathan Perkins http://www.cse.ohio-state.edu/~perkinjo From nilesha at cdac.in Mon Sep 3 11:36:29 2007 From: nilesha at cdac.in (Nilesh Awate) Date: Mon Sep 3 12:01:39 2007 Subject: [mvapich-discuss] Printing DEBUG_PRINT mesges Message-ID: <46DC29FD.6030003@cdac.in> Hi All, I'm using mvapich2-1.0-beta with ofed1.2 udapl interface. I want to print DEBUG_PRINT masseges for that i've included DDEBUG flag in make.mvapich.udapl make file but while making its give error in that perticular DEBUG_PRINT statements "variables not defined" is there anything else setting of flags ... waiting for reply Nilesh, C-DAC, India From panda at cse.ohio-state.edu Tue Sep 4 02:10:13 2007 From: panda at cse.ohio-state.edu (Dhabaleswar Panda) Date: Tue Sep 4 02:10:42 2007 Subject: [mvapich-discuss] Verify the application is really running In-Reply-To: <4832.218.242.127.98.1188625737.squirrel@mail.altair.com.cn> from "wgy@altair.com.cn" at Sep 01, 2007 01:48:57 PM Message-ID: <200709040610.l846ADAg006816@xi.cse.ohio-state.edu> Are you sure you are using the osu_latency.c file from mvapich web site? Your e-mail indicates about using a `latency.c' file. FYI, the osu_latency.c benchmark (latest version v2.2) is available from the following URL: https://mvapich.cse.ohio-state.edu/svn/mpi/mvapich/trunk/osu_benchmarks/osu_latency.c DK > > Hello, Jeff and Dr, Panda: > I get back with the following resluts and further ques: > 1)latency test reslut: > the latency.c I downloaded from mvapich website can be complied in the > cluster with errors: > [radioss@hpc-node-01 job1]$ /usr/local/topspin/mpi/mpich/bin/mpicc > latency.c -o lat > latency.c:66: error: syntax error before "for" > latency.c:68: error: `i' undeclared here (not in a function) > latency.c:68: warning: data definition has no type or storage class > latency.c:69: error: syntax error before '}' token > latency.c:73: error: `skip_large' undeclared here (not in a function) > latency.c:73: warning: data definition has no type or storage class > latency.c:74: error: syntax error before '}' token > latency.c:76: warning: parameter names (without types) in function > declaration > latency.c:76: warning: data definition has no type or storage class > latency.c:78: error: syntax error before "if" > latency.c:82: error: syntax error before numeric constant > latency.c:82: warning: data definition has no type or storage class > latency.c:83: error: syntax error before numeric constant > latency.c:84: warning: data definition has no type or storage class > latency.c:86: error: initializer element is not constant > latency.c:86: warning: data definition has no type or storage class > latency.c:88: error: syntax error before '}' token > latency.c:92: error: syntax error before numeric constant > latency.c:92: warning: data definition has no type or storage class > latency.c:98: error: `t_start' undeclared here (not in a function) > latency.c:98: error: `loop' undeclared here (not in a function) > latency.c:98: warning: data definition has no type or storage class > latency.c:99: error: syntax error before string constant > latency.c:99: warning: conflicting types for built-in function 'fprintf' > latency.c:99: warning: data definition has no type or storage class > latency.c:104: warning: data definition has no type or storage class > latency.c:105: error: syntax error before "return" > latency.c:107:2: warning: no newline at end of file > latency.c:68: error: storage size of `r_buf' isn't know > > I had to use mpi_latency.c shipped with mvapich in the cluster and got the > following latency test results. > > [radioss@hpc-node-01 job1]$ /usr/local/topspin/mpi/mpich/bin/mpirun_rsh > -np 2 -hostfile appfile ./lat 10000 1 > 1 6.288650 > [radioss@hpc-node-01 job1]$ /usr/local/topspin/mpi/mpich/bin/mpirun_rsh > -np 2 -hostfile appfile ./lat 10000 4 > 4 6.410350 > while Topspin's Host-Side Drivers User Guide for Linux Release 3.1.0 gives > the following latency test figure as an example: > [root@qa-bc1-blade2 root]# /usr/local/topspin/mpi/mpich/bin/mpirun_ssh -np > 2 qabc1- > blade2 qa-bc1-blade3 /usr/local/topspin/mpi/mpich/bin/mpi_latency 10000 1 > 1 6.684000 > 2) Jeff Squyres once asked me: > >> I have 4-cores nodes here.. > >> I would expect to run it as: > >> /usr/local/topspin/mpi/mpich/bin/mpirun_ssh -np 2 -hostfile hosts > > >^^ Is that the right path? Or is it "mvapich"? Regardless, I think > wherever you find mpirun_ssh under /usr/local/topspin/mpi is probably > the right one. > the path is right, and I am pretty sure it is mavapich because: > i)rpm -qf /usr/local/topspin/mpi/mpich/bin/mpirun_ssh gives: > topspin-ib-mpi-rhel4-3.2.0-118 > ii)[radioss@hpc-node-01 local]$ > /usr/local/topspin/mpi/mpich/bin/mpirun_rsh -v > OSU MVAPICH VERSION 0.9.5-SingleRail > > 3)when I try to use hp mpi 2.2.5 over the IB network I got the following: > [radioss@hpc-node-01 job1]$ /opt/hpmpi/bin/mpirun -stdio=i0 > -cpu_bind=cyclic -VAPI -f appfile < PFTANKD01 > dlopen test for MPI_ICLIB_VAPI__VAPI_MAIN could not open libs in list > libmtl_common.so libmpga.so libmosal.so libvapi.so: > /usr/local/topspin/lib64/libmosal.so: undefined symbol: pthread_create > dlopen test for MPI_ICLIB_VAPI__VAPI_CISCO could not open libs in list > libpthread.so libmosal.so libvapi.so: /usr/lib64/libpthread.so: > invalid ELF header > mpid: MPI BUG: VAPI requested but not available > what does it probably indicate? anything is wrong with the IB configuration? > > RPM packages installed there: > [radioss@hpc-node-01 job1]$ rpm -qa|grep topspin > topspin-ib-rhel4-3.2.0-118 > topspin-ib-mpi-rhel4-3.2.0-118 > topspin-ib-mod-rhel4-2.6.9-42.ELsmp-3.2.0-118 > > 4)You guys suggested me to use HP MPI (no native mvapich) and OFED IB > stack if possible. > now I have some questiosn hope you can have a quick comment or refer me > some website link so that I can read through: > i)how to verify which IB stack is used here, OFED or Cisco/Topspin IB > stack? what's the advantages of OFED IB stack over Cisco/Topspin IB stack? > ii)what's the advatages of HP HPI over "native mvapich"? what means by > "native mvapich"? the one shipped with Cisco/Topspin? is it enough to > upgrade mvapich to the latest one which is availabel on mvapich website? > > Thanks a lot for all of you for your kindly help! > > Henry, Wu. > > > > > | On Aug 29, 2007, at 12:54 PM, wgy@altair.com.cn wrote: > | > |> Yes, I think I used mavapich shipped with Topspin, but I am not sure > |> unless I know how to verify it. > | > | If it's in the /usr/local/topspin directory, it's the Topspin (later > | Cisco) MVAPICH. > | > |> about latency test, I downloaded > |> https://mvapich.cse.ohio-state.edu/svn/mpi/mvapich/trunk/ > |> osu_benchmarks/osu_latency.c > |> and will compile it to run a benchmark. Can you please tell me how > |> should > |> I run it? how many nodes should be used and how many cpus should be > |> involved? > | > | You typically run it with 2 MPI processes; one on each host. It > | measures the MPI network latency between those two hosts. > | > |> I have 4-cores nodes here.. > |> I would expect to run it as: > |> /usr/local/topspin/mpi/mpich/bin/mpirun_ssh -np 2 -hostfile hosts > | > | ^^ Is that the right path? Or is it "mvapich"? Regardless, I think > | wherever you find mpirun_ssh under /usr/local/topspin/mpi is probably > | the right one. > | > |> osu_latency.o > | > | Is your executable really named osu_latency.o? That's uncommon. > | Regardless, run the executable that you got when you compiled > | osu_latency.c with mpicc. > | > |> and include the following in the hosts file > |> hpc-node-01 > |> hpc-node-02 > | > | Sounds right. I'm not an MVAPICH expert, though -- so I defer to the > | maintainers here on this list for the finer details... > | > |> Is it right? > |> Thanks a lot, I am really a newbie with Infiniband.... > | > | If this is your own system, I do want to stress that OFED is really > | the way to go with HPC InfiniBand installations these days. The > | MPI's that are included are much more recent, and all new development > | work is happening in the OFED arena. > | > | I recommend that you upgrade if you can. > | > | > |> Henry, Wu > |> > |> > |> | On Aug 29, 2007, at 12:25 PM, wgy@altair.com.cn wrote: > |> | > |> |> Hello, Jeff: > |> |> The mvapich version is OSU mvapich0.95. > |> |> does it mean that it is Cisco IB stack and therefor the application > |> |> I run > |> |> with mvapich is really running over IB network? > |> | > |> | The version of MVAPICH, by itself, does not mean that it is or is > |> not > |> | running over IB. > |> | > |> | What *implies* that you are running over IB is: > |> | > |> | - You implied that you are using the MVAPICH shipped with the > |> Topspin > |> | IB stack (which is not OFED). Is that correct? > |> | - I *believe* that the Topspin MVAPICH did not have TCP support > |> | compiled into it (Topspin was before my time, but I am pretty sure > |> | that the Cisco MVAPICH shipped with the Cisco IB stack does not) > |> | > |> | What would *prove* that you are using IB (vs. gige) is: > |> | > |> | - Run a simple latency test, as Dr. Panda suggested. Your latency > |> | should be single-digit microseconds (exact numbers depend on your > |> | hardware -- this might be all older stuff since you mentioned > |> | "Topspin", not "Cisco"; Topspin was acquired by Cisco quite a while > |> | ago...). If your latency is much higher than that (e.g., 50 us), > |> | you're using gige. > |> | > |> | > |> | > |> |> Thanks. > |> |> > |> |> Henry, Wu. > |> |> | In addition to what Dr. Panda said, Cisco recommends that all HPC > |> |> | customers upgrade to the OFED IB driver stack if possible (some > |> |> | customers cannot upgrade for various reasons). FWIW: all new > |> HPC/ > |> |> MPI > |> |> | work is occurring in the OFED arena. > |> |> | > |> |> | I bring this up because you specifically mention Topspin > |> Infiniband, > |> |> | which I'm *assuming* is the Cisco IB stack (not the OFED IB > |> stack), > |> |> | and is therefore shipping with a somewhat older version of > |> MVAPICH > |> |> | that was derived from the OSU MVAPICH. The Cisco MVAPICH should > |> |> only > |> |> | be compiled with IB support enabled; a simple latency test should > |> |> | prove that you're running over IB and not ethernet. > |> |> | > |> |> | Much more recent versions of MPI implementations are included > |> with > |> |> | the OFED stack (Cisco provides binary distributions of OFED on > |> |> | www.cisco.com). > |> |> | > |> |> | > |> |> | On Aug 29, 2007, at 11:44 AM, Dhabaleswar Panda wrote: > |> |> | > |> |> |> > |> |> |> > |> |> |> On Wed, 29 Aug 2007 wgy@altair.com.cn wrote: > |> |> |> > |> |> |>> Hello, list: > |> |> |>> It might be a silly questions but I wonder how to verify run > |> with > |> |> |>> mvapich > |> |> |>> (come with Topspin Infiniband) over Infiniband, NOT Gigabite > |> |> network. > |> |> |>> Is there an option to force mvapich to use IB network otherwise > |> |> |>> just exits? > |> |> |> > |> |> |> MVAPICH has several underlying interfaces: Gen2, uDAPL, VAPI, > |> TCP/ > |> |> |> IP and > |> |> |> shared memory. Please take a look at the user guide > |> (available from > |> |> |> mvapich project page) to see the differences and capabilities of > |> |> these > |> |> |> interfaces. Gen2 interface (corresponding to OFED) will give you > |> |> |> the best performance and scalability. If you have OFED stack > |> |> |> installed, > |> |> |> you should be able to configure mvapich to run over Gen2 > |> interface > |> |> |> (as per the instructions indicated in the user guide). During > |> OFED > |> |> |> installation, you can also select mvapich from the package. > |> |> |> > |> |> |> On your existing installation, you can also run OSU benchmarks > |> |> (such > |> |> |> as OSU latency). If you get latency number in the range of 2~4 > |> |> |> microsec > |> |> |> for short messages (say 4 bytes), it is already running over the > |> |> |> native > |> |> |> IB. > |> |> |> > |> |> |> Hope this helps. > |> |> |> > |> |> |> DK > |> |> |> > |> |> |>> Thanks for your suggestion. > |> |> |>> Rdgs. > |> |> |>> Henry, Wu > |> |> |>> > |> |> |>> _______________________________________________ > |> |> |>> mvapich-discuss mailing list > |> |> |>> mvapich-discuss@cse.ohio-state.edu > |> |> |>> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > |> |> |>> > |> |> |> > |> |> |> _______________________________________________ > |> |> |> mvapich-discuss mailing list > |> |> |> mvapich-discuss@cse.ohio-state.edu > |> |> |> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > |> |> | > |> |> | > |> |> | -- > |> |> | Jeff Squyres > |> |> | Cisco Systems > |> |> | > |> |> | > |> | > |> | > |> | -- > |> | Jeff Squyres > |> | Cisco Systems > |> | > |> | > | > | > | -- > | Jeff Squyres > | Cisco Systems > | > | > > > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > From wgy at altair.com.cn Tue Sep 4 02:22:47 2007 From: wgy at altair.com.cn (wgy@altair.com.cn) Date: Tue Sep 4 02:23:20 2007 Subject: [mvapich-discuss] Verify the application is really running In-Reply-To: <200709040610.l846ADAg006816@xi.cse.ohio-state.edu> References: <200709040610.l846ADAg006816@xi.cse.ohio-state.edu> Message-ID: <25052.203.193.122.66.1188886967.squirrel@mail.altair.com.cn> Hello: I am quite sure I used the one you referred and got the compile error as you can see from the message.. I just renamed osu_latency.c to lantency.c while uploading... Thanks. Henry, Wu. | Are you sure you are using the osu_latency.c file from mvapich web | site? Your e-mail indicates about using a `latency.c' file. | | FYI, the osu_latency.c benchmark (latest version v2.2) is available | from the following URL: | | https://mvapich.cse.ohio-state.edu/svn/mpi/mvapich/trunk/osu_benchmarks/osu_latency.c | | DK | |> |> Hello, Jeff and Dr, Panda: |> I get back with the following resluts and further ques: |> 1)latency test reslut: |> the latency.c I downloaded from mvapich website can be complied in the |> cluster with errors: |> [radioss@hpc-node-01 job1]$ /usr/local/topspin/mpi/mpich/bin/mpicc |> latency.c -o lat |> latency.c:66: error: syntax error before "for" |> latency.c:68: error: `i' undeclared here (not in a function) |> latency.c:68: warning: data definition has no type or storage class |> latency.c:69: error: syntax error before '}' token |> latency.c:73: error: `skip_large' undeclared here (not in a function) |> latency.c:73: warning: data definition has no type or storage class |> latency.c:74: error: syntax error before '}' token |> latency.c:76: warning: parameter names (without types) in function |> declaration |> latency.c:76: warning: data definition has no type or storage class |> latency.c:78: error: syntax error before "if" |> latency.c:82: error: syntax error before numeric constant |> latency.c:82: warning: data definition has no type or storage class |> latency.c:83: error: syntax error before numeric constant |> latency.c:84: warning: data definition has no type or storage class |> latency.c:86: error: initializer element is not constant |> latency.c:86: warning: data definition has no type or storage class |> latency.c:88: error: syntax error before '}' token |> latency.c:92: error: syntax error before numeric constant |> latency.c:92: warning: data definition has no type or storage class |> latency.c:98: error: `t_start' undeclared here (not in a function) |> latency.c:98: error: `loop' undeclared here (not in a function) |> latency.c:98: warning: data definition has no type or storage class |> latency.c:99: error: syntax error before string constant |> latency.c:99: warning: conflicting types for built-in function 'fprintf' |> latency.c:99: warning: data definition has no type or storage class |> latency.c:104: warning: data definition has no type or storage class |> latency.c:105: error: syntax error before "return" |> latency.c:107:2: warning: no newline at end of file |> latency.c:68: error: storage size of `r_buf' isn't know |> |> I had to use mpi_latency.c shipped with mvapich in the cluster and got |> the |> following latency test results. |> |> [radioss@hpc-node-01 job1]$ /usr/local/topspin/mpi/mpich/bin/mpirun_rsh |> -np 2 -hostfile appfile ./lat 10000 1 |> 1 6.288650 |> [radioss@hpc-node-01 job1]$ /usr/local/topspin/mpi/mpich/bin/mpirun_rsh |> -np 2 -hostfile appfile ./lat 10000 4 |> 4 6.410350 |> while Topspin's Host-Side Drivers User Guide for Linux Release 3.1.0 |> gives |> the following latency test figure as an example: |> [root@qa-bc1-blade2 root]# /usr/local/topspin/mpi/mpich/bin/mpirun_ssh |> -np |> 2 qabc1- |> blade2 qa-bc1-blade3 /usr/local/topspin/mpi/mpich/bin/mpi_latency 10000 |> 1 |> 1 6.684000 |> 2) Jeff Squyres once asked me: |> >> I have 4-cores nodes here.. |> >> I would expect to run it as: |> >> /usr/local/topspin/mpi/mpich/bin/mpirun_ssh -np 2 -hostfile hosts |> |> >^^ Is that the right path? Or is it "mvapich"? Regardless, I think |> wherever you find mpirun_ssh under /usr/local/topspin/mpi is probably |> the right one. |> the path is right, and I am pretty sure it is mavapich because: |> i)rpm -qf /usr/local/topspin/mpi/mpich/bin/mpirun_ssh gives: |> topspin-ib-mpi-rhel4-3.2.0-118 |> ii)[radioss@hpc-node-01 local]$ |> /usr/local/topspin/mpi/mpich/bin/mpirun_rsh -v |> OSU MVAPICH VERSION 0.9.5-SingleRail |> |> 3)when I try to use hp mpi 2.2.5 over the IB network I got the |> following: |> [radioss@hpc-node-01 job1]$ /opt/hpmpi/bin/mpirun -stdio=i0 |> -cpu_bind=cyclic -VAPI -f appfile < PFTANKD01 |> dlopen test for MPI_ICLIB_VAPI__VAPI_MAIN could not open libs in list |> libmtl_common.so libmpga.so libmosal.so libvapi.so: |> /usr/local/topspin/lib64/libmosal.so: undefined symbol: pthread_create |> dlopen test for MPI_ICLIB_VAPI__VAPI_CISCO could not open libs in list |> libpthread.so libmosal.so libvapi.so: /usr/lib64/libpthread.so: |> invalid ELF header |> mpid: MPI BUG: VAPI requested but not available |> what does it probably indicate? anything is wrong with the IB |> configuration? |> |> RPM packages installed there: |> [radioss@hpc-node-01 job1]$ rpm -qa|grep topspin |> topspin-ib-rhel4-3.2.0-118 |> topspin-ib-mpi-rhel4-3.2.0-118 |> topspin-ib-mod-rhel4-2.6.9-42.ELsmp-3.2.0-118 |> |> 4)You guys suggested me to use HP MPI (no native mvapich) and OFED IB |> stack if possible. |> now I have some questiosn hope you can have a quick comment or refer me |> some website link so that I can read through: |> i)how to verify which IB stack is used here, OFED or Cisco/Topspin IB |> stack? what's the advantages of OFED IB stack over Cisco/Topspin IB |> stack? |> ii)what's the advatages of HP HPI over "native mvapich"? what means by |> "native mvapich"? the one shipped with Cisco/Topspin? is it enough to |> upgrade mvapich to the latest one which is availabel on mvapich website? |> |> Thanks a lot for all of you for your kindly help! |> |> Henry, Wu. |> |> |> |> |> | On Aug 29, 2007, at 12:54 PM, wgy@altair.com.cn wrote: |> | |> |> Yes, I think I used mavapich shipped with Topspin, but I am not sure |> |> unless I know how to verify it. |> | |> | If it's in the /usr/local/topspin directory, it's the Topspin (later |> | Cisco) MVAPICH. |> | |> |> about latency test, I downloaded |> |> https://mvapich.cse.ohio-state.edu/svn/mpi/mvapich/trunk/ |> |> osu_benchmarks/osu_latency.c |> |> and will compile it to run a benchmark. Can you please tell me how |> |> should |> |> I run it? how many nodes should be used and how many cpus should be |> |> involved? |> | |> | You typically run it with 2 MPI processes; one on each host. It |> | measures the MPI network latency between those two hosts. |> | |> |> I have 4-cores nodes here.. |> |> I would expect to run it as: |> |> /usr/local/topspin/mpi/mpich/bin/mpirun_ssh -np 2 -hostfile hosts |> | |> | ^^ Is that the right path? Or is it "mvapich"? Regardless, I think |> | wherever you find mpirun_ssh under /usr/local/topspin/mpi is probably |> | the right one. |> | |> |> osu_latency.o |> | |> | Is your executable really named osu_latency.o? That's uncommon. |> | Regardless, run the executable that you got when you compiled |> | osu_latency.c with mpicc. |> | |> |> and include the following in the hosts file |> |> hpc-node-01 |> |> hpc-node-02 |> | |> | Sounds right. I'm not an MVAPICH expert, though -- so I defer to the |> | maintainers here on this list for the finer details... |> | |> |> Is it right? |> |> Thanks a lot, I am really a newbie with Infiniband.... |> | |> | If this is your own system, I do want to stress that OFED is really |> | the way to go with HPC InfiniBand installations these days. The |> | MPI's that are included are much more recent, and all new development |> | work is happening in the OFED arena. |> | |> | I recommend that you upgrade if you can. |> | |> | |> |> Henry, Wu |> |> |> |> |> |> | On Aug 29, 2007, at 12:25 PM, wgy@altair.com.cn wrote: |> |> | |> |> |> Hello, Jeff: |> |> |> The mvapich version is OSU mvapich0.95. |> |> |> does it mean that it is Cisco IB stack and therefor the |> application |> |> |> I run |> |> |> with mvapich is really running over IB network? |> |> | |> |> | The version of MVAPICH, by itself, does not mean that it is or is |> |> not |> |> | running over IB. |> |> | |> |> | What *implies* that you are running over IB is: |> |> | |> |> | - You implied that you are using the MVAPICH shipped with the |> |> Topspin |> |> | IB stack (which is not OFED). Is that correct? |> |> | - I *believe* that the Topspin MVAPICH did not have TCP support |> |> | compiled into it (Topspin was before my time, but I am pretty sure |> |> | that the Cisco MVAPICH shipped with the Cisco IB stack does not) |> |> | |> |> | What would *prove* that you are using IB (vs. gige) is: |> |> | |> |> | - Run a simple latency test, as Dr. Panda suggested. Your latency |> |> | should be single-digit microseconds (exact numbers depend on your |> |> | hardware -- this might be all older stuff since you mentioned |> |> | "Topspin", not "Cisco"; Topspin was acquired by Cisco quite a while |> |> | ago...). If your latency is much higher than that (e.g., 50 us), |> |> | you're using gige. |> |> | |> |> | |> |> | |> |> |> Thanks. |> |> |> |> |> |> Henry, Wu. |> |> |> | In addition to what Dr. Panda said, Cisco recommends that all |> HPC |> |> |> | customers upgrade to the OFED IB driver stack if possible (some |> |> |> | customers cannot upgrade for various reasons). FWIW: all new |> |> HPC/ |> |> |> MPI |> |> |> | work is occurring in the OFED arena. |> |> |> | |> |> |> | I bring this up because you specifically mention Topspin |> |> Infiniband, |> |> |> | which I'm *assuming* is the Cisco IB stack (not the OFED IB |> |> stack), |> |> |> | and is therefore shipping with a somewhat older version of |> |> MVAPICH |> |> |> | that was derived from the OSU MVAPICH. The Cisco MVAPICH should |> |> |> only |> |> |> | be compiled with IB support enabled; a simple latency test |> should |> |> |> | prove that you're running over IB and not ethernet. |> |> |> | |> |> |> | Much more recent versions of MPI implementations are included |> |> with |> |> |> | the OFED stack (Cisco provides binary distributions of OFED on |> |> |> | www.cisco.com). |> |> |> | |> |> |> | |> |> |> | On Aug 29, 2007, at 11:44 AM, Dhabaleswar Panda wrote: |> |> |> | |> |> |> |> |> |> |> |> |> |> |> |> On Wed, 29 Aug 2007 wgy@altair.com.cn wrote: |> |> |> |> |> |> |> |>> Hello, list: |> |> |> |>> It might be a silly questions but I wonder how to verify run |> |> with |> |> |> |>> mvapich |> |> |> |>> (come with Topspin Infiniband) over Infiniband, NOT Gigabite |> |> |> network. |> |> |> |>> Is there an option to force mvapich to use IB network |> otherwise |> |> |> |>> just exits? |> |> |> |> |> |> |> |> MVAPICH has several underlying interfaces: Gen2, uDAPL, VAPI, |> |> TCP/ |> |> |> |> IP and |> |> |> |> shared memory. Please take a look at the user guide |> |> (available from |> |> |> |> mvapich project page) to see the differences and capabilities |> of |> |> |> these |> |> |> |> interfaces. Gen2 interface (corresponding to OFED) will give |> you |> |> |> |> the best performance and scalability. If you have OFED stack |> |> |> |> installed, |> |> |> |> you should be able to configure mvapich to run over Gen2 |> |> interface |> |> |> |> (as per the instructions indicated in the user guide). During |> |> OFED |> |> |> |> installation, you can also select mvapich from the package. |> |> |> |> |> |> |> |> On your existing installation, you can also run OSU benchmarks |> |> |> (such |> |> |> |> as OSU latency). If you get latency number in the range of 2~4 |> |> |> |> microsec |> |> |> |> for short messages (say 4 bytes), it is already running over |> the |> |> |> |> native |> |> |> |> IB. |> |> |> |> |> |> |> |> Hope this helps. |> |> |> |> |> |> |> |> DK |> |> |> |> |> |> |> |>> Thanks for your suggestion. |> |> |> |>> Rdgs. |> |> |> |>> Henry, Wu |> |> |> |>> |> |> |> |>> _______________________________________________ |> |> |> |>> mvapich-discuss mailing list |> |> |> |>> mvapich-discuss@cse.ohio-state.edu |> |> |> |>> |> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss |> |> |> |>> |> |> |> |> |> |> |> |> _______________________________________________ |> |> |> |> mvapich-discuss mailing list |> |> |> |> mvapich-discuss@cse.ohio-state.edu |> |> |> |> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss |> |> |> | |> |> |> | |> |> |> | -- |> |> |> | Jeff Squyres |> |> |> | Cisco Systems |> |> |> | |> |> |> | |> |> | |> |> | |> |> | -- |> |> | Jeff Squyres |> |> | Cisco Systems |> |> | |> |> | |> | |> | |> | -- |> | Jeff Squyres |> | Cisco Systems |> | |> | |> |> |> _______________________________________________ |> mvapich-discuss mailing list |> mvapich-discuss@cse.ohio-state.edu |> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss |> | | From narravul at cse.ohio-state.edu Tue Sep 4 10:29:53 2007 From: narravul at cse.ohio-state.edu (Sundeep Narravula) Date: Tue Sep 4 10:30:19 2007 Subject: [mvapich-discuss] [PATCH] ofed-1.2.5/mvapich2 bug fixes In-Reply-To: <1188576958.14461.137.camel@sale659> Message-ID: Hi Jim, Thank you for sending us these patches for mvapich2-0.9.8. We will make these fixes available for mvapich2-0.9.8 soon. Recently we have released mvapich2-1.0 beta2. These fixes are already present in the mvapich2-1.0 codebase. You are welcome to try this version. It can be downloaded from our web-page: http://mvapich.cse.ohio-state.edu/download/mvapich2/ Regards, --Sundeep. On Fri, 31 Aug 2007, Jim Schutt wrote: > Hi, > > I've been working with mvapich2 from the OFED-1.2.5 release (I used > http://www.openfabrics.org/downloads/OFED/ofed-1.2.5/OFED-1.2.5.tgz). > > I've found a couple bugs in that version of mvapich2, which seem to > also be present in the upstream SVN at > https://mvapich.cse.ohio-state.edu/svn/mpi/mvapich2/branches/0.9.8 > as of revision 1480. > > The first is an install tool bug: DESTDIR gets prepended twice in > some cases, once when calling FixInstallFile and once inside it. > > The second is a memory-scribbling bug: in the call-chain > rdma_cm_get_hostnames()->PMI_KVS_Get()->PMIU_getval() > PMIU_getval() overwrites byte PMI_vallen_max - 1 (in my test > case, PMI_vallen_max had value 2048) for a character array that > has length 16. > > I'm not sure these are the right fixes, but with these patches > applied, mvapich2 from ofed-1.2.5 installs correctly and runs tests > it wouldn't run without them. > > -- Jim > > -- > Jim Schutt > Sandia National Laboratories, Albuquerque, New Mexico USA > > diff -urN mvapich2-0.9.8.orig/src/mpe2/sbin/mpeinstall.in mvapich2-0.9.8/src/mpe2/sbin/mpeinstall.in > --- mvapich2-0.9.8.orig/src/mpe2/sbin/mpeinstall.in 2006-04-09 11:57:00.000000000 -0600 > +++ mvapich2-0.9.8/src/mpe2/sbin/mpeinstall.in 2007-08-31 10:02:10.000000000 -0600 > @@ -442,10 +442,10 @@ > echo "Copying MPE utility programs to $DESTDIR$bindir" > CopyDirRecurP $binbuild_dir $bindir $XMODE > if [ -s $binbuild_dir/mpecc -a -x $binbuild_dir/mpecc ] ; then > - FixInstallFile $binbuild_dir/mpecc $DESTDIR$bindir/mpecc $XMODE > + FixInstallFile $binbuild_dir/mpecc $bindir/mpecc $XMODE > fi > if [ -s $binbuild_dir/mpefc -a -x $binbuild_dir/mpefc ] ; then > - FixInstallFile $binbuild_dir/mpefc $DESTDIR$bindir/mpefc $XMODE > + FixInstallFile $binbuild_dir/mpefc $bindir/mpefc $XMODE > fi > fi > fi > @@ -457,7 +457,7 @@ > CopyDirRecurP $etcbuild_dir $sysconfdir $MODE > cd $etcbuild_dir && \ > for file in *.conf ; do \ > - FixInstallFile $file $DESTDIR$sysconfdir/$file ; \ > + FixInstallFile $file $sysconfdir/$file ; \ > done > fi > fi > diff -urN mvapich2-0.9.8.orig/src/pmi/simple/simple_pmi.c mvapich2-0.9.8/src/pmi/simple/simple_pmi.c > --- mvapich2-0.9.8.orig/src/pmi/simple/simple_pmi.c 2006-04-09 11:57:00.000000000 -0600 > +++ mvapich2-0.9.8/src/pmi/simple/simple_pmi.c 2007-08-31 10:02:26.000000000 -0600 > @@ -566,7 +566,7 @@ > PMIU_getval( "rc", buf, PMIU_MAXLINE ); > rc = atoi( buf ); > if ( rc == 0 ) { > - PMIU_getval( "value", value, PMI_vallen_max ); > + PMIU_getval( "value", value, length ); > return( 0 ); > } > else { > > > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > From perkinjo at cse.ohio-state.edu Tue Sep 4 11:27:00 2007 From: perkinjo at cse.ohio-state.edu (Jonathan L. Perkins) Date: Tue Sep 4 11:27:28 2007 Subject: [mvapich-discuss] Verify the application is really running In-Reply-To: <25052.203.193.122.66.1188886967.squirrel@mail.altair.com.cn> References: <200709040610.l846ADAg006816@xi.cse.ohio-state.edu> <25052.203.193.122.66.1188886967.squirrel@mail.altair.com.cn> Message-ID: <46DD7944.60507@cse.ohio-state.edu> Can you send us the output from /usr/local/topspin/mpi/mpich/bin/mpicc -v This will let us know what compiler and which version of it that you're using. We do not see this problem when trying in our environment using a somewhat recent version of gcc. Below I pasted line 66 with a few lines of context around it. There is no for loop here so I'm a bit confused as to why you're getting the errors that you posted. int main(int argc, char *argv[]) { int myid, numprocs, i; int size; MPI_Status reqstat; char *s_buf, *r_buf; int align_size; Also, as a sanity check, can you download the osu_latency.c file again from https://mvapich.cse.ohio-state.edu/svn/mpi/mvapich/trunk/osu_benchmarks/osu_latency.c to verify that we are referencing the same file. Thanks for your input and we hope with further information we can solve this compilation issue that you're having. wgy@altair.com.cn wrote: > Hello: > I am quite sure I used the one you referred and got the compile error as > you can see from the message.. I just renamed osu_latency.c to lantency.c > while uploading... > Thanks. > Henry, Wu. > > | Are you sure you are using the osu_latency.c file from mvapich web > | site? Your e-mail indicates about using a `latency.c' file. > | > | FYI, the osu_latency.c benchmark (latest version v2.2) is available > | from the following URL: > | > | > https://mvapich.cse.ohio-state.edu/svn/mpi/mvapich/trunk/osu_benchmarks/osu_latency.c > | > | DK > | > |> > |> Hello, Jeff and Dr, Panda: > |> I get back with the following resluts and further ques: > |> 1)latency test reslut: > |> the latency.c I downloaded from mvapich website can be complied in the > |> cluster with errors: > |> [radioss@hpc-node-01 job1]$ /usr/local/topspin/mpi/mpich/bin/mpicc > |> latency.c -o lat > |> latency.c:66: error: syntax error before "for" > |> latency.c:68: error: `i' undeclared here (not in a function) > |> latency.c:68: warning: data definition has no type or storage class > |> latency.c:69: error: syntax error before '}' token > |> latency.c:73: error: `skip_large' undeclared here (not in a function) > |> latency.c:73: warning: data definition has no type or storage class > |> latency.c:74: error: syntax error before '}' token > |> latency.c:76: warning: parameter names (without types) in function > |> declaration > |> latency.c:76: warning: data definition has no type or storage class > |> latency.c:78: error: syntax error before "if" > |> latency.c:82: error: syntax error before numeric constant > |> latency.c:82: warning: data definition has no type or storage class > |> latency.c:83: error: syntax error before numeric constant > |> latency.c:84: warning: data definition has no type or storage class > |> latency.c:86: error: initializer element is not constant > |> latency.c:86: warning: data definition has no type or storage class > |> latency.c:88: error: syntax error before '}' token > |> latency.c:92: error: syntax error before numeric constant > |> latency.c:92: warning: data definition has no type or storage class > |> latency.c:98: error: `t_start' undeclared here (not in a function) > |> latency.c:98: error: `loop' undeclared here (not in a function) > |> latency.c:98: warning: data definition has no type or storage class > |> latency.c:99: error: syntax error before string constant > |> latency.c:99: warning: conflicting types for built-in function 'fprintf' > |> latency.c:99: warning: data definition has no type or storage class > |> latency.c:104: warning: data definition has no type or storage class > |> latency.c:105: error: syntax error before "return" > |> latency.c:107:2: warning: no newline at end of file > |> latency.c:68: error: storage size of `r_buf' isn't know > |> > |> I had to use mpi_latency.c shipped with mvapich in the cluster and got > |> the > |> following latency test results. > |> > |> [radioss@hpc-node-01 job1]$ /usr/local/topspin/mpi/mpich/bin/mpirun_rsh > |> -np 2 -hostfile appfile ./lat 10000 1 > |> 1 6.288650 > |> [radioss@hpc-node-01 job1]$ /usr/local/topspin/mpi/mpich/bin/mpirun_rsh > |> -np 2 -hostfile appfile ./lat 10000 4 > |> 4 6.410350 > |> while Topspin's Host-Side Drivers User Guide for Linux Release 3.1.0 > |> gives > |> the following latency test figure as an example: > |> [root@qa-bc1-blade2 root]# /usr/local/topspin/mpi/mpich/bin/mpirun_ssh > |> -np > |> 2 qabc1- > |> blade2 qa-bc1-blade3 /usr/local/topspin/mpi/mpich/bin/mpi_latency 10000 > |> 1 > |> 1 6.684000 > |> 2) Jeff Squyres once asked me: > |> >> I have 4-cores nodes here.. > |> >> I would expect to run it as: > |> >> /usr/local/topspin/mpi/mpich/bin/mpirun_ssh -np 2 -hostfile hosts > |> > |> >^^ Is that the right path? Or is it "mvapich"? Regardless, I think > |> wherever you find mpirun_ssh under /usr/local/topspin/mpi is probably > |> the right one. > |> the path is right, and I am pretty sure it is mavapich because: > |> i)rpm -qf /usr/local/topspin/mpi/mpich/bin/mpirun_ssh gives: > |> topspin-ib-mpi-rhel4-3.2.0-118 > |> ii)[radioss@hpc-node-01 local]$ > |> /usr/local/topspin/mpi/mpich/bin/mpirun_rsh -v > |> OSU MVAPICH VERSION 0.9.5-SingleRail > |> > |> 3)when I try to use hp mpi 2.2.5 over the IB network I got the > |> following: > |> [radioss@hpc-node-01 job1]$ /opt/hpmpi/bin/mpirun -stdio=i0 > |> -cpu_bind=cyclic -VAPI -f appfile < PFTANKD01 > |> dlopen test for MPI_ICLIB_VAPI__VAPI_MAIN could not open libs in list > |> libmtl_common.so libmpga.so libmosal.so libvapi.so: > |> /usr/local/topspin/lib64/libmosal.so: undefined symbol: pthread_create > |> dlopen test for MPI_ICLIB_VAPI__VAPI_CISCO could not open libs in list > |> libpthread.so libmosal.so libvapi.so: /usr/lib64/libpthread.so: > |> invalid ELF header > |> mpid: MPI BUG: VAPI requested but not available > |> what does it probably indicate? anything is wrong with the IB > |> configuration? > |> > |> RPM packages installed there: > |> [radioss@hpc-node-01 job1]$ rpm -qa|grep topspin > |> topspin-ib-rhel4-3.2.0-118 > |> topspin-ib-mpi-rhel4-3.2.0-118 > |> topspin-ib-mod-rhel4-2.6.9-42.ELsmp-3.2.0-118 > |> > |> 4)You guys suggested me to use HP MPI (no native mvapich) and OFED IB > |> stack if possible. > |> now I have some questiosn hope you can have a quick comment or refer me > |> some website link so that I can read through: > |> i)how to verify which IB stack is used here, OFED or Cisco/Topspin IB > |> stack? what's the advantages of OFED IB stack over Cisco/Topspin IB > |> stack? > |> ii)what's the advatages of HP HPI over "native mvapich"? what means by > |> "native mvapich"? the one shipped with Cisco/Topspin? is it enough to > |> upgrade mvapich to the latest one which is availabel on mvapich website? > |> > |> Thanks a lot for all of you for your kindly help! > |> > |> Henry, Wu. > |> > |> > |> > |> > |> | On Aug 29, 2007, at 12:54 PM, wgy@altair.com.cn wrote: > |> | > |> |> Yes, I think I used mavapich shipped with Topspin, but I am not sure > |> |> unless I know how to verify it. > |> | > |> | If it's in the /usr/local/topspin directory, it's the Topspin (later > |> | Cisco) MVAPICH. > |> | > |> |> about latency test, I downloaded > |> |> https://mvapich.cse.ohio-state.edu/svn/mpi/mvapich/trunk/ > |> |> osu_benchmarks/osu_latency.c > |> |> and will compile it to run a benchmark. Can you please tell me how > |> |> should > |> |> I run it? how many nodes should be used and how many cpus should be > |> |> involved? > |> | > |> | You typically run it with 2 MPI processes; one on each host. It > |> | measures the MPI network latency between those two hosts. > |> | > |> |> I have 4-cores nodes here.. > |> |> I would expect to run it as: > |> |> /usr/local/topspin/mpi/mpich/bin/mpirun_ssh -np 2 -hostfile hosts > |> | > |> | ^^ Is that the right path? Or is it "mvapich"? Regardless, I think > |> | wherever you find mpirun_ssh under /usr/local/topspin/mpi is probably > |> | the right one. > |> | > |> |> osu_latency.o > |> | > |> | Is your executable really named osu_latency.o? That's uncommon. > |> | Regardless, run the executable that you got when you compiled > |> | osu_latency.c with mpicc. > |> | > |> |> and include the following in the hosts file > |> |> hpc-node-01 > |> |> hpc-node-02 > |> | > |> | Sounds right. I'm not an MVAPICH expert, though -- so I defer to the > |> | maintainers here on this list for the finer details... > |> | > |> |> Is it right? > |> |> Thanks a lot, I am really a newbie with Infiniband.... > |> | > |> | If this is your own system, I do want to stress that OFED is really > |> | the way to go with HPC InfiniBand installations these days. The > |> | MPI's that are included are much more recent, and all new development > |> | work is happening in the OFED arena. > |> | > |> | I recommend that you upgrade if you can. > |> | > |> | > |> |> Henry, Wu > |> |> > |> |> > |> |> | On Aug 29, 2007, at 12:25 PM, wgy@altair.com.cn wrote: > |> |> | > |> |> |> Hello, Jeff: > |> |> |> The mvapich version is OSU mvapich0.95. > |> |> |> does it mean that it is Cisco IB stack and therefor the > |> application > |> |> |> I run > |> |> |> with mvapich is really running over IB network? > |> |> | > |> |> | The version of MVAPICH, by itself, does not mean that it is or is > |> |> not > |> |> | running over IB. > |> |> | > |> |> | What *implies* that you are running over IB is: > |> |> | > |> |> | - You implied that you are using the MVAPICH shipped with the > |> |> Topspin > |> |> | IB stack (which is not OFED). Is that correct? > |> |> | - I *believe* that the Topspin MVAPICH did not have TCP support > |> |> | compiled into it (Topspin was before my time, but I am pretty sure > |> |> | that the Cisco MVAPICH shipped with the Cisco IB stack does not) > |> |> | > |> |> | What would *prove* that you are using IB (vs. gige) is: > |> |> | > |> |> | - Run a simple latency test, as Dr. Panda suggested. Your latency > |> |> | should be single-digit microseconds (exact numbers depend on your > |> |> | hardware -- this might be all older stuff since you mentioned > |> |> | "Topspin", not "Cisco"; Topspin was acquired by Cisco quite a while > |> |> | ago...). If your latency is much higher than that (e.g., 50 us), > |> |> | you're using gige. > |> |> | > |> |> | > |> |> | > |> |> |> Thanks. > |> |> |> > |> |> |> Henry, Wu. > |> |> |> | In addition to what Dr. Panda said, Cisco recommends that all > |> HPC > |> |> |> | customers upgrade to the OFED IB driver stack if possible (some > |> |> |> | customers cannot upgrade for various reasons). FWIW: all new > |> |> HPC/ > |> |> |> MPI > |> |> |> | work is occurring in the OFED arena. > |> |> |> | > |> |> |> | I bring this up because you specifically mention Topspin > |> |> Infiniband, > |> |> |> | which I'm *assuming* is the Cisco IB stack (not the OFED IB > |> |> stack), > |> |> |> | and is therefore shipping with a somewhat older version of > |> |> MVAPICH > |> |> |> | that was derived from the OSU MVAPICH. The Cisco MVAPICH should > |> |> |> only > |> |> |> | be compiled with IB support enabled; a simple latency test > |> should > |> |> |> | prove that you're running over IB and not ethernet. > |> |> |> | > |> |> |> | Much more recent versions of MPI implementations are included > |> |> with > |> |> |> | the OFED stack (Cisco provides binary distributions of OFED on > |> |> |> | www.cisco.com). > |> |> |> | > |> |> |> | > |> |> |> | On Aug 29, 2007, at 11:44 AM, Dhabaleswar Panda wrote: > |> |> |> | > |> |> |> |> > |> |> |> |> > |> |> |> |> On Wed, 29 Aug 2007 wgy@altair.com.cn wrote: > |> |> |> |> > |> |> |> |>> Hello, list: > |> |> |> |>> It might be a silly questions but I wonder how to verify run > |> |> with > |> |> |> |>> mvapich > |> |> |> |>> (come with Topspin Infiniband) over Infiniband, NOT Gigabite > |> |> |> network. > |> |> |> |>> Is there an option to force mvapich to use IB network > |> otherwise > |> |> |> |>> just exits? > |> |> |> |> > |> |> |> |> MVAPICH has several underlying interfaces: Gen2, uDAPL, VAPI, > |> |> TCP/ > |> |> |> |> IP and > |> |> |> |> shared memory. Please take a look at the user guide > |> |> (available from > |> |> |> |> mvapich project page) to see the differences and capabilities > |> of > |> |> |> these > |> |> |> |> interfaces. Gen2 interface (corresponding to OFED) will give > |> you > |> |> |> |> the best performance and scalability. If you have OFED stack > |> |> |> |> installed, > |> |> |> |> you should be able to configure mvapich to run over Gen2 > |> |> interface > |> |> |> |> (as per the instructions indicated in the user guide). During > |> |> OFED > |> |> |> |> installation, you can also select mvapich from the package. > |> |> |> |> > |> |> |> |> On your existing installation, you can also run OSU benchmarks > |> |> |> (such > |> |> |> |> as OSU latency). If you get latency number in the range of 2~4 > |> |> |> |> microsec > |> |> |> |> for short messages (say 4 bytes), it is already running over > |> the > |> |> |> |> native > |> |> |> |> IB. > |> |> |> |> > |> |> |> |> Hope this helps. > |> |> |> |> > |> |> |> |> DK > |> |> |> |> > |> |> |> |>> Thanks for your suggestion. > |> |> |> |>> Rdgs. > |> |> |> |>> Henry, Wu > |> |> |> |>> > |> |> |> |>> _______________________________________________ > |> |> |> |>> mvapich-discuss mailing list > |> |> |> |>> mvapich-discuss@cse.ohio-state.edu > |> |> |> |>> > |> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > |> |> |> |>> > |> |> |> |> > |> |> |> |> _______________________________________________ > |> |> |> |> mvapich-discuss mailing list > |> |> |> |> mvapich-discuss@cse.ohio-state.edu > |> |> |> |> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > |> |> |> | > |> |> |> | > |> |> |> | -- > |> |> |> | Jeff Squyres > |> |> |> | Cisco Systems > |> |> |> | > |> |> |> | > |> |> | > |> |> | > |> |> | -- > |> |> | Jeff Squyres > |> |> | Cisco Systems > |> |> | > |> |> | > |> | > |> | > |> | -- > |> | Jeff Squyres > |> | Cisco Systems > |> | > |> | > |> > |> > |> _______________________________________________ > |> mvapich-discuss mailing list > |> mvapich-discuss@cse.ohio-state.edu > |> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > |> > | > | > > > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss -- Jonathan Perkins http://www.cse.ohio-state.edu/~perkinjo From koop at cse.ohio-state.edu Tue Sep 4 12:28:28 2007 From: koop at cse.ohio-state.edu (Matthew Koop) Date: Tue Sep 4 12:28:55 2007 Subject: [mvapich-discuss] FATAL event IBV_EVENT_QP_LAST_WQE_REACHED In-Reply-To: <46D747E7.7090607@noaa.gov> Message-ID: Nathan, Thanks. Let us know if the user sees these errors with either of these settings. Also, if there is any test code you can give us it would be much appreciated. The IBV_WC_LOC_LEN_ERR error is more likely to be the real error (it is not normal), although tracking it without a reproducer or some idea as to the MPI operations being used will be difficult. Matt On Thu, 30 Aug 2007, Nathan Dauchy wrote: > Matt, > > I'll try to get the user who is having this problem to run with those > environment variables set. > > They did report this error as well: > > 0:w71] Abort: [w71:0] Got completion with error IBV_WC_LOC_LEN_ERR, > code=1, dest rank=8 at line 388 in file viacheck.c > > But I had assumed that was a "normal" exit condition for a failed run. > Is this not the case? > > Thanks, > Nathan > > > Matthew Koop wrote: > > Nathan, > > > > Do you ever see any other errors, or is it just > > IBV_EVENT_QP_LAST_WQE_REACHED? Sometimes when a job fails one process will > > have an error and then the rest of the processes can exit with another > > error status unrelated to the problem. > > > > Can you try running with VIADEV_USE_SHMEM_COLL=0 and see if that makes any > > difference? > > > > e.g. > > mpirun_rsh -np 4 h1 h1 h2 h2 VIADEV_USE_SHMEM_COLL=0 ./IMB-MPI1 > > > > If that doesn't work, you can also try: > > VIADEV_USE_COALESCE=0 > > > > These will help us narrow down the problem a bit. > > > > Thanks, > > Matt > > > > > > On Tue, 21 Aug 2007, Nathan Dauchy wrote: > > > >> Updated... > >> > >> Nathan Dauchy wrote: > >>> I finally had time to get back to this issue... > >>> > >>> The OSU benchmarks run fine. > >>> The presta benchmarks run fine. > >> > >> IMB now runs fine. I had the library path wrong when compiling. > >> > >>>> Sayantan Sur wrote: > >>>>> Thanks for reporting the problem. The event > >>>>> IBV_EVENT_QP_LAST_WQE_REACHED means that the QP (internal InfiniBand > >>>>> communication channel) is in an error state and all requests are > >>>>> consumed. Could it be related to a setup issue? Can you run any other > >>>>> MPI programs such as OSU benchmarks, IMB etc. on all these nodes? > >>>>> > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > From ttelford.groups at gmail.com Tue Sep 4 18:58:55 2007 From: ttelford.groups at gmail.com (Troy Telford) Date: Tue Sep 4 18:59:32 2007 Subject: [mvapich-discuss] libmpichf90.so? Message-ID: <200709041658.56311.ttelford.groups@gmail.com> When building MVAPICH 0.9.9, I notice that 'libmpichf90.a' is built, but I can't find a corresponding 'libmpichf90.so' This is a problem for me as I have a codebase that is expecting to link against libmpichf90.so (ie. a shared lib rather than a static lib). Is there a way to get a shared library for libmpichf90? Was there _ever_ a way to get a shared library for libmpichf90 (and it is just disabled?) Thanks. -- Troy Telford From singh.jasjit at yahoo.co.in Wed Sep 5 01:51:42 2007 From: singh.jasjit at yahoo.co.in (jasjit singh) Date: Wed Sep 5 01:52:20 2007 Subject: [mvapich-discuss] temporary files causing problem Message-ID: <837545.35648.qm@web94012.mail.in2.yahoo.com> hi I have been running uDAPL code with mvapich2-0.9.8p2. Corresponding makefile (make.mvapich.udapl) has all the default CFLAGS except ONE_SIDED. It ends up creating some temporary files in /tmp directory. But after mpiexec ends, it doesn't delete these temporary files. These files are not deleted even if i reboot the system. I run mpiexec infinitely. So i get my disks full after 10-12 hrs on all the four nodes which i am running the application on. Could anybody help me in this regard ? Is there any way out to get rid of these temporary files ? Regards, Jasjit Singh --------------------------------- Yahoo! Answers - Get better answers from someone who knows. Tryit now. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20070905/f18cf05c/attachment.html From potts at hpcapplications.com Wed Sep 5 11:14:04 2007 From: potts at hpcapplications.com (Mark Potts) Date: Wed Sep 5 11:16:32 2007 Subject: [mvapich-discuss] suggested minor feature Message-ID: <46DEC7BC.7040706@hpcapplications.com> Hi, Despite the many issues I've raised about MVAPICH job cleanup and timeouts (all resolved now it appears), I'd like to raise another related issue -- a suggestion. We've found that a job that correctly has all processes call MPI_Finalize() at the end of their communications stages, can not permit any processes to terminate if it is desired for even a single thread to continue to work. That is, after MPI_Finalize() is called and any processes correctly terminate there is only a 10 second window in which any remaining processes will be allowed to run before mpirun_rsh kills the remaining children. This presents a problem for codes that naturally complete the job's task in serial mode or codes in which debugging of a process is needed after MPI_Finalize(). The suggestion would be: to provide the timeout period (currently 10 seconds) as a VIADEV_* env variable, with default of 10, which users could then modify when 10 seconds was too little time for a remaining process. By the same token this env variable could be used to trim the timeout period to a smaller value, when a user deemed 10 seconds to not be agressive enough. regards, -- *********************************** >> Mark J. Potts, PhD >> >> HPC Applications Inc. >> phone: 410-992-8360 Bus >> 410-313-9318 Home >> 443-418-4375 Cell >> email: potts@hpcapplications.com >> potts@excray.com *********************************** From panda at cse.ohio-state.edu Wed Sep 5 13:25:57 2007 From: panda at cse.ohio-state.edu (Dhabaleswar Panda) Date: Wed Sep 5 13:26:22 2007 Subject: [mvapich-discuss] suggested minor feature In-Reply-To: <46DEC7BC.7040706@hpcapplications.com> from "Mark Potts" at Sep 05, 2007 11:14:04 AM Message-ID: <200709051725.l85HPvGb023400@xi.cse.ohio-state.edu> Hi Mark, > Despite the many issues I've raised about MVAPICH job cleanup and > timeouts (all resolved now it appears), I'd like to raise another > related issue -- a suggestion. Glad to know that the issues appear to be resolved now. Thanks for the following suggestion. We will work on this enhancement and will send you a patch soon. > We've found that a job that correctly has all processes call > MPI_Finalize() at the end of their communications stages, can > not permit any processes to terminate if it is desired for even > a single thread to continue to work. That is, after MPI_Finalize() > is called and any processes correctly terminate there is only a > 10 second window in which any remaining processes will be allowed > to run before mpirun_rsh kills the remaining children. This > presents a problem for codes that naturally complete the job's > task in serial mode or codes in which debugging of a process is > needed after MPI_Finalize(). > > The suggestion would be: > to provide the timeout period (currently 10 seconds) as a > VIADEV_* env variable, with default of 10, which users could > then modify when 10 seconds was too little time for a remaining > process. By the same token this env variable could be used > to trim the timeout period to a smaller value, when a user > deemed 10 seconds to not be agressive enough. > regards, Best Regards, DK > *********************************** > >> Mark J. Potts, PhD > >> > >> HPC Applications Inc. > >> phone: 410-992-8360 Bus > >> 410-313-9318 Home > >> 443-418-4375 Cell > >> email: potts@hpcapplications.com > >> potts@excray.com > *********************************** > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > From chai.15 at osu.edu Wed Sep 5 15:17:39 2007 From: chai.15 at osu.edu (LEI CHAI) Date: Wed Sep 5 15:18:06 2007 Subject: [mvapich-discuss] temporary files causing problem Message-ID: <254847a254aa22.254aa22254847a@osu.edu> Hi Jasjit, We have a few bug fixes and code enhancements in the latest releases. Could you please download mvapich2-0.9.8p3 or mvapich2-1.0 beta and try? http://mvapich.cse.ohio-state.edu/download/mvapich2/ Thanks, Lei Content-Type: multipart/alternative; boundary="0-537041755-1188971502=:35648" Content-Transfer-Encoding: 8bit --0-537041755-1188971502=:35648 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit hi I have been running uDAPL code with mvapich2-0.9.8p2. Corresponding makefile (make.mvapich.udapl) has all the default CFLAGS except ONE_SIDED. It ends up creating some temporary files in /tmp directory. But after mpiexec ends, it doesn't delete these temporary files. These files are not deleted even if i reboot the system. I run mpiexec infinitely. So i get my disks full after 10-12 hrs on all the four nodes which i am running the application on. Could anybody help me in this regard ? Is there any way out to get rid of these temporary files ? Regards, Jasjit Singh --------------------------------- Yahoo! Answers - Get better answers from someone who knows. Tryit now. --0-537041755-1188971502=:35648 Content-Type: text/html; charset=iso-8859-1 Content-Transfer-Encoding: 8bit hi
I have been running uDAPL code with mvapich2-0.9.8p2. Corresponding makefile (make.mvapich.udapl) has all the default CFLAGS except ONE_SIDED. It ends up creating some temporary files in /tmp directory. But after mpiexec ends, it doesn't delete these temporary files. These files are not deleted even if i reboot the system. I run mpiexec infinitely. So i get my disks full after 10-12 hrs on all the four nodes which i am running the application on. Could anybody help me in this regard ? Is there any way out to get rid of these temporary files ?


Regards,
Jasjit Singh

Yahoo! Answers - Get better answers from someone who knows. Try it now. --0-537041755-1188971502=:35648-- -------------- next part -------------- _______________________________________________ mvapich-discuss mailing list mvapich-discuss@cse.ohio-state.edu http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss From koop at cse.ohio-state.edu Wed Sep 5 16:36:13 2007 From: koop at cse.ohio-state.edu (Matthew Koop) Date: Wed Sep 5 16:36:40 2007 Subject: [mvapich-discuss] Printing DEBUG_PRINT mesges In-Reply-To: <46DC29FD.6030003@cdac.in> Message-ID: Nilesh, DEBUG_PRINT is used for internal development, so we cannot guarantee that the messages printed will be useful to others (or correct). As such, we do not support this option for the end-user. Is there a particular problem you are experiencing that we can address? Matt On Mon, 3 Sep 2007, Nilesh Awate wrote: > > Hi All, > > I'm using mvapich2-1.0-beta with ofed1.2 udapl interface. > > I want to print DEBUG_PRINT masseges for that i've included DDEBUG flag in > make.mvapich.udapl make file > > but while making its give error in that perticular DEBUG_PRINT statements > "variables not defined" > > is there anything else setting of flags ... > > waiting for reply > > Nilesh, > C-DAC, India > > > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > From wgy at altair.com.cn Wed Sep 5 22:21:51 2007 From: wgy at altair.com.cn (wgy@altair.com.cn) Date: Wed Sep 5 22:23:24 2007 Subject: [mvapich-discuss] Verify the application is really running In-Reply-To: <46DD7944.60507@cse.ohio-state.edu> References: <200709040610.l846ADAg006816@xi.cse.ohio-state.edu> <25052.203.193.122.66.1188886967.squirrel@mail.altair.com.cn> <46DD7944.60507@cse.ohio-state.edu> Message-ID: <21268.203.193.122.66.1189045311.squirrel@mail.altair.com.cn> Hello£º I am away from the machine these days but I will do as you request when I can. Thanks. Henry, Wu. | Can you send us the output from | | /usr/local/topspin/mpi/mpich/bin/mpicc -v | | This will let us know what compiler and which version of it that you're | using. We do not see this problem when trying in our environment using | a somewhat recent version of gcc. | | Below I pasted line 66 with a few lines of context around it. There is | no for loop here so I'm a bit confused as to why you're getting the | errors that you posted. | | | int main(int argc, char *argv[]) | { | | int myid, numprocs, i; | int size; | MPI_Status reqstat; | char *s_buf, *r_buf; | int align_size; | | | Also, as a sanity check, can you download the osu_latency.c file again | from | https://mvapich.cse.ohio-state.edu/svn/mpi/mvapich/trunk/osu_benchmarks/osu_latency.c | | to verify that we are referencing the same file. Thanks for your input | and we hope with further information we can solve this compilation issue | that you're having. | | | wgy@altair.com.cn wrote: |> Hello: |> I am quite sure I used the one you referred and got the compile error as |> you can see from the message.. I just renamed osu_latency.c to |> lantency.c |> while uploading... |> Thanks. |> Henry, Wu. |> |> | Are you sure you are using the osu_latency.c file from mvapich web |> | site? Your e-mail indicates about using a `latency.c' file. |> | |> | FYI, the osu_latency.c benchmark (latest version v2.2) is available |> | from the following URL: |> | |> | |> https://mvapich.cse.ohio-state.edu/svn/mpi/mvapich/trunk/osu_benchmarks/osu_latency.c |> | |> | DK |> | |> |> |> |> Hello, Jeff and Dr, Panda: |> |> I get back with the following resluts and further ques: |> |> 1)latency test reslut: |> |> the latency.c I downloaded from mvapich website can be complied in |> the |> |> cluster with errors: |> |> [radioss@hpc-node-01 job1]$ /usr/local/topspin/mpi/mpich/bin/mpicc |> |> latency.c -o lat |> |> latency.c:66: error: syntax error before "for" |> |> latency.c:68: error: `i' undeclared here (not in a function) |> |> latency.c:68: warning: data definition has no type or storage class |> |> latency.c:69: error: syntax error before '}' token |> |> latency.c:73: error: `skip_large' undeclared here (not in a function) |> |> latency.c:73: warning: data definition has no type or storage class |> |> latency.c:74: error: syntax error before '}' token |> |> latency.c:76: warning: parameter names (without types) in function |> |> declaration |> |> latency.c:76: warning: data definition has no type or storage class |> |> latency.c:78: error: syntax error before "if" |> |> latency.c:82: error: syntax error before numeric constant |> |> latency.c:82: warning: data definition has no type or storage class |> |> latency.c:83: error: syntax error before numeric constant |> |> latency.c:84: warning: data definition has no type or storage class |> |> latency.c:86: error: initializer element is not constant |> |> latency.c:86: warning: data definition has no type or storage class |> |> latency.c:88: error: syntax error before '}' token |> |> latency.c:92: error: syntax error before numeric constant |> |> latency.c:92: warning: data definition has no type or storage class |> |> latency.c:98: error: `t_start' undeclared here (not in a function) |> |> latency.c:98: error: `loop' undeclared here (not in a function) |> |> latency.c:98: warning: data definition has no type or storage class |> |> latency.c:99: error: syntax error before string constant |> |> latency.c:99: warning: conflicting types for built-in function |> 'fprintf' |> |> latency.c:99: warning: data definition has no type or storage class |> |> latency.c:104: warning: data definition has no type or storage class |> |> latency.c:105: error: syntax error before "return" |> |> latency.c:107:2: warning: no newline at end of file |> |> latency.c:68: error: storage size of `r_buf' isn't know |> |> |> |> I had to use mpi_latency.c shipped with mvapich in the cluster and |> got |> |> the |> |> following latency test results. |> |> |> |> [radioss@hpc-node-01 job1]$ |> /usr/local/topspin/mpi/mpich/bin/mpirun_rsh |> |> -np 2 -hostfile appfile ./lat 10000 1 |> |> 1 6.288650 |> |> [radioss@hpc-node-01 job1]$ |> /usr/local/topspin/mpi/mpich/bin/mpirun_rsh |> |> -np 2 -hostfile appfile ./lat 10000 4 |> |> 4 6.410350 |> |> while Topspin's Host-Side Drivers User Guide for Linux Release 3.1.0 |> |> gives |> |> the following latency test figure as an example: |> |> [root@qa-bc1-blade2 root]# |> /usr/local/topspin/mpi/mpich/bin/mpirun_ssh |> |> -np |> |> 2 qabc1- |> |> blade2 qa-bc1-blade3 /usr/local/topspin/mpi/mpich/bin/mpi_latency |> 10000 |> |> 1 |> |> 1 6.684000 |> |> 2) Jeff Squyres once asked me: |> |> >> I have 4-cores nodes here.. |> |> >> I would expect to run it as: |> |> >> /usr/local/topspin/mpi/mpich/bin/mpirun_ssh -np 2 -hostfile hosts |> |> |> |> >^^ Is that the right path? Or is it "mvapich"? Regardless, I think |> |> wherever you find mpirun_ssh under /usr/local/topspin/mpi is probably |> |> the right one. |> |> the path is right, and I am pretty sure it is mavapich because: |> |> i)rpm -qf /usr/local/topspin/mpi/mpich/bin/mpirun_ssh gives: |> |> topspin-ib-mpi-rhel4-3.2.0-118 |> |> ii)[radioss@hpc-node-01 local]$ |> |> /usr/local/topspin/mpi/mpich/bin/mpirun_rsh -v |> |> OSU MVAPICH VERSION 0.9.5-SingleRail |> |> |> |> 3)when I try to use hp mpi 2.2.5 over the IB network I got the |> |> following: |> |> [radioss@hpc-node-01 job1]$ /opt/hpmpi/bin/mpirun -stdio=i0 |> |> -cpu_bind=cyclic -VAPI -f appfile < PFTANKD01 |> |> dlopen test for MPI_ICLIB_VAPI__VAPI_MAIN could not open libs in list |> |> libmtl_common.so libmpga.so libmosal.so libvapi.so: |> |> /usr/local/topspin/lib64/libmosal.so: undefined symbol: |> pthread_create |> |> dlopen test for MPI_ICLIB_VAPI__VAPI_CISCO could not open libs in |> list |> |> libpthread.so libmosal.so libvapi.so: |> /usr/lib64/libpthread.so: |> |> invalid ELF header |> |> mpid: MPI BUG: VAPI requested but not available |> |> what does it probably indicate? anything is wrong with the IB |> |> configuration? |> |> |> |> RPM packages installed there: |> |> [radioss@hpc-node-01 job1]$ rpm -qa|grep topspin |> |> topspin-ib-rhel4-3.2.0-118 |> |> topspin-ib-mpi-rhel4-3.2.0-118 |> |> topspin-ib-mod-rhel4-2.6.9-42.ELsmp-3.2.0-118 |> |> |> |> 4)You guys suggested me to use HP MPI (no native mvapich) and OFED IB |> |> stack if possible. |> |> now I have some questiosn hope you can have a quick comment or refer |> me |> |> some website link so that I can read through: |> |> i)how to verify which IB stack is used here, OFED or Cisco/Topspin IB |> |> stack? what's the advantages of OFED IB stack over Cisco/Topspin IB |> |> stack? |> |> ii)what's the advatages of HP HPI over "native mvapich"? what means |> by |> |> "native mvapich"? the one shipped with Cisco/Topspin? is it enough to |> |> upgrade mvapich to the latest one which is availabel on mvapich |> website? |> |> |> |> Thanks a lot for all of you for your kindly help! |> |> |> |> Henry, Wu. |> |> |> |> |> |> |> |> |> |> | On Aug 29, 2007, at 12:54 PM, wgy@altair.com.cn wrote: |> |> | |> |> |> Yes, I think I used mavapich shipped with Topspin, but I am not |> sure |> |> |> unless I know how to verify it. |> |> | |> |> | If it's in the /usr/local/topspin directory, it's the Topspin |> (later |> |> | Cisco) MVAPICH. |> |> | |> |> |> about latency test, I downloaded |> |> |> https://mvapich.cse.ohio-state.edu/svn/mpi/mvapich/trunk/ |> |> |> osu_benchmarks/osu_latency.c |> |> |> and will compile it to run a benchmark. Can you please tell me how |> |> |> should |> |> |> I run it? how many nodes should be used and how many cpus should |> be |> |> |> involved? |> |> | |> |> | You typically run it with 2 MPI processes; one on each host. It |> |> | measures the MPI network latency between those two hosts. |> |> | |> |> |> I have 4-cores nodes here.. |> |> |> I would expect to run it as: |> |> |> /usr/local/topspin/mpi/mpich/bin/mpirun_ssh -np 2 -hostfile hosts |> |> | |> |> | ^^ Is that the right path? Or is it "mvapich"? Regardless, I |> think |> |> | wherever you find mpirun_ssh under /usr/local/topspin/mpi is |> probably |> |> | the right one. |> |> | |> |> |> osu_latency.o |> |> | |> |> | Is your executable really named osu_latency.o? That's uncommon. |> |> | Regardless, run the executable that you got when you compiled |> |> | osu_latency.c with mpicc. |> |> | |> |> |> and include the following in the hosts file |> |> |> hpc-node-01 |> |> |> hpc-node-02 |> |> | |> |> | Sounds right. I'm not an MVAPICH expert, though -- so I defer to |> the |> |> | maintainers here on this list for the finer details... |> |> | |> |> |> Is it right? |> |> |> Thanks a lot, I am really a newbie with Infiniband.... |> |> | |> |> | If this is your own system, I do want to stress that OFED is really |> |> | the way to go with HPC InfiniBand installations these days. The |> |> | MPI's that are included are much more recent, and all new |> development |> |> | work is happening in the OFED arena. |> |> | |> |> | I recommend that you upgrade if you can. |> |> | |> |> | |> |> |> Henry, Wu |> |> |> |> |> |> |> |> |> | On Aug 29, 2007, at 12:25 PM, wgy@altair.com.cn wrote: |> |> |> | |> |> |> |> Hello, Jeff: |> |> |> |> The mvapich version is OSU mvapich0.95. |> |> |> |> does it mean that it is Cisco IB stack and therefor the |> |> application |> |> |> |> I run |> |> |> |> with mvapich is really running over IB network? |> |> |> | |> |> |> | The version of MVAPICH, by itself, does not mean that it is or |> is |> |> |> not |> |> |> | running over IB. |> |> |> | |> |> |> | What *implies* that you are running over IB is: |> |> |> | |> |> |> | - You implied that you are using the MVAPICH shipped with the |> |> |> Topspin |> |> |> | IB stack (which is not OFED). Is that correct? |> |> |> | - I *believe* that the Topspin MVAPICH did not have TCP support |> |> |> | compiled into it (Topspin was before my time, but I am pretty |> sure |> |> |> | that the Cisco MVAPICH shipped with the Cisco IB stack does not) |> |> |> | |> |> |> | What would *prove* that you are using IB (vs. gige) is: |> |> |> | |> |> |> | - Run a simple latency test, as Dr. Panda suggested. Your |> latency |> |> |> | should be single-digit microseconds (exact numbers depend on |> your |> |> |> | hardware -- this might be all older stuff since you mentioned |> |> |> | "Topspin", not "Cisco"; Topspin was acquired by Cisco quite a |> while |> |> |> | ago...). If your latency is much higher than that (e.g., 50 |> us), |> |> |> | you're using gige. |> |> |> | |> |> |> | |> |> |> | |> |> |> |> Thanks. |> |> |> |> |> |> |> |> Henry, Wu. |> |> |> |> | In addition to what Dr. Panda said, Cisco recommends that all |> |> HPC |> |> |> |> | customers upgrade to the OFED IB driver stack if possible |> (some |> |> |> |> | customers cannot upgrade for various reasons). FWIW: all new |> |> |> HPC/ |> |> |> |> MPI |> |> |> |> | work is occurring in the OFED arena. |> |> |> |> | |> |> |> |> | I bring this up because you specifically mention Topspin |> |> |> Infiniband, |> |> |> |> | which I'm *assuming* is the Cisco IB stack (not the OFED IB |> |> |> stack), |> |> |> |> | and is therefore shipping with a somewhat older version of |> |> |> MVAPICH |> |> |> |> | that was derived from the OSU MVAPICH. The Cisco MVAPICH |> should |> |> |> |> only |> |> |> |> | be compiled with IB support enabled; a simple latency test |> |> should |> |> |> |> | prove that you're running over IB and not ethernet. |> |> |> |> | |> |> |> |> | Much more recent versions of MPI implementations are included |> |> |> with |> |> |> |> | the OFED stack (Cisco provides binary distributions of OFED |> on |> |> |> |> | www.cisco.com). |> |> |> |> | |> |> |> |> | |> |> |> |> | On Aug 29, 2007, at 11:44 AM, Dhabaleswar Panda wrote: |> |> |> |> | |> |> |> |> |> |> |> |> |> |> |> |> |> |> |> On Wed, 29 Aug 2007 wgy@altair.com.cn wrote: |> |> |> |> |> |> |> |> |> |>> Hello, list: |> |> |> |> |>> It might be a silly questions but I wonder how to verify |> run |> |> |> with |> |> |> |> |>> mvapich |> |> |> |> |>> (come with Topspin Infiniband) over Infiniband, NOT |> Gigabite |> |> |> |> network. |> |> |> |> |>> Is there an option to force mvapich to use IB network |> |> otherwise |> |> |> |> |>> just exits? |> |> |> |> |> |> |> |> |> |> MVAPICH has several underlying interfaces: Gen2, uDAPL, |> VAPI, |> |> |> TCP/ |> |> |> |> |> IP and |> |> |> |> |> shared memory. Please take a look at the user guide |> |> |> (available from |> |> |> |> |> mvapich project page) to see the differences and |> capabilities |> |> of |> |> |> |> these |> |> |> |> |> interfaces. Gen2 interface (corresponding to OFED) will give |> |> you |> |> |> |> |> the best performance and scalability. If you have OFED stack |> |> |> |> |> installed, |> |> |> |> |> you should be able to configure mvapich to run over Gen2 |> |> |> interface |> |> |> |> |> (as per the instructions indicated in the user guide). |> During |> |> |> OFED |> |> |> |> |> installation, you can also select mvapich from the package. |> |> |> |> |> |> |> |> |> |> On your existing installation, you can also run OSU |> benchmarks |> |> |> |> (such |> |> |> |> |> as OSU latency). If you get latency number in the range of |> 2~4 |> |> |> |> |> microsec |> |> |> |> |> for short messages (say 4 bytes), it is already running over |> |> the |> |> |> |> |> native |> |> |> |> |> IB. |> |> |> |> |> |> |> |> |> |> Hope this helps. |> |> |> |> |> |> |> |> |> |> DK |> |> |> |> |> |> |> |> |> |>> Thanks for your suggestion. |> |> |> |> |>> Rdgs. |> |> |> |> |>> Henry, Wu |> |> |> |> |>> |> |> |> |> |>> _______________________________________________ |> |> |> |> |>> mvapich-discuss mailing list |> |> |> |> |>> mvapich-discuss@cse.ohio-state.edu |> |> |> |> |>> |> |> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss |> |> |> |> |>> |> |> |> |> |> |> |> |> |> |> _______________________________________________ |> |> |> |> |> mvapich-discuss mailing list |> |> |> |> |> mvapich-discuss@cse.ohio-state.edu |> |> |> |> |> |> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss |> |> |> |> | |> |> |> |> | |> |> |> |> | -- |> |> |> |> | Jeff Squyres |> |> |> |> | Cisco Systems |> |> |> |> | |> |> |> |> | |> |> |> | |> |> |> | |> |> |> | -- |> |> |> | Jeff Squyres |> |> |> | Cisco Systems |> |> |> | |> |> |> | |> |> | |> |> | |> |> | -- |> |> | Jeff Squyres |> |> | Cisco Systems |> |> | |> |> | |> |> |> |> |> |> _______________________________________________ |> |> mvapich-discuss mailing list |> |> mvapich-discuss@cse.ohio-state.edu |> |> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss |> |> |> | |> | |> |> |> _______________________________________________ |> mvapich-discuss mailing list |> mvapich-discuss@cse.ohio-state.edu |> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss | | | -- | Jonathan Perkins | http://www.cse.ohio-state.edu/~perkinjo | From pasha at dev.mellanox.co.il Thu Sep 6 12:23:51 2007 From: pasha at dev.mellanox.co.il (Pavel Shamis (Pasha)) Date: Thu Sep 6 12:21:34 2007 Subject: [mvapich-discuss] Printing DEBUG_PRINT mesges In-Reply-To: References: Message-ID: <46E02997.2070909@dev.mellanox.co.il> You may to try to use V_PRINT macro (located in mpid/ch_gen2/ibverbs_header.h) it is disabled by default but if you will rebuild the code with "-DVIADEV_DEBUG" flag it will enable the macro and will create the debug build - as result the performance will be very bad. In the code when you call the macro you may setup on witch "debug level" you want to print the message: DEBUG00 - DEBUG03 For setting specific debug level during run you may use VIADEV_DEBUG_LEVEL parameter. For example - VIADEV_DEBUG_LEVEL=1 will say to mvapich to run in level 1 of debug. But As Matt say before all this stuff we use for internal development and you should understand very well what are you doing. Pasha. Matthew Koop wrote: > Nilesh, > > DEBUG_PRINT is used for internal development, so we cannot guarantee that > the messages printed will be useful to others (or correct). As such, we do > not support this option for the end-user. > > Is there a particular problem you are experiencing that we can address? > > Matt > > > On Mon, 3 Sep 2007, Nilesh Awate wrote: > > >> Hi All, >> >> I'm using mvapich2-1.0-beta with ofed1.2 udapl interface. >> >> I want to print DEBUG_PRINT masseges for that i've included DDEBUG flag in >> make.mvapich.udapl make file >> >> but while making its give error in that perticular DEBUG_PRINT statements >> "variables not defined" >> >> is there anything else setting of flags ... >> >> waiting for reply >> >> Nilesh, >> C-DAC, India >> >> >> _______________________________________________ >> mvapich-discuss mailing list >> mvapich-discuss@cse.ohio-state.edu >> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss >> >> > > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > > From perkinjo at cse.ohio-state.edu Thu Sep 6 15:31:34 2007 From: perkinjo at cse.ohio-state.edu (Jonathan L. Perkins) Date: Thu Sep 6 15:31:59 2007 Subject: [mvapich-discuss] suggested minor feature In-Reply-To: <46DEC7BC.7040706@hpcapplications.com> References: <46DEC7BC.7040706@hpcapplications.com> Message-ID: <46E05596.3080303@cse.ohio-state.edu> Mark Potts wrote: > Hi, > Despite the many issues I've raised about MVAPICH job cleanup and > timeouts (all resolved now it appears), I'd like to raise another > related issue -- a suggestion. > > We've found that a job that correctly has all processes call > MPI_Finalize() at the end of their communications stages, can > not permit any processes to terminate if it is desired for even > a single thread to continue to work. That is, after MPI_Finalize() > is called and any processes correctly terminate there is only a > 10 second window in which any remaining processes will be allowed > to run before mpirun_rsh kills the remaining children. This > presents a problem for codes that naturally complete the job's > task in serial mode or codes in which debugging of a process is > needed after MPI_Finalize(). > > The suggestion would be: > to provide the timeout period (currently 10 seconds) as a > VIADEV_* env variable, with default of 10, which users could > then modify when 10 seconds was too little time for a remaining > process. By the same token this env variable could be used > to trim the timeout period to a smaller value, when a user > deemed 10 seconds to not be agressive enough. > regards, Mark: In light of your suggestion we took a look at how mpirun_rsh handles the termination of its children processes. With a small change in the semantics we managed to remove the "timeout" entirely. We now allow processes that exit cleanly to not effect the lifespan of other processes. In the case that a process doesn't exit cleanly, the other processes will still destroyed like normal. Can you try out the attached patch and let us know whether everything works in the way that an end user would expect? We also welcome any further suggestions. Thanks. -- Jonathan Perkins http://www.cse.ohio-state.edu/~perkinjo -------------- next part -------------- A non-text attachment was scrubbed... Name: mpirun_rsh.patch Type: text/x-patch Size: 1589 bytes Desc: not available Url : http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20070906/1da828ed/mpirun_rsh.bin From potts at hpcapplications.com Thu Sep 6 15:59:37 2007 From: potts at hpcapplications.com (Mark Potts) Date: Thu Sep 6 16:00:48 2007 Subject: [mvapich-discuss] suggested minor feature In-Reply-To: <46E05596.3080303@cse.ohio-state.edu> References: <46DEC7BC.7040706@hpcapplications.com> <46E05596.3080303@cse.ohio-state.edu> Message-ID: <46E05C29.90006@hpcapplications.com> Jonathan, It'll take me a few days to be able to get to look at this. In the meantime you could probably save me a little searching by telling me if this patch is to the baseline mvapich-0.9.9 mpirun_rsh.c or to that same routine with the later patches. You guys have provided us with several patches that relate to timeouts and job cleanup in the past few weeks and I want to assure applying the patch to the right bits. Thanks. regards, Jonathan L. Perkins wrote: > Mark Potts wrote: >> Hi, >> Despite the many issues I've raised about MVAPICH job cleanup and >> timeouts (all resolved now it appears), I'd like to raise another >> related issue -- a suggestion. >> >> We've found that a job that correctly has all processes call >> MPI_Finalize() at the end of their communications stages, can >> not permit any processes to terminate if it is desired for even >> a single thread to continue to work. That is, after MPI_Finalize() >> is called and any processes correctly terminate there is only a >> 10 second window in which any remaining processes will be allowed >> to run before mpirun_rsh kills the remaining children. This >> presents a problem for codes that naturally complete the job's >> task in serial mode or codes in which debugging of a process is >> needed after MPI_Finalize(). >> >> The suggestion would be: >> to provide the timeout period (currently 10 seconds) as a >> VIADEV_* env variable, with default of 10, which users could >> then modify when 10 seconds was too little time for a remaining >> process. By the same token this env variable could be used >> to trim the timeout period to a smaller value, when a user >> deemed 10 seconds to not be agressive enough. >> regards, > > Mark: > In light of your suggestion we took a look at how mpirun_rsh handles the > termination of its children processes. With a small change in the > semantics we managed to remove the "timeout" entirely. > > We now allow processes that exit cleanly to not effect the lifespan of > other processes. In the case that a process doesn't exit cleanly, the > other processes will still destroyed like normal. > > Can you try out the attached patch and let us know whether everything > works in the way that an end user would expect? We also welcome any > further suggestions. Thanks. > -- *********************************** >> Mark J. Potts, PhD >> >> HPC Applications Inc. >> phone: 410-992-8360 Bus >> 410-313-9318 Home >> 443-418-4375 Cell >> email: potts@hpcapplications.com >> potts@excray.com *********************************** From perkinjo at cse.ohio-state.edu Thu Sep 6 16:53:30 2007 From: perkinjo at cse.ohio-state.edu (Jonathan L. Perkins) Date: Thu Sep 6 16:53:56 2007 Subject: [mvapich-discuss] suggested minor feature In-Reply-To: <46E05C29.90006@hpcapplications.com> References: <46DEC7BC.7040706@hpcapplications.com> <46E05596.3080303@cse.ohio-state.edu> <46E05C29.90006@hpcapplications.com> Message-ID: <46E068CA.7050204@cse.ohio-state.edu> Mark Potts wrote: > Jonathan, > It'll take me a few days to be able to get to look at this. > > In the meantime you could probably save me a little searching by > telling me if this patch is to the baseline mvapich-0.9.9 > mpirun_rsh.c or to that same routine with the later patches. You > guys have provided us with several patches that relate to > timeouts and job cleanup in the past few weeks and I want to > assure applying the patch to the right bits. This patch should be applied to the already patched version of mpirun_rsh.c. If you would like a patch against the mvapich-0.9.9 release version of mpirun_rsh.c just let me know. > Thanks. > > regards, > > Jonathan L. Perkins wrote: >> Mark Potts wrote: >>> Hi, >>> Despite the many issues I've raised about MVAPICH job cleanup and >>> timeouts (all resolved now it appears), I'd like to raise another >>> related issue -- a suggestion. >>> >>> We've found that a job that correctly has all processes call >>> MPI_Finalize() at the end of their communications stages, can >>> not permit any processes to terminate if it is desired for even >>> a single thread to continue to work. That is, after MPI_Finalize() >>> is called and any processes correctly terminate there is only a >>> 10 second window in which any remaining processes will be allowed >>> to run before mpirun_rsh kills the remaining children. This >>> presents a problem for codes that naturally complete the job's >>> task in serial mode or codes in which debugging of a process is >>> needed after MPI_Finalize(). >>> >>> The suggestion would be: >>> to provide the timeout period (currently 10 seconds) as a >>> VIADEV_* env variable, with default of 10, which users could >>> then modify when 10 seconds was too little time for a remaining >>> process. By the same token this env variable could be used >>> to trim the timeout period to a smaller value, when a user >>> deemed 10 seconds to not be agressive enough. >>> regards, >> >> Mark: >> In light of your suggestion we took a look at how mpirun_rsh handles >> the termination of its children processes. With a small change in the >> semantics we managed to remove the "timeout" entirely. >> >> We now allow processes that exit cleanly to not effect the lifespan of >> other processes. In the case that a process doesn't exit cleanly, the >> other processes will still destroyed like normal. >> >> Can you try out the attached patch and let us know whether everything >> works in the way that an end user would expect? We also welcome any >> further suggestions. Thanks. >> > -- Jonathan Perkins http://www.cse.ohio-state.edu/~perkinjo From singh.jasjit at yahoo.co.in Fri Sep 7 00:46:09 2007 From: singh.jasjit at yahoo.co.in (jasjit singh) Date: Fri Sep 7 00:46:40 2007 Subject: [mvapich-discuss] temporary files causing problem In-Reply-To: <254847a254aa22.254aa22254847a@osu.edu> Message-ID: <255280.17585.qm@web94007.mail.in2.yahoo.com> Hi Lei, Thanks for the info, i will try with the latest version. One thing i want to mention is when i run mpiexec, it doesn't exit cleanly. it is actually aborted due to some DTO failure. so is "abrupt close" the reason that the temporary files are not being removed ? whether a clean exit will also remove temporary files ? regards, Jasjit LEI CHAI wrote: Hi Jasjit, We have a few bug fixes and code enhancements in the latest releases. Could you please download mvapich2-0.9.8p3 or mvapich2-1.0 beta and try? http://mvapich.cse.ohio-state.edu/download/mvapich2/ Thanks, Lei Content-Type: multipart/alternative; boundary="0-537041755-1188971502=:35648" Content-Transfer-Encoding: 8bit --0-537041755-1188971502=:35648 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit hi I have been running uDAPL code with mvapich2-0.9.8p2. Corresponding makefile (make.mvapich.udapl) has all the default CFLAGS except ONE_SIDED. It ends up creating some temporary files in /tmp directory. But after mpiexec ends, it doesn't delete these temporary files. These files are not deleted even if i reboot the system. I run mpiexec infinitely. So i get my disks full after 10-12 hrs on all the four nodes which i am running the application on. Could anybody help me in this regard ? Is there any way out to get rid of these temporary files ? Regards, Jasjit Singh --------------------------------- Yahoo! Answers - Get better answers from someone who knows. Tryit now. --0-537041755-1188971502=:35648 Content-Type: text/html; charset=iso-8859-1 Content-Transfer-Encoding: 8bit hi I have been running uDAPL code with mvapich2-0.9.8p2. Corresponding makefile (make.mvapich.udapl) has all the default CFLAGS except ONE_SIDED. It ends up creating some temporary files in /tmp directory. But after mpiexec ends, it doesn't delete these temporary files. These files are not deleted even if i reboot the system. I run mpiexec infinitely. So i get my disks full after 10-12 hrs on all the four nodes which i am running the application on. Could anybody help me in this regard ? Is there any way out to get rid of these temporary files ? Regards, Jasjit Singh --------------------------------- Yahoo! Answers - Get better answers from someone who knows. href="http://uk.answers.yahoo.com/;_ylc=X3oDMTEydmViNG02BF9TAzIxMTQ3MTcxOTAEc2VjA21haWwEc2xrA3RhZ2xpbmU">Try it now. --0-537041755-1188971502=:35648-- _______________________________________________ mvapich-discuss mailing list mvapich-discuss@cse.ohio-state.edu http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss --------------------------------- For ideas on reducing your carbon footprint visit Yahoo! For Good this month. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20070907/4a1f6130/attachment-0001.html From koop at cse.ohio-state.edu Fri Sep 7 00:51:57 2007 From: koop at cse.ohio-state.edu (Matthew Koop) Date: Fri Sep 7 00:52:23 2007 Subject: [mvapich-discuss] Printing DEBUG_PRINT mesges In-Reply-To: <46E02997.2070909@dev.mellanox.co.il> Message-ID: Just to be clear, what Pasha has mentioned (-DVIADEV_DEBUG) is for the ch_gen2 device of MVAPICH. You appear to be using the uDAPL device of MVAPICH2, which has different flags. Matt On Thu, 6 Sep 2007, Pavel Shamis (Pasha) wrote: > You may to try to use V_PRINT macro (located in > mpid/ch_gen2/ibverbs_header.h) > it is disabled by default but if you will rebuild the code with > "-DVIADEV_DEBUG" flag it will enable the macro and will create the debug > build - as result the performance > will be very bad. > > In the code when you call the macro you may setup on witch "debug level" > you want to print the message: DEBUG00 - DEBUG03 > > For setting specific debug level during run you may use > VIADEV_DEBUG_LEVEL parameter. > For example - VIADEV_DEBUG_LEVEL=1 will say to mvapich to run in level > 1 of debug. > > But As Matt say before all this stuff we use for internal development > and you should understand very well what are you doing. > > Pasha. > > > Matthew Koop wrote: > > Nilesh, > > > > DEBUG_PRINT is used for internal development, so we cannot guarantee that > > the messages printed will be useful to others (or correct). As such, we do > > not support this option for the end-user. > > > > Is there a particular problem you are experiencing that we can address? > > > > Matt > > > > > > On Mon, 3 Sep 2007, Nilesh Awate wrote: > > > > > >> Hi All, > >> > >> I'm using mvapich2-1.0-beta with ofed1.2 udapl interface. > >> > >> I want to print DEBUG_PRINT masseges for that i've included DDEBUG flag in > >> make.mvapich.udapl make file > >> > >> but while making its give error in that perticular DEBUG_PRINT statements > >> "variables not defined" > >> > >> is there anything else setting of flags ... > >> > >> waiting for reply > >> > >> Nilesh, > >> C-DAC, India > >> > >> > >> _______________________________________________ > >> mvapich-discuss mailing list > >> mvapich-discuss@cse.ohio-state.edu > >> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > >> > >> > > > > _______________________________________________ > > mvapich-discuss mailing list > > mvapich-discuss@cse.ohio-state.edu > > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > > > > > From ibatis2 at 163.com Sun Sep 9 05:18:22 2007 From: ibatis2 at 163.com (jetspeed) Date: Sun Sep 9 05:19:03 2007 Subject: [mvapich-discuss] How to configure IPoIB when using OFED1.2 ? Message-ID: <20070909171822.18c6064a.ibatis2@163.com> Hi all === I installed OFED 1.2.5 on my InfiniBand connected cluster. but It dosn't work. I want the inifinBand network use IP 10.0.0.*, so I configure the ofed_net.conf file as follow LAN_INTERFACE_ib0=eth0 IPADDR_ib0=10.0.0.* NETMASK_ib0=255.255.255.0 NETWORK_ib0=10.0.0.0 BROADCAST_ib0=10.0.0.255 ONBOOT_ib0=1 === when I mpdboot on one machine, I got the message mpdboot_inode01 (handle_mpd_output 359): failed to ping mpd on inode01; recvd output={} and the /tmp/mpd2_logfile_me is logfile for mpd with pid 21779 inode01_32928: mpd_uncaught_except_tb handling: exceptions.TypeError: sequence item 0: expected string, int found /usr/mpi/gcc/mvapich2-0.9.8-15/bin/mpdlib.py 733 handle_ring_listener_connection newsock.correctChallengeResponse = \ /usr/mpi/gcc/mvapich2-0.9.8-15/bin/mpdlib.py 488 handle_active_streams handler(stream,*args) /usr/mpi/gcc/mvapich2-0.9.8-15/bin/mpd.py 266 runmainloop rv = self.streamHandler.handle_active_streams(timeout=8.0) /usr/mpi/gcc/mvapich2-0.9.8-15/bin/mpd.py 240 run self.runmainloop() /usr/mpi/gcc/mvapich2-0.9.8-15/bin/mpd.py 1344 ? mpd.run() === PS: when I start openibd, [root@inode01 ~]# /etc/init.d/openibd start Loading HCA driver and Access Layer: [ OK ] Setting up InfiniBand network interfaces: Bringing up interface ib0: [ OK ] Bringing up interface ib1: [ OK ] Setting up service network . . . [ done ] Setting up VNIC interfaces... Error in parsing VNIC configuration file. Please refer to /var/log/messages. Bringing up VNIC interfaces [FAILED] I don't know where the problem is, anyone has a suggestion? thanks in advance From panda at cse.ohio-state.edu Sun Sep 9 21:44:26 2007 From: panda at cse.ohio-state.edu (Dhabaleswar Panda) Date: Sun Sep 9 21:44:54 2007 Subject: [mvapich-discuss] How to configure IPoIB when using OFED1.2 ? In-Reply-To: <20070909171822.18c6064a.ibatis2@163.com> from "jetspeed" at Sep 09, 2007 05:18:22 PM Message-ID: <200709100144.l8A1iRqF028953@xi.cse.ohio-state.edu> Based on your e-mail, it appears that you are trying to run MVAPICH2 over IPoIB interface (not basic Gen2 interface) with OFED 1.2.5. If this is correct, please refer to section 5.6 of MVAPICH2 user guide. You can download the user guide from mvapich web page. DK > Hi all > > === I installed OFED 1.2.5 on my InfiniBand connected cluster. but It dosn't work. I want the inifinBand network use IP 10.0.0.*, so I configure the ofed_net.conf file as follow > > LAN_INTERFACE_ib0=eth0 > IPADDR_ib0=10.0.0.* > NETMASK_ib0=255.255.255.0 > NETWORK_ib0=10.0.0.0 > BROADCAST_ib0=10.0.0.255 > ONBOOT_ib0=1 > > > === when I mpdboot on one machine, I got the message > > mpdboot_inode01 (handle_mpd_output 359): failed to ping mpd on inode01; recvd output={} > > and the /tmp/mpd2_logfile_me is > > logfile for mpd with pid 21779 > inode01_32928: mpd_uncaught_except_tb handling: > exceptions.TypeError: sequence item 0: expected string, int found > /usr/mpi/gcc/mvapich2-0.9.8-15/bin/mpdlib.py 733 handle_ring_listener_connection newsock.correctChallengeResponse = \ > /usr/mpi/gcc/mvapich2-0.9.8-15/bin/mpdlib.py 488 handle_active_streams > handler(stream,*args) > /usr/mpi/gcc/mvapich2-0.9.8-15/bin/mpd.py 266 runmainloop > rv = self.streamHandler.handle_active_streams(timeout=8.0) > /usr/mpi/gcc/mvapich2-0.9.8-15/bin/mpd.py 240 run > self.runmainloop() > /usr/mpi/gcc/mvapich2-0.9.8-15/bin/mpd.py 1344 ? > mpd.run() > > > === PS: when I start openibd, > > [root@inode01 ~]# /etc/init.d/openibd start > Loading HCA driver and Access Layer: [ OK ] > Setting up InfiniBand network interfaces: > Bringing up interface ib0: [ OK ] > Bringing up interface ib1: [ OK ] > Setting up service network . . . [ done ] > Setting up VNIC interfaces... > Error in parsing VNIC configuration file. Please refer to /var/log/messages. > Bringing up VNIC interfaces [FAILED] > > I don't know where the problem is, anyone has a suggestion? > thanks in advance > > > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > From pasha at dev.mellanox.co.il Mon Sep 10 03:58:16 2007 From: pasha at dev.mellanox.co.il (Pavel Shamis (Pasha)) Date: Mon Sep 10 03:55:49 2007 Subject: [mvapich-discuss] Going for vacation on Sep 11- Oct 9 Message-ID: <46E4F918.7000703@dev.mellanox.co.il> For urgent issues please contact: Ishai - ishai@mellanox.co.il Regards, Pasha From tacke at uni-mainz.de Mon Sep 10 11:55:27 2007 From: tacke at uni-mainz.de (Tacke, Markus) Date: Mon Sep 10 11:55:57 2007 Subject: [mvapich-discuss] MVAPICH2 1.0 bet2 compile error Message-ID: Just started installing mvapich2 from the mvapich2-1.0-beta2.tar.gz tarball. Being impatient (and having 4 CPUs in my machine) I tried compiling it with MAKEFLAGS="-j 6" CC=gcc ./make.mvapich2.ofa the compilation crashed with ...... make[3]: Leaving directory `/usr/local/lc2_mgmt_1/src/mvapich2-1.0/src/binding/f77' make[3]: Entering directory `/usr/local/lc2_mgmt_1/src/mvapich2-1.0/src/binding/f90' rm -f mpifnoext.h gfortran -I. -c mpi.f90 gfortran -I. -c mpi_constants.f90 sed -e 's/^C/\!/g' -e '/EXTERNAL/d' \ -e '/DOUBLE PRECISION/d' ../f77/mpif.h > mpifnoext.h gfortran -I. -c mpi_sizeofs.f90 In file mpi.f90:6 USE MPI_SIZEOFS 1 Fatal Error: Can't open module file 'mpi_sizeofs.mod' for reading at (1): No such file or directory gfortran -I. -c ./mpi_constants.f90 make[3]: *** [mpi.o] Error 1 make[3]: *** Waiting for unfinished jobs.... gfortran -I. -c mpi_sizeofs.f90 make[3]: Leaving directory `/usr/local/lc2_mgmt_1/src/mvapich2-1.0/src/binding/f90' make[2]: *** [all-redirect] Error 1 make[2]: Leaving directory `/usr/local/lc2_mgmt_1/src/mvapich2-1.0/src/binding' make[1]: *** [all-redirect] Error 2 make[1]: Leaving directory `/usr/local/lc2_mgmt_1/src/mvapich2-1.0/src' make: *** [all-redirect] Error 2 gfortran -I. -c ./mpi_constants.f90 make[3]: *** [mpi.o] Error 1 make[3]: *** Waiting for unfinished jobs.... gfortran -I. -c mpi_sizeofs.f90 make[3]: Leaving directory `/usr/local/lc2_mgmt_1/src/mvapich2-1.0/src/binding/f90' make[2]: *** [all-redirect] Error 1 make[2]: Leaving directory `/usr/local/lc2_mgmt_1/src/mvapich2-1.0/src/binding' make[1]: *** [all-redirect] Error 2 make[1]: Leaving directory `/usr/local/lc2_mgmt_1/src/mvapich2-1.0/src' make: *** [all-redirect] Error 2 ... I think that there may be an error in the dependencies in the Makefiles, because without MAKEFLAGS="-j 6" everythings works. (there seems to be a similar problem in the make of mvapich-0.9.9) Markus Tacke From koop at cse.ohio-state.edu Mon Sep 10 15:33:32 2007 From: koop at cse.ohio-state.edu (Matthew Koop) Date: Mon Sep 10 15:33:58 2007 Subject: [mvapich-discuss] How to configure IPoIB when using OFED1.2 ? In-Reply-To: <20070909171822.18c6064a.ibatis2@163.com> Message-ID: Just to check, is your password in ~/.mpd.conf a number or does it contain letters as well? If it is only numbers, try adding some letters -- try "pass234" or something. Matt On Sun, 9 Sep 2007, jetspeed wrote: > Hi all > > === I installed OFED 1.2.5 on my InfiniBand connected cluster. but It > dosn't work. I want the inifinBand network use IP 10.0.0.*, so I > configure the ofed_net.conf file as follow > > LAN_INTERFACE_ib0=eth0 > IPADDR_ib0=10.0.0.* > NETMASK_ib0=255.255.255.0 > NETWORK_ib0=10.0.0.0 > BROADCAST_ib0=10.0.0.255 > ONBOOT_ib0=1 > > > === when I mpdboot on one machine, I got the message > > mpdboot_inode01 (handle_mpd_output 359): failed to ping mpd on inode01; recvd output={} > > and the /tmp/mpd2_logfile_me is > > logfile for mpd with pid 21779 > inode01_32928: mpd_uncaught_except_tb handling: > exceptions.TypeError: sequence item 0: expected string, int found > /usr/mpi/gcc/mvapich2-0.9.8-15/bin/mpdlib.py 733 handle_ring_listener_connection newsock.correctChallengeResponse = \ > /usr/mpi/gcc/mvapich2-0.9.8-15/bin/mpdlib.py 488 handle_active_streams > handler(stream,*args) > /usr/mpi/gcc/mvapich2-0.9.8-15/bin/mpd.py 266 runmainloop > rv = self.streamHandler.handle_active_streams(timeout=8.0) > /usr/mpi/gcc/mvapich2-0.9.8-15/bin/mpd.py 240 run > self.runmainloop() > /usr/mpi/gcc/mvapich2-0.9.8-15/bin/mpd.py 1344 ? > mpd.run() > > > === PS: when I start openibd, > > [root@inode01 ~]# /etc/init.d/openibd start > Loading HCA driver and Access Layer: [ OK ] > Setting up InfiniBand network interfaces: > Bringing up interface ib0: [ OK ] > Bringing up interface ib1: [ OK ] > Setting up service network . . . [ done ] > Setting up VNIC interfaces... > Error in parsing VNIC configuration file. Please refer to /var/log/messages. > Bringing up VNIC interfaces [FAILED] > > I don't know where the problem is, anyone has a suggestion? > thanks in advance > > > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > From rowland at cse.ohio-state.edu Mon Sep 10 19:26:22 2007 From: rowland at cse.ohio-state.edu (Shaun Rowland) Date: Mon Sep 10 19:26:52 2007 Subject: [mvapich-discuss] libmpichf90.so? In-Reply-To: <200709041658.56311.ttelford.groups@gmail.com> References: <200709041658.56311.ttelford.groups@gmail.com> Message-ID: <46E5D29E.6020208@cse.ohio-state.edu> Troy Telford wrote: > When building MVAPICH 0.9.9, I notice that 'libmpichf90.a' is built, but I > can't find a corresponding 'libmpichf90.so' > > This is a problem for me as I have a codebase that is expecting to link > against libmpichf90.so (ie. a shared lib rather than a static lib). > > Is there a way to get a shared library for libmpichf90? > > Was there _ever_ a way to get a shared library for libmpichf90 (and it is just > disabled?) It isn't disabled. The MPICH base does not build the F90 library as shared. I've taken a look at the top-level Makefile.in that builds the Fortran 77 shared library, and even though it builds a shared library, it still puts the farg.o initf.o objects into a static library named libfmpichfarg.a. I believe the note in the Makefile.in relevant to this says: # The ar at the end puts the Fortran MPI_Init and the routine that # it calls to get getarg and iargc into a separate library. This # is needed for shared lib support # Note that if there is no Fortran support, then we don't build the farg # library I am not sure what can be done about this right now. It might be possible to build the Fortran 90 library in a similar way, but the excluded farg90.o would probably need to be dealt with in some similar way. We can try and look into this, but I don't have a solution at the moment unfortunately. -- Shaun Rowland rowland@cse.ohio-state.edu http://www.cse.ohio-state.edu/~rowland/ From singh.jasjit at yahoo.co.in Tue Sep 11 10:39:37 2007 From: singh.jasjit at yahoo.co.in (jasjit singh) Date: Tue Sep 11 10:40:10 2007 Subject: [mvapich-discuss] Problem with more number of processes Message-ID: <887466.97647.qm@web94012.mail.in2.yahoo.com> Hi I have been running MVAPICH 1.0 over OFED 1.2 uDAPL interface on four nodes. i ran 64 processes, that came out to be 16 processes per node. it ran finely. but after increasing the number of processes further, i started getting error. here are some of the final lines of the output i got when i ran 68 processes on 4 nodes i.e 17 processes per node [rdma_udapl_init.c:1875] error(-2147024846): Could not reset ep rank 58 in job 12 in05_33381 caused collective abort of all ranks exit status of rank 58: return code 1 [rdma_udapl_init.c:1875] error(-2147024849): Could not reset ep [rdma_udapl_init.c:1875] error(-2147024849): Could not reset ep [rdma_udapl_init.c:1875] error(-2147024849): Could not reset ep rank 57 in job 12 in05_33381 caused collective abort of all ranks exit status of rank 57: killed by signal 9 [rdma_udapl_init.c:1875] error(-2147024849): Could not reset ep and here is the same when i ran 200 processes i.e 50 processes per node hello: dapl/common/dapl_evd_util.c:1012: dapli_evd_cqe_to_event: Assertion `(((void *)0) != cookie)' failed. hello: dapl/common/dapl_evd_util.c:1012: dapli_evd_cqe_to_event: Assertion `(((void *)0) != cookie)' failed. hello: dapl/common/dapl_evd_util.c:1012: dapli_evd_cqe_to_event: Assertion `(((void *)0) != cookie)' failed. rank 158 in job 16 in05_36664 caused collective abort of all ranks exit status of rank 158: killed by signal 9 [rdma_udapl_init.c:1875] error(-2147024849): Could not reset ep rank 86 in job 16 in05_36664 caused collective abort of all ranks exit status of rank 86: killed by signal 9 rank 46 in job 16 in05_36664 caused collective abort of all ranks exit status of rank 46: killed by signal 9 rank 6 in job 16 in05_36664 caused collective abort of all ranks exit status of rank 6: killed by signal 9 Could anybody please tell why increasing the number of processes results in an absurd behaviour ? Is any limit affecting this run, that needs to be changed ? What is the solution to get more number of processes run successfully ? thanks, Jasjit Singh --------------------------------- For ideas on reducing your carbon footprint visit Yahoo! For Good this month. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20070911/c532619d/attachment-0001.html From chai.15 at osu.edu Tue Sep 11 15:11:23 2007 From: chai.15 at osu.edu (LEI CHAI) Date: Tue Sep 11 15:11:53 2007 Subject: [mvapich-discuss] Problem with more number of processes Message-ID: <31ca3b131cd828.31cd82831ca3b1@osu.edu> Hi Jasjit, May we ask how many processors do you have on each node? Since 16 or 50 processes per node seems large :-) If you are running more processes than processors (oversubscription) then it needs `blocking' support to get good performance, which we have for OFED-gen2 layer, but not for the uDAPL layer yet. So we suggest you do not run it in oversubscription mode. And finally, could you try to increase the on demand connection mode threshold: $ mpiexec -n 64 -env MV2_ON_DEMAND_THRESHOLD 1024 ./a.out Thanks, Lei Content-Type: multipart/alternative; boundary="0-409717822-1189521577=:97647" Content-Transfer-Encoding: 8bit --0-409717822-1189521577=:97647 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Hi I have been running MVAPICH 1.0 over OFED 1.2 uDAPL interface on four nodes. i ran 64 processes, that came out to be 16 processes per node. it ran finely. but after increasing the number of processes further, i started getting error. here are some of the final lines of the output i got when i ran 68 processes on 4 nodes i.e 17 processes per node [rdma_udapl_init.c:1875] error(-2147024846): Could not reset ep rank 58 in job 12 in05_33381 caused collective abort of all ranks exit status of rank 58: return code 1 [rdma_udapl_init.c:1875] error(-2147024849): Could not reset ep [rdma_udapl_init.c:1875] error(-2147024849): Could not reset ep [rdma_udapl_init.c:1875] error(-2147024849): Could not reset ep rank 57 in job 12 in05_33381 caused collective abort of all ranks exit status of rank 57: killed by signal 9 [rdma_udapl_init.c:1875] error(-2147024849): Could not reset ep and here is the same when i ran 200 processes i.e 50 processes per node hello: dapl/common/dapl_evd_util.c:1012: dapli_evd_cqe_to_event: Assertion `(((void *)0) != cookie)' failed. hello: dapl/common/dapl_evd_util.c:1012: dapli_evd_cqe_to_event: Assertion `(((void *)0) != cookie)' failed. hello: dapl/common/dapl_evd_util.c:1012: dapli_evd_cqe_to_event: Assertion `(((void *)0) != cookie)' failed. rank 158 in job 16 in05_36664 caused collective abort of all ranks exit status of rank 158: killed by signal 9 [rdma_udapl_init.c:1875] error(-2147024849): Could not reset ep rank 86 in job 16 in05_36664 caused collective abort of all ranks exit status of rank 86: killed by signal 9 rank 46 in job 16 in05_36664 caused collective abort of all ranks exi