From patrice.martinez at bull.net Mon Oct 1 04:09:35 2007 From: patrice.martinez at bull.net (Patrice Martinez) Date: Mon Oct 1 08:32:15 2007 Subject: [mvapich-discuss] Problem with linpack/mvapich2/BLCR Message-ID: <4700AB3F.7030909@bull.net> An HTML attachment was scrubbed... URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20071001/34971914/attachment-0001.html From mamidala at cse.ohio-state.edu Mon Oct 1 14:21:12 2007 From: mamidala at cse.ohio-state.edu (amith rajith mamidala) Date: Mon Oct 1 14:21:43 2007 Subject: [mvapich-discuss] collectives fail under mvapich2-1.0 (fwd) In-Reply-To: <46FC2746.7010508@ualberta.ca> Message-ID: Hi Edmund, We were able to run the 12 process test for collectives on 3 nodes. Can you provide us some details as to how the processes were launched? e.g. block or cyclic or any other distrubution. thanks, -Amith. On Thu, 27 Sep 2007, Edmund Sumbar wrote: > Edmund Sumbar wrote: > > I'll try running the SKaMPI tests again. Maybe > > I missed something, as with the mvapich2 tests. > > I recompiled and reran SKaMPI pt2pt, coll, > onesided, and mmisc tests on 3 nodes, 4 > processors per node. > > pt2pt and mmisc succeeded, while coll and > onesided failed (stalled). Any ideas? > > For what it's worth, here are the tails of > the output files... > > > $ tail coll_ib-3x4.sko > # SKaMPI Version 5.0 rev. 191 > > begin result "MPI_Bcast-nodes-short" > nodes= 2 1024 3.8 0.2 39 2.9 3.6 > nodes= 3 1024 6.6 0.4 38 4.0 6.3 4.9 > nodes= 4 1024 9.2 0.2 32 4.6 7.7 7.6 8.6 > > > $ tail onesided_ib-3x4.sko > cpus= 8 4 50051.7 1.3 8 50051.7 --- --- --- --- --- --- --- > cpus= 9 4 50051.5 0.7 8 50051.5 --- --- --- --- --- --- --- --- > cpus= 10 4 50047.7 1.6 8 50047.7 --- --- --- --- --- --- --- > --- --- > cpus= 11 4 50058.2 2.7 8 50058.2 --- --- --- --- --- --- --- > --- --- --- > cpus= 12 4 50074.3 2.8 8 50074.3 --- --- --- --- --- --- --- > --- --- --- --- > end result "MPI_Win_wait delayed,small" > # duration = 9.00 sec > > begin result "MPI_Win_wait delayed without MPI_Put" > cpus= 2 1048576 50025.0 1.4 8 50025.0 --- > > > -- > Ed[mund [Sumbar]] > AICT Research Support, Univ of Alberta > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > From esumbar at ualberta.ca Mon Oct 1 15:12:35 2007 From: esumbar at ualberta.ca (Edmund Sumbar) Date: Mon Oct 1 15:15:55 2007 Subject: [mvapich-discuss] collectives fail under mvapich2-1.0 (fwd) In-Reply-To: References: Message-ID: <470146A3.9010508@ualberta.ca> amith rajith mamidala wrote: > We were able to run the 12 process test for collectives on 3 nodes. > Can you provide us some details as to how the processes were launched? > e.g. block or cyclic or any other distrubution. Hi Amith, I've been running the tests as batch jobs through Torque/Maui using the mpiexec program. All parameters are defaults, as far as I know. Typical job script is #!/bin/bash #PBS -S /bin/bash #PBS -l nodes=3:ppn=4 #PBS -l pvmem=1gb #PBS -W x=QOS:test test=coll size=3x4 mpiexec=/usr/local/mpiexec/bin/mpiexec skampi=/scratch/esumbar/mpi-test.d/skampi/mvapich/skampi-5.0.1-r0191/skampi cd $PBS_O_WORKDIR $mpiexec $skampi -i ${test}.ski -o ${test}_ib-${size}.sko Mpiexec details... $ /usr/local/mpiexec/bin/mpiexec --version Version 0.81, configure options: '--with-pbs=/opt/torque' '--with-default-comm=mpich2-pmi' '--prefix=/usr/local/mpiexec' System... $ uname -a Linux 2.6.21-smp #1 SMP Tue Aug 7 12:45:20 MDT 2007 GNU/Linux Routing table... $ netstat -r Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 255.255.255.255 * 255.255.255.255 UH 0 0 0 eth0 10.0.6.0 * 255.255.255.0 U 0 0 0 ib0 129.128.125.0 * 255.255.255.0 U 0 0 0 eth1 192.168.44.0 * 255.255.255.0 U 0 0 0 vmnet8 192.168.43.0 * 255.255.255.0 U 0 0 0 vmnet1 10.0.0.0 * 255.255.0.0 U 0 0 0 eth0 224.0.0.0 * 240.0.0.0 U 0 0 0 eth0 default gateway.nic.ual 0.0.0.0 UG 0 0 0 eth1 Please let me know if you need further info. Is there a diagnostic mode that can be enabled? Could there be some MVAPICH2 parameter that needs adjusting from its default value? -- Ed[mund [Sumbar]] AICT Research Support, Univ of Alberta From stevejones at stanford.edu Mon Oct 1 18:19:23 2007 From: stevejones at stanford.edu (Steve Jones) Date: Mon Oct 1 18:19:58 2007 Subject: [mvapich-discuss] IBV_EVENT_QP_LAST_WQE_REACHED error Message-ID: <20071001151923.qkt41usgro4w8w08@webmail.stanford.edu> Hi. We've moved from using Topspin Drivers to Cisco-OFED 1.2 and OSU MVAPICH on one cluster & have run into issues with one code when running on nodes >1, failing with an IBV_EVENT_QP_LAST_WQE_REACHED error. The application will compile and run correctly on a single node. *Application on nodes=1:ppn=8* [smjones@compute-3-11 Test_for_Steve]$ mpiexec ~/NGA/bin/arts No input file name was detected, using "input". Step Time CFLmax Umax Vmax Wmax Divergence 0 0.00000E+00 0.2920 2.190E+01 0.000E+00 0.000E+00 5.702E+04 1 5.00000E-06 1.0322 3.297E+01 1.984E+01 1.281E-04 5.742E-01 2 9.84412E-06 0.9742 3.896E+01 1.902E+01 1.366E-04 5.294E-01 3 1.47779E-05 0.9240 4.261E+01 1.816E+01 1.627E-04 1.824E-02 *Application on nodes=2:ppn=4* [smjones@compute-3-15 Test_for_Steve]$ mpiexec ~/NGA/bin/arts No input file name was detected, using "input". Step Time CFLmax Umax Vmax Wmax Divergence [0:compute-3-15.local] Abort: [0] Got FATAL event IBV_EVENT_QP_LAST_WQE_REACHED, code=16 at line 2551 in file viacheck.c mpiexec: Warning: accept_abort_conn: MPI_Abort from IP 10.255.255.169, rank 0, killing all. forrtl: error (78): process killed (SIGTERM) forrtl: error (78): process killed (SIGTERM) forrtl: error (78): process killed (SIGTERM) forrtl: error (78): process killed (SIGTERM) mpiexec: Warning: task 0 exited with status 255. It's been noted that the application crashes when calling MPI_COMM_SPLIT. I noticed traffic on the list about running checks with ib_rdma_bw tests. I've performed the test in advance and have included the output below. I've also appended the Makefile for MVAPICH in case it's a build related issue. Thanks. Steve run ib_rdma_bw test- [smjones@compute-3-21 ~]$ /usr/bin/ib_rdma_bw 25531: | port=18515 | ib_port=1 | size=65536 | tx_depth=100 | iters=1000 | duplex=0 | cma=0 | [smjones@compute-3-21 ~]$ /usr/bin/ib_rdma_bw -s1048576 -n100 25587: | port=18515 | ib_port=1 | size=1048576 | tx_depth=100 | iters=100 | duplex=0 | cma=0 | 25587: Local address: LID 0x229, QPN 0x960405, PSN 0xe28cbd RKey 0x2002400 VAddr 0x00002a959aa000 25587: Remote address: LID 0x231, QPN 0x6e0405, PSN 0xd141ca, RKey 0x28002400 VAddr 0x00002a95bbb000 [smjones@compute-3-20 ~]$ /usr/bin/ib_rdma_bw -s1048576 -n100 c3-21 25551: | port=18515 | ib_port=1 | size=1048576 | tx_depth=100 | iters=100 | duplex=0 | cma=0 | 25551: Local address: LID 0x231, QPN 0x6e0405, PSN 0xd141ca RKey 0x28002400 VAddr 0x00002a95bbb000 25551: Remote address: LID 0x229, QPN 0x960405, PSN 0xe28cbd, RKey 0x2002400 VAddr 0x00002a959aa000 25551: Bandwidth peak (#0 to #86): 1138.18 MB/sec 25551: Bandwidth average: 1136.94 MB/sec 25551: Service Demand peak (#0 to #86): 1997 cycles/KB 25551: Service Demand Avg : 1999 cycles/KB #!/bin/bash source ./make.mvapich.def arch # Mandatory variables. All are checked except CXX and F90. MTHOME=/usr PREFIX=/share/apps/mvapich/intel export CC=icc export CXX=icc export F77=ifort export F90=ifort export RSHCOMMAND=ssh IO_BUS="_PCI_EX_" ARCH="_EM64T_" LINKS="_DDR_" export LIBS="-L${MTHOME}/lib64 -libverbs -libumad -libcommon -lpthread" export FFLAGS="-L${MTHOME}/lib64 -xP -fPIC" export CFLAGS="-D${ARCH} -D__INTEL_COMPILER -g -DCH_GEN2 -DMEMORY_SCALE -D_AFFINITY_ \ -D_SMP_ -D_SMP_RNDV_ -DVIADEV_RPUT_SUPPORT \ -fPIC -DEARLY_SEND_COMPLETION -DLAZY_MEM_UNREGISTER \ -D${IO_BUS} -D${LINKS} \ -I${MTHOME}/include -I${MTHOME}/include/rdma \ -I/opt/panfs/include" export CCFLAGS="-lstdc++" # Prelogue make distclean &>/dev/null # Configure MVAPICH echo "Configuring MVAPICH..." ./configure --with-device=ch_gen2 --with-arch=LINUX -prefix=${PREFIX} \ --enable-cxx --enable-debug \ --enable-devdebug \ --enable-f77 --enable-f90 \ --with-romio --with-file-system=ufs+nfs+panfs \ --without-mpe \ -lib="-L${MTHOME}/lib64 -libverbs -libumad -libcommon -lpthread" From huanwei at cse.ohio-state.edu Tue Oct 2 11:34:11 2007 From: huanwei at cse.ohio-state.edu (wei huang) Date: Tue Oct 2 11:34:42 2007 Subject: [mvapich-discuss] collectives fail under mvapich2-1.0 (fwd) In-Reply-To: <46FC2746.7010508@ualberta.ca> Message-ID: Hi Ed, We look into the problem more wrt one sided issues. However, we don't see the program hang in the MPI library. Actually the program is not hanging. But somehow for MPI_Win_test, we find the following code: if (get_measurement_rank() == 0) { reduced_group = exclude_rank_from_group(0, onesided_group); mpiassert = extract_onesided_assertions(assertion, "MPI_Win_post"); MPI_Win_post(reduced_group, mpiassert, onesided_win); start_time = start_synchronization(); MPI_Win_test(onesided_win, &flag); end_time = stop_synchronization(); if (flag == 0) MPI_Win_wait(onesided_win); } else { reduced_group = exclude_all_ranks_except_from_group(0, onesided_group); mpiassert = extract_onesided_assertions(assertion, "MPI_Win_start"); MPI_Win_start(reduced_group, mpiassert, onesided_win); if (do_a_put) MPI_Put(get_send_buffer(), count, datatype, 0, get_measurement_rank(), count, datatype, onesided_win); MPI_Win_complete(onesided_win); start_synchronization(); stop_synchronization(); } And the test is spending more and more in in start_synchronization(), which seems to calculate a certain timestamp, and busily reads wtime() until we reach that timestamp. We find that start_synchronization() is taking longer and longer time, and finally will spend tens of seconds before it returns. We are not sure how the timestamp is calculated, so we cc this email to SkaMPI team and hope they can give some insights here. Dear SkaMPI team, we face a problem running SkaMPI using mvapich2-1.0 on 12 processes (3 nodes, 4 processes each, block distribution). We find that start_synchronization() in MPI_Win_test is taking very long time to return as the test goes on. As a result, the test appears to be hang. We are not sure how the timestamp is calculated and how you adjust this value. Could you please help give some insights here? Thanks. Regards, Wei Huang 774 Dreese Lab, 2015 Neil Ave, Dept. of Computer Science and Engineering Ohio State University OH 43210 Tel: (614)292-8501 On Thu, 27 Sep 2007, Edmund Sumbar wrote: > Edmund Sumbar wrote: > > I'll try running the SKaMPI tests again. Maybe > > I missed something, as with the mvapich2 tests. > > I recompiled and reran SKaMPI pt2pt, coll, > onesided, and mmisc tests on 3 nodes, 4 > processors per node. > > pt2pt and mmisc succeeded, while coll and > onesided failed (stalled). Any ideas? > > For what it's worth, here are the tails of > the output files... > > > $ tail coll_ib-3x4.sko > # SKaMPI Version 5.0 rev. 191 > > begin result "MPI_Bcast-nodes-short" > nodes= 2 1024 3.8 0.2 39 2.9 3.6 > nodes= 3 1024 6.6 0.4 38 4.0 6.3 4.9 > nodes= 4 1024 9.2 0.2 32 4.6 7.7 7.6 8.6 > > > $ tail onesided_ib-3x4.sko > cpus= 8 4 50051.7 1.3 8 50051.7 --- --- --- --- --- --- --- > cpus= 9 4 50051.5 0.7 8 50051.5 --- --- --- --- --- --- --- --- > cpus= 10 4 50047.7 1.6 8 50047.7 --- --- --- --- --- --- --- > --- --- > cpus= 11 4 50058.2 2.7 8 50058.2 --- --- --- --- --- --- --- > --- --- --- > cpus= 12 4 50074.3 2.8 8 50074.3 --- --- --- --- --- --- --- > --- --- --- --- > end result "MPI_Win_wait delayed,small" > # duration = 9.00 sec > > begin result "MPI_Win_wait delayed without MPI_Put" > cpus= 2 1048576 50025.0 1.4 8 50025.0 --- > > > -- > Ed[mund [Sumbar]] > AICT Research Support, Univ of Alberta > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > From mamidala at cse.ohio-state.edu Tue Oct 2 11:42:36 2007 From: mamidala at cse.ohio-state.edu (amith rajith mamidala) Date: Tue Oct 2 11:43:07 2007 Subject: [mvapich-discuss] IBV_EVENT_QP_LAST_WQE_REACHED error In-Reply-To: <20071001151923.qkt41usgro4w8w08@webmail.stanford.edu> Message-ID: Hi Steve, Can you do these couple of checks? 1. Making sure that the IB installation is the same on both the nodes. 2. Using mpirun_rsh instead of mpiexec. 3. Disabling shared memory collectives by using the environment variable VIADEV_USE_SHMEM_COLL=0, thanks, -Amith. On Mon, 1 Oct 2007, Steve Jones wrote: > Hi. > > We've moved from using Topspin Drivers to Cisco-OFED 1.2 and OSU > MVAPICH on one cluster & have run into issues with one code when > running on nodes >1, failing with an IBV_EVENT_QP_LAST_WQE_REACHED > error. The application will compile and run correctly on a single node. > > *Application on nodes=1:ppn=8* > > [smjones@compute-3-11 Test_for_Steve]$ mpiexec ~/NGA/bin/arts > No input file name was detected, using "input". > Step Time CFLmax Umax Vmax > Wmax Divergence > 0 0.00000E+00 0.2920 2.190E+01 0.000E+00 > 0.000E+00 5.702E+04 > 1 5.00000E-06 1.0322 3.297E+01 1.984E+01 > 1.281E-04 5.742E-01 > 2 9.84412E-06 0.9742 3.896E+01 1.902E+01 > 1.366E-04 5.294E-01 > 3 1.47779E-05 0.9240 4.261E+01 1.816E+01 > 1.627E-04 1.824E-02 > > > *Application on nodes=2:ppn=4* > > [smjones@compute-3-15 Test_for_Steve]$ mpiexec ~/NGA/bin/arts > No input file name was detected, using "input". > Step Time CFLmax Umax Vmax > Wmax Divergence > [0:compute-3-15.local] Abort: [0] Got FATAL event > IBV_EVENT_QP_LAST_WQE_REACHED, code=16 > at line 2551 in file viacheck.c > mpiexec: Warning: accept_abort_conn: MPI_Abort from IP > 10.255.255.169, rank 0, killing all. > forrtl: error (78): process killed (SIGTERM) > forrtl: error (78): process killed (SIGTERM) > forrtl: error (78): process killed (SIGTERM) > forrtl: error (78): process killed (SIGTERM) > mpiexec: Warning: task 0 exited with status 255. > > It's been noted that the application crashes when calling MPI_COMM_SPLIT. > > I noticed traffic on the list about running checks with ib_rdma_bw > tests. I've performed the test in advance and have included the output > below. I've also appended the Makefile for MVAPICH in case it's a > build related issue. > > Thanks. > > Steve > > > > > > run ib_rdma_bw test- > > [smjones@compute-3-21 ~]$ /usr/bin/ib_rdma_bw > 25531: | port=18515 | ib_port=1 | size=65536 | tx_depth=100 | > iters=1000 | duplex=0 | cma=0 | > > [smjones@compute-3-21 ~]$ /usr/bin/ib_rdma_bw -s1048576 -n100 > 25587: | port=18515 | ib_port=1 | size=1048576 | tx_depth=100 | > iters=100 | duplex=0 | cma=0 | > 25587: Local address: LID 0x229, QPN 0x960405, PSN 0xe28cbd RKey > 0x2002400 VAddr 0x00002a959aa000 > 25587: Remote address: LID 0x231, QPN 0x6e0405, PSN 0xd141ca, RKey > 0x28002400 VAddr 0x00002a95bbb000 > > > [smjones@compute-3-20 ~]$ /usr/bin/ib_rdma_bw -s1048576 -n100 c3-21 > 25551: | port=18515 | ib_port=1 | size=1048576 | tx_depth=100 | > iters=100 | duplex=0 | cma=0 | > 25551: Local address: LID 0x231, QPN 0x6e0405, PSN 0xd141ca RKey > 0x28002400 VAddr 0x00002a95bbb000 > 25551: Remote address: LID 0x229, QPN 0x960405, PSN 0xe28cbd, RKey > 0x2002400 VAddr 0x00002a959aa000 > > > 25551: Bandwidth peak (#0 to #86): 1138.18 MB/sec > 25551: Bandwidth average: 1136.94 MB/sec > 25551: Service Demand peak (#0 to #86): 1997 cycles/KB > 25551: Service Demand Avg : 1999 cycles/KB > > > #!/bin/bash > > source ./make.mvapich.def > arch > > # Mandatory variables. All are checked except CXX and F90. > MTHOME=/usr > PREFIX=/share/apps/mvapich/intel > export CC=icc > export CXX=icc > export F77=ifort > export F90=ifort > export RSHCOMMAND=ssh > IO_BUS="_PCI_EX_" > ARCH="_EM64T_" > LINKS="_DDR_" > > export LIBS="-L${MTHOME}/lib64 -libverbs -libumad -libcommon -lpthread" > export FFLAGS="-L${MTHOME}/lib64 -xP -fPIC" > export CFLAGS="-D${ARCH} -D__INTEL_COMPILER -g -DCH_GEN2 > -DMEMORY_SCALE -D_AFFINITY_ \ > -D_SMP_ -D_SMP_RNDV_ -DVIADEV_RPUT_SUPPORT \ > -fPIC -DEARLY_SEND_COMPLETION -DLAZY_MEM_UNREGISTER \ > -D${IO_BUS} -D${LINKS} \ > -I${MTHOME}/include -I${MTHOME}/include/rdma \ > -I/opt/panfs/include" > export CCFLAGS="-lstdc++" > # Prelogue > make distclean &>/dev/null > > # Configure MVAPICH > > echo "Configuring MVAPICH..." > > ./configure --with-device=ch_gen2 --with-arch=LINUX -prefix=${PREFIX} \ > --enable-cxx --enable-debug \ > --enable-devdebug \ > --enable-f77 --enable-f90 \ > --with-romio --with-file-system=ufs+nfs+panfs \ > --without-mpe \ > -lib="-L${MTHOME}/lib64 -libverbs -libumad -libcommon -lpthread" > > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > From stevejones at stanford.edu Tue Oct 2 12:53:55 2007 From: stevejones at stanford.edu (Steve Jones) Date: Tue Oct 2 12:54:28 2007 Subject: [mvapich-discuss] IBV_EVENT_QP_LAST_WQE_REACHED error In-Reply-To: References: Message-ID: <20071002095355.5f6ewyyjx6sok8so@webmail.stanford.edu> > Can you do these couple of checks? > 1. Making sure that the IB installation is the same on both the nodes. > 2. Using mpirun_rsh instead of mpiexec. > 3. Disabling shared memory collectives by using the environment variable > VIADEV_USE_SHMEM_COLL=0, Hi Amith. I checked the installation on nodes, switched to mpirun_rsh, and used the environment variable. I also ran ldd to verify libs that are being called within the session that's failing. No changes. Let me know what else I can do to debug. Thanks. Steve [smjones@compute-5-0 Test_for_Steve]$ env |grep VIA VIADEV_USE_SHMEM_COLL=0 [smjones@compute-5-0 Test_for_Steve]$ /share/apps/mvapich/intel/bin/mpirun_rsh -ssh -np 16 -hostfile $PBS_NODEFILE ~/NGA/bin/arts No input file name was detected, using "input". Step Time CFLmax Umax Vmax Wmax Divergence mpirun_rsh: Abort signaled from [0] [0:compute-5-0.local] Abort: [0] Got FATAL event IBV_EVENT_QP_LAST_WQE_REACHED, code=16 at line 2551 in file viacheck.c done. [smjones@compute-5-0 Test_for_Steve]$ /share/apps/mvapich/intel/bin/mpirun_rsh -np 16 -hostfile $PBS_NODEFILE ~/NGA/bin/arts No input file name was detected, using "input". Step Time CFLmax Umax Vmax Wmax Divergence [0:compute-5-0.local] Abort: [0] Got FATAL event IBV_EVENT_QP_LAST_WQE_REACHED, code=16 mpirun_rsh: Abort signaled from [0] at line 2551 in file viacheck.c done. [smjones@compute-5-0 Test_for_Steve]$ mpiexec -npernode 1 ldd ~/NGA/bin/arts libibverbs.so.1 => /usr/lib64/libibverbs.so.1 (0x0000002a95573000) libibumad.so.1 => /usr/lib64/libibumad.so.1 (0x0000002a9567f000) libibcommon.so.1 => /usr/lib64/libibcommon.so.1 (0x0000002a95789000) libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x0000003133900000) librt.so.1 => /lib64/tls/librt.so.1 (0x0000003134300000) libm.so.6 => /lib64/tls/libm.so.6 (0x0000003133700000) libc.so.6 => /lib64/tls/libc.so.6 (0x0000003133200000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003134100000) libdl.so.2 => /lib64/libdl.so.2 (0x0000003133500000) /lib64/ld-linux-x86-64.so.2 (0x0000003133000000) libibverbs.so.1 => /usr/lib64/libibverbs.so.1 (0x0000002a95573000) libibumad.so.1 => /usr/lib64/libibumad.so.1 (0x0000002a9567f000) libibcommon.so.1 => /usr/lib64/libibcommon.so.1 (0x0000002a95789000) libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x0000003817c00000) librt.so.1 => /lib64/tls/librt.so.1 (0x0000003818200000) libm.so.6 => /lib64/tls/libm.so.6 (0x0000003817a00000) libc.so.6 => /lib64/tls/libc.so.6 (0x0000003817500000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003818800000) libdl.so.2 => /lib64/libdl.so.2 (0x0000003817800000) /lib64/ld-linux-x86-64.so.2 (0x0000003817300000) From esumbar at ualberta.ca Tue Oct 2 13:19:05 2007 From: esumbar at ualberta.ca (Edmund Sumbar) Date: Tue Oct 2 13:22:26 2007 Subject: [mvapich-discuss] collectives fail under mvapich2-1.0 (fwd) In-Reply-To: References: Message-ID: <47027D89.1060602@ualberta.ca> wei huang wrote: > Hi Ed, > > We look into the problem more wrt one sided issues. However, we don't see > the program hang in the MPI library. Actually the program is not hanging. > But somehow for MPI_Win_test, we find the following code: > > if (get_measurement_rank() == 0) { > reduced_group = exclude_rank_from_group(0, onesided_group); > mpiassert = extract_onesided_assertions(assertion, "MPI_Win_post"); > MPI_Win_post(reduced_group, mpiassert, onesided_win); > > start_time = start_synchronization(); > MPI_Win_test(onesided_win, &flag); > end_time = stop_synchronization(); > if (flag == 0) > MPI_Win_wait(onesided_win); > } > else { > reduced_group = exclude_all_ranks_except_from_group(0, onesided_group); > mpiassert = extract_onesided_assertions(assertion, "MPI_Win_start"); > MPI_Win_start(reduced_group, mpiassert, onesided_win); > if (do_a_put) > MPI_Put(get_send_buffer(), count, datatype, 0, get_measurement_rank(), > count, datatype, onesided_win); > MPI_Win_complete(onesided_win); > start_synchronization(); > stop_synchronization(); > } > > And the test is spending more and more in in start_synchronization(), > which seems to calculate a certain timestamp, and busily reads wtime() > until we reach that timestamp. We find that start_synchronization() is > taking longer and longer time, and finally will spend tens of seconds > before it returns. We are not sure how the timestamp is calculated, so we > cc this email to SkaMPI team and hope they can give some insights here. Hi Wei, Thank you for looking into this issue. This may or may not be related, but the SKaMPI document describing their new timing approach for collectives can be found at http://www.springerlink.com/content/7ygll9u0h02t8mth/fulltext.pdf -- Ed[mund [Sumbar]] AICT Research Support, Univ of Alberta From kevin.ball at qlogic.com Tue Oct 2 13:01:20 2007 From: kevin.ball at qlogic.com (Kevin Ball) Date: Tue Oct 2 14:12:25 2007 Subject: [mvapich-discuss] collectives fail under mvapich2-1.0 (fwd) In-Reply-To: References: Message-ID: <1191344480.4868.48.camel@ammonite> Hi Wei and Ed, This email triggered a memory in me, and after digging through our bug tracking system, found that we ran into this exact same problem with PathScale's port of MPICH quite a while ago. Here is the analysis from one of our engineers at that time: ----------------------------------------------------------- he root of the problem is in the function syncol_pattern in skosfile.c. It tries to be smart about trying to get a uniform value for all its timings (100 max to be precise), and for that, for some unknown reason, it thinks that once it gets a flawed measurement, it should wait for an inordinately long time (1.7 seconds to be precise) between taking two samples. I tried to look around for what the reason for this number could be, but could not. And when the next sample is flawed as well, it increments the time it should wait. Here is the routine it calls: int wait_till(double time_stamp, double *last_time_stamp) { if( (*last_time_stamp = MPI_Wtime()) > time_stamp ) return 0; else { while( (*last_time_stamp = MPI_Wtime()) < time_stamp ) ; return 1; } } And the interval to wait for is determined by: double should_wait_till(int counter, double interval, double offset) { return (counter+1)*interval + first_time + offset; } As you can see, depending on which sample had a flawed measurement, it increases the wait period by that much !! And thats why the program crawls and crawls and feels like it is hanging. ---------------------------------------------------------------------- Another engineer added the following: ----------------------------------------------------------------------- Verified, thanks for the analysis and hard work. I made the following edits to encourage speedier forward progress but it is still dog slow: =================================================================== RCS file: RCS/skosfile.c,v retrieving revision 1.1 diff -r1.1 skosfile.c 16214c16214,16219 < if( max_tbms[i] >= interval ) flawed_measurements++; --- > if( max_tbms[i] > interval && > (max_tbms[i] - interval) / interval > 0.01 ) { > flawed_measurements++; > /* printf("FLAWED: %f >= %f, flawed_measurements = %d\n", > max_tbms[i], interval, flawed_measurements); */ > } 16216a16222 > /* printf("REALLY FLAWED : interval now %f\n", interval); */ ---------------------------------------------------------- On Tue, 2007-10-02 at 08:34, wei huang wrote: > Hi Ed, > > We look into the problem more wrt one sided issues. However, we don't see > the program hang in the MPI library. Actually the program is not hanging. > But somehow for MPI_Win_test, we find the following code: > > if (get_measurement_rank() == 0) { > reduced_group = exclude_rank_from_group(0, onesided_group); > mpiassert = extract_onesided_assertions(assertion, "MPI_Win_post"); > MPI_Win_post(reduced_group, mpiassert, onesided_win); > > start_time = start_synchronization(); > MPI_Win_test(onesided_win, &flag); > end_time = stop_synchronization(); > if (flag == 0) > MPI_Win_wait(onesided_win); > } > else { > reduced_group = exclude_all_ranks_except_from_group(0, onesided_group); > mpiassert = extract_onesided_assertions(assertion, "MPI_Win_start"); > MPI_Win_start(reduced_group, mpiassert, onesided_win); > if (do_a_put) > MPI_Put(get_send_buffer(), count, datatype, 0, get_measurement_rank(), > count, datatype, onesided_win); > MPI_Win_complete(onesided_win); > start_synchronization(); > stop_synchronization(); > } > > And the test is spending more and more in in start_synchronization(), > which seems to calculate a certain timestamp, and busily reads wtime() > until we reach that timestamp. We find that start_synchronization() is > taking longer and longer time, and finally will spend tens of seconds > before it returns. We are not sure how the timestamp is calculated, so we > cc this email to SkaMPI team and hope they can give some insights here. > > Dear SkaMPI team, we face a problem running SkaMPI using mvapich2-1.0 on > 12 processes (3 nodes, 4 processes each, block distribution). We find that > start_synchronization() in MPI_Win_test is taking very long time to return > as the test goes on. As a result, the test appears to be hang. We are not > sure how the timestamp is calculated and how you adjust this value. Could > you please help give some insights here? > > Thanks. > > Regards, > Wei Huang > > 774 Dreese Lab, 2015 Neil Ave, > Dept. of Computer Science and Engineering > Ohio State University > OH 43210 > Tel: (614)292-8501 > > > On Thu, 27 Sep 2007, Edmund Sumbar wrote: > > > Edmund Sumbar wrote: > > > I'll try running the SKaMPI tests again. Maybe > > > I missed something, as with the mvapich2 tests. > > > > I recompiled and reran SKaMPI pt2pt, coll, > > onesided, and mmisc tests on 3 nodes, 4 > > processors per node. > > > > pt2pt and mmisc succeeded, while coll and > > onesided failed (stalled). Any ideas? > > > > For what it's worth, here are the tails of > > the output files... > > > > > > $ tail coll_ib-3x4.sko > > # SKaMPI Version 5.0 rev. 191 > > > > begin result "MPI_Bcast-nodes-short" > > nodes= 2 1024 3.8 0.2 39 2.9 3.6 > > nodes= 3 1024 6.6 0.4 38 4.0 6.3 4.9 > > nodes= 4 1024 9.2 0.2 32 4.6 7.7 7.6 8.6 > > > > > > $ tail onesided_ib-3x4.sko > > cpus= 8 4 50051.7 1.3 8 50051.7 --- --- --- --- --- --- --- > > cpus= 9 4 50051.5 0.7 8 50051.5 --- --- --- --- --- --- --- --- > > cpus= 10 4 50047.7 1.6 8 50047.7 --- --- --- --- --- --- --- > > --- --- > > cpus= 11 4 50058.2 2.7 8 50058.2 --- --- --- --- --- --- --- > > --- --- --- > > cpus= 12 4 50074.3 2.8 8 50074.3 --- --- --- --- --- --- --- > > --- --- --- --- > > end result "MPI_Win_wait delayed,small" > > # duration = 9.00 sec > > > > begin result "MPI_Win_wait delayed without MPI_Put" > > cpus= 2 1048576 50025.0 1.4 8 50025.0 --- > > > > > > -- > > Ed[mund [Sumbar]] > > AICT Research Support, Univ of Alberta > > _______________________________________________ > > mvapich-discuss mailing list > > mvapich-discuss@cse.ohio-state.edu > > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > > > > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss From panda at cse.ohio-state.edu Tue Oct 2 14:20:10 2007 From: panda at cse.ohio-state.edu (Dhabaleswar Panda) Date: Tue Oct 2 14:20:43 2007 Subject: [mvapich-discuss] collectives fail under mvapich2-1.0 (fwd) In-Reply-To: <1191344480.4868.48.camel@ammonite> from "Kevin Ball" at Oct 02, 2007 10:01:20 AM Message-ID: <200710021820.l92IKAUf000616@xi.cse.ohio-state.edu> Hi Kevin, Thanks for providing further insights to this problem based on QLogic's experience. It will be good if SkaMPI team can take a look at this issue and make appropriate changes to their benchmarks. Thanks, DK ====== > Hi Wei and Ed, > > This email triggered a memory in me, and after digging through our bug > tracking system, found that we ran into this exact same problem with > PathScale's port of MPICH quite a while ago. Here is the analysis from > one of our engineers at that time: > > ----------------------------------------------------------- > he root of the problem is in the function syncol_pattern in > skosfile.c. It tries to be smart about trying to get a uniform value for all its > timings (100 max to be precise), and for that, for some unknown reason, it > thinks that once it gets a flawed measurement, it should wait for an > inordinately long time (1.7 seconds to be precise) between taking two samples. I > tried to look around for what the reason for this number could be, but could > not. And when the next sample is flawed as well, it increments the time it > should wait. Here is the routine it calls: > > int wait_till(double time_stamp, double *last_time_stamp) > { > if( (*last_time_stamp = MPI_Wtime()) > time_stamp ) > return 0; > else { > while( (*last_time_stamp = MPI_Wtime()) < time_stamp ) ; > return 1; > } > } > > > And the interval to wait for is determined by: > > double should_wait_till(int counter, double interval, double offset) > { > return (counter+1)*interval + first_time + offset; > } > > > As you can see, depending on which sample had a flawed measurement, it increases > the wait period by that much !! > > And thats why the program crawls and crawls and feels like it is hanging. > ---------------------------------------------------------------------- > > Another engineer added the following: > > ----------------------------------------------------------------------- > Verified, thanks for the analysis and hard work. I made the following edits to > encourage speedier forward progress but it is still dog slow: > > =================================================================== > RCS file: RCS/skosfile.c,v > retrieving revision 1.1 > diff -r1.1 skosfile.c > 16214c16214,16219 > < if( max_tbms[i] >= interval ) flawed_measurements++; > --- > > if( max_tbms[i] > interval && > > (max_tbms[i] - interval) / interval > 0.01 ) { > > flawed_measurements++; > > /* printf("FLAWED: %f >= %f, flawed_measurements = %d\n", > > max_tbms[i], interval, flawed_measurements); */ > > } > 16216a16222 > > /* printf("REALLY FLAWED : interval now %f\n", interval); */ > > ---------------------------------------------------------- > > On Tue, 2007-10-02 at 08:34, wei huang wrote: > > Hi Ed, > > > > We look into the problem more wrt one sided issues. However, we don't see > > the program hang in the MPI library. Actually the program is not hanging. > > But somehow for MPI_Win_test, we find the following code: > > > > if (get_measurement_rank() == 0) { > > reduced_group = exclude_rank_from_group(0, onesided_group); > > mpiassert = extract_onesided_assertions(assertion, "MPI_Win_post"); > > MPI_Win_post(reduced_group, mpiassert, onesided_win); > > > > start_time = start_synchronization(); > > MPI_Win_test(onesided_win, &flag); > > end_time = stop_synchronization(); > > if (flag == 0) > > MPI_Win_wait(onesided_win); > > } > > else { > > reduced_group = exclude_all_ranks_except_from_group(0, onesided_group); > > mpiassert = extract_onesided_assertions(assertion, "MPI_Win_start"); > > MPI_Win_start(reduced_group, mpiassert, onesided_win); > > if (do_a_put) > > MPI_Put(get_send_buffer(), count, datatype, 0, get_measurement_rank(), > > count, datatype, onesided_win); > > MPI_Win_complete(onesided_win); > > start_synchronization(); > > stop_synchronization(); > > } > > > > And the test is spending more and more in in start_synchronization(), > > which seems to calculate a certain timestamp, and busily reads wtime() > > until we reach that timestamp. We find that start_synchronization() is > > taking longer and longer time, and finally will spend tens of seconds > > before it returns. We are not sure how the timestamp is calculated, so we > > cc this email to SkaMPI team and hope they can give some insights here. > > > > Dear SkaMPI team, we face a problem running SkaMPI using mvapich2-1.0 on > > 12 processes (3 nodes, 4 processes each, block distribution). We find that > > start_synchronization() in MPI_Win_test is taking very long time to return > > as the test goes on. As a result, the test appears to be hang. We are not > > sure how the timestamp is calculated and how you adjust this value. Could > > you please help give some insights here? > > > > Thanks. > > > > Regards, > > Wei Huang > > > > 774 Dreese Lab, 2015 Neil Ave, > > Dept. of Computer Science and Engineering > > Ohio State University > > OH 43210 > > Tel: (614)292-8501 > > > > > > On Thu, 27 Sep 2007, Edmund Sumbar wrote: > > > > > Edmund Sumbar wrote: > > > > I'll try running the SKaMPI tests again. Maybe > > > > I missed something, as with the mvapich2 tests. > > > > > > I recompiled and reran SKaMPI pt2pt, coll, > > > onesided, and mmisc tests on 3 nodes, 4 > > > processors per node. > > > > > > pt2pt and mmisc succeeded, while coll and > > > onesided failed (stalled). Any ideas? > > > > > > For what it's worth, here are the tails of > > > the output files... > > > > > > > > > $ tail coll_ib-3x4.sko > > > # SKaMPI Version 5.0 rev. 191 > > > > > > begin result "MPI_Bcast-nodes-short" > > > nodes= 2 1024 3.8 0.2 39 2.9 3.6 > > > nodes= 3 1024 6.6 0.4 38 4.0 6.3 4.9 > > > nodes= 4 1024 9.2 0.2 32 4.6 7.7 7.6 8.6 > > > > > > > > > $ tail onesided_ib-3x4.sko > > > cpus= 8 4 50051.7 1.3 8 50051.7 --- --- --- --- --- --- --- > > > cpus= 9 4 50051.5 0.7 8 50051.5 --- --- --- --- --- --- --- --- > > > cpus= 10 4 50047.7 1.6 8 50047.7 --- --- --- --- --- --- --- > > > --- --- > > > cpus= 11 4 50058.2 2.7 8 50058.2 --- --- --- --- --- --- --- > > > --- --- --- > > > cpus= 12 4 50074.3 2.8 8 50074.3 --- --- --- --- --- --- --- > > > --- --- --- --- > > > end result "MPI_Win_wait delayed,small" > > > # duration = 9.00 sec > > > > > > begin result "MPI_Win_wait delayed without MPI_Put" > > > cpus= 2 1048576 50025.0 1.4 8 50025.0 --- > > > > > > > > > -- > > > Ed[mund [Sumbar]] > > > AICT Research Support, Univ of Alberta > > > _______________________________________________ > > > mvapich-discuss mailing list > > > mvapich-discuss@cse.ohio-state.edu > > > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > > > > > > > _______________________________________________ > > mvapich-discuss mailing list > > mvapich-discuss@cse.ohio-state.edu > > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > From vishnu at cse.ohio-state.edu Tue Oct 2 15:48:16 2007 From: vishnu at cse.ohio-state.edu (Abhinav Vishnu) Date: Tue Oct 2 15:49:13 2007 Subject: [mvapich-discuss] IBV_EVENT_QP_LAST_WQE_REACHED error In-Reply-To: <20071002095355.5f6ewyyjx6sok8so@webmail.stanford.edu> References: <20071002095355.5f6ewyyjx6sok8so@webmail.stanford.edu> Message-ID: <20071002194811.GA2987@cse.ohio-state.edu> Hi Steve, Thanks for your detailed reply. May i suggest you to do a fresh install of MVAPICH downloaded from our webpage. I am not sure what the dependencies are with respect to your installation, but it looks like it may be something simple. MVAPICH can be downloaded from our webpage: http://mvapich.cse.ohio-state.edu/ The build and usage instructions are available here: Build: http://mvapich.cse.ohio-state.edu/support/mvapich_user_guide.html#x1-90004.4 Usage: http://mvapich.cse.ohio-state.edu/support/mvapich_user_guide.html#x1-180005 There are a couple of other things, which i would suggest. To use an environment variable, please mention it explicitly on the command line: [smjones@compute-5-0 Test_for_Steve]$ /share/apps/mvapich/intel/bin/mpirun_rsh -ssh -np 16 -hostfile $PBS_NODEFILE VIADEV_USE_SHMEM_COLL=0 ~/NGA/bin/arts Also in addition to your application (which i am assuming you compiled with the version installed earlier), can you also compile simple MPI programs like osu_benchmarks and see if they work fine. Finally, it will be helpful if the application can be executed from 2 processes to 16 processes (2x1, 2x2, 2x4, 2x8) to see if there are some other issues with the code. Please keep us updated of your experimentation. Thanks and regards, :- Abhinav > >Can you do these couple of checks? > >1. Making sure that the IB installation is the same on both the nodes. > >2. Using mpirun_rsh instead of mpiexec. > >3. Disabling shared memory collectives by using the environment variable > > VIADEV_USE_SHMEM_COLL=0, > > Hi Amith. > > I checked the installation on nodes, switched to mpirun_rsh, and used > the environment variable. I also ran ldd to verify libs that are being > called within the session that's failing. No changes. > > Let me know what else I can do to debug. > > Thanks. > > Steve > > [smjones@compute-5-0 Test_for_Steve]$ env |grep VIA > VIADEV_USE_SHMEM_COLL=0 > > [smjones@compute-5-0 Test_for_Steve]$ > /share/apps/mvapich/intel/bin/mpirun_rsh -ssh -np 16 -hostfile > $PBS_NODEFILE ~/NGA/bin/arts > No input file name was detected, using "input". > Step Time CFLmax Umax Vmax > Wmax Divergence > mpirun_rsh: Abort signaled from [0] > [0:compute-5-0.local] Abort: [0] Got FATAL event > IBV_EVENT_QP_LAST_WQE_REACHED, code=16 > at line 2551 in file viacheck.c > done. > > [smjones@compute-5-0 Test_for_Steve]$ > /share/apps/mvapich/intel/bin/mpirun_rsh -np 16 -hostfile > $PBS_NODEFILE ~/NGA/bin/arts > No input file name was detected, using "input". > Step Time CFLmax Umax Vmax > Wmax Divergence > [0:compute-5-0.local] Abort: [0] Got FATAL event > IBV_EVENT_QP_LAST_WQE_REACHED, code=16 > mpirun_rsh: Abort signaled from [0] > at line 2551 in file viacheck.c > done. > > [smjones@compute-5-0 Test_for_Steve]$ mpiexec -npernode 1 ldd ~/NGA/bin/arts > libibverbs.so.1 => /usr/lib64/libibverbs.so.1 (0x0000002a95573000) > libibumad.so.1 => /usr/lib64/libibumad.so.1 (0x0000002a9567f000) > libibcommon.so.1 => /usr/lib64/libibcommon.so.1 (0x0000002a95789000) > libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x0000003133900000) > librt.so.1 => /lib64/tls/librt.so.1 (0x0000003134300000) > libm.so.6 => /lib64/tls/libm.so.6 (0x0000003133700000) > libc.so.6 => /lib64/tls/libc.so.6 (0x0000003133200000) > libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003134100000) > libdl.so.2 => /lib64/libdl.so.2 (0x0000003133500000) > /lib64/ld-linux-x86-64.so.2 (0x0000003133000000) > libibverbs.so.1 => /usr/lib64/libibverbs.so.1 (0x0000002a95573000) > libibumad.so.1 => /usr/lib64/libibumad.so.1 (0x0000002a9567f000) > libibcommon.so.1 => /usr/lib64/libibcommon.so.1 (0x0000002a95789000) > libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x0000003817c00000) > librt.so.1 => /lib64/tls/librt.so.1 (0x0000003818200000) > libm.so.6 => /lib64/tls/libm.so.6 (0x0000003817a00000) > libc.so.6 => /lib64/tls/libc.so.6 (0x0000003817500000) > libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003818800000) > libdl.so.2 => /lib64/libdl.so.2 (0x0000003817800000) > /lib64/ld-linux-x86-64.so.2 (0x0000003817300000) > > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss From huanwei at cse.ohio-state.edu Wed Oct 3 16:31:11 2007 From: huanwei at cse.ohio-state.edu (wei huang) Date: Wed Oct 3 16:31:43 2007 Subject: [mvapich-discuss] Problem with linpack/mvapich2/BLCR In-Reply-To: <4700AB3F.7030909@bull.net> Message-ID: Hi Patrice, We tried running hpl with BLCR support and it looks fine. We have run the test on two set of machines. One is dual processor Intel Xeon nodes, we run 4 processes with 2 processes on each node. We also ran the test on 880 Opteron (quad dual-core), hosting all 4 processes on one node. We try to use as similar HPL input as you are using, see below: ============================================================================ T/V N NB P Q Time Gflops ---------------------------------------------------------------------------- WR00C2L4 5000 112 4 1 6.64 1.255e+01 ---------------------------------------------------------------------------- ||Ax-b||_oo / ( eps * ||A||_1 * N ) = 0.0355903 ...... PASSED ||Ax-b||_oo / ( eps * ||A||_1 * ||x||_1 ) = 0.0234950 ...... PASSED ||Ax-b||_oo / ( eps * ||A||_oo * ||x||_oo ) = 0.0045862 ...... PASSED The difference is that we don't have intel-mkl installed. So we are using HPL with goto library. Could you let us know if you can reproduce the problem with goto? Thanks. Regards, Wei Huang 774 Dreese Lab, 2015 Neil Ave, Dept. of Computer Science and Engineering Ohio State University OH 43210 Tel: (614)292-8501 On Mon, 1 Oct 2007, Patrice Martinez wrote: > > Hello, > > I encounter problem running linpack benchmark with mvapich2 configured for BLCR support: computations are sometimes right, sometimes wrong. > Let me describe the context: > > > Hardware used: > > 1. > > Bull Novascale R422, 2xXeon Core 2 Duo 5150@ 2.66 Ghz, 8Gb de RAM > > 2. > > IB HCA Mellanox MT25208 dual-port > > Software used > > 1. > > RHEL4 U4, kernel 2.6.9.42-ELSmp, > > 2. > > gcc-3.4.6 > > 3. > > intel mkl 9.1 > > 4. > > blcr-0.6.0, > > 5. > > mvapich2-1.0, > > 6. > > OFED-1.2.5.1, > > 7. > > linpack-9.1 > > > > Tests > > > -For this test, the two ports of the IB HCA are connected together. > > -I made the following link to avoid problems forwarding environment variables: > > #l /lib64/libcr.so.0 > lrwxrwxrwx 1 root root 23 Sep 21 11:19 /lib64/libcr.so.0 -> /usr/local/lib/libcr.so > > - blcr modules are loaded: > > service blcr start > > - mpd daemon is run: > > mpdboot --ncpus=4 > > - And finally, linpack is configured to invert a small matrix (N=5000), and linpack is executed: > > mpiexec -n 4 ./xhpl > > > Analyse > > > Depending on the parameters P and Q given in the HPL.dat file, computations are always right or always wrong... > With P=4, Q=1: > ============================================================================ > T/V N NB P Q Time Gflops > ---------------------------------------------------------------------------- > W00C2L4 5000 112 4 1 4.28 1.948e+01 > ---------------------------------------------------------------------------- > ||Ax-b||_oo / ( eps * ||A||_1 * N ) = 25110713646301407346688.0000000 ...... FAILED > ||Ax-b||_oo / ( eps * ||A||_1 * ||x||_1 ) = 155458419119.8088379 ...... FAILED > ||Ax-b||_oo / ( eps * ||A||_oo * ||x||_oo ) = 17288875125.5442734 ...... FAILED > ||Ax-b||_oo . . . . . . . . . . . . . . . . . = 17973740643825.015625 > ||A||_oo . . . . . . . . . . . . . . . . . . . = 1283.266028 > ||A||_1 . . . . . . . . . . . . . . . . . . . = 1289.434188 > ||x||_oo . . . . . . . . . . . . . . . . . . . = 1459401545070.356201 > ||x||_1 . . . . . . . . . . . . . . . . . . . = 807634407595160.750000 > ============================================================================ > > With P=2, Q=2 > > ============================================================================ > T/V N NB P Q Time Gflops > ---------------------------------------------------------------------------- > W00C2L4 5000 112 2 2 3.39 2.459e+01 > ---------------------------------------------------------------------------- > ||Ax-b||_oo / ( eps * ||A||_1 * N ) = 0.0420265 ...... PASSED > ||Ax-b||_oo / ( eps * ||A||_1 * ||x||_1 ) = 0.0277438 ...... PASSED > ||Ax-b||_oo / ( eps * ||A||_oo * ||x||_oo ) = 0.0054156 ...... PASSED > ============================================================================ > > > It is interesting to see that computations are faster when they're right... > > When using mvapich2 compiled without BLCR support, computations are always right, of course. > Any idea? > > -- > > Cordialement/Best regards > > Patrice Martinez > > Linux Kernel Architect. > > OFFICE : B1-405 > PHONE : +33 (0)4 76 29 74 69 > EMAIL : Patrice.martinez@bull.net > ADDR : BULL, 1 rue de Provence, BP 208, 38432 Echirolles Cedex, FRANCE > > From huanwei at cse.ohio-state.edu Wed Oct 3 16:53:04 2007 From: huanwei at cse.ohio-state.edu (wei huang) Date: Wed Oct 3 16:53:36 2007 Subject: [mvapich-discuss] Problem with linpack/mvapich2/BLCR (fwd) Message-ID: Hi Patrice, Forgot to mention in my last email. We are using BLCR-0.6.1 (the latest version) and OFED-1.2.5.1. Thanks. Regards, Wei Huang 774 Dreese Lab, 2015 Neil Ave, Dept. of Computer Science and Engineering Ohio State University OH 43210 Tel: (614)292-8501 ---------- Forwarded message ---------- Date: Wed, 3 Oct 2007 16:31:11 -0400 (EDT) From: wei huang To: Patrice Martinez Cc: mvapich-discuss@cse.ohio-state.edu Subject: Re: [mvapich-discuss] Problem with linpack/mvapich2/BLCR Hi Patrice, We tried running hpl with BLCR support and it looks fine. We have run the test on two set of machines. One is dual processor Intel Xeon nodes, we run 4 processes with 2 processes on each node. We also ran the test on 880 Opteron (quad dual-core), hosting all 4 processes on one node. We try to use as similar HPL input as you are using, see below: ============================================================================ T/V N NB P Q Time Gflops ---------------------------------------------------------------------------- WR00C2L4 5000 112 4 1 6.64 1.255e+01 ---------------------------------------------------------------------------- ||Ax-b||_oo / ( eps * ||A||_1 * N ) = 0.0355903 ...... PASSED ||Ax-b||_oo / ( eps * ||A||_1 * ||x||_1 ) = 0.0234950 ...... PASSED ||Ax-b||_oo / ( eps * ||A||_oo * ||x||_oo ) = 0.0045862 ...... PASSED The difference is that we don't have intel-mkl installed. So we are using HPL with goto library. Could you let us know if you can reproduce the problem with goto? Thanks. Regards, Wei Huang 774 Dreese Lab, 2015 Neil Ave, Dept. of Computer Science and Engineering Ohio State University OH 43210 Tel: (614)292-8501 On Mon, 1 Oct 2007, Patrice Martinez wrote: > > Hello, > > I encounter problem running linpack benchmark with mvapich2 configured for BLCR support: computations are sometimes right, sometimes wrong. > Let me describe the context: > > > Hardware used: > > 1. > > Bull Novascale R422, 2xXeon Core 2 Duo 5150@ 2.66 Ghz, 8Gb de RAM > > 2. > > IB HCA Mellanox MT25208 dual-port > > Software used > > 1. > > RHEL4 U4, kernel 2.6.9.42-ELSmp, > > 2. > > gcc-3.4.6 > > 3. > > intel mkl 9.1 > > 4. > > blcr-0.6.0, > > 5. > > mvapich2-1.0, > > 6. > > OFED-1.2.5.1, > > 7. > > linpack-9.1 > > > > Tests > > > -For this test, the two ports of the IB HCA are connected together. > > -I made the following link to avoid problems forwarding environment variables: > > #l /lib64/libcr.so.0 > lrwxrwxrwx 1 root root 23 Sep 21 11:19 /lib64/libcr.so.0 -> /usr/local/lib/libcr.so > > - blcr modules are loaded: > > service blcr start > > - mpd daemon is run: > > mpdboot --ncpus=4 > > - And finally, linpack is configured to invert a small matrix (N=5000), and linpack is executed: > > mpiexec -n 4 ./xhpl > > > Analyse > > > Depending on the parameters P and Q given in the HPL.dat file, computations are always right or always wrong... > With P=4, Q=1: > ============================================================================ > T/V N NB P Q Time Gflops > ---------------------------------------------------------------------------- > W00C2L4 5000 112 4 1 4.28 1.948e+01 > ---------------------------------------------------------------------------- > ||Ax-b||_oo / ( eps * ||A||_1 * N ) = 25110713646301407346688.0000000 ...... FAILED > ||Ax-b||_oo / ( eps * ||A||_1 * ||x||_1 ) = 155458419119.8088379 ...... FAILED > ||Ax-b||_oo / ( eps * ||A||_oo * ||x||_oo ) = 17288875125.5442734 ...... FAILED > ||Ax-b||_oo . . . . . . . . . . . . . . . . . = 17973740643825.015625 > ||A||_oo . . . . . . . . . . . . . . . . . . . = 1283.266028 > ||A||_1 . . . . . . . . . . . . . . . . . . . = 1289.434188 > ||x||_oo . . . . . . . . . . . . . . . . . . . = 1459401545070.356201 > ||x||_1 . . . . . . . . . . . . . . . . . . . = 807634407595160.750000 > ============================================================================ > > With P=2, Q=2 > > ============================================================================ > T/V N NB P Q Time Gflops > ---------------------------------------------------------------------------- > W00C2L4 5000 112 2 2 3.39 2.459e+01 > ---------------------------------------------------------------------------- > ||Ax-b||_oo / ( eps * ||A||_1 * N ) = 0.0420265 ...... PASSED > ||Ax-b||_oo / ( eps * ||A||_1 * ||x||_1 ) = 0.0277438 ...... PASSED > ||Ax-b||_oo / ( eps * ||A||_oo * ||x||_oo ) = 0.0054156 ...... PASSED > ============================================================================ > > > It is interesting to see that computations are faster when they're right... > > When using mvapich2 compiled without BLCR support, computations are always right, of course. > Any idea? > > -- > > Cordialement/Best regards > > Patrice Martinez > > Linux Kernel Architect. > > OFFICE : B1-405 > PHONE : +33 (0)4 76 29 74 69 > EMAIL : Patrice.martinez@bull.net > ADDR : BULL, 1 rue de Provence, BP 208, 38432 Echirolles Cedex, FRANCE > > _______________________________________________ mvapich-discuss mailing list mvapich-discuss@cse.ohio-state.edu http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss From reussner at ipd.uka.de Thu Oct 4 05:20:30 2007 From: reussner at ipd.uka.de (Ralf Reussner) Date: Thu Oct 4 10:25:14 2007 Subject: [mvapich-discuss] collectives fail under mvapich2-1.0 (fwd) In-Reply-To: References: Message-ID: <4704B05E.2040209@ipd.uka.de> Dear Wei Huang, thanks for your report. We will have a look into this. Please include instead of my email adress the email of our developer & maintainer team skampi@ira.uka.de in the discussions, so that all of us are informend on the discussion. Best regards Ralf > Hi Ed, > > We look into the problem more wrt one sided issues. However, we don't see > the program hang in the MPI library. Actually the program is not hanging. > But somehow for MPI_Win_test, we find the following code: > > if (get_measurement_rank() == 0) { > reduced_group = exclude_rank_from_group(0, onesided_group); > mpiassert = extract_onesided_assertions(assertion, "MPI_Win_post"); > MPI_Win_post(reduced_group, mpiassert, onesided_win); > > start_time = start_synchronization(); > MPI_Win_test(onesided_win, &flag); > end_time = stop_synchronization(); > if (flag == 0) > MPI_Win_wait(onesided_win); > } > else { > reduced_group = exclude_all_ranks_except_from_group(0, onesided_group); > mpiassert = extract_onesided_assertions(assertion, "MPI_Win_start"); > MPI_Win_start(reduced_group, mpiassert, onesided_win); > if (do_a_put) > MPI_Put(get_send_buffer(), count, datatype, 0, get_measurement_rank(), > count, datatype, onesided_win); > MPI_Win_complete(onesided_win); > start_synchronization(); > stop_synchronization(); > } > > And the test is spending more and more in in start_synchronization(), > which seems to calculate a certain timestamp, and busily reads wtime() > until we reach that timestamp. We find that start_synchronization() is > taking longer and longer time, and finally will spend tens of seconds > before it returns. We are not sure how the timestamp is calculated, so we > cc this email to SkaMPI team and hope they can give some insights here. > > Dear SkaMPI team, we face a problem running SkaMPI using mvapich2-1.0 on > 12 processes (3 nodes, 4 processes each, block distribution). We find that > start_synchronization() in MPI_Win_test is taking very long time to return > as the test goes on. As a result, the test appears to be hang. We are not > sure how the timestamp is calculated and how you adjust this value. Could > you please help give some insights here? > > Thanks. > > Regards, > Wei Huang > > 774 Dreese Lab, 2015 Neil Ave, > Dept. of Computer Science and Engineering > Ohio State University > OH 43210 > Tel: (614)292-8501 > > > On Thu, 27 Sep 2007, Edmund Sumbar wrote: > > >> Edmund Sumbar wrote: >> >>> I'll try running the SKaMPI tests again. Maybe >>> I missed something, as with the mvapich2 tests. >>> >> I recompiled and reran SKaMPI pt2pt, coll, >> onesided, and mmisc tests on 3 nodes, 4 >> processors per node. >> >> pt2pt and mmisc succeeded, while coll and >> onesided failed (stalled). Any ideas? >> >> For what it's worth, here are the tails of >> the output files... >> >> >> $ tail coll_ib-3x4.sko >> # SKaMPI Version 5.0 rev. 191 >> >> begin result "MPI_Bcast-nodes-short" >> nodes= 2 1024 3.8 0.2 39 2.9 3.6 >> nodes= 3 1024 6.6 0.4 38 4.0 6.3 4.9 >> nodes= 4 1024 9.2 0.2 32 4.6 7.7 7.6 8.6 >> >> >> $ tail onesided_ib-3x4.sko >> cpus= 8 4 50051.7 1.3 8 50051.7 --- --- --- --- --- --- --- >> cpus= 9 4 50051.5 0.7 8 50051.5 --- --- --- --- --- --- --- --- >> cpus= 10 4 50047.7 1.6 8 50047.7 --- --- --- --- --- --- --- >> --- --- >> cpus= 11 4 50058.2 2.7 8 50058.2 --- --- --- --- --- --- --- >> --- --- --- >> cpus= 12 4 50074.3 2.8 8 50074.3 --- --- --- --- --- --- --- >> --- --- --- --- >> end result "MPI_Win_wait delayed,small" >> # duration = 9.00 sec >> >> begin result "MPI_Win_wait delayed without MPI_Put" >> cpus= 2 1048576 50025.0 1.4 8 50025.0 --- >> >> >> -- >> Ed[mund [Sumbar]] >> AICT Research Support, Univ of Alberta >> _______________________________________________ >> mvapich-discuss mailing list >> mvapich-discuss@cse.ohio-state.edu >> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss >> >> > > -- -------------------------------------------------------------- Prof. Dr. Ralf Reussner - Chair Software-Design and -Quality Institute for Program Structures and Data Organization Faculty of Informatics, Universitaet Karlsruhe (TH) Am Fasanengarten 5, D-76131 Karlsruhe, Germany Office 327, Main Computer Science Building (50.34) Tel. +49 721 608 5993, Fax. +49 721 608 5990 http://sdq.ipd.uka.de -------------------------------------------------------------- From patrice.martinez at bull.net Fri Oct 5 07:01:19 2007 From: patrice.martinez at bull.net (Patrice Martinez) Date: Fri Oct 5 08:33:13 2007 Subject: [mvapich-discuss] Problem with linpack/mvapich2/BLCR In-Reply-To: References: Message-ID: <4706197F.4090602@bull.net> An HTML attachment was scrubbed... URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20071005/cd22ea1f/attachment-0001.html From luitjens at cs.utah.edu Fri Oct 5 12:55:36 2007 From: luitjens at cs.utah.edu (Justin) Date: Fri Oct 5 12:56:16 2007 Subject: [mvapich-discuss] waitsome/testsome memory allocation Message-ID: <47066C88.50207@cs.utah.edu> Hi, I am tracking down some memory issues in our code. And I am finding strange memory allocations occurring within MPI_Waitsome and MPI_Testsome. In one section of our code we use MPI_Pack and MPI_Unpack to combine a bunch of small messages. We then send out the packed messages using isend. The receiving processors post irecvs. To complete the communication we use both testsome and waitsome. What we are seeing is processors start by allocating a small amount of memory but as the code marches forward in time processors will allocate more memory within one of these mpi calls. Processors continue allocating larger and larger amounts of memory as time goes on. For example early on the allocation might be a couple KB but eventually it will get to around 1MB and i've even seen it as high as 14MB. I predict that if I ran it further it would allocate a much larger amount that 14MB. Processors are not all allocating this memory at the same time. In other parts of the code we do not use packing and we do not see this allocation behavior. I'm guessing that somewhere we are either miss-using packing or some other MPI feature and are causing MPI to leak. I was wondering if you could tell me why testsome/waitsome would allocate memory as that could provide a good hint as to how we are miss-using mpi. Currently we are using mvapich version 0.9.9 on Atlas at LLNL Thanks, Justin From dog at lanl.gov Fri Oct 5 15:28:18 2007 From: dog at lanl.gov (David Gunter) Date: Fri Oct 5 15:29:02 2007 Subject: [mvapich-discuss] MVAPICH2-1.0 fails to complete IMB 3.0 sendrecv test (Was: Error in final mpicc, mpiCC, ...) In-Reply-To: <200709250256.l8P2uLBr021496@xi.cse.ohio-state.edu> References: <200709250256.l8P2uLBr021496@xi.cse.ohio-state.edu> Message-ID: I was finally able to free up some time and try to pinpoint what is going wrong and send some follow-up to this problem. We are trying to build and run MVAPICH2 1.0 but the IMB 3.0 tests fail to complete (MPI1 tests only), hanging in the IMB_sendrecv tests when the message size goes about 4MB. I have traced the problem to using the --enable-fast configure switch. If I do not include that switch, things run fine. Here is how I am configuring and building: $ ./configure --prefix=${PREFIX} --libdir=${PREFIX}/lib64 \ --enable-sharedlibs=gcc \ --enable-romio --with-file-system=ufs+nfs+panfs \ --enable-threads=multiple --enable-g=dbg \ --enable-totalview --enable-debuginfo \ --with-mpe --enable-f77 --enable-f90 --enable-cxx \ --with-openib=${OPEN_IB_HOME} \ --with-pbs=${PBS_HOME} --with-device=osu_ch3:mrail \ --with-rdma=gen2 --with-pm=mpd $ make $ make install That is the configuration that works on our IB cluster. Again, if I add in the --enable-fast flag, then the IMB 3.0 sendrecv test hangs. -david On Sep 24, 2007, at 8:56 PM, Dhabaleswar Panda wrote: > David - We tried mpi_sendrecv on message sizes > 4 MB on various > system configurations and platforms with mvapich2-1.0. Things seem to > be working fine. We are not able to reproduce the error you are > mentioning. Could you let us know details of the situation where you > are seeing the mpi_sendrecv failure. > > Thanks, > > DK > >> What's the best way to go about resolving this? Note: we are unable >> to run mvapich2-1.0 on our systems. There is a problem with that >> version which prevents mpi_sendrecv from working for message sizes >> above 4MB. >> >> -david >> -- >> David Gunter >> HPC-4: HPC Environments: Parallel Tools Team >> Los Alamos National Laboratory >> >> >> >> --Apple-Mail-6--585903557 >> Content-Transfer-Encoding: quoted-printable >> Content-Type: text/html; >> charset=ISO-8859-1 >> >> > space; = >> -khtml-line-break: after-white-space; ">After building >> mvapich2-0.98p3 = >> on one of our systems I noticed that the final lib install >> directory = >> looked like this:

> class=3D"khtml-block-placeholder">
/opt/mvapich2/ >> mvapich2-0.98p3= >> /lib64

> DIV>
whereas = >> inside the final "mpicc" and other compiler scripts was the = >> following:

> class=3D"khtml-block-placeholder">
libdir=3D$ >> {exec_prefix}/lib> DIV>

which >> maps out = >> to=A0/opt/mvapich2/mvapich2-0.98p3/lib.

> class=3D"khtml-block-placeholder">
Thus, any attempt to = >> compile using the default mpi compile scripts results in libraries >> not = >> being found.

> class=3D"khtml-block-placeholder">
What's the best way >> to go = >> about resolving this?=A0 Note: we are unable to run mvapich2-1.0 >> on our = >> systems.=A0 There is a problem with that version which prevents = >> mpi_sendrecv from working for message sizes above 4MB.> DIV>

> class=3D"khtml-block-placeholder">
-david
> class=3D"Apple-style-span" style=3D"border-collapse: separate; = >> border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: >> Helvetica; = >> font-size: 12px; font-style: normal; font-variant: normal; font- >> weight: = >> normal; letter-spacing: normal; line-height: normal; text-align: >> auto; = >> -khtml-text-decorations-in-effect: none; text-indent: 0px; = >> -apple-text-size-adjust: auto; text-transform: none; orphans: 2; = >> white-space: normal; widows: 2; word-spacing: 0px; ">> class=3D"Apple-style-span" style=3D"border-collapse: separate; = >> border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: >> Helvetica; = >> font-size: 12px; font-style: normal; font-variant: normal; font- >> weight: = >> normal; letter-spacing: normal; line-height: normal; text-align: >> auto; = >> -khtml-text-decorations-in-effect: none; text-indent: 0px; = >> -apple-text-size-adjust: auto; text-transform: none; orphans: 2; = >> white-space: normal; widows: 2; word-spacing: 0px; ">> class=3D"Apple-style-span" style=3D"border-collapse: separate; = >> border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: >> Helvetica; = >> font-size: 12px; font-style: normal; font-variant: normal; font- >> weight: = >> normal; letter-spacing: normal; line-height: normal; text-align: >> auto; = >> -khtml-text-decorations-in-effect: none; text-indent: 0px; = >> -apple-text-size-adjust: auto; text-transform: none; orphans: 2; = >> white-space: normal; widows: 2; word-spacing: 0px; ">> class=3D"Apple-style-span" style=3D"border-collapse: separate; = >> border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: >> Helvetica; = >> font-size: 12px; font-style: normal; font-variant: normal; font- >> weight: = >> normal; letter-spacing: normal; line-height: normal; text-align: >> auto; = >> -khtml-text-decorations-in-effect: none; text-indent: 0px; = >> -apple-text-size-adjust: auto; text-transform: none; orphans: 2; = >> white-space: normal; widows: 2; word-spacing: 0px; ">> class=3D"Apple-style-span" style=3D"border-collapse: separate; = >> border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: >> Helvetica; = >> font-size: 12px; font-style: normal; font-variant: normal; font- >> weight: = >> normal; letter-spacing: normal; line-height: normal; text-align: >> auto; = >> -khtml-text-decorations-in-effect: none; text-indent: 0px; = >> -apple-text-size-adjust: auto; text-transform: none; orphans: 2; = >> white-space: normal; widows: 2; word-spacing: 0px; = >> ">
--
David Gunter
HPC-4: HPC Environments: = >> Parallel Tools Team
Los Alamos National = >> Laboratory

> class=3D"Apple-interchange-newline"> = >>

= >> >> --Apple-Mail-6--585903557-- >> >> --===============1825395516== >> Content-Type: text/plain; charset="us-ascii" >> MIME-Version: 1.0 >> Content-Transfer-Encoding: 7bit >> Content-Disposition: inline >> >> _______________________________________________ >> mvapich-discuss mailing list >> mvapich-discuss@cse.ohio-state.edu >> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss >> >> --===============1825395516==-- >> > From vishnu at cse.ohio-state.edu Sat Oct 6 00:54:39 2007 From: vishnu at cse.ohio-state.edu (Abhinav Vishnu) Date: Sat Oct 6 00:55:11 2007 Subject: [mvapich-discuss] waitsome/testsome memory allocation In-Reply-To: <47066C88.50207@cs.utah.edu> References: <47066C88.50207@cs.utah.edu> Message-ID: <20071006045437.GA17544@cse.ohio-state.edu> Hi Justin, Thanks for reporting the problem. It seems strange that Testsome and Waitsome primitives are actually allocating memory. Is it possible to get access to the code snippet, which actually reproduces this problem? Please let us know. Thanks, :- Abhinav > Hi, > > I am tracking down some memory issues in our code. And I am finding > strange memory allocations occurring within MPI_Waitsome and > MPI_Testsome. In one section of our code we use MPI_Pack and MPI_Unpack > to combine a bunch of small messages. We then send out the packed > messages using isend. The receiving processors post irecvs. To > complete the communication we use both testsome and waitsome. What we > are seeing is processors start by allocating a small amount of memory > but as the code marches forward in time processors will allocate more > memory within one of these mpi calls. Processors continue allocating > larger and larger amounts of memory as time goes on. For example early > on the allocation might be a couple KB but eventually it will get to > around 1MB and i've even seen it as high as 14MB. I predict that if I > ran it further it would allocate a much larger amount that 14MB. > Processors are not all allocating this memory at the same time. In > other parts of the code we do not use packing and we do not see this > allocation behavior. I'm guessing that somewhere we are either > miss-using packing or some other MPI feature and are causing MPI to leak. > > I was wondering if you could tell me why testsome/waitsome would > allocate memory as that could provide a good hint as to how we are > miss-using mpi. > > Currently we are using mvapich version 0.9.9 on Atlas at LLNL > > Thanks, > Justin > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss From vishnu at cse.ohio-state.edu Sat Oct 6 17:11:09 2007 From: vishnu at cse.ohio-state.edu (Abhinav Vishnu) Date: Sat Oct 6 17:11:42 2007 Subject: [mvapich-discuss] MVAPICH2-1.0 fails to complete IMB 3.0 sendrecv test (Was: Error in final mpicc, mpiCC, ...) In-Reply-To: References: <200709250256.l8P2uLBr021496@xi.cse.ohio-state.edu> Message-ID: <20071006211108.GB26207@cse.ohio-state.edu> Hi David, Thanks for your mail. The enable-fast option pick the appropriate options for fast execution. This turns off error checking and timing collection. We typically do not enable this because it will not report errors. We tested the latest version with this option and we are not able to reproduce the problem for messages up to 32 MB. In our testbed: The CPUs support the EM64T technology and run in 64 bit mode. The nodes support 8x PCI Express interfaces and are equipped with MT25208 HCAs with PCI Express interfaces. Thanks and regards, :- Abhinav > I was finally able to free up some time and try to pinpoint what is > going wrong and send some follow-up to this problem. > > We are trying to build and run MVAPICH2 1.0 but the IMB 3.0 tests > fail to complete (MPI1 tests only), hanging in the IMB_sendrecv tests > when the message size goes about 4MB. > > I have traced the problem to using the --enable-fast configure > switch. If I do not include that switch, things run fine. Here is > how I am configuring and building: > > $ ./configure --prefix=${PREFIX} --libdir=${PREFIX}/lib64 \ > --enable-sharedlibs=gcc \ > --enable-romio --with-file-system=ufs+nfs+panfs \ > --enable-threads=multiple --enable-g=dbg \ > --enable-totalview --enable-debuginfo \ > --with-mpe --enable-f77 --enable-f90 --enable-cxx \ > --with-openib=${OPEN_IB_HOME} \ > --with-pbs=${PBS_HOME} --with-device=osu_ch3:mrail \ > --with-rdma=gen2 --with-pm=mpd > > $ make > $ make install > > That is the configuration that works on our IB cluster. Again, if I > add in the --enable-fast flag, then the IMB 3.0 sendrecv test hangs. > > -david > > > On Sep 24, 2007, at 8:56 PM, Dhabaleswar Panda wrote: > > >David - We tried mpi_sendrecv on message sizes > 4 MB on various > >system configurations and platforms with mvapich2-1.0. Things seem to > >be working fine. We are not able to reproduce the error you are > >mentioning. Could you let us know details of the situation where you > >are seeing the mpi_sendrecv failure. > > > >Thanks, > > > >DK > > > >>What's the best way to go about resolving this? Note: we are unable > >>to run mvapich2-1.0 on our systems. There is a problem with that > >>version which prevents mpi_sendrecv from working for message sizes > >>above 4MB. > >> > >>-david > >>-- > >>David Gunter > >>HPC-4: HPC Environments: Parallel Tools Team > >>Los Alamos National Laboratory > >> > >> > >> > >>--Apple-Mail-6--585903557 > >>Content-Transfer-Encoding: quoted-printable > >>Content-Type: text/html; > >> charset=ISO-8859-1 > >> > >> >>space; = > >>-khtml-line-break: after-white-space; ">After building > >>mvapich2-0.98p3 = > >>on one of our systems I noticed that the final lib install > >>directory = > >>looked like this:

>>class=3D"khtml-block-placeholder">
/opt/mvapich2/ > >>mvapich2-0.98p3= > >>/lib64

>>DIV>
whereas = > >>inside the final "mpicc" and other compiler scripts was the = > >>following:

>>class=3D"khtml-block-placeholder">
libdir=3D$ > >>{exec_prefix}/lib >>DIV>

which > >>maps out = > >>to=A0/opt/mvapich2/mvapich2-0.98p3/lib.

>>class=3D"khtml-block-placeholder">
Thus, any attempt to = > >>compile using the default mpi compile scripts results in libraries > >>not = > >>being found.

>>class=3D"khtml-block-placeholder">
What's the best way > >>to go = > >>about resolving this?=A0 Note: we are unable to run mvapich2-1.0 > >>on our = > >>systems.=A0 There is a problem with that version which prevents = > >>mpi_sendrecv from working for message sizes above 4MB. >>DIV>

>>class=3D"khtml-block-placeholder">
-david
>>class=3D"Apple-style-span" style=3D"border-collapse: separate; = > >>border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: > >>Helvetica; = > >>font-size: 12px; font-style: normal; font-variant: normal; font- > >>weight: = > >>normal; letter-spacing: normal; line-height: normal; text-align: > >>auto; = > >>-khtml-text-decorations-in-effect: none; text-indent: 0px; = > >>-apple-text-size-adjust: auto; text-transform: none; orphans: 2; = > >>white-space: normal; widows: 2; word-spacing: 0px; "> >>class=3D"Apple-style-span" style=3D"border-collapse: separate; = > >>border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: > >>Helvetica; = > >>font-size: 12px; font-style: normal; font-variant: normal; font- > >>weight: = > >>normal; letter-spacing: normal; line-height: normal; text-align: > >>auto; = > >>-khtml-text-decorations-in-effect: none; text-indent: 0px; = > >>-apple-text-size-adjust: auto; text-transform: none; orphans: 2; = > >>white-space: normal; widows: 2; word-spacing: 0px; "> >>class=3D"Apple-style-span" style=3D"border-collapse: separate; = > >>border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: > >>Helvetica; = > >>font-size: 12px; font-style: normal; font-variant: normal; font- > >>weight: = > >>normal; letter-spacing: normal; line-height: normal; text-align: > >>auto; = > >>-khtml-text-decorations-in-effect: none; text-indent: 0px; = > >>-apple-text-size-adjust: auto; text-transform: none; orphans: 2; = > >>white-space: normal; widows: 2; word-spacing: 0px; "> >>class=3D"Apple-style-span" style=3D"border-collapse: separate; = > >>border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: > >>Helvetica; = > >>font-size: 12px; font-style: normal; font-variant: normal; font- > >>weight: = > >>normal; letter-spacing: normal; line-height: normal; text-align: > >>auto; = > >>-khtml-text-decorations-in-effect: none; text-indent: 0px; = > >>-apple-text-size-adjust: auto; text-transform: none; orphans: 2; = > >>white-space: normal; widows: 2; word-spacing: 0px; "> >>class=3D"Apple-style-span" style=3D"border-collapse: separate; = > >>border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: > >>Helvetica; = > >>font-size: 12px; font-style: normal; font-variant: normal; font- > >>weight: = > >>normal; letter-spacing: normal; line-height: normal; text-align: > >>auto; = > >>-khtml-text-decorations-in-effect: none; text-indent: 0px; = > >>-apple-text-size-adjust: auto; text-transform: none; orphans: 2; = > >>white-space: normal; widows: 2; word-spacing: 0px; = > >>">
--
David Gunter
HPC-4: HPC Environments: = > >>Parallel Tools Team
Los Alamos National = > >>Laboratory

>>class=3D"Apple-interchange-newline"> = > >>

= > >> > >>--Apple-Mail-6--585903557-- > >> > >>--===============1825395516== > >>Content-Type: text/plain; charset="us-ascii" > >>MIME-Version: 1.0 > >>Content-Transfer-Encoding: 7bit > >>Content-Disposition: inline > >> > >>_______________________________________________ > >>mvapich-discuss mailing list > >>mvapich-discuss@cse.ohio-state.edu > >>http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > >> > >>--===============1825395516==-- > >> > > > > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss From worldeb at ukr.net Mon Oct 8 13:47:10 2007 From: worldeb at ukr.net (Egor Tur) Date: Mon Oct 8 13:47:46 2007 Subject: [mvapich-discuss] Abort: Error creating CQ Message-ID: Hi folk. I have problem when I try submit jobs with mpiexec under mvapich1. See errors below: [0:node001] Abort: Error creating CQ at line 358 in file viainit.c [7:node008] Abort: Error creating CQ at line 358 in file viainit.c [2:node003] Abort: Error creating CQ at line 358 in file viainit.c [1:node002] Abort: Error creating CQ at line 358 in file viainit.c [3:node004] Abort: Error creating CQ at line 358 in file viainit.c [4:node005] Abort: Error creating CQ at line 358 in file viainit.c [5:node006] Abort: Error creating CQ at line 358 in file viainit.c [6:node007] Abort: Error creating CQ at line 358 in file viainit.c mpiexec: Warning: tasks 0-7 exited before completing MPI startup. [2:node003] Abort: Error creating CQ at line 358 in file viainit.c [6:node007] Abort: Error creating CQ at line 358 in file viainit.c [4:node005] Abort: Error creating CQ at line 358 in file viainit.c [3:node004] Abort: Error creating CQ at line 358 in file viainit.c [5:node006] Abort: Error creating CQ at line 358 in file viainit.c [1:node002] Abort: Error creating CQ at line 358 in file viainit.c [7:node008] Abort: Error creating CQ at line 358 in file viainit.c [0:node001] Abort: Error creating CQ at line 358 in file viainit.c mpiexec: Error: read_ib_one: rank -3 out of bounds [0..8). Command line is: mpiexec -comm mpich-ib -n 8 ./progs mpiexec is 0.82 version. mvapich1 is 0.9.9 version When I use mpirun for mvapich1 than jobs running correctly. Also when I submit jobs under ethernet with mpiexec than its running without errors. ulimit is unlimited. How can i fix this situation? Thanx. From koop at cse.ohio-state.edu Mon Oct 8 14:16:44 2007 From: koop at cse.ohio-state.edu (Matthew Koop) Date: Mon Oct 8 14:17:17 2007 Subject: [mvapich-discuss] Abort: Error creating CQ In-Reply-To: Message-ID: MVAPICH 0.9.9 changed the startup protocol from previous versions, which requires a patch to mpiexec. The patch is at the top of the mpiexec webpage: http://www.osc.edu/~pw/mpiexec/index.php Let us know if you continue to have problems after applying this patch. Thanks, Matt On Mon, 8 Oct 2007, Egor Tur wrote: > > Hi folk. > > I have problem when I try submit jobs with mpiexec under mvapich1. > See errors below: > [0:node001] Abort: Error creating CQ > at line 358 in file viainit.c > [7:node008] Abort: Error creating CQ > at line 358 in file viainit.c > [2:node003] Abort: Error creating CQ > at line 358 in file viainit.c > [1:node002] Abort: Error creating CQ > at line 358 in file viainit.c > [3:node004] Abort: Error creating CQ > at line 358 in file viainit.c > [4:node005] Abort: Error creating CQ > at line 358 in file viainit.c > [5:node006] Abort: Error creating CQ > at line 358 in file viainit.c > [6:node007] Abort: Error creating CQ > at line 358 in file viainit.c > mpiexec: Warning: tasks 0-7 exited before completing MPI startup. > [2:node003] Abort: Error creating CQ > at line 358 in file viainit.c > [6:node007] Abort: Error creating CQ > at line 358 in file viainit.c > [4:node005] Abort: Error creating CQ > at line 358 in file viainit.c > [3:node004] Abort: Error creating CQ > at line 358 in file viainit.c > [5:node006] Abort: Error creating CQ > at line 358 in file viainit.c > [1:node002] Abort: Error creating CQ > at line 358 in file viainit.c > [7:node008] Abort: Error creating CQ > at line 358 in file viainit.c > [0:node001] Abort: Error creating CQ > at line 358 in file viainit.c > mpiexec: Error: read_ib_one: rank -3 out of bounds [0..8). > > Command line is: > mpiexec -comm mpich-ib -n 8 ./progs > > mpiexec is 0.82 version. mvapich1 is 0.9.9 version > > When I use mpirun for mvapich1 than jobs running correctly. > Also when I submit jobs under ethernet with mpiexec than its running without errors. > ulimit is unlimited. > > How can i fix this situation? > > Thanx. > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > From worldeb at ukr.net Mon Oct 8 14:56:44 2007 From: worldeb at ukr.net (Egor Tur) Date: Mon Oct 8 14:57:24 2007 Subject: [mvapich-discuss] Abort: Error creating CQ In-Reply-To: Message-ID: Hi Matt, This error occur after applying this patch. Now I get latest version of mpiexec from svn, compile and install but errors are the same. Thanx again. > MVAPICH 0.9.9 changed the startup protocol from previous versions, which > requires a patch to mpiexec. The patch is at the top of the mpiexec > webpage: > > http://www.osc.edu/~pw/mpiexec/index.php > > Let us know if you continue to have problems after applying this patch. > > Thanks, > Matt > > On Mon, 8 Oct 2007, Egor Tur wrote: > > > > > Hi folk. > > > > I have problem when I try submit jobs with mpiexec under mvapich1. > > See errors below: > > [0:node001] Abort: Error creating CQ > > at line 358 in file viainit.c > > [7:node008] Abort: Error creating CQ > > at line 358 in file viainit.c > > [2:node003] Abort: Error creating CQ > > at line 358 in file viainit.c > > [1:node002] Abort: Error creating CQ > > at line 358 in file viainit.c > > [3:node004] Abort: Error creating CQ > > at line 358 in file viainit.c > > [4:node005] Abort: Error creating CQ > > at line 358 in file viainit.c > > [5:node006] Abort: Error creating CQ > > at line 358 in file viainit.c > > [6:node007] Abort: Error creating CQ > > at line 358 in file viainit.c > > mpiexec: Warning: tasks 0-7 exited before completing MPI startup. > > [2:node003] Abort: Error creating CQ > > at line 358 in file viainit.c > > [6:node007] Abort: Error creating CQ > > at line 358 in file viainit.c > > [4:node005] Abort: Error creating CQ > > at line 358 in file viainit.c > > [3:node004] Abort: Error creating CQ > > at line 358 in file viainit.c > > [5:node006] Abort: Error creating CQ > > at line 358 in file viainit.c > > [1:node002] Abort: Error creating CQ > > at line 358 in file viainit.c > > [7:node008] Abort: Error creating CQ > > at line 358 in file viainit.c > > [0:node001] Abort: Error creating CQ > > at line 358 in file viainit.c > > mpiexec: Error: read_ib_one: rank -3 out of bounds [0..8). > > > > Command line is: > > mpiexec -comm mpich-ib -n 8 ./progs > > > > mpiexec is 0.82 version. mvapich1 is 0.9.9 version > > > > When I use mpirun for mvapich1 than jobs running correctly. > > Also when I submit jobs under ethernet with mpiexec than its running without errors. > > ulimit is unlimited. > > > > How can i fix this situation? From luitjens at cs.utah.edu Wed Oct 10 00:14:10 2007 From: luitjens at cs.utah.edu (Justin) Date: Wed Oct 10 00:15:09 2007 Subject: [mvapich-discuss] waitsome/testsome memory allocation In-Reply-To: <47066C88.50207@cs.utah.edu> References: <47066C88.50207@cs.utah.edu> Message-ID: <470C5192.2000309@cs.utah.edu> Hi, The relevant stack traces on these allocations is the following: 1. /g/g20/luitjens/SCIRunMemory/dbg/lib/libCore_Thread.so [0x2a95c42284] 2. /lib64/tls/libc.so.6 [0x2a9a7152b0] 3. /lib64/tls/libc.so.6(gsignal+0x3d) [0x2a9a71521d] 4. /lib64/tls/libc.so.6(abort+0xfe) [0x2a9a716a1e] 5. /usr/local/tools/gnu/gcc/3.4.4_RH_chaos_3_x86_64/usr/lib64/libstdc++.so.6 in __gnu_cxx::__verbose_terminate_handler() 6. /usr/local/tools/gnu/gcc/3.4.4_RH_chaos_3_x86_64/usr/lib64/libstdc++.so.6 [0x2a9a499076] 7. /usr/local/tools/gnu/gcc/3.4.4_RH_chaos_3_x86_64/usr/lib64/libstdc++.so.6 [0x2a9a4990a3] 8. /usr/local/tools/gnu/gcc/3.4.4_RH_chaos_3_x86_64/usr/lib64/libstdc++.so.6 [0x2a9a4990b6] 9. /usr/local/tools/gnu/gcc/3.4.4_RH_chaos_3_x86_64/usr/lib64/libstdc++.so.6(__cxa_call_unexpected+0x48) [0x2a9a498fc8] a. /g/g20/luitjens/SCIRunMemory/dbg/lib/libCore_Malloc.so(malloc+0x63) [0x2a980c92ff] b. /g/g20/luitjens/mpi//lib/libmpich.so.1.0(MPID_SBiAllocate+0x39) [0x2a98bdce39] c. /g/g20/luitjens/mpi//lib/libmpich.so.1.0(MPID_SBalloc+0x2b) [0x2a98bdcf8b] d. /g/g20/luitjens/mpi//lib/libmpich.so.1.0(MPID_Msg_arrived+0xe3) [0x2a98bda2b3] e. /g/g20/luitjens/mpi//lib/libmpich.so.1.0(viadev_incoming_eager_start+0x43) [0x2a98be8753] f. /g/g20/luitjens/mpi//lib/libmpich.so.1.0(viadev_process_recv+0x2ef) [0x2a98be9b6f] 10. /g/g20/luitjens/mpi//lib/libmpich.so.1.0(MPID_DeviceCheck+0xde) [0x2a98bea77e] 11. /g/g20/luitjens/mpi//lib/libmpich.so.1.0(MPI_Testsome+0x45) [0x2a98be1b35] And 1. /g/g20/luitjens/SCIRunMemory/dbg/lib/libCore_Thread.so [0x2a95c42284] 2. /lib64/tls/libc.so.6 [0x2a9a7152b0] 3. /lib64/tls/libc.so.6(gsignal+0x3d) [0x2a9a71521d] 4. /lib64/tls/libc.so.6(abort+0xfe) [0x2a9a716a1e] 5. /usr/local/tools/gnu/gcc/3.4.4_RH_chaos_3_x86_64/usr/lib64/libstdc++.so.6 in __gnu_cxx::__verbose_terminate_handler() 6. /usr/local/tools/gnu/gcc/3.4.4_RH_chaos_3_x86_64/usr/lib64/libstdc++.so.6 [0x2a9a499076] 7. /usr/local/tools/gnu/gcc/3.4.4_RH_chaos_3_x86_64/usr/lib64/libstdc++.so.6 [0x2a9a4990a3] 8. /usr/local/tools/gnu/gcc/3.4.4_RH_chaos_3_x86_64/usr/lib64/libstdc++.so.6 [0x2a9a4990b6] 9. /usr/local/tools/gnu/gcc/3.4.4_RH_chaos_3_x86_64/usr/lib64/libstdc++.so.6(__cxa_call_unexpected+0x48) [0x2a9a498fc8] a. /g/g20/luitjens/SCIRunMemory/dbg/lib/libCore_Malloc.so(malloc+0x63) [0x2a980c92ff] b. /g/g20/luitjens/mpi//lib/libmpich.so.1.0(MPID_SBiAllocate+0x39) [0x2a98bdce39] c. /g/g20/luitjens/mpi//lib/libmpich.so.1.0(MPID_SBalloc+0x2b) [0x2a98bdcf8b] d. /g/g20/luitjens/mpi//lib/libmpich.so.1.0(MPID_Msg_arrived+0xe3) [0x2a98bda2b3] e. /g/g20/luitjens/mpi//lib/libmpich.so.1.0(smpi_net_lookup+0xc24) [0x2a98bd3bd4] f. /g/g20/luitjens/mpi//lib/libmpich.so.1.0(MPID_SMP_Check_incoming+0x2d5) [0x2a98bd4ee5] 10. /g/g20/luitjens/mpi//lib/libmpich.so.1.0(MPID_DeviceCheck+0x185) [0x2a98bea825] 11. /g/g20/luitjens/mpi//lib/libmpich.so.1.0(MPI_Testsome+0x45) [0x2a98be1b35] I have tried turning of the ELAN optimizations and the allocations still occur. The commonality in the stack traces appears to be calls to SBalloc. Is it possible that there is a leak in the MPI library that we are running into? When is the memory allocated in this function freed? If the same communication pattern is occurring over and over what would cause this function to keep allocating memory instead of reusing the memory that has already been allocated? Thanks Justin Justin wrote: > Hi, > > I am tracking down some memory issues in our code. And I am finding > strange memory allocations occurring within MPI_Waitsome and > MPI_Testsome. In one section of our code we use MPI_Pack and > MPI_Unpack to combine a bunch of small messages. We then send out the > packed messages using isend. The receiving processors post irecvs. > To complete the communication we use both testsome and waitsome. What > we are seeing is processors start by allocating a small amount of > memory but as the code marches forward in time processors will > allocate more memory within one of these mpi calls. Processors > continue allocating larger and larger amounts of memory as time goes > on. For example early on the allocation might be a couple KB but > eventually it will get to around 1MB and i've even seen it as high as > 14MB. I predict that if I ran it further it would allocate a much > larger amount that 14MB. Processors are not all allocating this > memory at the same time. In other parts of the code we do not use > packing and we do not see this allocation behavior. I'm guessing that > somewhere we are either miss-using packing or some other MPI feature > and are causing MPI to leak. > > I was wondering if you could tell me why testsome/waitsome would > allocate memory as that could provide a good hint as to how we are > miss-using mpi. > > Currently we are using mvapich version 0.9.9 on Atlas at LLNL > > Thanks, > Justin > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss From marc at klingon.uab.es Wed Oct 10 05:05:59 2007 From: marc at klingon.uab.es (Marc Noguera) Date: Wed Oct 10 05:05:57 2007 Subject: [mvapich-discuss] MVAPICH2 IFC compile problem Message-ID: <470C95F7.1060702@klingon.uab.es> Dear list, I guess this is the correct place to ask for help. I am trying to compile MVAPICH2-1.0 on a Opteron Fedora Core 2 system, using Intel 10.0 fortran compilers, and the OFED software stack. I am using the make.mvapich.ofa script which, I think, is meant for that purpose. my compiling environment is as follows: CC=icc CXX=icc F77=ifort F90=ifort the error comes out from configure at the step where f90 checking takes place. It is probably a simple problem but I simply don't know what to do. Any idea? is this a compiler problem or am I forgettin some configuration option? please find attach the config.log file generated from the compile. Thanks in advance Marc -------------- next part -------------- This file contains any messages produced by compilers while running configure, to aid debugging if configure makes a mistake. It was created by configure, which was generated by GNU Autoconf 2.59. Invocation command line was $ ./configure --prefix=/QFsoft/MVAPICH2/mvapich2-1.0_IFC10.0_64 --with-device=osu_ch3:mrail --with-rdma=gen2 --with-pm=mpd --disable-romio --without-mpe ## --------- ## ## Platform. ## ## --------- ## hostname = borg70.uab.es uname -m = x86_64 uname -r = 2.6.9-34.ELsmp uname -s = Linux uname -v = #1 SMP Fri Feb 24 16:56:28 EST 2006 /usr/bin/uname -p = unknown /bin/uname -X = unknown /bin/arch = x86_64 /usr/bin/arch -k = unknown /usr/convex/getsysinfo = unknown hostinfo = unknown /bin/machine = unknown /usr/bin/oslevel = unknown /bin/universe = unknown PATH: /QFsoft/TURBOMOLE/turbomole5.6/bin/i686-pc-linux-gnu PATH: /QFsoft/TURBOMOLE/turbomole5.6/scripts PATH: /usr/local/torque/bin PATH: /usr/kerberos/bin PATH: /usr/local/bin PATH: /bin PATH: /usr/bin PATH: /usr/X11R6/bin PATH: /usr/local/ofed/bin PATH: /usr/local/ofed/sbin PATH: /QFcomm PATH: /QFsoft/java/bin PATH: /QFsoft/BABEL/openbabel-2.1.0/bin PATH: /QFsoft/midas/bin PATH: /QFsoft/mmabin PATH: /QFsoft/mmabin/mm3 PATH: /QFsoft/IMOMM/mm392 PATH: /QFsoft/gulp/Exe PATH: . PATH: /users/sysuser/bin PATH: /opt/intel/cce/10.0.023/bin PATH: /opt/intel/fce/10.0.023/bin ## ----------- ## ## Core tests. ## ## ----------- ## configure:2916: checking for gcc configure:2942: result: icc configure:3186: checking for C compiler version configure:3189: icc --version &5 icc (ICC) 10.0 20070426 Copyright (C) 1985-2007 Intel Corporation. All rights reserved. configure:3192: $? = 0 configure:3194: icc -v &5 Version 10.0 configure:3197: $? = 0 configure:3199: icc -V &5 Intel(R) C Compiler for applications running on Intel(R) 64, Version 10.0 Build 20070426 Package ID: l_cc_p_10.0.023 Copyright (C) 1985-2007 Intel Corporation. All rights reserved. configure:3202: $? = 0 configure:3225: checking for C compiler default output file name configure:3228: icc -D_X86_64_ -D_SMP_ -DUSE_HEADER_CACHING -DONE_SIDED -DMPIDI_CH3_CHANNEL_RNDV -DMPID_USE_SEQUENCE_NUMBERS -DRDMA_CM -I/usr/local/ofed/include -O2 conftest.c -L/usr/local/ofed/lib64 -lrdmacm -libverbs -libumad -lpthread >&5 configure:3231: $? = 0 configure:3277: result: a.out configure:3282: checking whether the C compiler works configure:3288: ./a.out configure:3291: $? = 0 configure:3308: result: yes configure:3315: checking whether we are cross compiling configure:3317: result: no configure:3320: checking for suffix of executables configure:3322: icc -o conftest -D_X86_64_ -D_SMP_ -DUSE_HEADER_CACHING -DONE_SIDED -DMPIDI_CH3_CHANNEL_RNDV -DMPID_USE_SEQUENCE_NUMBERS -DRDMA_CM -I/usr/local/ofed/include -O2 conftest.c -L/usr/local/ofed/lib64 -lrdmacm -libverbs -libumad -lpthread >&5 configure:3325: $? = 0 configure:3350: result: configure:3356: checking for suffix of object files configure:3377: icc -c -D_X86_64_ -D_SMP_ -DUSE_HEADER_CACHING -DONE_SIDED -DMPIDI_CH3_CHANNEL_RNDV -DMPID_USE_SEQUENCE_NUMBERS -DRDMA_CM -I/usr/local/ofed/include -O2 conftest.c >&5 configure:3380: $? = 0 configure:3402: result: o configure:3406: checking whether we are using the GNU C compiler configure:3430: icc -c -D_X86_64_ -D_SMP_ -DUSE_HEADER_CACHING -DONE_SIDED -DMPIDI_CH3_CHANNEL_RNDV -DMPID_USE_SEQUENCE_NUMBERS -DRDMA_CM -I/usr/local/ofed/include -O2 conftest.c >&5 configure:3436: $? = 0 configure:3439: test -z || test ! -s conftest.err configure:3442: $? = 0 configure:3445: test -s conftest.o configure:3448: $? = 0 configure:3461: result: yes configure:3467: checking whether icc accepts -g configure:3488: icc -c -g conftest.c >&5 configure:3494: $? = 0 configure:3497: test -z || test ! -s conftest.err configure:3500: $? = 0 configure:3503: test -s conftest.o configure:3506: $? = 0 configure:3517: result: yes configure:3534: checking for icc option to accept ANSI C configure:3604: icc -c -D_X86_64_ -D_SMP_ -DUSE_HEADER_CACHING -DONE_SIDED -DMPIDI_CH3_CHANNEL_RNDV -DMPID_USE_SEQUENCE_NUMBERS -DRDMA_CM -I/usr/local/ofed/include -O2 conftest.c >&5 configure:3610: $? = 0 configure:3613: test -z || test ! -s conftest.err configure:3616: $? = 0 configure:3619: test -s conftest.o configure:3622: $? = 0 configure:3640: result: none needed configure:3658: icc -c -D_X86_64_ -D_SMP_ -DUSE_HEADER_CACHING -DONE_SIDED -DMPIDI_CH3_CHANNEL_RNDV -DMPID_USE_SEQUENCE_NUMBERS -DRDMA_CM -I/usr/local/ofed/include -O2 conftest.c >&5 conftest.c(2): error: identifier "choke" is undefined choke me ^ conftest.c(3): error: expected a ";" compilation aborted for conftest.c (code 2) configure:3664: $? = 2 configure: failed program was: | #ifndef __cplusplus | choke me | #endif configure:4170: checking for type of weak symbol support configure:4199: icc -o conftest -D_X86_64_ -D_SMP_ -DUSE_HEADER_CACHING -DONE_SIDED -DMPIDI_CH3_CHANNEL_RNDV -DMPID_USE_SEQUENCE_NUMBERS -DRDMA_CM -I/usr/local/ofed/include -O2 conftest.c -L/usr/local/ofed/lib64 -lrdmacm -libverbs -libumad -lpthread >&5 configure:4205: $? = 0 configure:4208: test -z || test ! -s conftest.err configure:4211: $? = 0 configure:4214: test -s conftest configure:4217: $? = 0 configure:4395: result: pragma weak configure:4422: checking whether __attribute__ ((weak)) allowed configure:4444: icc -c -D_X86_64_ -D_SMP_ -DUSE_HEADER_CACHING -DONE_SIDED -DMPIDI_CH3_CHANNEL_RNDV -DMPID_USE_SEQUENCE_NUMBERS -DRDMA_CM -I/usr/local/ofed/include -O2 conftest.c >&5 configure:4450: $? = 0 configure:4453: test -z || test ! -s conftest.err configure:4456: $? = 0 configure:4459: test -s conftest.o configure:4462: $? = 0 configure:4473: result: yes configure:4487: checking for multiple weak symbol support configure:4524: result: yes configure:4681: checking for Fortran 77 compiler version configure:4685: ifort --version &5 ifort (IFORT) 10.0 20070426 Copyright (C) 1985-2007 Intel Corporation. All rights reserved. configure:4688: $? = 0 configure:4690: ifort -v &5 Version 10.0 configure:4693: $? = 0 configure:4695: ifort -V &5 Intel(R) Fortran Compiler for applications running on Intel(R) 64, Version 10.0 Build 20070426 Package ID: l_fc_p_10.0.023 Copyright (C) 1985-2007 Intel Corporation. All rights reserved. configure:4698: $? = 0 configure:4706: checking whether we are using the GNU Fortran 77 compiler configure:4720: ifort -c -L/usr/local/ofed/lib64 conftest.F >&5 fortcom: Error: conftest.F, line 3: Syntax error, found END-OF-STATEMENT when expecting one of: => = . ( : % choke me ---------------^ fortcom: Error: conftest.F, line 3: This statement is positioned incorrectly and/or has syntax errors. choke me ---------------^ compilation aborted for conftest.F (code 1) configure:4726: $? = 1 configure: failed program was: | program main | #ifndef __GNUC__ | choke me | #endif | | end configure:4751: result: no configure:4757: checking whether ifort accepts -g configure:4769: ifort -c -g conftest.f >&5 configure:4775: $? = 0 configure:4778: test -z || test ! -s conftest.err configure:4781: $? = 0 configure:4784: test -s conftest.o configure:4787: $? = 0 configure:4799: result: yes configure:5079: checking how to get verbose linking output from ifort configure:5090: ifort -c -L/usr/local/ofed/lib64 conftest.f >&5 configure:5096: $? = 0 configure:5099: test -z || test ! -s conftest.err configure:5102: $? = 0 configure:5105: test -s conftest.o configure:5108: $? = 0 configure:5124: ifort -o conftest -L/usr/local/ofed/lib64 -v conftest.f -L/usr/local/ofed/lib64 -lrdmacm -libverbs -libumad -lpthread >&5 Version 10.0 /opt/intel/fce/10.0.023/bin/fortcom -D__INTEL_COMPILER=1000 -D_MT -D__ELF__ -D__INTEL_COMPILER_BUILD_DATE=20070426 -D__unix__ -D__unix -D__linux__ -D__linux -D__gnu_linux__ -Dunix -Dlinux -D__x86_64 -D__x86_64__ -mGLOB_pack_sort_init_list -I. -I/opt/intel/fce/10.0.023/include -I/opt/intel/fce/10.0.023/substitute_headers -I/usr/lib/gcc-lib/x86_64-redhat-linux/3.3.3/include -I/usr/local/include -I/usr/include -I/usr/lib/gcc-lib/x86_64-redhat-linux/3.3.3/include "-fp_modspec fp_speculation_FAST" -O2 -mP1OPT_version=1000 -mGLOB_source_language=GLOB_SOURCE_LANGUAGE_F90 -mGLOB_tune_for_fort -mGLOB_use_fort_dope_vector -mP2OPT_static_promotion -mP1OPT_print_version=FALSE -mP3OPT_use_mspp_call_convention -mCG_use_gas_got_workaround=F -mP2OPT_align_option_used=TRUE "-mGLOB_options_string=-o conftest -L/usr/local/ofed/lib64 -v -L/usr/local/ofed/lib64 -lrdmacm -libverbs -libumad -lpthread" -mGLOB_cxx_limited_range=FALSE -mP2OPT_eh_nirvana -mGLOB_diag_file=/tmp/ifort67Opjw.diag -mGLOB_as_output_backup_file_name=/tmp/ifortzlrNx2as_.s -mGLOB_machine_model=GLOB_MACHINE_MODEL_EFI2 -mGLOB_fp_speculation=GLOB_FP_SPECULATION_FAST -mGLOB_extended_instructions=0x8 -mP2OPT_subs_out_of_bound=FALSE -mGLOB_ansi_alias -mIPOPT_ninl_user_level=2 -mIPOPT_args_in_regs=0 -mPGOPTI_value_profile_use=T -mIPOPT_activate -mIPOPT_lite -mP2OPT_hlo_level=2 -mP2OPT_hlo -mPAROPT_par_report=1 -mIPOPT_obj_output_file_name=/tmp/ifort67Opjw.o "-mGLOB_linker_version=2.15.90.0.3 20040415" -mP3OPT_asm_target=P3OPT_ASM_TARGET_GAS -mGLOB_obj_output_file=/tmp/ifort67Opjw.o -mGLOB_source_dialect=GLOB_SOURCE_DIALECT_FORTRAN -mP1OPT_source_file_name=conftest.f conftest.f ld /usr/lib64/crt1.o /usr/lib64/crti.o /usr/lib/gcc-lib/x86_64-redhat-linux/3.3.3/crtbegin.o --eh-frame-hdr -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o conftest /opt/intel/fce/10.0.023/lib/for_main.o -L/usr/local/ofed/lib64 /tmp/ifort67Opjw.o -L/usr/local/ofed/lib64 -lrdmacm -libverbs -libumad -lpthread -L/opt/intel/fce/10.0.023/lib -L/usr/lib/gcc-lib/x86_64-redhat-linux/3.3.3/ -L/usr/lib64 -Bstatic -lifport -lifcore -limf -lsvml -Bdynamic -lm -Bstatic -lipgo -lirc -Bdynamic -lc -lgcc_s -lgcc -Bstatic -lirc_s -Bdynamic -ldl -lc /usr/lib/gcc-lib/x86_64-redhat-linux/3.3.3/crtend.o /usr/lib64/crtn.o rm /tmp/ifortLeTpcggnudirs rm /tmp/ifortLkDqqMgas rm /tmp/ifortzlrNx2as_.s rm /tmp/ifort08YaFildashv rm /tmp/ifortKNY3Myarg rm /tmp/ifortCsvtYOgnudirs rm /tmp/ifort67Opjw.o configure:5180: result: -v configure:5182: checking for Fortran libraries of ifort configure:5202: ifort -o conftest -L/usr/local/ofed/lib64 -v conftest.f -L/usr/local/ofed/lib64 -lrdmacm -libverbs -libumad -lpthread >&5 Version 10.0 /opt/intel/fce/10.0.023/bin/fortcom -D__INTEL_COMPILER=1000 -D_MT -D__ELF__ -D__INTEL_COMPILER_BUILD_DATE=20070426 -D__unix__ -D__unix -D__linux__ -D__linux -D__gnu_linux__ -Dunix -Dlinux -D__x86_64 -D__x86_64__ -mGLOB_pack_sort_init_list -I. -I/opt/intel/fce/10.0.023/include -I/opt/intel/fce/10.0.023/substitute_headers -I/usr/lib/gcc-lib/x86_64-redhat-linux/3.3.3/include -I/usr/local/include -I/usr/include -I/usr/lib/gcc-lib/x86_64-redhat-linux/3.3.3/include "-fp_modspec fp_speculation_FAST" -O2 -mP1OPT_version=1000 -mGLOB_source_language=GLOB_SOURCE_LANGUAGE_F90 -mGLOB_tune_for_fort -mGLOB_use_fort_dope_vector -mP2OPT_static_promotion -mP1OPT_print_version=FALSE -mP3OPT_use_mspp_call_convention -mCG_use_gas_got_workaround=F -mP2OPT_align_option_used=TRUE "-mGLOB_options_string=-o conftest -L/usr/local/ofed/lib64 -v -L/usr/local/ofed/lib64 -lrdmacm -libverbs -libumad -lpthread" -mGLOB_cxx_limited_range=FALSE -mP2OPT_eh_nirvana -mGLOB_diag_file=/tmp/ifortOtTzkW.diag -mGLOB_as_output_backup_file_name=/tmp/ifortmPeWDuas_.s -mGLOB_machine_model=GLOB_MACHINE_MODEL_EFI2 -mGLOB_fp_speculation=GLOB_FP_SPECULATION_FAST -mGLOB_extended_instructions=0x8 -mP2OPT_subs_out_of_bound=FALSE -mGLOB_ansi_alias -mIPOPT_ninl_user_level=2 -mIPOPT_args_in_regs=0 -mPGOPTI_value_profile_use=T -mIPOPT_activate -mIPOPT_lite -mP2OPT_hlo_level=2 -mP2OPT_hlo -mPAROPT_par_report=1 -mIPOPT_obj_output_file_name=/tmp/ifortOtTzkW.o "-mGLOB_linker_version=2.15.90.0.3 20040415" -mP3OPT_asm_target=P3OPT_ASM_TARGET_GAS -mGLOB_obj_output_file=/tmp/ifortOtTzkW.o -mGLOB_source_dialect=GLOB_SOURCE_DIALECT_FORTRAN -mP1OPT_source_file_name=conftest.f conftest.f ld /usr/lib64/crt1.o /usr/lib64/crti.o /usr/lib/gcc-lib/x86_64-redhat-linux/3.3.3/crtbegin.o --eh-frame-hdr -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o conftest /opt/intel/fce/10.0.023/lib/for_main.o -L/usr/local/ofed/lib64 /tmp/ifortOtTzkW.o -L/usr/local/ofed/lib64 -lrdmacm -libverbs -libumad -lpthread -L/opt/intel/fce/10.0.023/lib -L/usr/lib/gcc-lib/x86_64-redhat-linux/3.3.3/ -L/usr/lib64 -Bstatic -lifport -lifcore -limf -lsvml -Bdynamic -lm -Bstatic -lipgo -lirc -Bdynamic -lc -lgcc_s -lgcc -Bstatic -lirc_s -Bdynamic -ldl -lc /usr/lib/gcc-lib/x86_64-redhat-linux/3.3.3/crtend.o /usr/lib64/crtn.o rm /tmp/ifortrnO5aFgnudirs rm /tmp/ifortbf54tdgas rm /tmp/ifortmPeWDuas_.s rm /tmp/ifortWOTNNLldashv rm /tmp/ifortbdd8X2arg rm /tmp/ifortEPz3bkgnudirs rm /tmp/ifortOtTzkW.o configure:5364: result: -L/usr/local/ofed/lib64 -lrdmacm -libverbs -libumad -lpthread -L/opt/intel/fce/10.0.023/lib -L/usr/lib/gcc-lib/x86_64-redhat-linux/3.3.3/ -L/usr/lib64 -lifport -lifcore -limf -lsvml -lm -lipgo -lirc -lgcc_s -lirc_s -ldl configure:5375: checking whether C can link with -L/usr/local/ofed/lib64 -lrdmacm -libverbs -libumad -lpthread -L/opt/intel/fce/10.0.023/lib -L/usr/lib/gcc-lib/x86_64-redhat-linux/3.3.3/ -L/usr/lib64 -lifport -lifcore -limf -lsvml -lm -lipgo -lirc -lgcc_s -lirc_s -ldl configure:5396: icc -o conftest -D_X86_64_ -D_SMP_ -DUSE_HEADER_CACHING -DONE_SIDED -DMPIDI_CH3_CHANNEL_RNDV -DMPID_USE_SEQUENCE_NUMBERS -DRDMA_CM -I/usr/local/ofed/include -O2 conftest.c -L/usr/local/ofed/lib64 -lrdmacm -libverbs -libumad -lpthread -L/usr/local/ofed/lib64 -lrdmacm -libverbs -libumad -lpthread -L/opt/intel/fce/10.0.023/lib -L/usr/lib/gcc-lib/x86_64-redhat-linux/3.3.3/ -L/usr/lib64 -lifport -lifcore -limf -lsvml -lm -lipgo -lirc -lgcc_s -lirc_s -ldl >&5 configure:5402: $? = 0 configure:5405: test -z || test ! -s conftest.err configure:5408: $? = 0 configure:5411: test -s conftest configure:5414: $? = 0 configure:5426: result: yes configure:5516: icc -c -D_X86_64_ -D_SMP_ -DUSE_HEADER_CACHING -DONE_SIDED -DMPIDI_CH3_CHANNEL_RNDV -DMPID_USE_SEQUENCE_NUMBERS -DRDMA_CM -I/usr/local/ofed/include -O2 conftest.c >&5 configure:5522: $? = 0 configure:5525: test -z || test ! -s conftest.err configure:5528: $? = 0 configure:5531: test -s conftest.o configure:5534: $? = 0 configure:5547: checking for linker for Fortran main programs configure:5565: icc -c -D_X86_64_ -D_SMP_ -DUSE_HEADER_CACHING -DONE_SIDED -DMPIDI_CH3_CHANNEL_RNDV -DMPID_USE_SEQUENCE_NUMBERS -DRDMA_CM -I/usr/local/ofed/include -O2 conftest.c >&5 configure:5568: $? = 0 configure:5587: ifort -c -L/usr/local/ofed/lib64 conftest.f >&5 configure:5590: $? = 0 configure:5598: result: Use Fortran to link programs configure:5623: checking for Fortran 77 name mangling configure:5648: ifort -c -L/usr/local/ofed/lib64 conftest.f 1>&5 configure:5651: $? = 0 configure:5685: icc -o conftest -D_X86_64_ -D_SMP_ -DUSE_HEADER_CACHING -DONE_SIDED -DMPIDI_CH3_CHANNEL_RNDV -DMPID_USE_SEQUENCE_NUMBERS -DRDMA_CM -I/usr/local/ofed/include -O2 conftest.c fconftestf.o -L/usr/local/ofed/lib64 -lrdmacm -libverbs -libumad -lpthread -L/opt/intel/fce/10.0.023/lib -L/usr/lib/gcc-lib/x86_64-redhat-linux/3.3.3/ -L/usr/lib64 -lifport -lifcore -limf -lsvml -lm -lipgo -lirc -lgcc_s -lirc_s -ldl -L/usr/local/ofed/lib64 -lrdmacm -libverbs -libumad -lpthread >&5 conftest.c(23): warning #266: function "my_name" declared implicitly my_name(); ^ /tmp/icc7ux2Wh.o(.text+0x1d): In function `main': : undefined reference to `my_name' configure:5691: $? = 1 configure: failed program was: | /* confdefs.h. */ | | #define PACKAGE_NAME "" | #define PACKAGE_TARNAME "" | #define PACKAGE_VERSION "" | #define PACKAGE_STRING "" | #define PACKAGE_BUGREPORT "" | #define HAVE_ERROR_CHECKING MPID_ERROR_LEVEL_ALL | #define MPICH_ERROR_MSG_LEVEL MPICH_ERROR_MSG_ALL | #define USE_LOGGING MPID_LOGGING_NONE | #define HAVE_RUNTIME_THREADCHECK 1 | #define MPICH_THREAD_LEVEL MPI_THREAD_MULTIPLE | #define USE_THREAD_IMPL MPICH_THREAD_IMPL_GLOBAL_MUTEX | #define HAVE_PRAGMA_WEAK 1 | #define USE_WEAK_SYMBOLS 1 | #define HAVE_MULTIPLE_PRAGMA_WEAK 1 | #define HAVE_LONG_LONG 1 | /* end confdefs.h. */ | | int | main () | { | my_name(); | ; | return 0; | } configure:5730: icc -o conftest -D_X86_64_ -D_SMP_ -DUSE_HEADER_CACHING -DONE_SIDED -DMPIDI_CH3_CHANNEL_RNDV -DMPID_USE_SEQUENCE_NUMBERS -DRDMA_CM -I/usr/local/ofed/include -O2 conftest.c fconftestf.o -L/usr/local/ofed/lib64 -lrdmacm -libverbs -libumad -lpthread -L/opt/intel/fce/10.0.023/lib -L/usr/lib/gcc-lib/x86_64-redhat-linux/3.3.3/ -L/usr/lib64 -lifport -lifcore -limf -lsvml -lm -lipgo -lirc -lgcc_s -lirc_s -ldl -L/usr/local/ofed/lib64 -lrdmacm -libverbs -libumad -lpthread >&5 conftest.c(23): warning #266: function "my_name_" declared implicitly my_name_(); ^ configure:5736: $? = 0 configure:5739: test -z || test ! -s conftest.err configure:5742: $? = 0 configure:5745: test -s conftest configure:5748: $? = 0 configure:5999: result: lower underscore configure:6092: checking what libraries are needed to link Fortran programs with C routines that use stdio configure:6111: icc -c -D_X86_64_ -D_SMP_ -DUSE_HEADER_CACHING -DONE_SIDED -DMPIDI_CH3_CHANNEL_RNDV -DMPID_USE_SEQUENCE_NUMBERS -DRDMA_CM -I/usr/local/ofed/include -O2 conftestc.c 1>&5 configure:6114: $? = 0 configure:6123: ifort -L/usr/local/ofed/lib64 -o conftest conftest.f conftestc.o 1>&5 configure:6126: $? = 0 configure:6144: result: none configure:6300: checking that f works as the extension for Fortran 90 program configure:6306: ifort -c conftest.f >&5 configure:6309: $? = 0 configure:6311: result: yes configure:6358: checking for Fortran 90 compiler version configure:6362: ifort --version &5 ifort (IFORT) 10.0 20070426 Copyright (C) 1985-2007 Intel Corporation. All rights reserved. configure:6365: $? = 0 configure:6367: ifort -v &5 Version 10.0 configure:6370: $? = 0 configure:6372: ifort -V &5 Intel(R) Fortran Compiler for applications running on Intel(R) 64, Version 10.0 Build 20070426 Package ID: l_fc_p_10.0.023 Copyright (C) 1985-2007 Intel Corporation. All rights reserved. configure:6375: $? = 0 configure:6383: checking whether we are using the GNU Fortran 90 compiler configure:6397: ifort -c conftest.F >&5 fortcom: Error: conftest.F, line 3: Syntax error, found END-OF-STATEMENT when expecting one of: => = . ( : % choke me ---------------^ fortcom: Error: conftest.F, line 3: This statement is positioned incorrectly and/or has syntax errors. choke me ---------------^ compilation aborted for conftest.F (code 1) configure:6403: $? = 1 configure: failed program was: | program main | #ifndef __GNUC__ | choke me | #endif | | end configure:6428: result: no configure:6435: checking whether ifort accepts -g configure:6447: ifort -c -g conftest.f >&5 configure:6453: $? = 0 configure:6456: test -z || test ! -s conftest.err configure:6459: $? = 0 configure:6462: test -s conftest.o configure:6465: $? = 0 configure:6477: result: yes configure:6755: checking for extension for Fortran 90 programs configure:6763: ifort -c conftest.f90 1>&5 configure:6766: $? = 0 configure:6768: result: f90 configure:6789: checking whether the Fortran 90 compiler (ifort ) works configure:6807: ifort -o conftest conftest.f90 -L/usr/local/ofed/lib64 -lrdmacm -libverbs -libumad -lpthread 1>&5 configure:6810: $? = 0 configure:6832: result: yes configure:6838: checking whether the Fortran 90 compiler (ifort ) is a cross-compiler configure:6840: result: no configure:6845: checking whether Fortran 90 works with Fortran 77 configure:6878: ifort -c -L/usr/local/ofed/lib64 conftest2.f 1>&5 configure:6881: $? = 0 configure:6883: ifort -o conftest conftest1.f90 conftest2.o -L/usr/local/ofed/lib64 -lrdmacm -libverbs -libumad -lpthread 1>&5 configure:6886: $? = 0 configure:6898: result: yes configure:6938: checking whether Fortran accepts ! for comments configure:6957: ifort -c -L/usr/local/ofed/lib64 conftest.f >&5 configure:6963: $? = 0 configure:6966: test -z || test ! -s conftest.err configure:6969: $? = 0 configure:6972: test -s conftest.o configure:6975: $? = 0 configure:6992: result: yes configure:7002: checking for include directory flag for Fortran configure:7022: ifort -c -L/usr/local/ofed/lib64 -Isrc conftest.f 1>&5 configure:7025: $? = 0 configure:7035: result: -I configure:7053: checking for Fortran 77 flag for library directories configure:7071: ifort -c -L/usr/local/ofed/lib64 conftest1.f 1>&5 configure:7074: $? = 0 configure:7082: ar cr conftest2/libconftest.a conftest1.o configure:7085: $? = 0 configure:7088: ranlib conftest2/libconftest.a configure:7091: $? = 0 configure:7095: ifort -o conftest -L/usr/local/ofed/lib64 -Lconftest2 conftest.f -lconftest 1>&5 configure:7098: $? = 0 configure:7112: result: -L configure:7184: checking for which Fortran libraries are needed to link C with Fortran configure:7195: ifort -c -L/usr/local/ofed/lib64 conftest.f 1>&5 configure:7198: $? = 0 configure:7234: icc -o conftest -D_X86_64_ -D_SMP_ -DUSE_HEADER_CACHING -DONE_SIDED -DMPIDI_CH3_CHANNEL_RNDV -DMPID_USE_SEQUENCE_NUMBERS -DRDMA_CM -I/usr/local/ofed/include -O2 conftest.c mconftestf.o -L/usr/local/ofed/lib64 -lrdmacm -libverbs -libumad -lpthread >&5 conftest.c(31): warning #266: function "ftest_" declared implicitly ftest_(); ^ configure:7240: $? = 0 configure:7243: test -z || test ! -s conftest.err configure:7246: $? = 0 configure:7249: test -s conftest configure:7252: $? = 0 configure:7405: result: none configure:7501: checking whether Fortran compiler processes .F files with C preprocessor configure:7512: ifort -c -L/usr/local/ofed/lib64 conftest.F 1>&5 configure:7515: $? = 0 configure:7537: result: yes configure:7682: checking that f works as the extension for Fortran 90 program configure:7688: ifort -c conftest.f >&5 configure:7691: $? = 0 configure:7693: result: yes configure:7734: checking for Fortran 90 compiler version configure:7744: ifort --version &5 ifort (IFORT) 10.0 20070426 Copyright (C) 1985-2007 Intel Corporation. All rights reserved. configure:7747: $? = 0 configure:7749: ifort -v &5 Version 10.0 configure:7752: $? = 0 configure:7754: ifort -V &5 Intel(R) Fortran Compiler for applications running on Intel(R) 64, Version 10.0 Build 20070426 Package ID: l_fc_p_10.0.023 Copyright (C) 1985-2007 Intel Corporation. All rights reserved. configure:7757: $? = 0 configure:7765: checking whether we are using the GNU Fortran 90 compiler configure:7810: result: no configure:7817: checking whether ifort accepts -g configure:7859: result: yes configure:7882: checking for extension for Fortran 90 programs configure:7890: ifort -c conftest.f90 1>&5 configure:7893: $? = 0 configure:7895: result: f90 configure:7916: checking whether the Fortran 90 compiler (ifort ) works configure:7934: ifort -o conftest conftest.f90 -L/usr/local/ofed/lib64 -lrdmacm -libverbs -libumad -lpthread 1>&5 configure:7937: $? = 0 configure:7958: result: yes configure:7964: checking whether the Fortran 90 compiler (ifort ) is a cross-compiler configure:7966: result: no configure:7995: checking for Fortran 90 module extension configure:8014: ifort -c conftest.f >&5 configure:8017: $? = 0 configure:8069: result: mod configure:8079: checking for Fortran 90 module include flag configure:8104: ifort -c conftest.f >&5 configure:8107: $? = 0 configure:8182: result: -I configure:8217: checking whether Fortran 90 accepts f90 suffix configure:8236: ifort -c conftest.f90 >&5 configure:8242: $? = 0 configure:8245: test -z || test ! -s conftest.err configure:8248: $? = 0 configure:8251: test -s conftest.o configure:8254: $? = 0 configure:8271: result: yes configure:8288: checking whether Fortran 90 compiler processes .F90 files with C preprocessor configure:8299: ifort -c conftest.F90 1>&5 configure:8302: $? = 0 configure:8324: result: yes configure:8354: checking what libraries are needed to link Fortran90 programs with C routines that use stdio configure:8374: icc -c -D_X86_64_ -D_SMP_ -DUSE_HEADER_CACHING -DONE_SIDED -DMPIDI_CH3_CHANNEL_RNDV -DMPID_USE_SEQUENCE_NUMBERS -DRDMA_CM -I/usr/local/ofed/include -O2 conftestc.c 1>&5 configure:8377: $? = 0 configure:8386: ifort -o conftest conftest.f90 conftestc.o 1>&5 configure:8389: $? = 0 configure:8407: result: none configure:8427: checking for f90 compiler vendor configure:8467: result: intel configure:8544: checking for c++ configure:8570: result: icc configure:8688: checking for C++ compiler version configure:8691: icc --version &5 icc (ICC) 10.0 20070426 Copyright (C) 1985-2007 Intel Corporation. All rights reserved. configure:8694: $? = 0 configure:8696: icc -v &5 Version 10.0 configure:8699: $? = 0 configure:8701: icc -V &5 Intel(R) C Compiler for applications running on Intel(R) 64, Version 10.0 Build 20070426 Package ID: l_cc_p_10.0.023 Copyright (C) 1985-2007 Intel Corporation. All rights reserved. configure:8704: $? = 0 configure:8707: checking whether we are using the GNU C++ compiler configure:8731: icc -c conftest.cc >&5 configure:8737: $? = 0 configure:8740: test -z || test ! -s conftest.err configure:8743: $? = 0 configure:8746: test -s conftest.o configure:8749: $? = 0 configure:8762: result: yes configure:8768: checking whether icc accepts -g configure:8789: icc -c -g conftest.cc >&5 configure:8795: $? = 0 configure:8798: test -z || test ! -s conftest.err configure:8801: $? = 0 configure:8804: test -s conftest.o configure:8807: $? = 0 configure:8818: result: yes configure:8860: icc -c -g -O2 conftest.cc >&5 configure:8866: $? = 0 configure:8869: test -z || test ! -s conftest.err configure:8872: $? = 0 configure:8875: test -s conftest.o configure:8878: $? = 0 configure:8904: icc -c -g -O2 conftest.cc >&5 conftest.cc(26): error: identifier "exit" is undefined exit (42); ^ compilation aborted for conftest.cc (code 2) configure:8910: $? = 2 configure: failed program was: | /* confdefs.h. */ | | #define PACKAGE_NAME "" | #define PACKAGE_TARNAME "" | #define PACKAGE_VERSION "" | #define PACKAGE_STRING "" | #define PACKAGE_BUGREPORT "" | #define HAVE_ERROR_CHECKING MPID_ERROR_LEVEL_ALL | #define MPICH_ERROR_MSG_LEVEL MPICH_ERROR_MSG_ALL | #define USE_LOGGING MPID_LOGGING_NONE | #define HAVE_RUNTIME_THREADCHECK 1 | #define MPICH_THREAD_LEVEL MPI_THREAD_MULTIPLE | #define USE_THREAD_IMPL MPICH_THREAD_IMPL_GLOBAL_MUTEX | #define HAVE_PRAGMA_WEAK 1 | #define USE_WEAK_SYMBOLS 1 | #define HAVE_MULTIPLE_PRAGMA_WEAK 1 | #define HAVE_LONG_LONG 1 | #define F77_NAME_LOWER_USCORE 1 | #define STDCALL | #define HAVE_FORTRAN_BINDING 1 | /* end confdefs.h. */ | | int | main () | { | exit (42); | ; | return 0; | } configure:8860: icc -c -g -O2 conftest.cc >&5 conftest.cc(22): error: namespace "std" has no member "exit" extern "C" void std::exit (int) throw (); using std::exit; ^ conftest.cc(22): error: namespace "std" has no member "exit" extern "C" void std::exit (int) throw (); using std::exit; ^ compilation aborted for conftest.cc (code 2) configure:8866: $? = 2 configure: failed program was: | /* confdefs.h. */ | | #define PACKAGE_NAME "" | #define PACKAGE_TARNAME "" | #define PACKAGE_VERSION "" | #define PACKAGE_STRING "" | #define PACKAGE_BUGREPORT "" | #define HAVE_ERROR_CHECKING MPID_ERROR_LEVEL_ALL | #define MPICH_ERROR_MSG_LEVEL MPICH_ERROR_MSG_ALL | #define USE_LOGGING MPID_LOGGING_NONE | #define HAVE_RUNTIME_THREADCHECK 1 | #define MPICH_THREAD_LEVEL MPI_THREAD_MULTIPLE | #define USE_THREAD_IMPL MPICH_THREAD_IMPL_GLOBAL_MUTEX | #define HAVE_PRAGMA_WEAK 1 | #define USE_WEAK_SYMBOLS 1 | #define HAVE_MULTIPLE_PRAGMA_WEAK 1 | #define HAVE_LONG_LONG 1 | #define F77_NAME_LOWER_USCORE 1 | #define STDCALL | #define HAVE_FORTRAN_BINDING 1 | /* end confdefs.h. */ | extern "C" void std::exit (int) throw (); using std::exit; | #include | int | main () | { | exit (42); | ; | return 0; | } configure:8860: icc -c -g -O2 conftest.cc >&5 conftest.cc(22): error: namespace "std" has no member "exit" extern "C" void std::exit (int); using std::exit; ^ conftest.cc(22): error: namespace "std" has no member "exit" extern "C" void std::exit (int); using std::exit; ^ compilation aborted for conftest.cc (code 2) configure:8866: $? = 2 configure: failed program was: | /* confdefs.h. */ | | #define PACKAGE_NAME "" | #define PACKAGE_TARNAME "" | #define PACKAGE_VERSION "" | #define PACKAGE_STRING "" | #define PACKAGE_BUGREPORT "" | #define HAVE_ERROR_CHECKING MPID_ERROR_LEVEL_ALL | #define MPICH_ERROR_MSG_LEVEL MPICH_ERROR_MSG_ALL | #define USE_LOGGING MPID_LOGGING_NONE | #define HAVE_RUNTIME_THREADCHECK 1 | #define MPICH_THREAD_LEVEL MPI_THREAD_MULTIPLE | #define USE_THREAD_IMPL MPICH_THREAD_IMPL_GLOBAL_MUTEX | #define HAVE_PRAGMA_WEAK 1 | #define USE_WEAK_SYMBOLS 1 | #define HAVE_MULTIPLE_PRAGMA_WEAK 1 | #define HAVE_LONG_LONG 1 | #define F77_NAME_LOWER_USCORE 1 | #define STDCALL | #define HAVE_FORTRAN_BINDING 1 | /* end confdefs.h. */ | extern "C" void std::exit (int); using std::exit; | #include | int | main () | { | exit (42); | ; | return 0; | } configure:8860: icc -c -g -O2 conftest.cc >&5 configure:8866: $? = 0 configure:8869: test -z || test ! -s conftest.err configure:8872: $? = 0 configure:8875: test -s conftest.o configure:8878: $? = 0 configure:8904: icc -c -g -O2 conftest.cc >&5 configure:8910: $? = 0 configure:8913: test -z || test ! -s conftest.err configure:8916: $? = 0 configure:8919: test -s conftest.o configure:8922: $? = 0 configure:8965: checking whether the compiler supports exceptions configure:8994: icc -c conftest.cc >&5 configure:9000: $? = 0 configure:9003: test -z || test ! -s conftest.err configure:9006: $? = 0 configure:9009: test -s conftest.o configure:9012: $? = 0 configure:9030: result: yes configure:9040: checking whether the compiler recognizes bool as a built-in type configure:9073: icc -c conftest.cc >&5 configure:9079: $? = 0 configure:9082: test -z || test ! -s conftest.err configure:9085: $? = 0 configure:9088: test -s conftest.o configure:9091: $? = 0 configure:9109: result: yes configure:9119: checking whether the compiler implements namespaces configure:9148: icc -c conftest.cc >&5 configure:9154: $? = 0 configure:9157: test -z || test ! -s conftest.err configure:9160: $? = 0 configure:9163: test -s conftest.o configure:9166: $? = 0 configure:9184: result: yes configure:9205: checking whether available configure:9229: icc -c conftest.cc >&5 configure:9235: $? = 0 configure:9238: test -z || test ! -s conftest.err configure:9241: $? = 0 configure:9244: test -s conftest.o configure:9247: $? = 0 configure:9258: result: yes configure:9264: checking whether the compiler implements the namespace std configure:9297: icc -c conftest.cc >&5 configure:9303: $? = 0 configure:9306: test -z || test ! -s conftest.err configure:9309: $? = 0 configure:9312: test -s conftest.o configure:9315: $? = 0 configure:9334: result: yes configure:9348: checking whether available configure:9372: icc -c conftest.cc >&5 conftest.cc(29): catastrophic error: could not open source file "math" #include ^ compilation aborted for conftest.cc (code 4) configure:9378: $? = 4 configure: failed program was: | /* confdefs.h. */ | | #define PACKAGE_NAME "" | #define PACKAGE_TARNAME "" | #define PACKAGE_VERSION "" | #define PACKAGE_STRING "" | #define PACKAGE_BUGREPORT "" | #define HAVE_ERROR_CHECKING MPID_ERROR_LEVEL_ALL | #define MPICH_ERROR_MSG_LEVEL MPICH_ERROR_MSG_ALL | #define USE_LOGGING MPID_LOGGING_NONE | #define HAVE_RUNTIME_THREADCHECK 1 | #define MPICH_THREAD_LEVEL MPI_THREAD_MULTIPLE | #define USE_THREAD_IMPL MPICH_THREAD_IMPL_GLOBAL_MUTEX | #define HAVE_PRAGMA_WEAK 1 | #define USE_WEAK_SYMBOLS 1 | #define HAVE_MULTIPLE_PRAGMA_WEAK 1 | #define HAVE_LONG_LONG 1 | #define F77_NAME_LOWER_USCORE 1 | #define STDCALL | #define HAVE_FORTRAN_BINDING 1 | #ifdef __cplusplus | extern "C" void exit (int) throw (); | #endif | #define HAVE_CXX_EXCEPTIONS | #define HAVE_NAMESPACES | #define HAVE_NAMESPACE_STD | /* end confdefs.h. */ | | #include | | int | main () | { | using namespace std; | ; | return 0; | } configure:9401: result: no configure:9450: checking for GNU g++ version configure:9481: icc -o conftest conftest.cc -L/usr/local/ofed/lib64 -lrdmacm -libverbs -libumad -lpthread >&5 configure:9484: $? = 0 configure:9486: ./conftest ./conftest: error while loading shared libraries: libcxaguard.so.5: cannot open shared object file: No such file or directory configure:9489: $? = 127 configure: program exited with status 127 configure: failed program was: | /* confdefs.h. */ | | #define PACKAGE_NAME "" | #define PACKAGE_TARNAME "" | #define PACKAGE_VERSION "" | #define PACKAGE_STRING "" | #define PACKAGE_BUGREPORT "" | #define HAVE_ERROR_CHECKING MPID_ERROR_LEVEL_ALL | #define MPICH_ERROR_MSG_LEVEL MPICH_ERROR_MSG_ALL | #define USE_LOGGING MPID_LOGGING_NONE | #define HAVE_RUNTIME_THREADCHECK 1 | #define MPICH_THREAD_LEVEL MPI_THREAD_MULTIPLE | #define USE_THREAD_IMPL MPICH_THREAD_IMPL_GLOBAL_MUTEX | #define HAVE_PRAGMA_WEAK 1 | #define USE_WEAK_SYMBOLS 1 | #define HAVE_MULTIPLE_PRAGMA_WEAK 1 | #define HAVE_LONG_LONG 1 | #define F77_NAME_LOWER_USCORE 1 | #define STDCALL | #define HAVE_FORTRAN_BINDING 1 | #ifdef __cplusplus | extern "C" void exit (int) throw (); | #endif | #define HAVE_CXX_EXCEPTIONS | #define HAVE_NAMESPACES | #define HAVE_NAMESPACE_STD | /* end confdefs.h. */ | #include | int main() { | int v = -1, m = -1; | FILE *fp = fopen("conftest.out","w"); | #ifdef __GNUC_MINOR__ | m = __GNUC_MINOR__; | #endif | #ifdef __GNUC__ | v = __GNUC__; | #endif | fprintf( fp, "v=%d, m=%d\n", v, m ); | fclose( fp ); | return 0; | } configure:9508: result: unknown configure:9583: checking for perl configure:9601: found /usr/bin/perl configure:9613: result: /usr/bin/perl configure:9625: checking for ar configure:9641: found /usr/bin/ar configure:9651: result: ar configure:9672: checking for ranlib configure:9688: found /usr/bin/ranlib configure:9698: result: ranlib configure:9720: checking for etags configure:9749: result: no configure:9875: checking whether global variables handled properly configure:9932: result: yes configure:9953: checking for a BSD-compatible install configure:10008: result: /usr/bin/install -c configure:10028: checking whether install works configure:10036: result: yes configure:10052: checking whether install breaks libraries configure:10093: result: no configure:10115: checking whether mkdir -p works configure:10131: result: yes configure:10149: checking for make configure:10165: found /usr/bin/make configure:10175: result: make configure:10188: checking whether clock skew breaks make configure:10209: result: no configure:10219: checking whether make supports include configure:10243: result: no configure:10252: checking whether make allows comments in actions configure:10275: result: no configure:10280: WARNING: Your make does not allow comments in target code. Using this make may cause problems when building programs. You should consider using gnumake instead. configure:10289: checking for virtual path format configure:10328: result: VPATH configure:10338: checking whether make sets CFLAGS configure:10360: result: yes configure:10407: checking for bash configure:10437: result: /bin/sh configure:10458: checking whether /bin/sh supports arrays configure:10467: result: yes con