From nilesha at cdac.in Fri Nov 2 09:33:11 2007 From: nilesha at cdac.in (Nilesh Awate) Date: Fri Nov 2 09:36:13 2007 Subject: [mvapich-discuss] getting error with mpdboot Message-ID: <472B2717.3050801@cdac.in> Hi, I'm using mvapivh2-1.0 with tcp/ip stack on 8 nodes, but this nodes have different updates of RHEL4 & i've compiled mvapich2 on each node. 2 nodes out of them are on different subnet I've configured everything on each node which is required. i got following error > mpdboot -n 8 mpdboot_in01 (handle_mpd_output 382): failed to handshake with mpd on in09; recvd output={} i tried excluding 'in09' which work very fine . . . > mpdboot -n 7 any clue . . . Nilesh From koop at cse.ohio-state.edu Fri Nov 2 12:04:03 2007 From: koop at cse.ohio-state.edu (Matthew Koop) Date: Fri Nov 2 12:04:39 2007 Subject: [mvapich-discuss] getting error with mpdboot In-Reply-To: <472B2717.3050801@cdac.in> Message-ID: Nilesh, You may want to follow the MPICH2 setup guide for MPD to diagnose the problem: http://www-unix.mcs.anl.gov/mpi/mpich2/downloads/mpich2-doc-user.pdf You will likely want to try starting the mpd daemons by hand rather than through mpdboot to figure out setup issues since it will give you more output. Matt On Fri, 2 Nov 2007, Nilesh Awate wrote: > Hi, > I'm using mvapivh2-1.0 with tcp/ip stack on 8 nodes, > but this nodes have different updates of RHEL4 & i've compiled mvapich2 > on each node. 2 nodes out of them are on different subnet > I've configured everything on each node which is required. > > i got following error > > > mpdboot -n 8 > mpdboot_in01 (handle_mpd_output 382): failed to handshake with mpd on > in09; recvd output={} > > > i tried excluding 'in09' which work very fine . . . > > mpdboot -n 7 > > any clue . . . > > Nilesh > > > > > > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > From marc at klingon.uab.es Tue Nov 6 06:01:38 2007 From: marc at klingon.uab.es (Marc Noguera) Date: Tue Nov 6 06:16:25 2007 Subject: [mvapich-discuss] MVAPICH2 "cannot create cq" error In-Reply-To: References: Message-ID: <47304992.3000405@klingon.uab.es> Matt, list sorry for the delayed answer. I have been out these days. I have just run the osu benchmarks. I attach the output for all of them. At first sight, everything is ok except for the Multi-Threaded Latency Test which gives no output. Could you confirm this? If that is the case, then I guess the problem is coming from the application ARMCI libs, so I'll try to get the newest version of Global Arrays an recompile. Thank you Marc En/na Matthew Koop ha escrit: > Marc, > > Good to hear simple programs are now working. Can you try the included > osu_benchmarks and verify those are running as well? That will verify that > the memlock issues are solved. > > It appears that the "create qp returned NULL" is coming from the ARMCI > library rather than MVAPICH2, which suggests that the problem may be there > instead. > > Matt > > On Wed, 31 Oct 2007, Marc Noguera wrote: > > >> Dear all, >> thanks again for the suggestions, >> I did reboot both test nodes after modifying settings, and the "cannot >> create cq" error disappeared. >> Now I can compile test applications, like hellow.c and obtain the a >> hello world output from all the process: >> However I want to use the NWchem application, which I can compile with >> not many troubles now. When I try to run this application using two >> nodes I obtain the following: >> >> borg70.uab.es:/tmp/T3>~/mvapich2/bin/mpiexec -n 2 >> ~/soft/nwchem-5.0/bin/LINUX64/nwchem >> ARMCI configured for 2 cluster nodes. Network protocol is 'OpenIB Verbs >> API'. >> 1:create qp returned NULL: 0 >> 1:create qp returned NULL: 0 >> Last System Error Message from Task 1:: Invalid argument >> application called MPI_Abort(MPI_COMM_WORLD, 0) - process 10:create qp >> returned NULL: 0 >> 0:create qp returned NULL: 0 >> Last System Error Message from Task 0:: Invalid argument >> application called MPI_Abort(MPI_COMM_WORLD, 0) - process 0rank 1 in job >> 4 borg70.uab.es_50642 caused collective abort of all ranks >> exit status of rank 1: killed by signal 9 >> borg70.uab.es:/tmp/T3> >> >> Of course, the mpdring is up. I understand from the userguide that this >> error is also related to the memlock issue, similary to the "cannot >> create cq" one. Is that correct? >> Ulimit -l gives "unlimited" output in both nodes, so I am stuck here. >> Any clue? >> Thanks in advance >> Marc >> >> En/na Matthew Koop ha escrit: >> >>> Marc, >>> >>> Did you perhaps update the lockable memory settings after starting the MPD >>> ring? If so, try exiting the ring using mpdallexit and then booting it >>> again with mpdboot so that mpd gets the new ulimit settings. >>> >>> Also, have you tried the ibv_rc_pingpong test that comes with the OFED >>> distribution? It will allow you to verify that your IB installation is >>> correct. >>> >>> Let us know if restarting the ring helps at all. >>> >>> Matt >>> >>> >>> On Tue, 30 Oct 2007, Marc Noguera wrote: >>> >>> >>> >>>> Dear list, >>>> I am trying to use mvapich2 on our cluster. I am making some tests on >>>> two dual opteron nodes running fedora core 6, using mvapich2 and >>>> portland compilers. >>>> I have successfully compiled mvapich2 using these compilers, at least I >>>> think so. I have used make.mvapich.ofa script as I have OFED 1.2.5 >>>> software stack installed on infiniband hardware. >>>> Environment at mvapich2 compile time was: >>>> CC=pgcc >>>> CXX=pgCC >>>> F77=pgf77 >>>> F90=pgf90 >>>> OPEN_IB_HOME=/usr/local/ofed >>>> PREFIX=~/mvapich2 >>>> RDMA_CM_SUPPORT="no" >>>> >>>> After that, I have compiled the pi3f90.f test program (mpif90 pi3f90) >>>> and I am trying to execute the a.out binary using mpdboot and mpiexec. >>>> >>>> I have done as said userguide, and have the .mpd.conf file (wiht 600 >>>> permissions) in $HOME. I have also created a mpd.hosts in my workdir, >>>> with these two lines containing: >>>> >>>> 10.10.1.170 ifhn=10.10.1.170 >>>> 10.10.1.171 ifhn=10.10.1.171 >>>> >>>> Moreover, I have modified /etc/security/limits.conf and /etc/init.d/sshd >>>> to ensure unlimited mem_lock values, also as mentioned by the userguide. >>>> That is, "ulimit -l" command gives a "unlimited" output on both test >>>> machines. >>>> Finally when trying to run the a.out test application, I obtain: >>>> >>>> borg70.uab.es:/users/sysuser/test/T3>~/mvapich2/bin/mpdboot -n 2 >>>> --ifhn=10.10.1.170 >>>> borg70.uab.es:/users/sysuser/test/T3>~/mvapich2/bin/mpdtrace -l >>>> borg70.uab.es_43715 (10.10.1.170) >>>> borg71.uab.es_37091 (10.10.1.171) >>>> borg70.uab.es:/users/sysuser/test/T3>~/mvapich2/bin/mpiexec -n 2 ./a.out >>>> Fatal error in MPI_Init: >>>> Other MPI error, error stack: >>>> MPIR_Init_thread(259)....: Initialization failed >>>> MPID_Init(102)...........: channel initialization failed >>>> MPIDI_CH3_Init(178)......: >>>> MPIDI_CH3I_RMDA_init(203): Failed to Initialize HCA type >>>> rdma_iba_hca_init(639)...: cannot create cq >>>> rank 1 in job 1 borg70.uab.es_43715 caused collective abort of all ranks >>>> exit status of rank 1: killed by signal 9 >>>> borg70.uab.es:/users/sysuser/test/T3> >>>> >>>> >>>> >>>> In the troubleshooting section of the userguid I find that "cannot >>>> create cq" are possibly due to mem_lock limits, but I think I have fixed >>>> these, or at least I think so. >>>> I am really stuck at this point. >>>> Can you give me any hint on what am I doing wrong? >>>> >>>> Thanks in advance >>>> Marc >>>> >>>> >>>> >>>> _______________________________________________ >>>> mvapich-discuss mailing list >>>> mvapich-discuss@cse.ohio-state.edu >>>> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss >>>> >>>> >>>> >>> >>> >>> > > > > > -------------- next part -------------- A non-text attachment was scrubbed... Name: osu_output.tgz Type: application/octet-stream Size: 2358 bytes Desc: not available Url : http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20071106/c62fa0aa/osu_output.obj From xmxmxie at gmail.com Thu Nov 8 21:46:44 2007 From: xmxmxie at gmail.com (Xie Min) Date: Thu Nov 8 21:47:27 2007 Subject: [mvapich-discuss] question about checkpoint/restart Message-ID: <91bd441b0711081846m465fa7d5yff7bdb93dca5064f@mail.gmail.com> hi, We have an infiniband cluster and use mvapich2-1.0.1 on it. Recently I want to try the checkoint/restart functions, but I met a problem at the startup of MPI programs. I am using NPB 3.2, below is the output of the error: root@node2 tmp]# /usr/local/mvapich2-ckpt/bin/mpiexec -l -np 4 ./bt.A.4 1: Fatal error in MPI_Init: 1: Other MPI error, error stack: 1: MPIR_Init_thread(259)...: Initialization failed 0: Fatal error in MPI_Init: 0: Other MPI error, error stack: 0: MPIR_Init_thread(259)...: Initialization failed 0: MPID_Init(102)..........: channel initialization failed 0: MPIDI_CH3_Init(178).....: 0: MPIDI_CH3I_CM_Init(1034): MPICM_Init_UD 0: MPICM_Init_UD(900)......: Couldn't create completion channel 1: MPID_Init(102)..........: channel initialization failed 1: MPIDI_CH3_Init(178).....: 1: MPIDI_CH3I_CM_Init(1034): MPICM_Init_UD 1: MPICM_Init_UD(900)......: Couldn't create completion channel rank 1 in job 6 node2_33394 caused collective abort of all ranks exit status of rank 1: killed by signal 9 rank 0 in job 6 node2_33394 caused collective abort of all ranks exit status of rank 0: killed by signal 9 We use BLCR 0.5.6 as the checkpointer, and OFED-1.2 stack, node in our cluster is based on Intel Xeon EMT64. I recompiled a new mvapich2 without checkpoint/restart in it, and don't met the error above. How to cope with this problem? Thanks. XM From huanwei at cse.ohio-state.edu Thu Nov 8 22:05:19 2007 From: huanwei at cse.ohio-state.edu (wei huang) Date: Thu Nov 8 22:05:54 2007 Subject: [mvapich-discuss] question about checkpoint/restart In-Reply-To: <91bd441b0711081846m465fa7d5yff7bdb93dca5064f@mail.gmail.com> Message-ID: Hi Xie, It looks to me that your environment is not setup properly. Can you run normal mvapich2 (without CR) using: mpiexec -np 4 -env MV2_ON_DEMAND_THRESHOLD 1 ./bt.A.4 Also, if you have multiple ofed installations can you make sure that bt.A.4 is linked with the correct ofed libraries. BTW, blcr 0.5.6 is a bit old. We would suggest you install the newest version 0.6.1, Although your problem does not seem to be related with the blcr version. Thanks Regards, Wei Huang 774 Dreese Lab, 2015 Neil Ave, Dept. of Computer Science and Engineering Ohio State University OH 43210 Tel: (614)292-8501 On Fri, 9 Nov 2007, Xie Min wrote: > hi, > We have an infiniband cluster and use mvapich2-1.0.1 on it. > Recently I want to try the checkoint/restart functions, but I met a > problem at the startup of MPI programs. I am using NPB 3.2, below is > the output of the error: > > root@node2 tmp]# /usr/local/mvapich2-ckpt/bin/mpiexec -l -np 4 ./bt.A.4 > 1: Fatal error in MPI_Init: > 1: Other MPI error, error stack: > 1: MPIR_Init_thread(259)...: Initialization failed > 0: Fatal error in MPI_Init: > 0: Other MPI error, error stack: > 0: MPIR_Init_thread(259)...: Initialization failed > 0: MPID_Init(102)..........: channel initialization failed > 0: MPIDI_CH3_Init(178).....: > 0: MPIDI_CH3I_CM_Init(1034): MPICM_Init_UD > 0: MPICM_Init_UD(900)......: Couldn't create completion channel > 1: MPID_Init(102)..........: channel initialization failed > 1: MPIDI_CH3_Init(178).....: > 1: MPIDI_CH3I_CM_Init(1034): MPICM_Init_UD > 1: MPICM_Init_UD(900)......: Couldn't create completion channel > rank 1 in job 6 node2_33394 caused collective abort of all ranks > exit status of rank 1: killed by signal 9 > rank 0 in job 6 node2_33394 caused collective abort of all ranks > exit status of rank 0: killed by signal 9 > > > We use BLCR 0.5.6 as the checkpointer, and OFED-1.2 stack, node in our > cluster is based on Intel Xeon EMT64. > > I recompiled a new mvapich2 without checkpoint/restart in it, and > don't met the error above. > > How to cope with this problem? > > Thanks. > > XM > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > From xmxmxie at gmail.com Sat Nov 10 02:08:11 2007 From: xmxmxie at gmail.com (Xie Min) Date: Sat Nov 10 02:08:48 2007 Subject: [mvapich-discuss] question about checkpoint/restart In-Reply-To: References: <91bd441b0711081846m465fa7d5yff7bdb93dca5064f@mail.gmail.com> Message-ID: <91bd441b0711092308n233ba350r69d1eefa98165182@mail.gmail.com> Hi, I compiled a new mvapich2 with these setting: ENABLE_CKPT=no PTMALLOC=yes RDMA_CM_SUPPORT=no and run the bt.A.4 using your method, but still met the problem, below I attach some more information, hopes they will be helpful. [root@node1 tmp]# ldd ./bt.A.4 libibverbs.so.1 => /usr/lib64/libibverbs.so.1 (0x00002aaaaabc1000) libibumad.so.1 => /usr/lib64/libibumad.so.1 (0x00002aaaaaccd000) libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x00002aaaaadd7000) libm.so.6 => /lib64/tls/libm.so.6 (0x00002aaaaaeec000) libc.so.6 => /lib64/tls/libc.so.6 (0x00002aaaab073000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002aaaab2a7000) libdl.so.2 => /lib64/libdl.so.2 (0x00002aaaab3b2000) libibcommon.so.1 => /usr/lib64/libibcommon.so.1 (0x00002aaaab4b6000) /lib64/ld-linux-x86-64.so.2 (0x00002aaaaaaab000) [root@node1 tmp]# rpm -qf /usr/lib64/libibverbs.so.1 libibverbs-1.1.1-0 [root@node1 tmp]# rpm -qi libibverbs-1.1.1-0 Name : libibverbs Relocations: (not relocatable) Version : 1.1.1 Vendor: OpenFabrics Release : 0 Build Date: Mon Sep 24 22:36:37 2007 Install Date: Mon Sep 24 23:31:38 2007 Build Host: star99 Group : System Environment/Libraries Source RPM: ofa_user-1.2-0.src.rpm Size : 201033 License: GPL/BSD Signature : (none) URL : http://www.openfabrics.org/ Summary : A library for direct userspace use of InfiniBand Description : libibverbs is a library that allows userspace processes to use InfiniBand "verbs" as described in the InfiniBand Architecture Specification. This includes direct hardware access for fast path operations. [root@node1 tmp]# /usr/local/mvapich2-ckpt/bin/mpiexec -l -np 4 -env MV2_ON_DEMAND_THRESHOLD 1 ./bt.A.4 2: Fatal error in MPI_Init: 2: Other MPI error, error stack: 0: Fatal error in MPI_Init: 0: Other MPI error, error stack: 0: MPIR_Init_thread(259)...: Initialization failed 0: MPID_Init(102)..........: channel initialization failed 0: MPIDI_CH3_Init(178).....: 0: MPIDI_CH3I_CM_Init(1034): MPICM_Init_UD 0: MPICM_Init_UD(900)......: Couldn't create completion channel 2: MPIR_Init_thread(259)...: Initialization failed 2: MPID_Init(102)..........: channel initialization failed 2: MPIDI_CH3_Init(178).....: 2: MPIDI_CH3I_CM_Init(1034): MPICM_Init_UD 2: MPICM_Init_UD(900)......: Couldn't create completion channel rank 0 in job 1 node1_43661 caused collective abort of all ranks exit status of rank 0: killed by signal 9 but when I run using "/usr/local/mvapich2-ckpt/bin/mpiexec -l -np 4 ./bt.A.4", it is OK. XM 2007/11/9, wei huang : > Hi Xie, > > It looks to me that your environment is not setup properly. Can you run > normal mvapich2 (without CR) using: > > mpiexec -np 4 -env MV2_ON_DEMAND_THRESHOLD 1 ./bt.A.4 > > Also, if you have multiple ofed installations can you make sure that > bt.A.4 is linked with the correct ofed libraries. > > BTW, blcr 0.5.6 is a bit old. We would suggest you install the newest > version 0.6.1, Although your problem does not seem to be related with the > blcr version. > > Thanks > > Regards, > Wei Huang > > 774 Dreese Lab, 2015 Neil Ave, > Dept. of Computer Science and Engineering > Ohio State University > OH 43210 > Tel: (614)292-8501 > > From xmxmxie at gmail.com Sat Nov 10 03:29:00 2007 From: xmxmxie at gmail.com (Xie Min) Date: Sat Nov 10 03:29:39 2007 Subject: [mvapich-discuss] question about checkpoint/restart In-Reply-To: References: <91bd441b0711081846m465fa7d5yff7bdb93dca5064f@mail.gmail.com> Message-ID: <91bd441b0711100029v79f6ea32q2be99abdd7ee7ea1@mail.gmail.com> Hi, Maybe something wrong in our configuration, I compile mvapich2 on another server, and with checkpoint/restart in it, this version works fine. XM 2007/11/9, wei huang : > Hi Xie, > > It looks to me that your environment is not setup properly. Can you run > normal mvapich2 (without CR) using: > > mpiexec -np 4 -env MV2_ON_DEMAND_THRESHOLD 1 ./bt.A.4 > > Also, if you have multiple ofed installations can you make sure that > bt.A.4 is linked with the correct ofed libraries. > > BTW, blcr 0.5.6 is a bit old. We would suggest you install the newest > version 0.6.1, Although your problem does not seem to be related with the > blcr version. > > Thanks > > Regards, > Wei Huang > > 774 Dreese Lab, 2015 Neil Ave, > Dept. of Computer Science and Engineering > Ohio State University > OH 43210 > Tel: (614)292-8501 > > > On Fri, 9 Nov 2007, Xie Min wrote: > > > hi, > > We have an infiniband cluster and use mvapich2-1.0.1 on it. > > Recently I want to try the checkoint/restart functions, but I met a > > problem at the startup of MPI programs. I am using NPB 3.2, below is > > the output of the error: > > > > root@node2 tmp]# /usr/local/mvapich2-ckpt/bin/mpiexec -l -np 4 ./bt.A.4 > > 1: Fatal error in MPI_Init: > > 1: Other MPI error, error stack: > > 1: MPIR_Init_thread(259)...: Initialization failed > > 0: Fatal error in MPI_Init: > > 0: Other MPI error, error stack: > > 0: MPIR_Init_thread(259)...: Initialization failed > > 0: MPID_Init(102)..........: channel initialization failed > > 0: MPIDI_CH3_Init(178).....: > > 0: MPIDI_CH3I_CM_Init(1034): MPICM_Init_UD > > 0: MPICM_Init_UD(900)......: Couldn't create completion channel > > 1: MPID_Init(102)..........: channel initialization failed > > 1: MPIDI_CH3_Init(178).....: > > 1: MPIDI_CH3I_CM_Init(1034): MPICM_Init_UD > > 1: MPICM_Init_UD(900)......: Couldn't create completion channel > > rank 1 in job 6 node2_33394 caused collective abort of all ranks > > exit status of rank 1: killed by signal 9 > > rank 0 in job 6 node2_33394 caused collective abort of all ranks > > exit status of rank 0: killed by signal 9 > > > > > > We use BLCR 0.5.6 as the checkpointer, and OFED-1.2 stack, node in our > > cluster is based on Intel Xeon EMT64. > > > > I recompiled a new mvapich2 without checkpoint/restart in it, and > > don't met the error above. > > > > How to cope with this problem? > > > > Thanks. > > > > XM > > _______________________________________________ > > mvapich-discuss mailing list > > mvapich-discuss@cse.ohio-state.edu > > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > > > > From sopib1am at ss1000.ms.mff.cuni.cz Mon Nov 12 19:55:29 2007 From: sopib1am at ss1000.ms.mff.cuni.cz (Bretislav Sopik) Date: Tue Nov 13 10:48:24 2007 Subject: [mvapich-discuss] problems with installing Message-ID: <20071113005528.GA15912@u-pl22.ms.mff.cuni.cz> Hello, I'm very sorry for my question, but I'm not able to run mvampich2. I've installed package mvapich2 through yast and I'm trying to go step by step acording your installation guide pdf. However I've found that many imporat files like mpdboot in my /bin directory are not executable and I do not even see any possibility making executable files from them. Am I so silly I can not go over it? I guess I am :( Please help me. Thank you. Bob From jsquyres at cisco.com Mon Nov 19 07:52:23 2007 From: jsquyres at cisco.com (Jeff Squyres) Date: Mon Nov 19 07:53:16 2007 Subject: [mvapich-discuss] MPI 2.1: Getting the band back together again Message-ID: <541FC1F3-8D30-40F1-B44D-B1541578D67F@cisco.com> All MPI users / developers: At both the Euro PVM/MPI'07 and the SC'07 conferences, we had productive meetings about how to progress the MPI standard: what to do about pending bugs in the MPI-1 and MPI-2 specs, new ideas that are emerging for possible successors to MPI-2, etc. It has been determined that the MPI Forum is formally going to re-convene and start having both electronic and physical meetings to resolve these issues. The first meeting is going to be in Chicago, IL, USA in early January 2008. I strongly encourage all interested parties to participate in the process. Please see the mpi-21 mailing list archives (particularly over the past week) and join the list yourself: http://www.mpi-forum.org/mpi2_1/index.htm Please spread the word. Thanks. -- Jeff Squyres Cisco Systems From koop at cse.ohio-state.edu Tue Nov 20 16:16:56 2007 From: koop at cse.ohio-state.edu (Matthew Koop) Date: Tue Nov 20 16:17:35 2007 Subject: [mvapich-discuss] problems with installing In-Reply-To: <20071113005528.GA15912@u-pl22.ms.mff.cuni.cz> Message-ID: Bob, Can you try downloading the MVAPICH2 package from our website and see if that works? (I'm also not sure what you mean by mpdboot not being able to have it be executable? Can you change the permissions on the files?) Matt On Tue, 13 Nov 2007, Bretislav Sopik wrote: > Hello, > I'm very sorry for my question, > but I'm not able to run mvampich2. > > I've installed package mvapich2 through yast and I'm trying to go step > by step acording your installation guide pdf. However I've found that many > imporat files like mpdboot in my /bin directory are not executable and I > do not even see any possibility making executable files from them. Am I > so silly I can not go over it? I guess I am :( Please help me. > > Thank you. > Bob > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > From sopib1am at ss1000.ms.mff.cuni.cz Wed Nov 21 03:56:46 2007 From: sopib1am at ss1000.ms.mff.cuni.cz (Bretislav Sopik) Date: Wed Nov 21 08:30:44 2007 Subject: [mvapich-discuss] MVAPICH Message-ID: <20071121085549.GA13139@u-pl22.ms.mff.cuni.cz> Dear Matt, I guess it's all problem of linux repositories I'm using. I've tried to install some MPI implementations and none of them worked me. When I've finally tried to download source from home page and compile it at computer it worked beautifully. So that's it. Thanks for your interest! Have a nice day! Bob From koop at cse.ohio-state.edu Wed Nov 21 08:34:58 2007 From: koop at cse.ohio-state.edu (Matthew Koop) Date: Wed Nov 21 08:35:40 2007 Subject: [mvapich-discuss] MVAPICH In-Reply-To: <20071121085549.GA13139@u-pl22.ms.mff.cuni.cz> Message-ID: Bob, Good to hear that it's working! Let us know if you have any other questions. Matt On Wed, 21 Nov 2007, Bretislav Sopik wrote: > Dear Matt, > I guess it's all problem of linux repositories I'm using. I've tried to > install some MPI implementations and none of them worked me. > > When I've finally tried to download source from home page and compile it > at computer it worked beautifully. > > So that's it. > > Thanks for your interest! > > Have a nice day! > Bob > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > From rezamirani at yahoo.com Thu Nov 22 00:43:18 2007 From: rezamirani at yahoo.com (Reza Mirani) Date: Thu Nov 22 00:43:57 2007 Subject: [mvapich-discuss] Installing MVAPICH2 Message-ID: <25756.78256.qm@web56605.mail.re3.yahoo.com> Dear Sir, I have Two Node with onboard infiniband cards(Mellanox),I installed IBGOLD from mellanox on it, but I don't know realy how can I setup a simple network with infiniband cards, after installing MVAPICH2 it seems it needs ssh on nodes, but in infinibands how can I run ssh? How can I ping nodes ? Please help me if possible. ************** Reza Mirani ____________________________________________________________________________________ Be a better sports nut! Let your teams follow you with Yahoo Mobile. Try it now. http://mobile.yahoo.com/sports;_ylt=At9_qDKvtAbMuh1G1SQtBI7ntAcJ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20071121/148cafd5/attachment.html From biswajit at crlindia.com Thu Nov 22 06:45:35 2007 From: biswajit at crlindia.com (Biswajit Mishra) Date: Thu Nov 22 06:46:18 2007 Subject: [mvapich-discuss] checkpointing failure ... Message-ID: <0AF7442124F01C49A6A93D8F04E5E3CF067C38@CHNEXVS01.VSNLXCHANGE.COM> I have installed mvapich2-1.0.1 with blcr-0.6.1 . While restarting a process after a check pointing , only few processes(Not all processes ) are starting up . What could be the reason ...??? My second problem is : While compiling a programme with application initiated synchronous checkpointing (using MVAPICH2_Sync_Checkpoint() ) getting following error messages . : undefined reference to `MVAPICH2_Sync_Checkpoint' Is there any header file I need to include or link with any library ...?? This message (including any attachment) is confidential and may be legally privileged. Access to this message by anyone other than the intended recipient(s) listed above is unauthorized. If you are not the intended recipient you are hereby notified that any disclosure, copying, or distribution of the message, or any action taken or omission of action by you in reliance upon it, is prohibited and may be unlawful. Please immediately notify the sender by reply e-mail and permanently delete all copies of the message if you have received this message in error. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20071122/9dd97551/attachment.html From panda at cse.ohio-state.edu Thu Nov 22 08:51:09 2007 From: panda at cse.ohio-state.edu (Dhabaleswar Panda) Date: Thu Nov 22 08:51:47 2007 Subject: [mvapich-discuss] Installing MVAPICH2 In-Reply-To: <25756.78256.qm@web56605.mail.re3.yahoo.com> from "Reza Mirani" at Nov 21, 2007 09:43:18 PM Message-ID: <200711221351.lAMDp9qD020413@xi.cse.ohio-state.edu> > Dear Sir, > I have Two Node with onboard infiniband cards(Mellanox),I installed IBGOLD from mellanox on it, but I don't know realy how can I setup a simple network with infiniband cards, after installing MVAPICH2 it seems it needs ssh on nodes, but in infinibands how can I run ssh? How can I ping nodes ? IBGOLD uses the VAPI stack which is becoming obsolete. Most InfiniBand users are switching to the OpenFabrics stack. You can get more information on this from www.openfabrics.org. You need to use the Gen2 interface from MVAPICH2 stack to work with the OpenFabrics stack. Please refer to the MVAPICH2 user guide for details. The OFED stack also comes with MVAPICH and MVAPICH2 with it and you can directly install it. Regarding ssh on nodes and pinging, you can refer to your system administror for details. DK > Please help me if possible. > > > ************** > Reza Mirani > > > > > ____________________________________________________________________________________ > Be a better sports nut! Let your teams follow you > with Yahoo Mobile. Try it now. http://mobile.yahoo.com/sports;_ylt=At9_qDKvtAbMuh1G1SQtBI7ntAcJ > --0-1631980982-1195710198=:78256 > Content-Type: text/html; charset=us-ascii > >
Dear Sir,
I have Two Node with onboard infiniband cards(Mellanox),I installed IBGOLD from mellanox on it, but I don't know realy how can I setup a simple network with infiniband cards, after installing MVAPICH2 it seems it needs ssh on nodes, but in infinibands how can I run ssh? How can I ping nodes ?

Please help me if possible.

 
**************
Reza Mirani


> >
Never miss a thing. Make Yahoo your homepage. > > > --0-1631980982-1195710198=:78256-- > > --===============1922654678== > Content-Type: text/plain; charset="us-ascii" > MIME-Version: 1.0 > Content-Transfer-Encoding: 7bit > Content-Disposition: inline > > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > > --===============1922654678==-- > From huanwei at cse.ohio-state.edu Fri Nov 23 10:29:58 2007 From: huanwei at cse.ohio-state.edu (wei huang) Date: Fri Nov 23 10:30:37 2007 Subject: [mvapich-discuss] checkpointing failure ... (fwd) In-Reply-To: <200711221411.lAMEBMX2020691@xi.cse.ohio-state.edu> Message-ID: Hi Biswajit, > > While restarting a process after a check pointing , > > only few processes(Not all processes ) are starting up . > > What could be the reason ...??? We never see this problem on our platform. Could you please give us more detailed information? Such as, detailed steps you do C/R. Also, is mpd running on the node you restart? Is BLCR installed properly on all nodes? Everytime (assume you've tried multiple times) do the started processes randomly distributed across all nodes, or just on few specific ones? > > My second problem is : > > While compiling a programme with application initiated synchronous chec= > > kpointing > > (using MVAPICH2_Sync_Checkpoint() ) getting following error messages . > > : undefined reference to `MVAPICH2_Sync_Checkpoint' > > > > Is there any header file I need to include or link with any library ...?? Currently it is protected by a CFLAG SYNC_CKPT. Please put -DSYNC_CKPT in the CFLAGS part of make.mvapich2.ofa and recompile mvapich2. Thanks. -- Wei From poobahtim at gmail.com Mon Nov 26 12:42:58 2007 From: poobahtim at gmail.com (Tim Hartley) Date: Mon Nov 26 12:43:42 2007 Subject: [mvapich-discuss] pthreads and mvapich? Message-ID: Hi all, I'm new to the list and am having a strange problem when running pthreads in an MPI process. The MPI calls are made safely, i.e. only from the initial process. The pthreads are just workers. However, on a multicore system, the pthreads always get scheduled on the same core. That is, multiple pthreads do not get scheduled on more than one core concurrently. I'm mainly curious if anyone has run into this before. This is mvapich 0.9.9, gcc 4.1.1, AMD Athlon(tm) 64 X2 Dual Core Processor 4600+. I can provide more information if necessary. I can provide small sample programs, and screenshots of top, too. Sadly, I won't be able to recompile mvapich, because I can't find the infiniband driver on the system. Thanks, Tim Hartley From gil at mellanox.co.il Mon Nov 26 17:16:46 2007 From: gil at mellanox.co.il (Gil Bloch) Date: Mon Nov 26 17:17:41 2007 Subject: [mvapich-discuss] pthreads and mvapich? In-Reply-To: Message-ID: <1E3DCD1C63492545881FACB6063A57C101636F32@mtiexch01.mti.com> > -----Original Message----- > From: mvapich-discuss-bounces@cse.ohio-state.edu [mailto:mvapich-discuss- > bounces@cse.ohio-state.edu] On Behalf Of Tim Hartley > Sent: Monday, November 26, 2007 9:43 AM > To: mvapich-discuss@cse.ohio-state.edu > Subject: [mvapich-discuss] pthreads and mvapich? > > Hi all, > > I'm new to the list and am having a strange problem when running > pthreads in an MPI process. The MPI calls are made safely, i.e. only > from the initial process. The pthreads are just workers. However, on > a multicore system, the pthreads always get scheduled on the same > core. That is, multiple pthreads do not get scheduled on more than > one core concurrently. I'm mainly curious if anyone has run into this > before. It seems like the MVAPICH process affinity feature prevent your process from using all cores in the node. Can you try using the following flag to disable affinity? VIADEV_USE_AFFINITY=0 - Gil > > This is mvapich 0.9.9, gcc 4.1.1, AMD Athlon(tm) 64 X2 Dual Core > Processor 4600+. I can provide more information if necessary. I can > provide small sample programs, and screenshots of top, too. > > Sadly, I won't be able to recompile mvapich, because I can't find the > infiniband driver on the system. > > Thanks, > Tim Hartley > _______________________________________________ > mvapich-discuss mailing list > mvapich-discuss@cse.ohio-state.edu > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss From keshetti85-student at yahoo.co.in Wed Nov 28 06:03:26 2007 From: keshetti85-student at yahoo.co.in (Keshetti Mahesh) Date: Wed Nov 28 07:49:04 2007 Subject: [mvapich-discuss] MVAPICH+IB] mvapich2-1.0.1 in IB network when openSM with LASH is running Message-ID: <829ded920711280303q6429d1fdxa09d807ff97c25a0@mail.gmail.com> Has anyone in the list ever tested MVAPICH in infiniband network in which openSM is running with LASH routing algorithm enabled? I also haven't tested the above case but i can foresee a problem because LASH routing algorithm in openSM uses virtual lanes (VL) which are directly mapped with service levels (SL) of infiniband network.And LASH routing algorithm assigns different VLs ( thus SLs) to different paths in the network. This SL <-> path association information is available only through the subnet manager (openSM) during connection establishment.But AFAIK, mvapich don't use the services of subnet manager for connection establishment between nodes. So I want to know whether anyone thought about it and working on it or not. regards, Mahesh From christian.guggenberger at rzg.mpg.de Thu Nov 29 10:36:20 2007 From: christian.guggenberger at rzg.mpg.de (Christian Guggenberger) Date: Thu Nov 29 10:37:01 2007 Subject: [mvapich-discuss] Fatal error in MPI_Allreduce Message-ID: <20071129153620.GG26051@daltons.rzg.mpg.de> Hi, attach is a small piece of Fortran code, which crashes with Fatal error in MPI_Allreduce: Invalid MPI_Op, error stack: MPI_Allreduce(707)..........: MPI_Allreduce(sbuf=0x610be0, rbuf=0x610c40, count=5, MPI_2DOUBLE_PRECISION, MPI_MINLOC, MPI_COMM_WORLD) failed MPIR_MINLOC_check_dtype(215): MPI_Op MPI_MINLOC operation not defined for this datatype as soon as it is compiled/linked with a MPI library (both mpich2 and mvapich2), which was compiled with Intel C/C++ 10.0.026 and/or 10.1.008 on Linux x86_64 machines. Intel 9.1, GNU or mixtures of GNU C/C++ and Lahey/Fujitsu Fortran work well. My question - is anyone aware of miscompilation(s) of mpich2/mvapich2 caused by Intel C/C++ 10.0 and 10.1 ? thanks a lot, - Christian -------------- next part -------------- program ringtopo include 'mpif.h' integer :: numtasks, rank, ierr integer, parameter :: n = 5 integer :: i real*8 :: dtL(2,n), dtG(2,n) call MPI_INIT(ierr) call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierr) call MPI_COMM_SIZE(MPI_COMM_WORLD, numtasks, ierr) do i=1,n dtL(1,i) = sin(13.*rank+i) dtL(2,i) = rank end do write(*,*)"----------------",rank do i=1,n write(*,'(a,i3,a,i3,a,2g12.5)')"- PE",rank,"/",numtasks," : ",dtL(1:2,i) end do call MPI_AllReduce (dtL(1,1), dtG(1,1), n, MPI_2Double_Precision, MPI_MinLoc, MPI_Comm_World, ierr) write(*,*)"################",rank do i=1,n write(*,'(a,i3,a,i3,a,2g12.5)')"+ PE",rank,"/",numtasks," : ",dtG(1:2,i) end do call MPI_FINALIZE(ierr) end From mamidala at cse.ohio-state.edu Fri Nov 30 12:57:02 2007 From: mamidala at cse.ohio-state.edu (amith rajith mamidala) Date: Fri Nov 30 12:57:40 2007 Subject: [mvapich-discuss] Fatal error in MPI_Allreduce In-Reply-To: <20071129153620.GG26051@daltons.rzg.mpg.de> Message-ID: Hi Christian, We are not very sure about the exact reason why Intel 10.0 compiler causes the miscompilation. However, the source of the problem could be not defining the fortran bindings properly. You can take a look at the file: trunk/src/mpi/coll/opminloc.c which causes this error to occur. I would suggest to contact the mpich team to see if this is a known error. Thanks, Amith. On Thu, 29 Nov 2007, Christian Guggenberger wrote: > Hi, > > attach is a small piece of Fortran code, which crashes with > > Fatal error in MPI_Allreduce: Invalid MPI_Op, error stack: > MPI_Allreduce(707)..........: MPI_Allreduce(sbuf=0x610be0, > rbuf=0x610c40, count=5, MPI_2DOUBLE_PRECISION, MPI_MINLOC, > MPI_COMM_WORLD) failed > MPIR_MINLOC_check_dtype(215): MPI_Op MPI_MINLOC operation not defined > for this datatype > > as soon as it is compiled/linked with a MPI library (both mpich2 and mvapich2), > which was compiled with Intel C/C++ 10.0.026 and/or 10.1.008 on Linux > x86_64 machines. > > Intel 9.1, GNU or mixtures of GNU C/C++ and Lahey/Fujitsu Fortran work > well. > > My question - is anyone aware of miscompilation(s) of mpich2/mvapich2 caused by Intel C/C++ > 10.0 and 10.1 ? > > thanks a lot, > > - Christian > From thakur at mcs.anl.gov Fri Nov 30 13:32:46 2007 From: thakur at mcs.anl.gov (Rajeev Thakur) Date: Fri Nov 30 13:33:28 2007 Subject: [MPICH] Re: [mvapich-discuss] Fatal error in MPI_Allreduce In-Reply-To: References: <20071129153620.GG26051@daltons.rzg.mpg.de> Message-ID: <004c01c8337f$67f65840$70013b0a@thakurlaptop> This is caused by a bug in the Intel 10.0 compiler. It cannot handle case statements of the form below. It needs a "return MPI_SUCCESS" after each case statement. You can fix it in your code by editing src/mpi/coll/opminloc.c and adding "return MPI_SUCCESS" after each of the case statements below. Rajeev int MPIR_MINLOC_check_dtype( MPI_Datatype type ) { static const char FCNAME[] = "MPIR_MINLOC_check_dtype"; switch (type) { /* first the C types */ case MPI_2INT: case MPI_FLOAT_INT: case MPI_LONG_INT: case MPI_SHORT_INT: case MPI_DOUBLE_INT: #if defined(HAVE_LONG_DOUBLE) case MPI_LONG_DOUBLE_INT: #endif /* now the Fortran types */ #ifdef HAVE_FORTRAN_BINDING #ifndef HAVE_NO_FORTRAN_MPI_TYPES_IN_C case MPI_2INTEGER: case MPI_2REAL: case MPI_2DOUBLE_PRECISION: #endif #endif return MPI_SUCCESS; /* --BEGIN ERROR HANDLING-- */ default: return MPIR_Err_create_code( MPI_SUCCESS, MPIR_ERR_RECOVERABLE, FCNAME, __LINE__, M PI_ERR_OP, "**opundefined","**opundefined %s", "MPI_MINLOC" ); /* --END ERROR HANDLING-- */ } } > -----Original Message----- > From: owner-mpich-discuss@mcs.anl.gov > [mailto:owner-mpich-discuss@mcs.anl.gov] On Behalf Of amith > rajith mamidala > Sent: Friday, November 30, 2007 11:57 AM > To: Christian Guggenberger > Cc: mvapich-discuss@cse.ohio-state.edu; mpich-discuss@mcs.anl.gov > Subject: [MPICH] Re: [mvapich-discuss] Fatal error in MPI_Allreduce > > Hi Christian, > > We are not very sure about the exact reason why Intel 10.0 > compiler causes the miscompilation. However, the source of > the problem could be not defining the fortran bindings > properly. You can take a look at the file: > trunk/src/mpi/coll/opminloc.c which causes this error to occur. > I would suggest to contact the mpich team to see if this is a > known error. > > Thanks, > > Amith. > > On Thu, 29 Nov 2007, Christian Guggenberger wrote: > > > Hi, > > > > attach is a small piece of Fortran code, which crashes with > > > > Fatal error in MPI_Allreduce: Invalid MPI_Op, error stack: > > MPI_Allreduce(707)..........: MPI_Allreduce(sbuf=0x610be0, > > rbuf=0x610c40, count=5, MPI_2DOUBLE_PRECISION, MPI_MINLOC, > > MPI_COMM_WORLD) failed > > MPIR_MINLOC_check_dtype(215): MPI_Op MPI_MINLOC operation > not defined > > for this datatype > > > > as soon as it is compiled/linked with a MPI library (both > mpich2 and > > mvapich2), which was compiled with Intel C/C++ 10.0.026 and/or > > 10.1.008 on Linux > > x86_64 machines. > > > > Intel 9.1, GNU or mixtures of GNU C/C++ and Lahey/Fujitsu > Fortran work > > well. > > > > My question - is anyone aware of miscompilation(s) of > mpich2/mvapich2 > > caused by Intel C/C++ 10.0 and 10.1 ? > > > > thanks a lot, > > > > - Christian > > > > From chan at mcs.anl.gov Fri Nov 30 13:30:27 2007 From: chan at mcs.anl.gov (Anthony Chan) Date: Fri Nov 30 13:41:43 2007 Subject: [MPICH] Re: [mvapich-discuss] Fatal error in MPI_Allreduce In-Reply-To: References: Message-ID: I recall I have seen a report of intel-10 compiler cannot handle switch statement correctly. I wonder if this is the case for you ? A.Chan On Fri, 30 Nov 2007, amith rajith mamidala wrote: > Hi Christian, > > We are not very sure about the exact reason why Intel 10.0 compiler causes > the miscompilation. However, the source of the problem could be not > defining the fortran bindings properly. You can take a look at the file: > trunk/src/mpi/coll/opminloc.c which causes this error to occur. > I would suggest to contact the mpich team to see if this is a known error. > > Thanks, > > Amith. > > On Thu, 29 Nov 2007, Christian Guggenberger wrote: > >> Hi, >> >> attach is a small piece of Fortran code, which crashes with >> >> Fatal error in MPI_Allreduce: Invalid MPI_Op, error stack: >> MPI_Allreduce(707)..........: MPI_Allreduce(sbuf=0x610be0, >> rbuf=0x610c40, count=5, MPI_2DOUBLE_PRECISION, MPI_MINLOC, >> MPI_COMM_WORLD) failed >> MPIR_MINLOC_check_dtype(215): MPI_Op MPI_MINLOC operation not defined >> for this datatype >> >> as soon as it is compiled/linked with a MPI library (both mpich2 and mvapich2), >> which was compiled with Intel C/C++ 10.0.026 and/or 10.1.008 on Linux >> x86_64 machines. >> >> Intel 9.1, GNU or mixtures of GNU C/C++ and Lahey/Fujitsu Fortran work >> well. >> >> My question - is anyone aware of miscompilation(s) of mpich2/mvapich2 caused by Intel C/C++ >> 10.0 and 10.1 ? >> >> thanks a lot, >> >> - Christian >> > > From chan at mcs.anl.gov Fri Nov 30 13:47:45 2007 From: chan at mcs.anl.gov (Anthony Chan) Date: Fri Nov 30 13:48:27 2007 Subject: [MPICH] Re: [mvapich-discuss] Fatal error in MPI_Allreduce In-Reply-To: <004c01c8337f$67f65840$70013b0a@thakurlaptop> References: <20071129153620.GG26051@daltons.rzg.mpg.de> <004c01c8337f$67f65840$70013b0a@thakurlaptop> Message-ID: http://softwarecommunity.intel.com/isn/Community/en-US/forums/thread/30237719.aspx A.Chan On Fri, 30 Nov 2007, Rajeev Thakur wrote: > This is caused by a bug in the Intel 10.0 compiler. It cannot handle case > statements of the form below. It needs a "return MPI_SUCCESS" after each > case statement. You can fix it in your code by editing > src/mpi/coll/opminloc.c and adding "return MPI_SUCCESS" after each of the > case statements below. > > Rajeev > > > > int MPIR_MINLOC_check_dtype( MPI_Datatype type ) > { > static const char FCNAME[] = "MPIR_MINLOC_check_dtype"; > > switch (type) { > /* first the C types */ > case MPI_2INT: > case MPI_FLOAT_INT: > case MPI_LONG_INT: > case MPI_SHORT_INT: > case MPI_DOUBLE_INT: > #if defined(HAVE_LONG_DOUBLE) > case MPI_LONG_DOUBLE_INT: > #endif > /* now the Fortran types */ > #ifdef HAVE_FORTRAN_BINDING > #ifndef HAVE_NO_FORTRAN_MPI_TYPES_IN_C > case MPI_2INTEGER: > case MPI_2REAL: > case MPI_2DOUBLE_PRECISION: > #endif > #endif > return MPI_SUCCESS; > /* --BEGIN ERROR HANDLING-- */ > default: > return MPIR_Err_create_code( MPI_SUCCESS, MPIR_ERR_RECOVERABLE, > FCNAME, __LINE__, M > PI_ERR_OP, "**opundefined","**opundefined %s", "MPI_MINLOC" ); > /* --END ERROR HANDLING-- */ > } > } > > >> -----Original Message----- >> From: owner-mpich-discuss@mcs.anl.gov >> [mailto:owner-mpich-discuss@mcs.anl.gov] On Behalf Of amith >> rajith mamidala >> Sent: Friday, November 30, 2007 11:57 AM >> To: Christian Guggenberger >> Cc: mvapich-discuss@cse.ohio-state.edu; mpich-discuss@mcs.anl.gov >> Subject: [MPICH] Re: [mvapich-discuss] Fatal error in MPI_Allreduce >> >> Hi Christian, >> >> We are not very sure about the exact reason why Intel 10.0 >> compiler causes the miscompilation. However, the source of >> the problem could be not defining the fortran bindings >> properly. You can take a look at the file: >> trunk/src/mpi/coll/opminloc.c which causes this error to occur. >> I would suggest to contact the mpich team to see if this is a >> known error. >> >> Thanks, >> >> Amith. >> >> On Thu, 29 Nov 2007, Christian Guggenberger wrote: >> >>> Hi, >>> >>> attach is a small piece of Fortran code, which crashes with >>> >>> Fatal error in MPI_Allreduce: Invalid MPI_Op, error stack: >>> MPI_Allreduce(707)..........: MPI_Allreduce(sbuf=0x610be0, >>> rbuf=0x610c40, count=5, MPI_2DOUBLE_PRECISION, MPI_MINLOC, >>> MPI_COMM_WORLD) failed >>> MPIR_MINLOC_check_dtype(215): MPI_Op MPI_MINLOC operation >> not defined >>> for this datatype >>> >>> as soon as it is compiled/linked with a MPI library (both >> mpich2 and >>> mvapich2), which was compiled with Intel C/C++ 10.0.026 and/or >>> 10.1.008 on Linux >>> x86_64 machines. >>> >>> Intel 9.1, GNU or mixtures of GNU C/C++ and Lahey/Fujitsu >> Fortran work >>> well. >>> >>> My question - is anyone aware of miscompilation(s) of >> mpich2/mvapich2 >>> caused by Intel C/C++ 10.0 and 10.1 ? >>> >>> thanks a lot, >>> >>> - Christian >>> >> >> > >