IB0054606-02 AOFED+ Host SoftwareRelease 1.5.4User Guide

x IB0054606-02 AOFED+ Host Software Release 1.5.4User GuideOpen MPI Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5–Using Other MPIsMVAPICH25-4 IB0054606-02 AHere is an example of a simple mpirun command running with four processes:$ mpirun -np 4 -hostfile mpihos

5–Using Other MPIsManaging MVAPICH, and MVAPICH2 with the mpi-selector UtilityIB0054606-02 A 5-5Running MVAPICH2 ApplicationsBy default, the MVAPICH2

5–Using Other MPIsPlatform MPI 85-6 IB0054606-02 AThe example shell scripts mpivars.sh and mpivars.csh, for registering with mpi-selector, are provid

5–Using Other MPIsIntel MPIIB0054606-02 A 5-7to,MPI_ICMOD_PSM__PSM_PATH = "^"Compiling Platform MPI 8 ApplicationsAs with Open MPI, QLogic

5–Using Other MPIsIntel MPI5-8 IB0054606-02 AInstallationFollow the instructions for download and installation of Intel MPI from the Intel web site.S

5–Using Other MPIsIntel MPIIB0054606-02 A 5-9Using DAPL 2.0.$ rpm -qa | grep dapldapl-devel-static-2.0.19-1compat-dapl-1.2.14-1dapl-2.0.19-1dapl-debu

5–Using Other MPIsIntel MPI5-10 IB0054606-02 ASubstitute bin if using 32-bit.Compiling Intel MPI ApplicationsAs with Open MPI, QLogic recommended tha

5–Using Other MPIsIntel MPIIB0054606-02 A 5-11uDAPL 1.2:-genv I_MPI_DEVICE rdma:OpenIB-cma uDAPL 2.0:-genv I_MPI_DEVICE rdma:ofa-v2-ib To help with d

5–Using Other MPIsImproving Performance of Other MPIs Over IB Verbs5-12 IB0054606-02 AImproving Performance of Other MPIs Over IB VerbsPerformance of

IB0054606-02 A 6-16 SHMEM Description and ConfigurationOverviewQLogic SHMEM is a user-level communications library for one-sided operations. It imple

IB0054606-02 A xiOFED+ Host Software Release 1.5.4User Guideiba_packet_capture. . . . . . . . . . . . . . . . G-21ibhosts . . . . . . . . . . . . .

6–SHMEM Description and ConfigurationInstallation6-2 IB0054606-02 AThe -qlc suffix denotes that this is the QLogic PSM version. MVAPICH version 1.2.

6–SHMEM Description and ConfigurationSHMEM ProgramsIB0054606-02 A 6-3By default QLogic SHMEM is installed with a prefix of /usr/shmem/qlogic into the

6–SHMEM Description and ConfigurationSHMEM Programs6-4 IB0054606-02 AThe % character in the previous example is used to indicate the shell prompt and

6–SHMEM Description and ConfigurationSHMEM ProgramsIB0054606-02 A 6-5-Wl,--export-dynamic,--allow-shlib-undefined-L $SHMEM_DIR/lib64/default-lqlogic_

6–SHMEM Description and ConfigurationSHMEM Programs6-6 IB0054606-02 ABy default mpirun is picked up from the path and is assumed to be called mpirun.

6–SHMEM Description and ConfigurationQLogic SHMEM Relationship with MPIIB0054606-02 A 6-7QLogic SHMEM Relationship with MPIQLogic SHMEM requires the

6–SHMEM Description and ConfigurationSlurm Integration6-8 IB0054606-02 ASlurm IntegrationQLogic SHMEM relies on an MPI implementation to provide a ru

6–SHMEM Description and ConfigurationSizing Global Shared MemoryIB0054606-02 A 6-9The salloc allocates 16 nodes and runs one copy of shmemrun on the

6–SHMEM Description and ConfigurationSizing Global Shared Memory6-10 IB0054606-02 AThe QLogic SHMEM library pre-allocates room in the virtual address

6–SHMEM Description and ConfigurationProgress ModelIB0054606-02 A 6-11Alternatively, if $SHMEM_SHMALLOC_BASE_ADDR is specified as 0, then each SHMEM

xii IB0054606-02 AOFED+ Host Software Release 1.5.4User GuideList of Figures3-1 QLogic OFED+ Software Structure . . . . . . . . . . . . . . . . . .

6–SHMEM Description and ConfigurationProgress Model6-12 IB0054606-02 AActive ProgressIn the active progress mode SHMEM progress is achieved when the

6–SHMEM Description and ConfigurationEnvironment VariablesIB0054606-02 A 6-13 SHMEM's long message protocol is disabled. This is because the lo

6–SHMEM Description and ConfigurationEnvironment Variables6-14 IB0054606-02 A$SHMEM_SHMALLOC_CHECK on Shared memory consistency checks set for 0 to

6–SHMEM Description and ConfigurationImplementation BehaviorIB0054606-02 A 6-15The command shmemrun automatically propagates SHMEM* environment varia

6–SHMEM Description and ConfigurationImplementation Behavior6-16 IB0054606-02 AFor a put operation, these descriptions use the terms "local comp

6–SHMEM Description and ConfigurationApplication Programming InterfaceIB0054606-02 A 6-17 8 byte put to a sync location Target side: Wait for the

6–SHMEM Description and ConfigurationApplication Programming Interface6-18 IB0054606-02 ATable 6-3. SHMEM Application Programming Interface CallsOper

6–SHMEM Description and ConfigurationApplication Programming InterfaceIB0054606-02 A 6-19shmem_putshmem_put32shmem_put64shmem_put128shmem_putmemNon-b

6–SHMEM Description and ConfigurationApplication Programming Interface6-20 IB0054606-02 Ashmem_short_iputIndexed Put Operations shmem_ixputshmem_ixpu

6–SHMEM Description and ConfigurationApplication Programming InterfaceIB0054606-02 A 6-21shmem_get32shmem_get64shmem_get128shmem_getmemNon-blocking G

IB0054606-02 A xiiiOFED+ Host Software Release 1.5.4User GuideList of Tables3-1 ibmtu Values. . . . . . . . . . . . . . . . . . . . . . . . . . . .

6–SHMEM Description and ConfigurationApplication Programming Interface6-22 IB0054606-02 AIndexed Get Operations shmem_ixgetshmem_ixget32shmem_ixget64

6–SHMEM Description and ConfigurationApplication Programming InterfaceIB0054606-02 A 6-23shmem_short_swapshmem_int_swapshmem_long_swapshmem_longlong_

6–SHMEM Description and ConfigurationApplication Programming Interface6-24 IB0054606-02 Ashmem_long_faddshmem_longlong_faddReductions shmem_int_and_t

6–SHMEM Description and ConfigurationApplication Programming InterfaceIB0054606-02 A 6-25shmem_short_max_to_allshmem_complexd_sum_to_allcomplex colle

6–SHMEM Description and ConfigurationApplication Programming Interface6-26 IB0054606-02 Ashmem_clear_lockshmem_test_lockEvents clear_eventset_eventwa

6–SHMEM Description and ConfigurationSHMEM Benchmark ProgramsIB0054606-02 A 6-27SHMEM Benchmark ProgramsThe following SHMEM micro-benchmark programs

6–SHMEM Description and ConfigurationSHMEM Benchmark Programs6-28 IB0054606-02 AAdditional SHMEM micro-benchmark programs are included to measure get

6–SHMEM Description and ConfigurationSHMEM Benchmark ProgramsIB0054606-02 A 6-29QLogic SHMEM all-to-all benchmarkshmem-alltoall: all-to-all put bench

6–SHMEM Description and ConfigurationSHMEM Benchmark Programs6-30 IB0054606-02 AQLogic SHMEM barrier benchmarkshmem-barrier: barrier benchmarkUsage:

6–SHMEM Description and ConfigurationSHMEM Benchmark ProgramsIB0054606-02 A 6-31Table 6-8. QLogic SHMEM reduce benchmark optionsOption Description-b

xiv IB0054606-02 AOFED+ Host Software Release 1.5.4User Guide

6–SHMEM Description and ConfigurationSHMEM Benchmark Programs6-32 IB0054606-02 A

IB0054606-02 A 7-17 Virtual Fabric support in PSMIntroductionPerformance Scaled Messaging (PSM) provides support for full Virtual Fabric (vFabric) in

7–Virtual Fabric support in PSMVirtual Fabric Support7-2 IB0054606-02 AVirtual Fabric SupportVirtual Fabric (vFabric) in PSM is supported with the QL

7–Virtual Fabric support in PSMUsing Service IDIB0054606-02 A 7-3Using Service ID Full vFabric integration with PSM is available, allowing the user t

7–Virtual Fabric support in PSMVerifying SL2VL tables on QLogic 7300 Series Adapters7-4 IB0054606-02 AVerifying SL2VL tables on QLogic 7300 Series Ad

IB0054606-02 A 8-18 Dispersive RoutingInfiniband® uses deterministic routing that is keyed from the Destination LID (DLID) of a port. The Fabric Mana

8–Dispersive Routing8-2 IB0054606-02 AInternally, PSM utilizes dispersive routing differently for small and large messages. Large messages are any me

8–Dispersive RoutingIB0054606-02 A 8-3 Static_Dest: The path selection is based on the CPU index of the destination process. Multiple paths can be u

8–Dispersive Routing8-4 IB0054606-02 A

IB0054606-02 A 9-19 gPXEgPXE is an open source (GPL) network bootloader. It provides a direct replacement for proprietary PXE ROMs. See http://etherb

IB0054606-02 A xvPrefaceThe QLogic OFED+ Host Software User Guide shows end users how to use the installed software to setup the fabric. End users in

9–gPXEPreparing the DHCP Server in Linux9-2 IB0054606-02 A A Linux install image like kickstart, which then installs software to the local hard driv

9–gPXEPreparing the DHCP Server in LinuxIB0054606-02 A 9-3Installing DHCPgPXE requires that the DHCP server runs on a machine that supports IP over I

9–gPXEPreparing the DHCP Server in Linux9-4 IB0054606-02 AConfiguring DHCP1. From the client host, find the GUID of the HCA by using p1info or look a

9–gPXENetbooting Over IBIB0054606-02 A 9-54. Restart the DHCP serverNetbooting Over IBThe following procedures are an example of netbooting over IB,

9–gPXENetbooting Over IB9-6 IB0054606-02 A1. Install Apache.2. Create an images.conf file and a kernels.conf file and place them in the /etc/httpd/co

9–gPXENetbooting Over IBIB0054606-02 A 9-7To add an IB driver into the initrd file, The IB modules need to be copied to the diskless image. The host

9–gPXENetbooting Over IB9-8 IB0054606-02 Ab. The infinipath rpm will install the file /usr/share/infinipath/gPXE/gpxe-qib-modify-initrd with contents

9–gPXENetbooting Over IBIB0054606-02 A 9-9# extract previous contentsgunzip -dc ../initrd-ib-${kern}.img | cpio --quiet -id# add infiniband modulesmk

9–gPXENetbooting Over IB9-10 IB0054606-02 A IFS=' ' v6cmd='/sbin/insmod /lib/'${xfrm}'.ko'"$v6cmd" cryp

9–gPXENetbooting Over IBIB0054606-02 A 9-11/sbin/insmod /lib/ib/ib_sa.ko/sbin/insmod /lib/ib/ib_cm.ko/sbin/insmod /lib/ib/ib_uverbs.ko/sbin/insmod /l

PrefaceLicense Agreementsxvi IB0054606-02 A Tex t i n blue font indicates a hyperlink (jump) to a figure, table, or section in this guide, and links

9–gPXENetbooting Over IB9-12 IB0054606-02 A# and show the differences.echo -e '\nChanges in files in initrd image\n'diff Orig-listing New-l

9–gPXENetbooting Over IBIB0054606-02 A 9-13The following is an example of a uniboot.php file:<?header ( 'Content-type: text/plain' );fun

9–gPXEHTTP Boot Setup9-14 IB0054606-02 AThis is the kernel that will boot.This file can be copied from any machine that has RHEL5.3 installed.2. Star

9–gPXEHTTP Boot SetupIB0054606-02 A 9-155. Create an images.conf file and a kernels.conf file using the examples in Step 2 of Boot Server Setup and p

9–gPXEHTTP Boot Setup9-16 IB0054606-02 A

IB0054606-02 A A-1A Benchmark ProgramsSeveral MPI performance measurement programs are installed by default with the MPIs you choose to install (such

A–Benchmark ProgramsBenchmark 1: Measuring MPI Latency Between Two NodesA-2 IB0054606-02 AThe program osu_latency, from Ohio State University, measur

A–Benchmark ProgramsBenchmark 1: Measuring MPI Latency Between Two NodesIB0054606-02 A A-3-H (or --hosts) allows the specification of the host list o

A–Benchmark ProgramsBenchmark 2: Measuring MPI Bandwidth Between Two NodesA-4 IB0054606-02 ABenchmark 2: Measuring MPI Bandwidth Between Two NodesThe

A–Benchmark ProgramsBenchmark 2: Measuring MPI Bandwidth Between Two NodesIB0054606-02 A A-5Typical output might look like:# OSU MPI Bandwidth Test v

PrefaceTechnical SupportIB0054606-02 A xviiTechnical SupportCustomers should contact their authorized maintenance provider for technical support of t

A–Benchmark ProgramsBenchmark 3: Messaging Rate MicrobenchmarksA-6 IB0054606-02 ABenchmark 3: Messaging Rate MicrobenchmarksOSU Multiple Bandwidth /

A–Benchmark ProgramsBenchmark 3: Messaging Rate MicrobenchmarksIB0054606-02 A A-7This was run on 12-core compute nodes, so we used Open MPI's -n

A–Benchmark ProgramsBenchmark 3: Messaging Rate MicrobenchmarksA-8 IB0054606-02 A N/2 is dynamically calculated at the end of the run. You can use

A–Benchmark ProgramsBenchmark 3: Messaging Rate MicrobenchmarksIB0054606-02 A A-9The benchmark has been updated with code to dynamically determine wh

A–Benchmark ProgramsBenchmark 3: Messaging Rate MicrobenchmarksA-10 IB0054606-02 ANote the improved message rate at small message sizes of ~25 millio

A–Benchmark ProgramsBenchmark 3: Messaging Rate MicrobenchmarksIB0054606-02 A A-11The higher peak bi-directional messaging rate of 34.6 million messa

A–Benchmark ProgramsBenchmark 3: Messaging Rate MicrobenchmarksA-12 IB0054606-02 A

IB0054606-02 A B-1B SRP ConfigurationSRP Configuration OverviewSRP stands for SCSI RDMA Protocol. It allows the SCSI protocol to run over IB for Stor

B–SRP ConfigurationQLogic SRP ConfigurationB-2 IB0054606-02 A A SRP Initiator Extension is a 64-bit numeric value that is appended to the port GUID

B–SRP ConfigurationQLogic SRP ConfigurationIB0054606-02 A B-3Stopping, Starting and Restarting the SRP DriverTo stop the qlgc_srp driver, use the fol

PrefaceTechnical Supportxviii IB0054606-02 AKnowledge DatabaseThe QLogic knowledge database is an extensive collection of QLogic product information

B–SRP ConfigurationQLogic SRP ConfigurationB-4 IB0054606-02 A1. By the port GUID of the IOC, or2. By the IOC profile string that is created by the VI

B–SRP ConfigurationQLogic SRP ConfigurationIB0054606-02 A B-5The system returns input similar to the following:st187:~/qlgc-srp-1_3_0_0_1 # ib_qlgc_s

B–SRP ConfigurationQLogic SRP ConfigurationB-6 IB0054606-02 A0x0000494353535250 service 3 : name SRP.T10:0000000000000004 id 0x0000494353535250

B–SRP ConfigurationQLogic SRP ConfigurationIB0054606-02 A B-7Enter ib_qlgc_srp_build_cfg. The system provides output similar to the following:# qlgc_

B–SRP ConfigurationQLogic SRP ConfigurationB-8 IB0054606-02 A noverify: 0 description: "SRP Virtual HBA 0"endThe ib_qlgc_srp_build_cfg co

B–SRP ConfigurationQLogic SRP ConfigurationIB0054606-02 A B-9Specifying a SRP Target PortThe SRP target can be specified in two different ways. To co

B–SRP ConfigurationQLogic SRP ConfigurationB-10 IB0054606-02 ASpecifying a SRP Target Port of a Session by IOCGUIDThe following example specifies a t

B–SRP ConfigurationQLogic SRP ConfigurationIB0054606-02 A B-11Restarting the SRP ModuleFor changes to take effect, including changes to the SRP map o

B–SRP ConfigurationQLogic SRP ConfigurationB-12 IB0054606-02 AWhen the qlgc_srp module encounters an adapter command, that adapter is assigned all pr

B–SRP ConfigurationQLogic SRP ConfigurationIB0054606-02 A B-13endadapterbegindescription: "Test Device 1"endConfiguring Fibre Channel Failo

IB0054606-02 A 1-11 IntroductionHow this Guide is OrganizedThe QLogic OFED+ Host Software User Guide is organized into these sections: Section 1, pr

B–SRP ConfigurationQLogic SRP ConfigurationB-14 IB0054606-02 AFailover Configuration File 1: Failing over from one SRP Initiator port to anotherIn th

B–SRP ConfigurationQLogic SRP ConfigurationIB0054606-02 A B-15 adapterIODepth: 1000 lunIODepth: 16 adapterMaxIO: 128 adapterMaxLUNs: 512 adapter

B–SRP ConfigurationQLogic SRP ConfigurationB-16 IB0054606-02 AOn the VIO hardware side, the following needs to be ensured: The target device is disc

B–SRP ConfigurationQLogic SRP ConfigurationIB0054606-02 A B-17On the VIO hardware side, the following need to be ensured on each FVIC involved in the

B–SRP ConfigurationQLogic SRP ConfigurationB-18 IB0054606-02 A The target device is discovered and configured through the appropriate FC port The S

B–SRP ConfigurationQLogic SRP ConfigurationIB0054606-02 A B-192 Adapter Ports and 2 Ports on a Single VIO ModuleIn this example, traffic is load bala

B–SRP ConfigurationQLogic SRP ConfigurationB-20 IB0054606-02 AUsing the roundrobinmode ParameterIn this example, the two sessions use different VIO h

B–SRP ConfigurationQLogic SRP ConfigurationIB0054606-02 A B-21Configuring SRP for Native IB Storage1. Review ib_qlgc_srp_query.QLogic Corporation. Vi

B–SRP ConfigurationQLogic SRP ConfigurationB-22 IB0054606-02 A2. Edit /etc/sysconfig/qlgc_srp.cfg to add this information.# service : name SRP.T10:0

B–SRP ConfigurationQLogic SRP ConfigurationIB0054606-02 A B-23 roundrobinmode: 0 # set to 1 if you do not want target connectivity verification no

ii IB0054606-02 AOFED+ Host Software Release 1.5.4User GuideInformation furnished in this manual is believed to be accurate and reliable. However, QL

1–IntroductionOverview1-2 IB0054606-02 A Appendix C, describes two methods the administrator can use to allow users to submit MPI jobs through batch

B–SRP ConfigurationOFED SRP ConfigurationB-24 IB0054606-02 AAdditional Details All LUNs found are reported to the Linux SCSI mid-layer. Linux may n

B–SRP ConfigurationOFED SRP ConfigurationIB0054606-02 A B-253. Choose the device you want to use, and run the command again with the -c option (as a

B–SRP ConfigurationOFED SRP ConfigurationB-26 IB0054606-02 ANotes

IB0054606-02 A C-1C Integration with a Batch Queuing SystemMost cluster systems use some kind of batch queuing system as an orderly way to provide us

C–Integration with a Batch Queuing SystemClean-up PSM Shared Memory FilesC-2 IB0054606-02 AThis command displays a list of processes using InfiniPath

C–Integration with a Batch Queuing SystemClean-up PSM Shared Memory FilesIB0054606-02 A C-3#!/bin/shfiles=`/bin/ls /dev/shm/psm_shm.* 2> /dev/null

C–Integration with a Batch Queuing SystemClean-up PSM Shared Memory FilesC-4 IB0054606-02 A

IB6054601-00 F Preliminary Using LEDs to Check the State of the Adapter-1D TroubleshootingThis appendix describes some of the tools you can use to di

D–TroubleshootingBIOS SettingsD-2 IB0054606-02 ABIOS SettingsThis section covers issues related to BIOS settings.The most important setting is Advanc

D–TroubleshootingKernel and Initialization IssuesIB0054606-02 A D-3Driver Load Fails Due to Unsupported KernelIf you try to load the InfiniPath drive

1–IntroductionInteroperabilityIB0054606-02 A 1-3 An embedded subnet manager can be used in one or more managed switches. QLogic offers the QLogic Em

D–TroubleshootingKernel and Initialization IssuesD-4 IB0054606-02 AA zero count in all CPU columns means that no InfiniPath interrupts have been deli

D–TroubleshootingKernel and Initialization IssuesIB0054606-02 A D-5InfiniPath ib_qib Initialization FailureThere may be cases where ib_qib was not pr

D–TroubleshootingOpenFabrics and InfiniPath IssuesD-6 IB0054606-02 AMPI Job Failures Due to Initialization ProblemsIf one or more nodes do not have t

D–TroubleshootingOpenFabrics and InfiniPath IssuesIB0054606-02 A D-7Manual Shutdown or Restart May Hang if NFS in UseIf you are using NFS over IPoIB

D–TroubleshootingSystem Administration TroubleshootingD-8 IB0054606-02 Aibsrpdm Command Hangs when Two Host Channel Adapters are Installed but Only U

D–TroubleshootingPerformance IssuesIB0054606-02 A D-9Broken Intermediate LinkSometimes message traffic passes through the fabric while other traffic

D–TroubleshootingPerformance IssuesD-10 IB0054606-02 AErratic PerformanceSometimes erratic performance is seen on applications that use interrupts. A

D–TroubleshootingPerformance IssuesIB0054606-02 A D-11This method is not the first choice because, on some systems, there may be two rows of ib_qib o

D–TroubleshootingOpen MPI TroubleshootingD-12 IB0054606-02 APerformance Warning if ib_qib Shares Interrupts with eth0When ib_qib shares interrupts wi

IB0054606-02 A E-1E ULP TroubleshootingTroubleshooting VirtualNIC and VIO Hardware IssuesTo verify that an IB host can access an Ethernet system thro

1–IntroductionInteroperability1-4 IB0054606-02 A

E–ULP TroubleshootingTroubleshooting VirtualNIC and VIO Hardware IssuesE-2 IB0054606-02 AVerify that the proper VirtualNIC driver is runningCheck tha

E–ULP TroubleshootingTroubleshooting VirtualNIC and VIO Hardware IssuesIB0054606-02 A E-3Verifying that the host can communicate with the I/O Control

E–ULP TroubleshootingTroubleshooting VirtualNIC and VIO Hardware IssuesE-4 IB0054606-02 A ID: Chassis 0x00066A00010003F2, Slot 1, IOC 3

E–ULP TroubleshootingTroubleshooting VirtualNIC and VIO Hardware IssuesIB0054606-02 A E-5If the host can not see applicable IOCs, there are two thing

E–ULP TroubleshootingTroubleshooting VirtualNIC and VIO Hardware IssuesE-6 IB0054606-02 AAnother reason why the host might not be able to see the nec

E–ULP TroubleshootingTroubleshooting VirtualNIC and VIO Hardware IssuesIB0054606-02 A E-7DEVICE=eioc1BOOTPROTO=staticIPADDR=172.26.48.132BROADCAST=17

E–ULP TroubleshootingTroubleshooting VirtualNIC and VIO Hardware IssuesE-8 IB0054606-02 AThere are up to 6 IOC GUIDs on each VIO hardware module (6 f

E–ULP TroubleshootingTroubleshooting SRP IssuesIB0054606-02 A E-9Troubleshooting SRP Issuesib_qlgc_srp_stats showing session in disconnected statePro

E–ULP TroubleshootingTroubleshooting SRP IssuesE-10 IB0054606-02 A: 0x0000000000000000 Completed Receives : 0x00000000000002c0 | Receive Error

E–ULP TroubleshootingTroubleshooting SRP IssuesIB0054606-02 A E-11Solution: Perhaps an interswitch cable has been disconnected, or the VIO hardware i

IB0054606-02 A 2-12 Step-by-Step Cluster Setup and MPI Usage ChecklistsThis section describes how to set up your cluster to run high-performance Mess

E–ULP TroubleshootingTroubleshooting SRP IssuesE-12 IB0054606-02 AFollowing is an example:SCSI Host # : 17 | Mode :

E–ULP TroubleshootingTroubleshooting SRP IssuesIB0054606-02 A E-13SWUs : 0x00000000000003e8 Busy SWUs : 0x0000000000000000 | SRP Req

E–ULP TroubleshootingTroubleshooting SRP IssuesE-14 IB0054606-02 ASolution 1:The host initiator has not been configured as an SRP initiator on the VI

E–ULP TroubleshootingTroubleshooting SRP IssuesIB0054606-02 A E-15Solution:This indicates a problem in the path between the VIO hardware and the targ

E–ULP TroubleshootingTroubleshooting SRP IssuesE-16 IB0054606-02 ASolution 2:Make certain that all sessions have a map to the same disk defined. The

E–ULP TroubleshootingTroubleshooting SRP IssuesIB0054606-02 A E-17In a failover configuration, if everything is configured correctly, one session wil

E–ULP TroubleshootingTroubleshooting SRP IssuesE-18 IB0054606-02 AThe system displays information similar to the following:st106:~ # ibv_devinfo -i 1

E–ULP TroubleshootingTroubleshooting SRP IssuesIB0054606-02 A E-19Need to determine the SRP driver version.Solution:To determine the SRP driver versi

E–ULP TroubleshootingTroubleshooting SRP IssuesE-20 IB0054606-02 A

IB0054606-02 A F-1F Write CombiningIntroductionWrite Combining improves write bandwidth to the QLogic driver by writing multiple words in a single bu

2–Step-by-Step Cluster Setup and MPI Usage ChecklistsUsing MPI2-2 IB0054606-02 A8. Set up the host environment to use ssh. Two methods are discussed

F–Write CombiningMTRR Mapping and Write CombiningF-2 IB0054606-02 ARevert to using MTRR-only behavior by following one of the two suggestions in MTRR

F–Write CombiningVerify Write Combining is WorkingIB0054606-02 A F-3The test results will list any problems, if they exist, and provide suggestions o

F–Write CombiningVerify Write Combining is WorkingF-4 IB0054606-02 ANotes

IB0054606-02 A G-1G Commands and FilesThe most useful commands and files for debugging, and common tasks, are presented in the following sections. Ma

G–Commands and FilesRestarting InfiniPathG-2 IB0054606-02 ARestarting InfiniPathWhen the driver status appears abnormal on any node, you can try rest

G–Commands and FilesSummary and Descriptions of CommandsIB0054606-02 A G-3ibtracertaDetermines the path that IB packets travel between two nodesibv_d

G–Commands and FilesSummary and Descriptions of CommandsG-4 IB0054606-02 Admesgdmesg prints out bootup messages. It is useful for checking for initia

G–Commands and FilesSummary and Descriptions of CommandsIB0054606-02 A G-5-S/--sgid GID — Source GID. (Can be in GID (“0x########:0x########”) or ine

G–Commands and FilesSummary and Descriptions of CommandsG-6 IB0054606-02 ASample output:# iba_opp_query --slid 0x31 --dlid 0x75 --sid 0x107Query Para

G–Commands and FilesSummary and Descriptions of CommandsIB0054606-02 A G-7 resv2 0x0 resv3 0x0Explanation o

IB0054606-02 A 3-13 InfiniBand® Cluster Setup and AdministrationThis section describes what the cluster administrator needs to know about the QLogic

G–Commands and FilesSummary and Descriptions of CommandsG-8 IB0054606-02 AExamples:Query by LID and SID:iba_opp_query -s 0x31 -d 0x75 -i 0x107iba_opp

G–Commands and FilesSummary and Descriptions of CommandsIB0054606-02 A G-9iba_hca_revThis command scans the system and reports hardware and firmware

G–Commands and FilesSummary and Descriptions of CommandsG-10 IB0054606-02 A[ADAPTER]PSID = MT_0D80120009pcie_gen2_speed_supported = trueadapter_dev_i

G–Commands and FilesSummary and Descriptions of CommandsIB0054606-02 A G-11port1_sd2_ob_preemp_pre_qdr = 0x0port2_sd2_ob_preemp_pre_qdr = 0x0port1_sd

G–Commands and FilesSummary and Descriptions of CommandsG-12 IB0054606-02 Aport2_sd3_muxmain_qdr = 0x1fmellanox_qdr_ib_support = truemellanox_ddr_ib_

G–Commands and FilesSummary and Descriptions of CommandsIB0054606-02 A G-13auto_ddr_option_4.tx_preemp = 0x0auto_ddr_option_5.tx_preemp_pre = 0x5auto

G–Commands and FilesSummary and Descriptions of CommandsG-14 IB0054606-02 Aauto_ddr_option_11.tx_preemp_msb = 0x0auto_ddr_option_11.tx_preemp_post =

G–Commands and FilesSummary and Descriptions of CommandsIB0054606-02 A G-15auto_ddr_option_6.rx_offs_lowpass_en = 0x0auto_ddr_option_7.rx_offs_lowpas

G–Commands and FilesSummary and Descriptions of CommandsG-16 IB0054606-02 Aauto_ddr_option_5.rx_main = 0xeauto_ddr_option_6.rx_main = 0xfauto_ddr_opt

G–Commands and FilesSummary and Descriptions of CommandsIB0054606-02 A G-17auto_ddr_option_11.rx_muxeq = 0x04auto_ddr_option_11.rx_muxmain = 0x1fauto

3–InfiniBand® Cluster Setup and AdministrationInstalled Layout3-2 IB0054606-02 AInstalled LayoutThis section describes the default installed layout f

G–Commands and FilesSummary and Descriptions of CommandsG-18 IB0054606-02 Albist_shift_freq = 3pll_stabilize = 0x13flash_div = 0x3lbist_array_bypass

G–Commands and FilesSummary and Descriptions of CommandsIB0054606-02 A G-19FW image verification succeeded. Image is bootable.######################i

G–Commands and FilesSummary and Descriptions of CommandsG-20 IB0054606-02 Alinkwidth (link width supported) – use -i for integer value (1=1X, 2=4X, 3

G–Commands and FilesSummary and Descriptions of CommandsIB0054606-02 A G-21Exampleiba_manage_switch -t 0x00066a00e3001234 -f QLogic_12000_V1_firmware

G–Commands and FilesSummary and Descriptions of CommandsG-22 IB0054606-02 A-a alarm – number of seconds for alarm trigger to dump capture and exit-s

G–Commands and FilesSummary and Descriptions of CommandsIB0054606-02 A G-23Following is a sample output for the DDR adapters:# ibstatusInfiniband dev

G–Commands and FilesSummary and Descriptions of CommandsG-24 IB0054606-02 Aibv_devinfoThis program displays information about IB devices, including v

G–Commands and FilesSummary and Descriptions of CommandsIB0054606-02 A G-25If the /lib/modules/OS_version/updates directory is not present, then the

G–Commands and FilesSummary and Descriptions of CommandsG-26 IB0054606-02 Aipath_checkout performs the following seven tests on the cluster:1. Execut

G–Commands and FilesSummary and Descriptions of CommandsIB0054606-02 A G-27In most cases of failure, the script suggests recommended actions. Also re

3–InfiniBand® Cluster Setup and AdministrationIB and OpenFabrics Driver OverviewIB0054606-02 A 3-3IB and OpenFabrics Driver OverviewThe ib_qib module

G–Commands and FilesSummary and Descriptions of CommandsG-28 IB0054606-02 AHere is sample usage and output:% ipath_control -i$Id: QLogic OFED Release

G–Commands and FilesSummary and Descriptions of CommandsIB0054606-02 A G-29MTRR is used by the InfiniPath driver to enable write combining to the QLo

G–Commands and FilesSummary and Descriptions of CommandsG-30 IB0054606-02 A Test the IB link and bandwidth between two InfiniPath IB adapters. Usin

G–Commands and FilesSummary and Descriptions of CommandsIB0054606-02 A G-31mpirunmpirun determines whether the program is being run against a QLogic

G–Commands and FilesCommon Tasks and CommandsG-32 IB0054606-02 AThis option poisons receive buffers at initialization and after each receive; pre-ini

G–Commands and FilesCommon Tasks and CommandsIB0054606-02 A G-33Table G-3. Common Tasks and Commands Summary Function CommandCheck the system state i

G–Commands and FilesSummary and Descriptions of Useful FilesG-34 IB0054606-02 ASummary and Descriptions of Useful FilesUseful files are summarized in

G–Commands and FilesSummary and Descriptions of Useful FilesIB0054606-02 A G-35This information is useful for reporting problems to Technical Support

G–Commands and FilesSummary of Configuration FilesG-36 IB0054606-02 AThis same directory contains other files with information related to status. The

G–Commands and FilesSummary of Configuration FilesIB0054606-02 A G-37Table G-7. Configuration Files Configuration File Name Description/etc/modprobe

3–InfiniBand® Cluster Setup and AdministrationIPoIB Network Interface Configuration3-4 IB0054606-02 AThis example assumes that no hosts files exist,

G–Commands and FilesSummary of Configuration FilesG-38 IB0054606-02 A

IB0054606-02 A H-1H Recommended ReadingReference material for further reading is provided in this appendix.References for MPIThe MPI Standard specifi

H–Recommended ReadingOpenFabricsH-2 IB0054606-02 AOpenFabricsInformation about the OpenFabrics Alliance (OFA) is located at:http://www.openfabrics.or

Corporate Headquarters QLogic Corporation 26650 Aliso Viejo Parkway Aliso Viejo, CA 92656 949.389.6000 www.qlogic.comInternational Office

3–InfiniBand® Cluster Setup and AdministrationIPoIB AdministrationIB0054606-02 A 3-5IPoIB AdministrationAdministering IPoIBStopping, Starting and Res

IB0054606-02 A iiiTable of ContentsPrefaceIntended Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x

3–InfiniBand® Cluster Setup and AdministrationIB Bonding3-6 IB0054606-02 ANAME field specified in the CREATE block. The following is an example of th

3–InfiniBand® Cluster Setup and AdministrationIB BondingIB0054606-02 A 3-7Red Hat EL5 and EL6The following is an example for bond0 (master). The file

3–InfiniBand® Cluster Setup and AdministrationIB Bonding3-8 IB0054606-02 ASuSE Linux Enterprise Server (SLES) 10 and 11The following is an example fo

3–InfiniBand® Cluster Setup and AdministrationIB BondingIB0054606-02 A 3-9Verify the following line is set to the value of yes in /etc/sysconfig/boot

3–InfiniBand® Cluster Setup and AdministrationSubnet Manager Configuration3-10 IB0054606-02 AExample of ifconfig output:st2169:/etc/sysconfig # ifcon

3–InfiniBand® Cluster Setup and AdministrationSubnet Manager ConfigurationIB0054606-02 A 3-11OpenSM is a component of the OpenFabrics project that pr

3–InfiniBand® Cluster Setup and AdministrationQLogic Distributed Subnet Administration3-12 IB0054606-02 AQLogic Distributed Subnet AdministrationAs I

3–InfiniBand® Cluster Setup and AdministrationQLogic Distributed Subnet AdministrationIB0054606-02 A 3-13Virtual Fabrics and the Distributed SAThe IB

3–InfiniBand® Cluster Setup and AdministrationQLogic Distributed Subnet Administration3-14 IB0054606-02 AIf you are using the QLogic Fabric Manager i

3–InfiniBand® Cluster Setup and AdministrationQLogic Distributed Subnet AdministrationIB0054606-02 A 3-15Figure 3-4. Distributed SA Multiple Virtual

iv IB0054606-02 AOFED+ Host Software Release 1.5.4User GuideSubnet Manager Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3–InfiniBand® Cluster Setup and AdministrationQLogic Distributed Subnet Administration3-16 IB0054606-02 AFigure 3-6. Virtual Fabrics with PSM_MPI Vir

3–InfiniBand® Cluster Setup and AdministrationQLogic Distributed Subnet AdministrationIB0054606-02 A 3-17Second, the Distributed SA handles overlaps

3–InfiniBand® Cluster Setup and AdministrationQLogic Distributed Subnet Administration3-18 IB0054606-02 ASIDThe SID is the primary configuration sett

3–InfiniBand® Cluster Setup and AdministrationQLogic Distributed Subnet AdministrationIB0054606-02 A 3-19DbgThis parameter controls how much logging

3–InfiniBand® Cluster Setup and AdministrationChanging the MTU Size3-20 IB0054606-02 AChanging the MTU SizeThe Maximum Transfer Unit (MTU) size enabl

3–InfiniBand® Cluster Setup and AdministrationManaging the ib_qib DriverIB0054606-02 A 3-21Managing the ib_qib DriverThe startup script for ib_qib is

3–InfiniBand® Cluster Setup and AdministrationManaging the ib_qib Driver3-22 IB0054606-02 ASee the ib_qib man page for more details.Configure the ib_

3–InfiniBand® Cluster Setup and AdministrationManaging the ib_qib DriverIB0054606-02 A 3-23You can check to see if opensmd is configured to autostart

3–InfiniBand® Cluster Setup and AdministrationMore Information on Configuring and Loading Drivers3-24 IB0054606-02 A/ipathfs/1/counter_names/ipathfs/

3–InfiniBand® Cluster Setup and AdministrationPerformance Settings and Management TipsIB0054606-02 A 3-25Performance TuningTuning compute or storage

IB0054606-02 A vOFED+ Host Software Release 1.5.4User GuideConfiguring for ssh . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3–InfiniBand® Cluster Setup and AdministrationPerformance Settings and Management Tips3-26 IB0054606-02 AIf cpuspeed or powersaved are being used as

3–InfiniBand® Cluster Setup and AdministrationPerformance Settings and Management TipsIB0054606-02 A 3-27Increasing the number of kernel receive queu

3–InfiniBand® Cluster Setup and AdministrationPerformance Settings and Management Tips3-28 IB0054606-02 AIn the rare case that the node has more than

3–InfiniBand® Cluster Setup and AdministrationPerformance Settings and Management TipsIB0054606-02 A 3-29For setting all C-States to 0 where there is

3–InfiniBand® Cluster Setup and AdministrationPerformance Settings and Management Tips3-30 IB0054606-02 AHigh Risk Tuning for Intel Harpertown CPUsFo

3–InfiniBand® Cluster Setup and AdministrationPerformance Settings and Management TipsIB0054606-02 A 3-31Additional Driver Module Parameter Tunings A

3–InfiniBand® Cluster Setup and AdministrationPerformance Settings and Management Tips3-32 IB0054606-02 A value is the parameter value for the parti

3–InfiniBand® Cluster Setup and AdministrationPerformance Settings and Management TipsIB0054606-02 A 3-33This command lets the driver automatically d

3–InfiniBand® Cluster Setup and AdministrationPerformance Settings and Management Tips3-34 IB0054606-02 AFor example:# cat /etc/modprobe.d/ib_ipoib.c

3–InfiniBand® Cluster Setup and AdministrationPerformance Settings and Management TipsIB0054606-02 A 3-35The values picked for the various checks and

vi IB0054606-02 AOFED+ Host Software Release 1.5.4User GuideDebugging MPI Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3–InfiniBand® Cluster Setup and AdministrationPerformance Settings and Management Tips3-36 IB0054606-02 AAUTOMATIC vs. INTERACTIVE MODEThe tool perfo

3–InfiniBand® Cluster Setup and AdministrationPerformance Settings and Management TipsIB0054606-02 A 3-37Applying the ResultsAutomatic mode versus in

3–InfiniBand® Cluster Setup and AdministrationPerformance Settings and Management Tips3-38 IB0054606-02 A rpm (see “rpm” on page G-32)strings (see

3–InfiniBand® Cluster Setup and AdministrationPerformance Settings and Management TipsIB0054606-02 A 3-39Remove Unneeded ServicesThe cluster administ

3–InfiniBand® Cluster Setup and AdministrationHost Environment Setup for MPI3-40 IB0054606-02 AOther services may be required by your batch queuing s

3–InfiniBand® Cluster Setup and AdministrationHost Environment Setup for MPIIB0054606-02 A 3-41“Configuring for ssh Using ssh-agent” on page 3-43 sho

3–InfiniBand® Cluster Setup and AdministrationHost Environment Setup for MPI3-42 IB0054606-02 A3. On each of the IB node systems, create or edit the

3–InfiniBand® Cluster Setup and AdministrationHost Environment Setup for MPIIB0054606-02 A 3-43At this point, any end user should be able to login to

3–InfiniBand® Cluster Setup and AdministrationChecking Cluster and Software Status3-44 IB0054606-02 A5. Finally, test by logging into the front end n

3–InfiniBand® Cluster Setup and AdministrationChecking Cluster and Software StatusIB0054606-02 A 3-45iba_opp_query iba_opp_query is used to check the

IB0054606-02 A viiOFED+ Host Software Release 1.5.4User GuideRunning SHMEM Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3–InfiniBand® Cluster Setup and AdministrationChecking Cluster and Software Status3-46 IB0054606-02 A mtu 0x4 rate

3–InfiniBand® Cluster Setup and AdministrationChecking Cluster and Software StatusIB0054606-02 A 3-47ibv_devinfo ibv_devinfo queries RDMA devices. Us

3–InfiniBand® Cluster Setup and AdministrationChecking Cluster and Software Status3-48 IB0054606-02 A

IB0054606-02 A 4-14 Running MPI on QLogic AdaptersThis section provides information on using the Message-Passing Interface (MPI) on QLogic IB HCAs. E

4–Running MPI on QLogic AdaptersOpen MPI4-2 IB0054606-02 AInstallationFollow the instructions in the QLogic Fabric Software Installation Guide for in

4–Running MPI on QLogic AdaptersOpen MPIIB0054606-02 A 4-3These wrapper scripts pass most options on to the underlying compiler. Use the documentatio

4–Running MPI on QLogic AdaptersOpen MPI4-4 IB0054606-02 AThe first choice will use verbs by default, and any with the _qlc string will use PSM by de

4–Running MPI on QLogic AdaptersOpen MPIIB0054606-02 A 4-5Configuring MPI Programs for Open MPIWhen configuring an MPI program (generating header fil

4–Running MPI on QLogic AdaptersOpen MPI4-6 IB0054606-02 AThe easiest way to use other compilers with any MPI that comes with QLogic OFED+ is to use

4–Running MPI on QLogic AdaptersOpen MPIIB0054606-02 A 4-7For Fortran 90 programs:$ mpif90 -f90=pgf90 -show pi3f90.f90 -o pi3f90pgf90 -I/usr/include/

viii IB0054606-02 AOFED+ Host Software Release 1.5.4User GuideA Benchmark ProgramsBenchmark 1: Measuring MPI Latency Between Two Nodes . . . . . . .

4–Running MPI on QLogic AdaptersOpen MPI4-8 IB0054606-02 A The default hardware context/CPU mappings can be changed on the QDR IB Adapters (QLE734x).

4–Running MPI on QLogic AdaptersOpen MPIIB0054606-02 A 4-9Performance can be improved in some cases by disabling IB hardware contexts when they are n

4–Running MPI on QLogic AdaptersOpen MPI4-10 IB0054606-02 ATo explicitly disable context sharing, set this environment variable in one of the two fol

4–Running MPI on QLogic AdaptersOpen MPIIB0054606-02 A 4-11Context Sharing Error MessagesThe error message when the context limit is exceeded is:No f

4–Running MPI on QLogic AdaptersOpen MPI4-12 IB0054606-02 Ampihosts File DetailsAs noted in “Create the mpihosts File” on page 4-3, a hostfile (also

4–Running MPI on QLogic AdaptersOpen MPIIB0054606-02 A 4-13 The command line option -hostfile can be used as shown in the following command line:$mp

4–Running MPI on QLogic AdaptersOpen MPI4-14 IB0054606-02 AThis option spawns n instances of program-name. These instances are called node programs.G

4–Running MPI on QLogic AdaptersOpen MPIIB0054606-02 A 4-15Open MPI directs UNIX standard output and error from remote nodes to the node that invoked

4–Running MPI on QLogic AdaptersOpen MPI4-16 IB0054606-02 AOpen MPI adds the base-name of the current node’s bindir (the directory where Open MPI’s e

4–Running MPI on QLogic AdaptersOpen MPIIB0054606-02 A 4-17Setting MCA ParametersThe -mca switch allows the passing of parameters to various Modular

IB0054606-02 A ixOFED+ Host Software Release 1.5.4User GuideConfiguring SRP for Native IB Storage . . . . . . . . . . . . . . . . . . . . . . . . B-

4–Running MPI on QLogic AdaptersOpen MPI4-18 IB0054606-02 AEnvironment VariablesTable 4-6 contains a summary of the environment variables that are re

4–Running MPI on QLogic AdaptersOpen MPIIB0054606-02 A 4-19IPATH_NO_CPUAFFINITY When set to 1, the PSM library will skip trying to set processor affi

4–Running MPI on QLogic AdaptersOpen MPI4-20 IB0054606-02 AJob Blocking in Case of Temporary IB Link FailuresBy default, as controlled by mpirun’s qu

4–Running MPI on QLogic AdaptersOpen MPI and Hybrid MPI/OpenMP ApplicationsIB0054606-02 A 4-21Open MPI and Hybrid MPI/OpenMP ApplicationsOpen MPI sup

4–Running MPI on QLogic AdaptersDebugging MPI Programs4-22 IB0054606-02 ADebugging MPI ProgramsDebugging parallel programs is substantially more diff

4–Running MPI on QLogic AdaptersDebugging MPI ProgramsIB0054606-02 A 4-23NOTEThe TotalView® debugger can be used with the Open MPI supplied in this r

4–Running MPI on QLogic AdaptersDebugging MPI Programs4-24 IB0054606-02 A

IB0054606-02 A 5-15 Using Other MPIsThis section provides information on using other MPI implementations. Detailed information on using Open MPI is p

5–Using Other MPIsInstalled Layout5-2 IB0054606-02 AThese MPI implementations run on multiple interconnects, and have their own mechanisms for select

5–Using Other MPIsOpen MPIIB0054606-02 A 5-3Open MPI Open MPI is an open source MPI-2 implementation from the Open MPI Project. Pre-compiled versions