How-To: Using Myrinet on the KeckII Cluster

Introduction
Scaling of parallel codes on the cluster
Sample scripts for submitting Myrinet jobs to PBS

Introduction

KeckII cluster has Myrinet interconnects (low latency, high bandwidth networking) installed. This high performance network should be preferentially used for your parallel applications instead of ethernet since it guarantees much better scalability and efficiency of parallel jobs on the cluster (most evident for codes with high communication time/CPU time ratio). The cluster's measured performance with Myrinet is 29 Gflops.

Scaling of parallel codes on the cluster

This is an example of how Myrinet improves scaling of parallel applications on the cluster. Three popular MD codes - Amber, Charmm and NAMD were tested using both fast ethernet (eth) and Myrinet (myri) networking interface. Although NAMD scaling on fast ethernet is very impressive (for commodity hardware) the Myrinet version achieves still better scaling. Amber and Charmm poor fast ethernet performance is significantly improved using Myrinet.

For more details about the benchmarks see this page.

Sample scripts for submitting Myrinet jobs to PBS

When submitting MPI jobs on the cluster using Myrinet the application must be compiled with Myrinet version of MPICH (located in /usr/mpich) and launched with /usr/bin/mpi-launch command.

Amber

PBS script for submitting 4 CPU Myrinet Amber job:


#!/bin/sh
#PBS -l nodes=2:ppn=2,walltime=4:00:00
#PBS -k eo
HOSTLIST=`cat $PBS_NODEFILE`
echo $HOSTLIST > $PBS_O_WORKDIR/.hostlist

ambercmd=$PBS_O_WORKDIR/amber.sh

cd $PBS_O_WORKDIR
export $PBS_O_WORKDIR

chmod +x $ambercmd

ssh-agent /usr/bin/mpi-launch --machines-file=${PBS_NODEFILE} $ambercmd </dev/null

And the actual amber script (called amber.sh in the PBS script):


#!/bin/sh

cd $PBS_O_WORKDIR

cat <<eof >mdin
 short md, nve ensemble
 &cntrl
   ntx=7, irest=1,
   ntc=2, ntf=2, tol=0.0000001,
   nstlim=1000,
   ntpr=10, ntwr=10000,
   dt=0.001, vlimit=10.0,
   cut=9.,
   ntt=0, temp0=300.,
 &end
 &ewald
  a=62.23, b=62.23, c=62.23,
  nfft1=64,nfft2=64,nfft3=64,
  skinnb=2.,
 &end
eof

/soft/linux/share/amber7/bin/sander -O -i mdin -c inpcrd.equil \
-o out < /dev/null

NAMD

Please direct any questions or comments to keck-help @ keck2.ucsd.edu
Last modified: April 27 2005 07:08:54 pm.