modified on 6 November 2017 at 14:18 ••• 467,134 views

Compiling NWChem

From NWChem

(Difference between revisions)
Jump to: navigation, search
(How to: Mac platforms)
(Compilation on Mac OS X Lion 10.7)
Line 376: Line 376:
=How to: Mac platforms=
=How to: Mac platforms=
-
== Compilation on Mac OS X Lion 10.7 ==
+
== Compilation of NWChem 6.1.1 release Mac OS X Lion 10.7 ==
-
Will be updated soon.
+
* Download XCode from the App store http://itunes.apple.com/us/app/xcode/id497799835?mt=12
=How-to: Cray platforms=
=How-to: Cray platforms=

Revision as of 12:00, 27 June 2012


Contents

Compiling NWChem from source

On this page, a step-by-step description of the build process and necessary and optional environment variables is outlined. In addition, based on the experiences of developers and users how-to's for various platforms have been created. These how-to's will be updated with additional platforms and better environment variables over time.

Setting up the proper environment variables

  • NWCHEM_TOP defines the top directory of the NWChem source tree, e.g.
  % setenv NWCHEM_TOP <your path>/nwchem
  • NWCHEM_TARGET defines your target platform, e.g.
  % setenv NWCHEM_TARGET LINUX64
The platforms that available are:
NWCHEM_TARGET Platform OS/Version Compilers
LINUX x86
ppc
RedHat, MDK, SLES
YD2.1, SLES
GNU, Intel, PGI
GNU, xlf
LINUX64 ia64
x86_64
ppc64
RedHat
SLES, RedHat
SLES, RedHat
Intel
GNU, PGI, PathScale, Intel
xlf
BGL
BGP
Bluegene/L
Bluegene/P
SLES blrts_xlf
bgxlf
LAPI
LAPI64
IBM SP AIX/LAPI
IBM
IBM64
IBM RS/6000 AIX
MACX
MACX64
Apple MacOSX OSX, Lion GNU, xlf, Intel
CYGWIN
WIN32
Intel x86 Windows with Cygwin
Windows
GNU
Compaq
SGI_N32
SGITFP
SGI 64 bit OS using 32 bit integers
SGI 64 bit OS
IRIX
HPUX
HPUX64
PA-RISC
PA-RISC, ia64
HPUX


  • ARMCI_NETWORK must be defined in order to achieve best performance on high performance networks, .e.g.
  % setenv ARMCI_NETWORK OPENIB

For a single processor system, this environment variable does not have to be defined.

Supported combination of ARMCI_NETWORK and NWCHEM_TARGET variables:
ARMCI_NETWORK NWCHEM_TARGET Network Protocol
OPENIB LINUX, LINUX64 InfiniBand OpenIB
MPI-MT
MPI-SPAWN
LINUX64 Myrinet MX, Infiniband MPI2
MELLANOX LINUX, LINUX64 InfiniBand VAPI
GM LINUX, LINUX64 Myrinet GM
GEMINI LINUX64 Cray Gemini Gemini
PORTALS LINUX64 Cray SeaStar/HyperTransport PORTALS
BGMLMPI BGL IBM BlueGene Torus BGLMPI
DCMFMPI BGP Global Tree/Interrupt DCMF, MPI
VIA LINUX Giganet/CLAN VIA


  • MPI variables needed to compile. For a single processor system, these environment variables do not have to be defined.


USE_MPI Set to "y" to indicate that NWChem should be compiled with MPI
USE_MPIF Set to "y" for the NWPW module to use fortran-bindings of MPI (Generally set when USE_MPI is set)
USE_MPIF4 Set to "y" for the NWPW module to use Integer*4 fortran-bindings of MPI. (Generally set when USE_MPI is set on most platforms)
LIBMPI Name of the MPI library that should be linked with -l (eg. -lmpich)
MPI_LIB Directory where the MPI library resides
MPI_INCLUDE Directory where the MPI include files reside

E.g. for MPICH2, this looks like:

  % setenv USE_MPI y
  % setenv USE_MPIF y
  % setenv USE_MPIF4 y
  % setenv LIBMPI "-lmpich -lopa -lmpl -lrt -lpthread"
  % setenv MPI_LIB <Your path to MPICH2 libraries>/lib 
  % setenv MPI_INCLUDE <Your path to MPICH2 libraries>/include
For some specific implementations the settings for MPI_LIB, MPI_INCLUDE, and LIBMPI look like:
MPI Implementation Environment variables
MPICH setenv MPI_LOC /usr/local #location of mpich installation
setenv MPI_LIB $MPI_LOC/lib
setenv MPI_INCLUDE $MPI_LOC/include
setenv LIBMPI "-lfmpich -lmpich -lpmpich"
MPICH2 setenv MPI_LOC /usr/local #location of mpich2 installation
setenv MPI_LIB $MPI_LOC/lib
setenv MPI_INCLUDE $MPI_LOC/include
setenv LIBMPI "-lmpich -lopa -lmpl -lrt -lpthread"
OPENMPI setenv MPI_LOC /usr/local #location of openmpi installation
setenv MPI_LIB $MPI_LOC/lib
setenv MPI_INCLUDE $MPI_LOC/include
setenv LIBMPI "-lmpi_f90 -lmpi_f77 -lmpi -ldl -Wl,--export-dynamic -lnsl -lutil"


Note:

When MPI is used, the appropriate MPI run command should be used to start an NWChem calculation, e.g.

  % mpirun -np 8 $NWCHEM_TOP/bin/${NWCHEM_TARGET}}/nwchem h2o.nw

When all nodes are connected via shared memory and the ch_shmem version of MPICH is installed and used, NWChem can be called directly, e.g.

  % $NWCHEM_TOP/bin/${NWCHEM_TARGET}/nwchem -np 8 h2o.nw
  • NWCHEM_MODULES defines the modules to be compiled, e.g.
  % setenv NWCHEM_MODULES "all python"
The following modules are available:
Module Description
all Everything useful
all python Everything useful plus python
qm All quantum mechanics modules
md MD only build

Note that additional environment variables need to be defined to specify the location of the Python libraries, when the python module is compiled. See the optional environmental variables section for specifics.

Adding optional environmental variables

LARGE_FILES can be set circumvent the 2 GB limit where possible (note that your system administrator must also enable large files in the file system), e.g.

  % setenv LARGE_FILES TRUE

USE_NOFSCHECK can be set to avoid NWChem creating files for each process when testing the size of the scratch directory (a.k.a. creation of junk files), e.g.

  % setenv USE_NOFSCHECK TRUE

LIB_DEFINES can be set to pass additional defines to the C preprocessor (for both Fortran and C), e.g.

  % setenv LIB_DEFINES -DDFLT_TOT_MEM=16777216

Note: -DDFLT_TOT_MEM sets the default dynamic memory available for NWChem to run, where the units are in doubles. Instead of manually defining these one can optionally use the "getmem.nwchem" script in the $NWCHEM_TOP/contrib directory. This script should be run after an initial build of the binary has been completed. The script will assess memory availability and make an educated guess, recompile the appropriate files and relink.

Setting Python environment variables

Python programs may be embedded into the NWChem input and used to control the execution of NWChem. To build with Python, Python needs to be available on your machine. The software can be download from http://www.python.org . Follow the Python instructions for installation and testing. NWChem has been tested with Python versions 1.5.1, 1.5.2, 1.6 and 2.0.

The following environment variables need to be set when compiling with Python:

  % setenv PYTHONHOME /usr/local/Python-1.5.1
  % setenv PYTHONVERSION 1.5
  % setenv USE_PYTHON64  y

Note that the third number in the version should not be kept: 2.2.3 should be set as 2.2

To run with Python, make sure that PYTHONHOME is set as mentioned above. You will also need to set PYTHONPATH to include any modules that you are using in your input. Examples of Python within NWChem are in the $NWCHEM_TOP/QA/tests/pyqa and $NWCHEM_TOP/contrib/python directories.

Using optimized math libraries

By default NWChem uses its own basic linear algebra subroutines (BLAS). To include faster BLAS routines, the environment variable BLASOPT needs to be set before building the code. For example, with ATLAS

  % setenv BLASOPT "-L/usr/local/ATLAS -lf77blas -latlas"
Good choices of optimized BLAS libraries on x86 (e.g. LINUX and LINUX64) hardware include:
GotoBLAS http://www.tacc.utexas.edu/tacc-projects/gotoblas2
Intel MKL http://www.intel.com/software/products/mkl/
AMD ACML http://developer.amd.com/libraries/acml/pages/default.aspx
ATLAS http://math-atlas.sf.net
Cray LibSci Available only on Cray x86_64 hardware, it is automatically linked when compiling on Cray XT and XE computers.


NWChem can also take advantage of the ScaLAPACK library if it is installed on your system. The following environment variables need to be set:

  % setenv USE_SCALAPACK y

  % setenv SCALAPACK "location of Scalapack and BLACS library"

WARNING: In the case of 64-bit platforms, most vendors optimized BLAS libraries cannot be used. This is due to the fact that while NWChem uses 64-bit integers (i.e. integer*8) on 64-bit platforms, most of the vendors optimized BLAS libraries used 32-bit integers. BLAS libraries not supporting 64-bit integers (at least in their default options/installations) include CXML (DECOSF), ESSL (LAPI64), MKL (LINUX64/ia64 and x86_64), ACML(LINUX64/x86_64), and GotoBLAS2(LINUX64). The same holds for the ScaLAPACK libraries, which internally use 32-bit integers.

A method is now available to link against the libraries mentioned above, using the following procedure:

  % cd $NWCHEM_TOP/src
  % make clean
  % make 64_to_32
  % make USE_64TO32=y HAS_BLAS=yes BLASOPT=" optimized BLAS"

E.g., for IBM64 this looks like

  % make  USE_64TO32=y HAS_BLAS=yes  BLASOPT="-lessl -lmass"

Notes:

  • GotoBLAS2 can be installed with 64bit integers. This is accomplished by compiling the GotoBLAS2 library after having by edited the GotoBLAS2 Makefile.rule file and un-commenting the line containing the INTERFACE64 definition. In other words, the line
          #INTERFACE64 = 1

                needs to be changed to

          INTERFACE64 = 1
  • ACML and MKL can support 64-bit integers if the appropriate library is chosen. For MKL, one can choose the ILP64 Version of Intel® MKL, while for ACML the int64 libraries should be chosen, e.g. in the case of ACML 4.4.0 using a PGI compiler /opt/acml/4.4.0/pgi64_int64/lib/libacml.a

Linking in NBO

The current versions of NBO provide a utility to generate source code that can be linked into computational chemistry packages such as NWChem. To utilize this functionality, follow the instructions in the NBO package to generate an nwnbo.f file. Linking NBO into NWChem can be done using the following procedure:

  % cd $NWCHEM_TOP/src
  % cp nwnbo.f $NWCHEM_TOP/src/nbo/.
  % make nwchem_config NWCHEM_MODULES="all nbo"
  % make

One can now use "task nbo" and incorporate NBO input into the NWChem input file directly:

 nbo
   $NBO NRT $END
   ...
 end

 task nbo

Building the NWChem binary

Once all required and optional environment variables have been set, NWChem can be compiled:

  % cd $NWCHEM_TOP/src

  % make nwchem_config

  % make >& make.log

The make above will use the standard compilers available on your system. To use compilers different from the default one can either set environment variables:

  % setenv FC <fortran compiler>
  % setenv CC <c compiler>

Or one can supply the compiler options to the make command, e.g:

  % make FC=ifort CC=icc

For example, on Linux FC could be set either equal to ifort, gfortran or pgf90

Note 1: If in a Linux environment, FC is set equal to anything other than the tested compilers, there is no guarantee of a successful installation, since the makefile structure has not been tested to process other settings. In other words, please avoid make FC="ifort -O3 -xhost" and stick to make FC="ifort", instead

Note 2: It's better to avoid redefining CC, since a) NWChem does not have C source that is a computational bottleneck and b) we typically test just the default C compiler. In other words, the recommendation is to compile with make FC=ifort

How-to: Linux workstation platforms

  • Common environmental variables for building in serial or in parallel with MPI
 % setenv NWCHEM_TOP <your path>/nwchem
 % setenv NWCHEM_TARGET LINUX64
 % setenv NWCHEM_MODULES  all
  • Common environmental variables for building with MPI

The following environment variables need to be set when NWChem is compiled with MPI:

 % setenv USE_MPI y
 % setenv USE_MPIF y
 % setenv USE_MPIF4 y
 % setenv MPI_LOC <your path>/openmpi-1.4.3  (for example, if you are using OpenMPI)
 % setenv MPI_LIB <your path>/openmpi-1.4.3/lib
 % setenv MPI_INCLUDE <your path>/openmpi-1.4.3/include
 % setenv LIBMPI "-lmpi_f90 -lmpi_f77 -lmpi -lpthread"
  • Compiling the code once all variables are set
  % cd $NWCHEM_TOP/src

  % make nwchem_config

  % make FC=gfortran >& make.log

How to: Mac platforms

Compilation of NWChem 6.1.1 release Mac OS X Lion 10.7

How-to: Cray platforms

Common environmental variables for building and running on the Cray XT and XE:

  % setenv NWCHEM_TOP <your path>/nwchem
  % setenv NWCHEM_TARGET LINUX64
  % setenv NWCHEM_MODULES all
  % setenv USE_MPI y
  % setenv USE_MPIF y
  % setenv USE_MPIF4 y
  % setenv USE_SCALAPACK y
  % setenv USE_64TO32 y
  % setenv LIBMPI " "
  • Portals, e.g. XT3, XT4, XT5

Set the following environmental variable for compilation:

  % setenv ARMCI_NETWORK PORTALS
  • Gemini, e.g. XE6

Load the onsided module by executing the command

  % module load onesided

Note: Preferred version of onesided is 1.5.0 or later ones.

Set the environmental variable for compilation:

  % setenv ARMCI_NETWORK GEMINI
  % setenv ONESIDED_USE_UDREG 1
  • Compiling the code on Cray once all variables are set
  % cd $NWCHEM_TOP/src

  % make nwchem_config

  % make 64_to_32

  % make FC=ftn >& make.log

How-to: IBM platforms

  • Compiling NWChem on BLUEGENE/L

The following environment variables need to be set

  % setenv NWCHEM_TOP <your path>/nwchem
  % setenv NWCHEM_TARGET BGL
  % setenv ARMCI_NETWORK BGMLMPI
  % setenv BGLSYS_DRIVER /bgl/BlueLight/ppcfloor
  % setenv BGLSYS_ROOT ${BGLSYS_DRIVER}/bglsys
  % setenv BLRTS_GNU_ROOT ${BGLSYS_DRIVER}/blrts-gnu
  % setenv BGDRIVER ${BGLSYS_DRIVER}
  % setenv BGCOMPILERS ${BLRTS_GNU_ROOT}/bin
  % setenv USE_MPI y
  % setenv LARGE_FILES TRUE
  % setenv MPI_LIB ${BGLSYS_ROOT}/lib
  % setenv MPI_INCLUDE ${BGLSYS_ROOT}/include
  % setenv LIBMPI "-lfmpich_.rts -lmpich.rts -lmsglayer.rts -lrts.rts -ldevices.rts"
  % setenv BGMLMPI_INCLUDE /bgl/BlueLight/ppcfloor/bglsys/include
  % setenv BGMLLIBS /bgl/BlueLight/ppcfloor/bglsys/lib

To compile, the following commands should be used:

 % cd $NWCHEM_TOP/src

 % make nwchem_config

 % make FC=blrts_xlf >& make.log
  • Compiling NWChem on BLUEGENE/P

The following environment variables need to be set

  % setenv NWCHEM_TARGET BGP
  % setenv ARMCI_NETWORK DCMFMPI
  % setenv MSG_COMMS DCMFMPI
  % setenv USE_MPI y
  % setenv LARGE_FILES TRUE
  % setenv BGP_INSTALLDIR /bgsys/drivers/ppcfloor
  % setenv BGCOMPILERS /bgsys/drivers/ppcfloor/gnu-linux/bin
  % setenv BGP_RUNTIMEPATH  /bgsys/drivers/ppcfloor/runtime
  % setenv ARMCIDRV ${BGP_INSTALLDIR}
  % setenv BGDRIVER ${ARMCIDRV}
  % setenv MPI_LIB ${BGDRIVER}/comm/lib
  % setenv MPI_INCLUDE ${BGDRIVER}/comm/include
  % setenv LIBMPI "-L${MPI_LIB} -lfmpich_.cnk -lmpich.cnk -ldcmfcoll.cnk -ldcmf.cnk -lpthread -lrt -L${BGP_RUNTIMEPATH}/SPI -lSPI.cna"
  % setenv BGMLMPI_INCLUDE ${MPI_INCLUDE}

To compile, the following commands should be used:

 % cd $NWCHEM_TOP/src

 % make nwchem_config

 % make FC=bgxlf >& make.log
  • Compiling NWChem on IBM PowerPC architectures

The following environment variables should be set:

  % setenv NWCHEM_TOP <your path>/nwchem
  % setenv NWCHEM_TARGET IBM64
  % setenv ARMCI_NETWORK MPI-MT
  % setenv OBJECT_MODE 64
  % setenv USE_MPI y
  % setenv LARGE_FILES TRUE
  % setenv MPI_LIB /usr/lpp/ppe.poe/lib
  % setenv MPI_INCLUDE /usr/lpp/ppe.poe/include
  % setenv LIBMPI "-lmpi -lpthreads"

To compile, the following commands should be used:

 % cd $NWCHEM_TOP/src

 % make nwchem_config

 % make FC=xlf >& make.log

How-to: Commodity clusters with Infiniband

Common environmental variables for building and running on most Infiniband clusters are:

  % setenv NWCHEM_TOP <your path>/nwchem
  % setenv NWCHEM_TARGET LINUX64
  % setenv NWCHEM_MODULES "all
  % setenv USE_MPI y
  % setenv USE_MPIF y
  % setenv USE_MPIF4 y
  % setenv MPI_LIB <Location of MPI library>/lib
  % setenv MPI_INCLUDE <Location of MPI library>/include
  % setenv LIBMPI <MPI library, e.g -lmtmpi or -lmpich>
  • On Infiniband clusters with the OpenIB software stack, the following environment variables should be defined
  % setenv ARMCI_NETWORK OPENIB
  % setenv IB_INCLUDE <Location of Infiniband libraries>/include
  % setenv MSG_COMMS MPI
  • On Infiniband clusters that do not support OpenIB, such as Myrinet MX, the MPI2 protocol can be used
  % setenv ARMCI_NETWORK MPI-MT
  • Compiling the code on an Infiniband cluster once all variables are set
  % cd $NWCHEM_TOP/src

  % make nwchem_config

  % make >& make.log

How-to: Windows Platforms

The current recommended approach for building a NWChem binary for a Windows platform is to build within Cygwin with make, perl, and gcc/gfortran version 4 installed.

  % setenv NWCHEM_TOP <your path>/nwchem
  % setenv NWCHEM_TARGET CYGWIN
  % setenv NWCHEM_MODULES all
  % cd $NWCHEM_TOP/src

  % make nwchem_config

  % make >& make.log

It used to be possible to build a version for native Windows using the Compaq Visual Fortran compiler. This has not been tested for the last couple of releases as the NWChem.

  % setenv NWCHEM_TOP <your path>/nwchem
  % setenv NWCHEM_TARGET WIN32
  % setenv NWCHEM_MODULES all

To start the compilation, start the Microsoft makefile utility from the top level source directory by typing "nmake". Reminder: For Compaq visual fortran don't forget to execute the "dfvars" script.

General site installation

The build procedures outlined above will allow use of NWChem within the NWChem directory structure. The code will look for the basis set library file in a default place within that directory structure. To install the code in a general, public place (e.g., /usr/local/NWChem) the following procedure can be applied:

  • Determine the local storage path for the install files. (e.g., /usr/local/NWChem).
  • Make directories
  mkdir /usr/local/NWChem
  mkdir /usr/local/NWChem/bin
  mkdir /usr/local/NWChem/data
  • Copy binary
  cp $NWCHEM_TOP/bin/${NWCHEM_TARGET}/nwchem /usr/local/NWChem/bin
  cd /usr/local/NWChem/bin 
  chmod 755 nwchem
  • Set links to data files (basis sets, force fields, etc.)
  cd $NWCHEM_TOP/src/basis
  cp -r libraries /usr/local/NWChem/data

  cd $NWCHEM_TOP/src/
  cp -r data /usr/local/NWChem

  cd $NWCHEM_TOP/src/nwpw
  cp -r libraryps /usr/local/NWChem/data
  • Each user will need a .nwchemrc file to point to these default data files. A global one could be put in /usr/local/NWChem/data and a symbolic link made in each users $HOME directory is probably the best plan for new installs. Users would have to issue the following command prior to using NWChem: ln -s /usr/local/NWChem/data/default.nwchemrc $HOME/.nwchemrc

Contents of the default.nwchemrc file based on the above information should be:

  nwchem_basis_library /usr/local/NWChem/data/libraries/
  nwchem_nwpw_library /usr/local/NWChem/data/libraryps/
  ffield amber
  amber_1 /usr/local/NWChem/data/amber_s/
  amber_2 /usr/local/NWChem/data/amber_q/
  amber_3 /usr/local/NWChem/data/amber_x/
  amber_4 /usr/local/NWChem/data/amber_u/
  spce    /usr/local/NWChem/data/solvents/spce.rst
  charmm_s /usr/local/NWChem/data/charmm_s/
  charmm_x /usr/local/NWChem/data/charmm_x/

Of course users can copy this file instead of making the symbolic link described above and change these defaults at their discretion.

It is can also be useful to use the NWCHEM_BASIS_LIBRARY environment variable when testing a new installation when an old one exists. This will allow you to overwrite the value of nwchem_basis_library in your .nwchemrc file and point to the new basis library. For example:

    % setenv NWCHEM_BASIS_LIBRARY "$NWCHEM/data-5.0/libraries/"

Do not forget the trailing "/".