modified on 30 September 2015 at 07:41 ••• 20,194 views


From NWChem

Revision as of 07:41, 30 September 2015 by Jhammond (Talk | contribs)
Jump to: navigation, search

Choosing the Right ARMCI Library

The Global Arrays parallel environment relies upon a one-sided communication runtime system. There are at least three options currently available:

  • ARMCI - This is the original one-sided library developed with Global Arrays by PNNL. It supports the widest range of platforms. See for details.
  • ComEx - This library is being developed by PNNL to replace ARMCI. It supports the fewer platforms than ARMCI but may be more robust on modern ones like InfiniBand. See for details.
  • ARMCI-MPI - This library is a completely separate implementation of the ARMCI interface by Argonne and Intel. It uses the one-sided communication features of MPI, rather than a platform-specific conduit. It currents provides both MPI-2 and MPI-3 implementations; the MPI-3 one is supported by nearly all platforms. See for details.

By default, Global Arrays will choose ARMCI or ComEx, based upon the environment variables selected by the user (see GA documentation for details). When a native implementation is available and works reliably, this is the best option for the NWChem user. However, in cases where an native implementation of ARMCI is not available or is not reliable, the user should consider using one of the MPI-based implementations.

There are many different ways to use MPI as the communication runtime of Global Arrays:

ARMCI_NETWORK Additional options Result Notes
MPI-PR ARMCI with progress rank Recommended
MPI-PT ARMCI with progress thread
MPI-MT ARMCI over multi-threaded MPI Do not use Open-MPI.
MPI-TS ARMCI over MPI without data server
MPI-SPAWN ARMCI over MPI using dynamic processes Requires MPI_Comm_spawn support.
ARMCI EXTERNAL_ARMCI_PATH Uses ARMCI-MPI (please use mpi3rma branch) See the ARMCI-MPI NWChem page for details.

It is difficult to provide complete guidance to the user as to which option to choose. However, it is observed that ARMCI_NETWORK=MPI-PR is stable and performs well on Cray XC platforms (e.g. NERSC Edison). On Intel True Scale systems, MPI-MT is more reliable than OPENIB or MPI-SPAWN. On IBM Blue Gene/Q, one must use the mpi2rma branch of ARMCI-MPI because MPI-3 is not fully supported. On all other platforms, the ARMCI-MPI branch mpi3rma is recommended. Please make sure to use the most recent version of MPI when using ARMCI-MPI.