Run Singularity on HPC

From cpl-wiki
Jump to: navigation, search

In order to use the Docker containers on HPC, you can pull the Docker container using singularity,

 singularity pull docker://cpllibrary/cpl-openfoam-lammps

Then we can run two example applications in two separate containers which link and exchange informations, you will need to run this in a directory which contains the ./cpl/COUPLER.in input files. As an example, you can get just this file as follows,

mkdir cpl
cd ./cpl
wget https://raw.githubusercontent.com/Crompulence/cpl-library/master/examples/minimal_send_recv_mocks/cpl/COUPLER.in
cd ../

The command to run these examples uses the path inside the singularity container to the files,

mpirun -np 1 singularity exec cpl-openfoam-lammps.simg /cpl-library/examples/minimal_send_recv_mocks/minimal_CFD.py : \
       -np 1 singularity exec cpl-openfoam-lammps.simg /cpl-library/examples/minimal_send_recv_mocks/minimal_MD.py

This should run and output,

 MPMD mode, CFD and MD both share MPI_COMM_WORLD
 Completed CPL communicator init for MD  , CPL_WORLD_COMM ID:           1
 Completed CPL communicator init for CFD , CPL_WORLD_COMM ID:           0
                                                                  
   ________/\\\\\\\\\__/\\\\\\\\\\\\\____/\\\_____________        
    _____/\\\////////__\/\\\/////////\\\_\/\\\_____________       
     ___/\\\/___________\/\\\_______\/\\\_\/\\\_____________      
      __/\\\_____________\/\\\\\\\\\\\\\/__\/\\\_____________     
       _\/\\\_____________\/\\\/////////____\/\\\_____________    
        _\//\\\____________\/\\\_____________\/\\\_____________   
         __\///\\\__________\/\\\_____________\/\\\_____________  
          ____\////\\\\\\\\\_\/\\\_____________\/\\\\\\\\\\\\\\\_ 
           _______\/////////__\///______________\///////////////__
                                                                  
                      C P L  -  L I B R A R Y                     
                                                                  
('CFD', 0, 0.0)
('MD', 0, 0.0)
('CFD', 1, 5.0)
('MD', 1, 2.0)
('CFD', 2, 10.0)
('MD', 2, 4.0)
('CFD', 3, 15.0)
('MD', 3, 6.0)
('CFD', 4, 20.0)
('MD', 4, 8.0)

Notice that we are running mpi outside the container, as recommended on the singularity website. This replies on ABI compatibility of the version of MPI you have on your system and the version installed in the docker container (MPICH3.2 at time of writing).

You can run using the MPI Port system by using two terminal, each with an mpiexec call, or simply put the first processes into the background using &,

mpirun -np 1 singularity exec cpl-openfoam-lammps.simg /cpl-library/examples/minimal_send_recv_mocks/minimal_CFD.py & \
mpirun -np 1 singularity exec cpl-openfoam-lammps.simg /cpl-library/examples/minimal_send_recv_mocks/minimal_MD.py

which runs as before with the

 Only CFD realm present in MPI_COMM_WORLD
 opened port: tag#0$description#meflow$port#44444$ifname#127.0.0.1$ Attempting to write to file:./port
 Portname written to file ./port                                                          
 Only MD realm present in MPI_COMM_WORLD
 accepted connection on port to root of            1  procs.
 connection accepted to root of            1  procs.
 Rank on realm            2  is            1  of            1   and rank on intercomm is            0  of            2
 Completed CPL communicator init for MD  , CPL_WORLD_COMM ID:           0
 Rank on realm            1  is            1  of            1   and rank on intercomm is            1  of            2
 Completed CPL communicator init for CFD , CPL_WORLD_COMM ID:           1
                                                                  
   ________/\\\\\\\\\__/\\\\\\\\\\\\\____/\\\_____________        
    _____/\\\////////__\/\\\/////////\\\_\/\\\_____________       
     ___/\\\/___________\/\\\_______\/\\\_\/\\\_____________      
      __/\\\_____________\/\\\\\\\\\\\\\/__\/\\\_____________     
       _\/\\\_____________\/\\\/////////____\/\\\_____________    
        _\//\\\____________\/\\\_____________\/\\\_____________   
         __\///\\\__________\/\\\_____________\/\\\_____________  
          ____\////\\\\\\\\\_\/\\\_____________\/\\\\\\\\\\\\\\\_ 
           _______\/////////__\///______________\///////////////__
                                                                  
                      C P L  -  L I B R A R Y                     
                                                                  
('CFD', 0, 0.0)
('MD', 0, 0.0)
('CFD', 1, 5.0)
('MD', 1, 2.0)
('CFD', 2, 10.0)
('MD', 2, 4.0)
('CFD', 3, 15.0)
('MD', 3, 6.0)
('CFD', 4, 20.0)
('MD', 4, 8.0)


Because of the way Singularity works, running MPI outside the container means it is not possible to run inside a single container using cplexec to create multiple MPI jobs,

mpirun -n 1 singularity exec cpl-openfoam-lammps.simg \
       cplexec -c 1 /cpl-library/examples/minimal_send_recv_mocks/minimal_CFD.py \
               -m 1 /cpl-library/examples/minimal_send_recv_mocks/minimal_MD.py

The problem is mpirun creates as many containers as specified with the -n flag, here one, so a container running on a single process tried to run two MPI processes inside. We no longer have the ability to specify how many processes are allocated to each of the coupled applications.

If you really want to use cplexec, then it is best to get a version of the cplexec file, which is just a python script, by git cloning cpl-library or even just getting the cplexec file and making it executable,

wget https://raw.githubusercontent.com/Crompulence/cpl-library/master/bin/cplexec
chmod +x ./cplexec
./cplexec -c 1 "singularity exec cpl-openfoam-lammps.simg /cpl-library/examples/minimal_send_recv_mocks/minimal_CFD.py"  -m 1 "singularity exec cpl-openfoam-lammps.simg /cpl-library/examples/minimal_send_recv_mocks/minimal_MD.py"

This then runs the two singularity containers each as an MPI process. As much of cplexec is dedicated to checking the libraries are all consistent and MPI is setup as expected, which is redundant for a containerised application, it is recommended to simply run with mpirun (or mpiexec) as above.