Before this case, take a look at the CPL library in Docker to see an introduction to running the MPI processes inside a single Docker container. However, a far more interesting application is to create two containers and then link the processes running in them with MPI sending information between them. First we need to get the CPL library docker image:
sudo docker pull cpllibrary/cpl-library
Next, we want to start a coupled case with a persistent shared volume The shared volume:
To starts an interactive bash session called cplrun with volume cpl-vol mounted under folder /cplruns
sudo docker run -it --name cplrun1 --mount source=cpl-vol,target=/cplruns cpllibrary/cpl-library
In the first docker container, copy an example to cplruns
rsync -avP /cpl-library/examples/minimal_send_recv_mocks/ /cplruns/
Change directory to cplruns and run the CFD example
cd /cplruns/ mpiexec -n 1 python ./minimal_CFD.py
This will create a PORT file in /cplruns/ which will be used to link to another executable running in a second container. To understand how this work, check out the discussion in Run CPL library which outlines the use of MPI PORT as a way to avoid patching the two coupled codes. As a result, we need to open A NEW TERMINAL and create a second docker container running in this but with access to the same file system
sudo docker run -it --name cplrun2 --mount source=cpl-vol,target=/cplruns cpllibrary/cpl-library
In this container, we change to the cplruns directory and simply start the corresponding MD run we want to couple,
cd /cplruns/ mpiexec -n 8 python ./minimal_MD.py
The output from the coupled codes should appear in both terminals as they exchange information. This has only been tested on a single machine, ssh keys and a shared file system would need to be established between computers.
To see the location of the volume on the host system
sudo docker volume inspect cpl-vol
We can use this approach to couple OpenFOAM and LAMMPS. In this case, both Docker containers are built FROM the same cpllibrary/cpl-library so have identical MPI and CPL library. As before, open two separate terminals as we'll be creating a Docker container for each program.
Starting with the OpenFOAM docker container in terminal 1,
sudo docker run -it --name cplopenfoam --mount source=cpl-vol,target=/cplruns cpllibrary/cplopenfoam
Now, we need to copy as example to the directory shared between containers, we'll take the granular example which uses an fcc lattice with fixed particles,
rsync -avP /cpl-library/examples/fcc_fixedParticles /cplruns/ cd /cplruns/fcc_fixedParticles
To setup OpenFOAM, check out the contents of
run.sh which does the setup on the mesh in parallel and runs the case.
cd openfoam blockMesh decomposePar cd ../ mpiexec -n 1 CPLSediFOAM -case openfoam/
This will wait after writing a portfile which allows connection when read by the corresponding LAMMPS instance that want to be coupled,
Only CFD realm present in MPI_COMM_WORLD opened port: tag#0$description#9b5483f241c$port#41258$ifname#172.21.0.2$ Attempting to write to file:./port Portname written to file ./port
Next, in the other terminal start a LAMMPS Docker container,
sudo docker run -it --name cpllammps --mount source=cpl-vol,target=/cplruns cpllibrary/cpllammps
In the container, we need to change to the run directory cplruns and run,
cd /cplruns mpiexec -n 1 lmp_cpl < lammps/fcc.in
the LAMMPS instance runs, links to OpenFOAM in a different Docker container with communication through the MPI port system and runs a case.