By calling `mpirun` outside of the container, we solve several very complicated work-flow aspects. For example, if `mpirun` is called from within the container it must have a method for spawning processes on remote nodes. Historically the SSH is used for this which means that there must be an `sshd` running within the container on the remote nodes, and this `sshd` process must not conflict with the `sshd` running on that host! It is also possible for the resource manager to launch the job and (in OpenMPI’s case) the Orted processes on the remote system, but that then requires resource manager modification and container awareness.
By calling `mpirun` outside of the container, we solve several very complicated work-flow aspects. For example, if `mpirun` is called from within the container it must have a method for spawning processes on remote nodes. Historically the SSH is used for this which means that there must be an `sshd` running within the container on the remote nodes, and this `sshd` process must not conflict with the `sshd` running on that host! It is also possible for the resource manager to launch the job and (in OpenMPI’s case) the Orted (Open RTE User-Level Daemon) processes on the remote system, but that then requires resource manager modification and container awareness.
In the end, we do not gain anything by calling `mpirun` from within the container except for increasing the complexity levels and possibly losing out on some added
performance benefits (e.g. if a container wasn’t built with the proper OFED as the host).
