{"id":403669,"date":"2025-01-06T08:34:54","date_gmt":"2025-01-06T08:34:54","guid":{"rendered":"https:\/\/innovationspace.ansys.com\/forum\/forums\/topic\/trouble-running-mpi-with-ansys-fluent-on-hpc\/"},"modified":"2025-01-06T08:35:52","modified_gmt":"2025-01-06T08:35:52","slug":"trouble-running-mpi-with-ansys-fluent-on-hpc","status":"publish","type":"topic","link":"https:\/\/innovationspace.ansys.com\/forum\/forums\/topic\/trouble-running-mpi-with-ansys-fluent-on-hpc\/","title":{"rendered":"Trouble Running MPI with ANSYS Fluent on HPC"},"content":{"rendered":"

<p><div><div>Hi,<\/div><br><div>I’m encountering issues with running ANSYS Fluent using MPI on HPC.<\/div><div>Below, I’ve included my SLURM job script and a snippet of the log output for reference.<\/div><br><div>I’m unsure how to proceed at this point and would appreciate any guidance or suggestions.<\/div><br><div>Job Script:<\/strong><\/div><br><div>#!\/bin\/bash<\/div><div>#SBATCH -J Fluent <\/div><div>#SBATCH -o run.out <\/div><div>#SBATCH -N 2 <\/div><div>#SBATCH -n 256 <\/div><div>#SBATCH -p development<\/div><div>#SBATCH -t 2:00:00<\/div><div>#SBATCH -A MYPROJECT<\/div><br><div>set echo on<\/div><div>total_tasks=256<\/div><div>tasks_per_node=128<\/div><br><div>fluent232=\/scratch\/tacc\/apps\/ANSYS\/2023R2\/v232\/fluent\/bin\/fluent<\/div><br><div>module load ansys<\/div><br><div>echo “Generating PNODES, removing log files!”<\/div><div>rm -f pnodes<\/div><div>nlist=$(scontrol show hostname $SLURM_NODELIST | paste -d, -s)<\/div><div>echo $nlist<\/div><div>echo $SLURM_CPUS_ON_NODE<\/div><div>for node in $(echo $nlist | tr “,” ” “); do<\/div><div>for i in $(seq 1 $tasks_per_node); do<\/div><div>echo $node >> pnodes<\/div><div>done<\/div><div>done<\/div><br><div>$fluent232 3ddp -t$total_tasks -g -cnf=pnodes -mpi=intel -pib.infinipath -ssh -g < run.inp >> run.log<\/div><br><br><div>Log Output (Snippet):<\/strong><\/div><br><div>Host spawning Node 0 on machine “c303-005.ls6.tacc.utexas.edu” (unix).<\/div><div>\/scratch\/tacc\/apps\/ANSYS\/2023R2\/v232\/fluent\/fluent23.2.0\/bin\/fluent -r23.2.0 3ddp -flux -node -t256 -pinfiniband -mpi=intel -cnf=pnodes -ssh -mport 129.114.41.53:129.114.41.53:40663:0<\/div><div>Starting \/scratch\/tacc\/apps\/ANSYS\/2023R2\/v232\/fluent\/fluent23.2.0\/multiport\/mpi\/lnamd64\/intel2021\/bin\/mpirun -f \/tmp\/fluent-appfile.MYID.919430 –rsh=ssh -genv FLUENT_ARCH lnamd64 -genv I_MPI_DEBUG 0 -genv I_MPI_ADJUST_GATHERV 3 -genv I_MPI_ADJUST_ALLREDUCE 2 -genv I_MPI_PLATFORM auto -genv PYTHONHOME \/scratch\/tacc\/apps\/ANSYS\/2023R2\/v232\/fluent\/fluent23.2.0\/..\/..\/commonfiles\/CPython\/3_10\/linx64\/Release\/python -genv FLUENT_PROD_DIR \/scratch\/tacc\/apps\/ANSYS\/2023R2\/v232\/fluent\/fluent23.2.0 -genv FLUENT_AFFINITY 0 -genv I_MPI_PIN enable -genv KMP_AFFINITY disabled -machinefile \/tmp\/fluent-appfile.MYID.919430 -np 256 \/scratch\/tacc\/apps\/ANSYS\/2023R2\/v232\/fluent\/fluent23.2.0\/lnamd64\/3ddp_node\/fluent_mpi.23.2.0 node -mpiw intel -pic infiniband -mport 129.114.41.53:129.114.41.53:40663:0<\/div><div>[mpiexec@c303-005.ls6.tacc.utexas.edu] check_exit_codes (..\/..\/..\/..\/..\/src\/pm\/i_hydra\/libhydra\/demux\/hydra_demux_poll.c:117): unable to run bstrap_proxy on c303-006 (pid 925626, exit code 65280)<\/div><div>[mpiexec@c303-005.ls6.tacc.utexas.edu] poll_for_event (..\/..\/..\/..\/..\/src\/pm\/i_hydra\/libhydra\/demux\/hydra_demux_poll.c:159): check exit codes error<br>[mpiexec@c303-005.ls6.tacc.utexas.edu] HYD_dmx_poll_wait_for_proxy_event (..\/..\/..\/..\/..\/src\/pm\/i_hydra\/libhydra\/demux\/hydra_demux_poll.c:212): poll for event error<br>[mpiexec@c303-005.ls6.tacc.utexas.edu] HYD_bstrap_setup (..\/..\/..\/..\/..\/src\/pm\/i_hydra\/libhydra\/bstrap\/src\/intel\/i_hydra_bstrap.c:1061): error waiting for event<br>[mpiexec@c303-005.ls6.tacc.utexas.edu] HYD_print_bstrap_setup_error_message (..\/..\/..\/..\/..\/src\/pm\/i_hydra\/mpiexec\/intel\/i_mpiexec.c:1027): error setting up the bootstrap proxies<\/div><br><br><div>I suspect there might be an issue with how MPI is set up or how the nodes are being utilized, but I’m not sure where to start troubleshooting.<\/div><br><div>Could someone help me:<\/div><\/p><\/p>\n

    \n
  1. Identify possible issues in my SLURM job script.<\/li>\n
  2. Understand if the MPI configuration might be causing this issue.<\/li>\n
  3. Suggest any debug or diagnostic steps I can take.<\/li>\n<\/ol>\n

    <p>Thank You!<\/p><p><br><br><\/div><\/p><\/p>\n","protected":false},"template":"","class_list":["post-403669","topic","type-topic","status-publish","hentry"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t