{"id":390807,"date":"2024-10-22T10:58:49","date_gmt":"2024-10-22T10:58:49","guid":{"rendered":"https:\/\/innovationspace.ansys.com\/forum\/forums\/reply\/390807\/"},"modified":"2024-10-22T10:58:49","modified_gmt":"2024-10-22T10:58:49","slug":"390807","status":"publish","type":"reply","link":"https:\/\/innovationspace.ansys.com\/forum\/forums\/reply\/390807\/","title":{"rendered":"Reply To: Fluent fails with Intel MPI protocol on 2 nodes"},"content":{"rendered":"

<p><span style=”font-size: 10pt;”>Morning George, thank you for replying.<\/span><\/p><\/p>\n

    <li style=”font-size: 10pt;”><span style=”font-size: 10pt;”>Tested both Fluent 2023R1 and 2024R1. <\/span><\/li>\n

    <li style=”font-size: 10pt;”><span style=”font-size: 10pt;”>Following a similar discussion on this Forum we tried to set environmental variable “I_MPI_PLATFORM” to “none”. Starting Fluent from the same terminal (either in text or graphic interface) resulted in Fluent output “I_MPI_PLATFORM=auto” anyway.<\/span><\/li>\n

    <li style=”font-size: 10pt;”><span style=”font-size: 10pt;”> B<\/span>oth Fluent versions crashed with the same output:<\/li>\n<\/ol>\n

    <p><span style=”font-size: 10pt;”>——————————————————————————————————————–<\/span><\/p><p class=”MsoNormal” style=”margin-bottom: 0cm; text-align: justify; line-height: normal;”><span style=”font-size: 10pt; font-family: Arial, sans-serif;”>Opening input\/output transcript to file “\/users\/ssivaraman\/fluent-20241022-100917-1713090.trn”.<\/span><\/p><p class=”MsoNormal” style=”margin-bottom: 0cm; text-align: justify; line-height: normal;”><span style=”font-size: 10pt; font-family: Arial, sans-serif;”>Auto-Transcript Start Time:<span style=”mso-spacerun: yes;”>  <\/span>10:09:17, 22 Oct 2024 <\/span><\/p><p class=”MsoNormal” style=”margin-bottom: 0cm; text-align: justify; line-height: normal;”><span style=”font-size: 10pt; font-family: Arial, sans-serif;”>\/opt\/apps\/ansys\/v241\/fluent\/fluent24.1.0\/bin\/fluent -r24.1.0 3ddp -host -t256 -cnf=node[187-188] -path\/opt\/apps\/ansys\/v241\/fluent -ssh -cx node187.pri.kelvin2.alces.network:43609:33349<\/span><\/p><p class=”MsoNormal” style=”margin-bottom: 0cm; text-align: justify; line-height: normal;”><span style=”font-size: 10pt; font-family: Arial, sans-serif;”>Starting \/opt\/apps\/ansys\/v241\/fluent\/fluent24.1.0\/lnamd64\/3ddp_host\/fluent.24.1.0 host -cx node187.pri.kelvin2.alces.network:43609:33349 “(list (rpsetvar (QUOTE parallel\/function) “fluent 3ddp -flux -node -r24.1.0 -t256 -pdefault -mpi=intel -cnf=node[187-188] -ssh”) (rpsetvar (QUOTE parallel\/rhost) “”) (rpsetvar (QUOTE parallel\/ruser) “”) (rpsetvar (QUOTE parallel\/nprocs_string) “256”) (rpsetvar (QUOTE parallel\/auto-spawn?) #t) (rpsetvar (QUOTE parallel\/trace-level) 0) (rpsetvar (QUOTE parallel\/remote-shell) 1) (rpsetvar (QUOTE parallel\/path) “\/opt\/apps\/ansys\/v241\/fluent”) (rpsetvar (QUOTE parallel\/hostsfile) “node[187-188]”) (rpsetvar (QUOTE gpuapp\/devices) “”))”<\/span><\/p><p class=”MsoNormal” style=”margin-bottom: 0cm; text-align: justify; line-height: normal;”><span style=”font-size: 10pt; font-family: Arial, sans-serif;”><span style=”mso-spacerun: yes;”>              <\/span>Welcome to ANSYS Fluent 2024 R1<\/span><\/p><p class=”MsoNormal” style=”margin-bottom: 0cm; text-align: justify; line-height: normal;”><span style=”font-size: 10pt; font-family: Arial, sans-serif;”> <\/span><\/p><p class=”MsoNormal” style=”margin-bottom: 0cm; text-align: justify; line-height: normal;”><span style=”font-size: 10pt; font-family: Arial, sans-serif;”><span style=”mso-spacerun: yes;”>              <\/span>Copyright 1987-2024 ANSYS, Inc. All Rights Reserved.<\/span><\/p><p class=”MsoNormal” style=”margin-bottom: 0cm; text-align: justify; line-height: normal;”><span style=”font-size: 10pt; font-family: Arial, sans-serif;”><span style=”mso-spacerun: yes;”>              <\/span>Unauthorized use, distribution or duplication is prohibited.<\/span><\/p><p class=”MsoNormal” style=”margin-bottom: 0cm; text-align: justify; line-height: normal;”><span style=”font-size: 10pt; font-family: Arial, sans-serif;”><span style=”mso-spacerun: yes;”>              <\/span>This product is subject to U.S. laws governing export and re-export.<\/span><\/p><p class=”MsoNormal” style=”margin-bottom: 0cm; text-align: justify; line-height: normal;”><span style=”font-size: 10pt; font-family: Arial, sans-serif;”><span style=”mso-spacerun: yes;”>              <\/span>For full Legal Notice, see documentation.<\/span><\/p><p class=”MsoNormal” style=”margin-bottom: 0cm; text-align: justify; line-height: normal;”><span style=”font-size: 10pt; font-family: Arial, sans-serif;”> <\/span><\/p><p class=”MsoNormal” style=”margin-bottom: 0cm; text-align: justify; line-height: normal;”><span style=”font-size: 10pt; font-family: Arial, sans-serif;”>Build Time: Nov 22 2023 10:07:25 EST<span style=”mso-spacerun: yes;”>  <\/span>Build Id: 10184<span style=”mso-spacerun: yes;”>  <\/span><\/span><\/p><p class=”MsoNormal” style=”margin-bottom: 0cm; text-align: justify; line-height: normal;”><span style=”font-size: 10pt; font-family: Arial, sans-serif;”><span style=”mso-spacerun: yes;”> <\/span><\/span><\/p><p class=”MsoNormal” style=”margin-bottom: 0cm; text-align: justify; line-height: normal;”><span style=”font-size: 10pt; font-family: Arial, sans-serif;”>Connected License Server List:<span style=”mso-tab-count: 1;”>  <\/span>1055@193.61.145.219<\/span><\/p><p class=”MsoNormal” style=”margin-bottom: 0cm; text-align: justify; line-height: normal;”><span style=”font-size: 10pt; font-family: Arial, sans-serif;”> <\/span><\/p><p class=”MsoNormal” style=”margin-bottom: 0cm; text-align: justify; line-height: normal;”><span style=”font-size: 10pt; font-family: Arial, sans-serif;”><span style=”mso-spacerun: yes;”>     <\/span>————————————————————–<\/span><\/p><p class=”MsoNormal” style=”margin-bottom: 0cm; text-align: justify; line-height: normal;”><span style=”font-size: 10pt; font-family: Arial, sans-serif;”><span style=”mso-spacerun: yes;”>     <\/span>This is an academic version of ANSYS FLUENT. Usage of this product<\/span><\/p><p class=”MsoNormal” style=”margin-bottom: 0cm; text-align: justify; line-height: normal;”><span style=”font-size: 10pt; font-family: Arial, sans-serif;”><span style=”mso-spacerun: yes;”>     <\/span>license is limited to the terms and conditions specified in your ANSYS<\/span><\/p><p class=”MsoNormal” style=”margin-bottom: 0cm; text-align: justify; line-height: normal;”><span style=”font-size: 10pt; font-family: Arial, sans-serif;”><span style=”mso-spacerun: yes;”>     <\/span>license form, additional terms section.<\/span><\/p><p class=”MsoNormal” style=”margin-bottom: 0cm; text-align: justify; line-height: normal;”><span style=”font-size: 10pt; font-family: Arial, sans-serif;”><span style=”mso-spacerun: yes;”>     <\/span>————————————————————–<\/span><\/p><p class=”MsoNormal” style=”margin-bottom: 0cm; text-align: justify; line-height: normal;”><span style=”font-size: 10pt; font-family: Arial, sans-serif;”>Host spawning Node 0 on machine “node187.pri.kelvin2.alces.network” (unix).<\/span><\/p><p class=”MsoNormal” style=”margin-bottom: 0cm; text-align: justify; line-height: normal;”><span style=”font-size: 10pt; font-family: Arial, sans-serif;”>\/opt\/apps\/ansys\/v241\/fluent\/fluent24.1.0\/bin\/fluent -r24.1.0 3ddp -flux -node -t256 -pdefault -mpi=intel -cnf=node[187-188] -ssh -mport 10.10.15.27:10.10.15.27:41037:0<\/span><\/p><p class=”MsoNormal” style=”margin-bottom: 0cm; text-align: justify; line-height: normal;”><span style=”font-size: 10pt; font-family: Arial, sans-serif;”>Starting \/opt\/apps\/ansys\/v241\/fluent\/fluent24.1.0\/multiport\/mpi\/lnamd64\/intel2021\/bin\/mpirun -f \/tmp\/fluent-appfile.ssivaraman.1713921 –rsh=ssh -genv FI_PROVIDER tcp -genv FLUENT_ARCH lnamd64 -genv I_MPI_DEBUG 0 -genv I_MPI_ADJUST_GATHERV 3 -genv I_MPI_ADJUST_ALLREDUCE 2 -genv <span style=”color: red;”>I_MPI_PLATFORM auto <\/span>-genv PYTHONHOME \/opt\/apps\/ansys\/v241\/fluent\/fluent24.1.0\/..\/..\/commonfiles\/CPython\/3_10\/linx64\/Release\/python -genv FLUENT_PROD_DIR \/opt\/apps\/ansys\/v241\/fluent\/fluent24.1.0 -genv FLUENT_AFFINITY 0 -genv I_MPI_PIN enable -genv KMP_AFFINITY disabled -machinefile \/tmp\/fluent-appfile.ssivaraman.1713921 -np 256 \/opt\/apps\/ansys\/v241\/fluent\/fluent24.1.0\/lnamd64\/3ddp_node\/fluent_mpi.24.1.0 node -mpiw intel -pic default -mport 10.10.15.27:10.10.15.27:41037:0<\/span><\/p><p class=”MsoNormal” style=”margin-bottom: 0cm; text-align: justify; line-height: normal;”><span style=”font-size: 10pt; font-family: Arial, sans-serif;”>[mpiexec@node187.pri.kelvin2.alces.network] check_exit_codes (..\/..\/..\/..\/..\/src\/pm\/i_hydra\/libhydra\/demux\/hydra_demux_poll.c:117): unable to run bstrap_proxy on node[187-188] (pid 1714765, exit code 65280)<\/span><\/p><p class=”MsoNormal” style=”margin-bottom: 0cm; text-align: justify; line-height: normal;”><span style=”font-size: 10pt; font-family: Arial, sans-serif;”>[mpiexec@node187.pri.kelvin2.alces.network] poll_for_event (..\/..\/..\/..\/..\/src\/pm\/i_hydra\/libhydra\/demux\/hydra_demux_poll.c:159): check exit codes error<\/span><\/p><p class=”MsoNormal” style=”margin-bottom: 0cm; text-align: justify; line-height: normal;”><span style=”font-size: 10pt; font-family: Arial, sans-serif;”>[mpiexec@node187.pri.kelvin2.alces.network] HYD_dmx_poll_wait_for_proxy_event (..\/..\/..\/..\/..\/src\/pm\/i_hydra\/libhydra\/demux\/hydra_demux_poll.c:212): poll for event error<\/span><\/p><p class=”MsoNormal” style=”margin-bottom: 0cm; text-align: justify; line-height: normal;”><span style=”font-size: 10pt; font-family: Arial, sans-serif;”>[mpiexec@node187.pri.kelvin2.alces.network] HYD_bstrap_setup (..\/..\/..\/..\/..\/src\/pm\/i_hydra\/libhydra\/bstrap\/src\/intel\/i_hydra_bstrap.c:1061): error waiting for event<\/span><\/p><p class=”MsoNormal” style=”margin-bottom: 0cm; text-align: justify; line-height: normal;”><span style=”font-size: 10pt; font-family: Arial, sans-serif;”>[mpiexec@node187.pri.kelvin2.alces.network] HYD_print_bstrap_setup_error_message (..\/..\/..\/..\/..\/src\/pm\/i_hydra\/mpiexec\/intel\/i_mpiexec.c:1027): error setting up the bootstrap proxies<\/span><\/p><p class=”MsoNormal” style=”margin-bottom: 0cm; text-align: justify; line-height: normal;”><span style=”font-size: 10pt; font-family: Arial, sans-serif;”>[mpiexec@node187.pri.kelvin2.alces.network] Possible reasons:<\/span><\/p><p class=”MsoNormal” style=”margin-bottom: 0cm; text-align: justify; line-height: normal;”><span style=”font-size: 10pt; font-family: Arial, sans-serif;”>[mpiexec@node187.pri.kelvin2.alces.network] 1. Host is unavailable. Please check that all hosts are available.<\/span><\/p><p class=”MsoNormal” style=”margin-bottom: 0cm; text-align: justify; line-height: normal;”><span style=”font-size: 10pt; font-family: Arial, sans-serif;”>[mpiexec@node187.pri.kelvin2.alces.network] 2. Cannot launch hydra_bstrap_proxy or it crashed on one of the hosts. Make sure hydra_bstrap_proxy is available on all hosts and it has right permissions.<\/span><\/p><p class=”MsoNormal” style=”margin-bottom: 0cm; text-align: justify; line-height: normal;”><span style=”font-size: 10pt; font-family: Arial, sans-serif;”>[mpiexec@node187.pri.kelvin2.alces.network] 3. Firewall refused connection. Check that enough ports are allowed in the firewall and specify them with the I_MPI_PORT_RANGE variable.<\/span><\/p><p class=”MsoNormal” style=”margin-bottom: 0cm; text-align: justify; line-height: normal;”><span style=”font-size: 10pt; font-family: Arial, sans-serif;”>[mpiexec@node187.pri.kelvin2.alces.network] 4. Ssh bootstrap cannot launch processes on remote host. Make sure that passwordless ssh connection is established across compute hosts.<\/span><\/p><p class=”MsoNormal” style=”margin-bottom: 0cm; text-align: justify; line-height: normal;”><span style=”font-size: 10pt; font-family: Arial, sans-serif;”>[mpiexec@node187.pri.kelvin2.alces.network]<span style=”mso-spacerun: yes;”>    <\/span>You may try using -bootstrap option to select alternative launcher.<\/span><\/p><p><span style=”font-size: 10pt;”>——————————————————————————————————————–<\/span><\/p><\/p>\n","protected":false},"template":"","class_list":["post-390807","reply","type-reply","status-publish","hentry"],"aioseo_notices":[],"acf":[],"_links":{"self":[{"href":"https:\/\/innovationspace.ansys.com\/forum\/wp-json\/wp\/v2\/replies\/390807","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/innovationspace.ansys.com\/forum\/wp-json\/wp\/v2\/replies"}],"about":[{"href":"https:\/\/innovationspace.ansys.com\/forum\/wp-json\/wp\/v2\/types\/reply"}],"version-history":[{"count":0,"href":"https:\/\/innovationspace.ansys.com\/forum\/wp-json\/wp\/v2\/replies\/390807\/revisions"}],"wp:attachment":[{"href":"https:\/\/innovationspace.ansys.com\/forum\/wp-json\/wp\/v2\/media?parent=390807"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}