{"id":166634,"date":"2022-02-22T13:54:31","date_gmt":"2022-02-22T13:54:31","guid":{"rendered":"\/forum\/forums\/topic\/ansys-fluent-running-in-parallel-issues\/"},"modified":"2022-04-06T14:02:20","modified_gmt":"2022-04-06T14:02:20","slug":"ansys-fluent-running-in-parallel-issues","status":"closed","type":"topic","link":"https:\/\/innovationspace.ansys.com\/forum\/forums\/topic\/ansys-fluent-running-in-parallel-issues\/","title":{"rendered":"Ansys Fluent Running in Parallel issues"},"content":{"rendered":"
\n
\n

OS: Centos 7 <\/p>\n

We use Slurm as a scheduler and have never had issues with MPI. By default our nodes use openmpi3\/3.1.4. When Requesting nodes I use the command: <\/p>\n

salloc -N 3 -n 20 –mem=80G -C ib<\/p>\n

This gives me 3 nodes, 20 cores each, 80G of memory and uses infiniband. I then SSH to the host node (with x11 forwarding) and then run ansys (runwb2 command) and load my Fluent workbench file. When clicking "Setup" I change the processing options to "Parallel Per Machine File" with 3 processes. Then click "Show More Options" -> "Parallel Settings". Interconnects = infiniband, MPI Types = openmpi, I then supply the file containing the machine names I was allocated with the salloc command.<\/p>\n

<\/p>\n<\/p>\n

This is the console log I recieve: <\/p>\n

\/opt\/ohpc\/pub\/apps\/ansys\/2019R3_Fluent\/v195\/fluent\/fluent19.5.0\/bin\/fluent -r19.5.0 3d -pinfiniband -host -alnamd64 -t3 -mpi=openmpi -cnf=\/mnt\/beegfs\/home\/testUser\/testNode.txt -path\/opt\/ohpc\/pub\/apps\/ansys\/2019R3_Fluent\/v195\/fluent -ssh -cx node007.hpc.fau.edu:39136:43808<\/p>\n

Starting \/opt\/ohpc\/pub\/apps\/ansys\/2019R3_Fluent\/v195\/fluent\/fluent19.5.0\/lnamd64\/3d_host\/fluent.19.5.0 host -cx node007.hpc.fau.edu:39136:43808 "(list (rpsetvar (QUOTE parallel\/function) "fluent 3d -flux -node -alnamd64 -r19.5.0 -t3 -pinfiniband -mpi=openmpi -cnf=\/mnt\/beegfs\/home\/testUser\/testNode.txt -ssh") (rpsetvar (QUOTE parallel\/rhost) "") (rpsetvar (QUOTE parallel\/ruser) "") (rpsetvar (QUOTE parallel\/nprocs_string) "3") (rpsetvar (QUOTE parallel\/auto-spawn?) #t) (rpsetvar (QUOTE parallel\/trace-level) 0) (rpsetvar (QUOTE parallel\/remote-shell) 1) (rpsetvar (QUOTE parallel\/path) "\/opt\/ohpc\/pub\/apps\/ansys\/2019R3_Fluent\/v195\/fluent") (rpsetvar (QUOTE parallel\/hostsfile) "\/mnt\/beegfs\/home\/testUser\/testNode.txt") )"<\/p>\n

<\/p>\n<\/p>\n

Welcome to ANSYS Fluent 2019 R3<\/p>\n

<\/p>\n<\/p>\n

Copyright 1987-2019 ANSYS, Inc. All Rights Reserved.<\/p>\n

Unauthorized use, distribution or duplication is prohibited.<\/p>\n

This product is subject to U.S. laws governing export and re-export.<\/p>\n

For full Legal Notice, see documentation.<\/p>\n

<\/p>\n<\/p>\n

Build Time: Aug 05 2019 15:40:42 EDT Build Id: 10249<\/p>\n

<\/p>\n<\/p>\n

*********************************************<\/p>\n

Info: Your license enables 4-way parallel execution.<\/p>\n

For faster simulations, please start the application with the appropriate parallel options.<\/p>\n

*********************************************<\/p>\n

<\/p>\n<\/p>\n

————————————————————–<\/p>\n

This is an academic version of ANSYS FLUENT. Usage of this product<\/p>\n

license is limited to the terms and conditions specified in your ANSYS<\/p>\n

license form, additional terms section.<\/p>\n

————————————————————–<\/p>\n

Host spawning Node 0 on machine "node007.hpc.fau.edu" (unix).<\/p>\n

\/opt\/ohpc\/pub\/apps\/ansys\/2019R3_Fluent\/v195\/fluent\/fluent19.5.0\/bin\/fluent -r19.5.0 3d -flux -node -alnamd64 -t3 -pinfiniband -mpi=openmpi -cnf=\/mnt\/beegfs\/home\/testUser\/testNode.txt -ssh -mport 10.116.1.7:10.116.1.7:43013:0<\/p>\n

Starting fixfiledes \/opt\/ohpc\/pub\/apps\/ansys\/2019R3_Fluent\/v195\/fluent\/fluent19.5.0\/multiport\/mpi\/lnamd64\/openmpi\/bin\/mpirun –mca btl self,vader,mvapi –prefix \/opt\/ohpc\/pub\/apps\/ansys\/2019R3_Fluent\/v195\/fluent\/fluent19.5.0\/multiport\/mpi\/lnamd64\/openmpi -x LD_LIBRARY_PATH -x KMP_AFFINITY=disabled -x FLUENT_ARCH=lnamd64 -x FLUENT_PROD_DIR=\/opt\/ohpc\/pub\/apps\/ansys\/2019R3_Fluent\/v195\/fluent\/fluent19.5.0 -x PYTHONHOME=\/opt\/ohpc\/pub\/apps\/ansys\/2019R3_Fluent\/v195\/fluent\/fluent19.5.0\/..\/..\/commonfiles\/CPython\/2_7_15\/linx64\/Release\/python -np 3 –hostfile \/tmp\/fluent-appfile.testUser.32486 \/opt\/ohpc\/pub\/apps\/ansys\/2019R3_Fluent\/v195\/fluent\/fluent19.5.0\/lnamd64\/3d_node\/fluent_mpi.19.5.0 node -mpiw openmpi -pic infiniband -mport 10.116.1.7:10.116.1.7:43013:0<\/p>\n

————————————————————————–<\/p>\n

Failed to create a completion queue (CQ):<\/p>\n

<\/p>\n<\/p>\n

Hostname: node007<\/p>\n

Requested CQE: 16384<\/p>\n

Error: Cannot allocate memory<\/p>\n

<\/p>\n<\/p>\n

Check the CQE attribute.<\/p>\n

————————————————————————–<\/p>\n

————————————————————————–<\/p>\n

Open MPI has detected that there are UD-capable Verbs devices on your<\/p>\n

system, but none of them were able to be setup properly. This may<\/p>\n

indicate a problem on this system.<\/p>\n

<\/p>\n<\/p>\n

You job will continue, but Open MPI will ignore the "ud" oob component<\/p>\n

in this run.<\/p>\n

<\/p>\n<\/p>\n

Hostname: node007<\/p>\n

————————————————————————–<\/p>\n

————————————————————————–<\/p>\n

Failed to create a completion queue (CQ):<\/p>\n

<\/p>\n<\/p>\n

Hostname: node082<\/p>\n

Requested CQE: 16384<\/p>\n

Error: Cannot allocate memory<\/p>\n

<\/p>\n<\/p>\n

Check the CQE attribute.<\/p>\n

————————————————————————–<\/p>\n

————————————————————————–<\/p>\n

Open MPI has detected that there are UD-capable Verbs devices on your<\/p>\n

system, but none of them were able to be setup properly. This may<\/p>\n

indicate a problem on this system.<\/p>\n

<\/p>\n<\/p>\n

You job will continue, but Open MPI will ignore the "ud" oob component<\/p>\n

in this run.<\/p>\n

<\/p>\n<\/p>\n

Hostname: node082<\/p>\n

————————————————————————–<\/p>\n

————————————————————————–<\/p>\n

Failed to create a completion queue (CQ):<\/p>\n

<\/p>\n<\/p>\n

Hostname: node081<\/p>\n

Requested CQE: 16384<\/p>\n

Error: Cannot allocate memory<\/p>\n

<\/p>\n<\/p>\n

Check the CQE attribute.<\/p>\n

————————————————————————–<\/p>\n

————————————————————————–<\/p>\n

A requested component was not found, or was unable to be opened. This<\/p>\n

means that this component is either not installed or is unable to be<\/p>\n

used on your system (e.g., sometimes this means that shared libraries<\/p>\n

that the component requires are unable to be found\/loaded). Note that<\/p>\n

Open MPI stopped checking at the first component that it did not find.<\/p>\n

<\/p>\n<\/p>\n

Host: node007.hpc.fau.edu<\/p>\n

Framework: btl<\/p>\n

Component: mvapi<\/p>\n

————————————————————————–<\/p>\n

————————————————————————–<\/p>\n

It looks like MPI_INIT failed for some reason; your parallel process is<\/p>\n

likely to abort. There are many reasons that a parallel process can<\/p>\n

fail during MPI_INIT; some of which are due to configuration or environment<\/p>\n

problems. This failure appears to be an internal failure; here's some<\/p>\n

additional information (which may only be relevant to an Open MPI<\/p>\n

developer):<\/p>\n

<\/p>\n<\/p>\n

mca_bml_base_open() failed<\/p>\n

–> Returned "Not found" (-13) instead of "Success" (0)<\/p>\n

————————————————————————–<\/p>\n

[node007:32653] *** An error occurred in MPI_Init<\/p>\n

[node007:32653] *** reported by process [4164222977,2]<\/p>\n

[node007:32653] *** on a NULL communicator<\/p>\n

[node007:32653] *** Unknown error<\/p>\n

[node007:32653] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,<\/p>\n

[node007:32653] *** and potentially your MPI job)<\/p>\n

[node007.hpc.fau.edu:32614] 3 more processes have sent help message help-oob-ud.txt \/ create-cq-failed<\/p>\n

[node007.hpc.fau.edu:32614] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help \/ error messages<\/p>\n

[node007.hpc.fau.edu:32614] 3 more processes have sent help message help-oob-ud.txt \/ no-ports-usable<\/p>\n

[node007.hpc.fau.edu:32614] 2 more processes have sent help message help-mca-base.txt \/ find-available:not-valid<\/p>\n

[node007.hpc.fau.edu:32614] 2 more processes have sent help message help-mpi-runtime.txt \/ mpi_init:startup:internal-failure<\/p>\n

[node007.hpc.fau.edu:32614] 1 more process has sent help message help-mpi-errors.txt \/ mpi_errors_are_fatal unknown handle<\/p>\n

<\/p>\n<\/p>\n

Any help would be greatly appreciated. Thanks<\/p>\n","protected":false},"template":"","class_list":["post-166634","topic","type-topic","status-closed","hentry","topic-tag-ansys-fluent","topic-tag-fluent"],"aioseo_notices":[],"acf":[],"custom_fields":[{"0":{"_btv_view_count":["2684"],"_bbp_likes_count":["0"],"_bbp_subscription":["257224"],"_bbp_topic_status":["unanswered"],"_bbp_status":["publish"],"_bbp_topic_id":["166634"],"_bbp_forum_id":["27796"],"_bbp_engagement":["157474","183088"],"_bbp_voice_count":["2"],"_bbp_reply_count":["3"],"_bbp_last_reply_id":["206270"],"_bbp_last_active_id":["206270"],"_bbp_last_active_time":["2022-04-06 14:02:20"]},"test":"skylerp"}],"_links":{"self":[{"href":"https:\/\/innovationspace.ansys.com\/forum\/wp-json\/wp\/v2\/topics\/166634","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/innovationspace.ansys.com\/forum\/wp-json\/wp\/v2\/topics"}],"about":[{"href":"https:\/\/innovationspace.ansys.com\/forum\/wp-json\/wp\/v2\/types\/topic"}],"version-history":[{"count":0,"href":"https:\/\/innovationspace.ansys.com\/forum\/wp-json\/wp\/v2\/topics\/166634\/revisions"}],"wp:attachment":[{"href":"https:\/\/innovationspace.ansys.com\/forum\/wp-json\/wp\/v2\/media?parent=166634"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}