{"id":158414,"date":"2021-05-12T05:57:48","date_gmt":"2021-05-12T05:57:48","guid":{"rendered":"\/forum\/forums\/topic\/multiple-nodes-using-fluent-on-hpc-under-slurm\/"},"modified":"2021-05-18T13:02:52","modified_gmt":"2021-05-18T13:02:52","slug":"multiple-nodes-using-fluent-on-hpc-under-slurm","status":"closed","type":"topic","link":"https:\/\/innovationspace.ansys.com\/forum\/forums\/topic\/multiple-nodes-using-fluent-on-hpc-under-slurm\/","title":{"rendered":"Multiple Nodes Using Fluent on HPC under SLURM"},"content":{"rendered":"
\n
\n

Hello <\/p>\n

I have a question regarding running fluent under slurm asking for 2 nodes and 32 core per node having in total 64. The Batch code i am submitting to SLURM is:<\/p>\n

#!\/bin\/bash<\/p>\n

#SBATCH –job-name=64cores     ## Name of the job.<\/p>\n

#SBATCH -A hetaha_lab    ## account to charge <\/p>\n

#SBATCH -p standard         ## partition\/queue name<\/p>\n

#SBATCH –nodes=2           ## (-N) number of nodes to use<\/p>\n

#SBATCH –mem=15G             ## request NGB of memory<\/p>\n

#SBATCH –ntasks-per-node=32 ## number of processes to launch per node<\/p>\n

#SBATCH –cpus-per-task=1   ## number of MPI threads<\/p>\n

#SBATCH –error=slurm-%J.err ## error log file<\/p>\n

module load ansys\/2019R2<\/p>\n

fluent 3ddp -t$SLURM_NTASKS -g -i jou_64.jou<\/p>\n

=================================<\/p>\n

When i do so fluent runs only on one machine and limits the number of cores to the max of this machine, here is what i get from fluent console<\/p>\n

======Console=====<\/p>\n

data\/opt\/apps\/ansys\/2019R2\/v194\/fluent\/fluent19.4.0\/bin\/fluent -r19.4.0 3ddp -t64 -g -i jou_64.jou<\/p>\n

\/data\/opt\/apps\/ansys\/2019R2\/v194\/fluent\/fluent19.4.0\/cortex\/lnamd64\/cortex.19.4.0 -f fluent -g -i jou_64.jou (fluent "3ddp -pshmem -host -r19.4.0 -t64 -mpi=ibmmpi -path\/data\/opt\/apps\/ansys\/2019R2\/v194\/fluent -ssh")<\/p>\n

\/data\/opt\/apps\/ansys\/2019R2\/v194\/fluent\/fluent19.4.0\/bin\/fluent -r19.4.0 3ddp -pshmem -host -t64 -mpi=ibmmpi -path\/data\/opt\/apps\/ansys\/2019R2\/v194\/fluent -ssh -cx hpc3-14-09.local:34889:44422<\/p>\n

Starting \/data\/opt\/apps\/ansys\/2019R2\/v194\/fluent\/fluent19.4.0\/lnamd64\/3ddp_host\/fluent.19.4.0 host -cx hpc3-14-09.local:34889:44422 "(list (rpsetvar (QUOTE parallel\/function) "fluent 3ddp -flux -node -r19.4.0 -t64 -pshmem -mpi=ibmmpi -ssh") (rpsetvar (QUOTE parallel\/rhost) "") (rpsetvar (QUOTE parallel\/ruser) "") (rpsetvar (QUOTE parallel\/nprocs_string) "64") (rpsetvar (QUOTE parallel\/auto-spawn?) #t) (rpsetvar (QUOTE parallel\/trace-level) 0) (rpsetvar (QUOTE parallel\/remote-shell) 1) (rpsetvar (QUOTE parallel\/path) "\/data\/opt\/apps\/ansys\/2019R2\/v194\/fluent") (rpsetvar (QUOTE parallel\/hostsfile) "") )"<\/p>\n

<\/p>\n<\/p>\n

             Welcome to ANSYS Fluent 2019 R2<\/p>\n

<\/p>\n<\/p>\n

             Copyright 1987-2019 ANSYS, Inc. All Rights Reserved.<\/p>\n

             Unauthorized use, distribution or duplication is prohibited.<\/p>\n

             This product is subject to U.S. laws governing export and re-export.<\/p>\n

             For full Legal Notice, see documentation.<\/p>\n

<\/p>\n<\/p>\n

Build Time: Apr 17 2019 13:39:08 EDT Build Id: 10133 <\/p>\n

<\/p>\n

<\/p>\n<\/p>\n

    ————————————————————–<\/p>\n

    This is an academic version of ANSYS FLUENT. Usage of this product<\/p>\n

    license is limited to the terms and conditions specified in your ANSYS<\/p>\n

    license form, additional terms section.<\/p>\n

    ————————————————————–<\/p>\n

Host spawning Node 0 on machine "hpc3-14-09" (unix).<\/p>\n

\/data\/opt\/apps\/ansys\/2019R2\/v194\/fluent\/fluent19.4.0\/bin\/fluent -r19.4.0 3ddp -flux -node -t64 -pshmem -mpi=ibmmpi -ssh -mport 10.240.58.22:10.240.58.22:36789:0<\/p>\n

Starting \/data\/opt\/apps\/ansys\/2019R2\/v194\/fluent\/fluent19.4.0\/multiport\/mpi\/lnamd64\/ibmmpi\/bin\/mpirun -e MPI_IBV_NO_FORK_SAFE=1 -e MPI_USE_MALLOPT_MMAP_MAX=0 -np 64 \/data\/opt\/apps\/ansys\/2019R2\/v194\/fluent\/fluent19.4.0\/lnamd64\/3ddp_node\/fluent_mpi.19.4.0 node -mpiw ibmmpi -pic shmem -mport 10.240.58.22:10.240.58.22:36789:0<\/p>\n

<\/p>\n<\/p>\n

——————————————————————————-<\/p>\n

ID    Hostname   Core  O.S.     PID         Vendor                    <\/p>\n

——————————————————————————-<\/p>\n

n0-63 hpc3-14-09 64\/40 Linux-64 19514-19577 Intel(R) Xeon(R) Gold 6148 <\/p>\n

host  hpc3-14-09        Linux-64 19292       Intel(R) Xeon(R) Gold 6148 <\/p>\n

<\/p>\n<\/p>\n

MPI Option Selected: ibmmpi<\/p>\n

Selected system interconnect: shared-memory<\/p>\n

——————————————————————————-<\/p>\n

<\/p>\n

<\/p>\n<\/p>\n

Reading journal file jou_64.jou…<\/p>\n

<\/p>\n<\/p>\n

====================<\/p>\n

Can some one help, how to run fluent on multiple node under SLURM.<\/p>\n","protected":false},"template":"","class_list":["post-158414","topic","type-topic","status-closed","hentry"],"aioseo_notices":[],"acf":[],"custom_fields":[{"0":{"_bbp_author_ip":[""],"_bbp_old_reply_author_name_id":["Anonymous"],"_bbp_old_is_reply_anonymous_id":["false"],"_btv_view_count":["5168"],"_bbp_likes_count":["0"],"_bbp_subscription":["249654"],"_bbp_topic_status":["unanswered"],"_bbp_status":["publish"],"_bbp_topic_id":["158414"],"_bbp_forum_id":["27796"],"_bbp_engagement":["157443","175484"],"_bbp_voice_count":["2"],"_bbp_reply_count":["1"],"_bbp_last_reply_id":["178555"],"_bbp_last_active_id":["178555"],"_bbp_last_active_time":["2021-05-18 13:02:52"]},"test":"nabilkhalifa"}],"_links":{"self":[{"href":"https:\/\/innovationspace.ansys.com\/forum\/wp-json\/wp\/v2\/topics\/158414","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/innovationspace.ansys.com\/forum\/wp-json\/wp\/v2\/topics"}],"about":[{"href":"https:\/\/innovationspace.ansys.com\/forum\/wp-json\/wp\/v2\/types\/topic"}],"version-history":[{"count":0,"href":"https:\/\/innovationspace.ansys.com\/forum\/wp-json\/wp\/v2\/topics\/158414\/revisions"}],"wp:attachment":[{"href":"https:\/\/innovationspace.ansys.com\/forum\/wp-json\/wp\/v2\/media?parent=158414"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}