{"id":369236,"date":"2024-05-30T14:35:06","date_gmt":"2024-05-30T14:35:06","guid":{"rendered":"\/forum\/forums\/topic\/slurm-integration\/"},"modified":"2024-05-31T13:49:07","modified_gmt":"2024-05-31T13:49:07","slug":"slurm-integration","status":"closed","type":"topic","link":"https:\/\/innovationspace.ansys.com\/forum\/forums\/topic\/slurm-integration\/","title":{"rendered":"SLURM integration"},"content":{"rendered":"

We have not been able to get slurm integration working with electronics desktop components, e.g. hfss on multiple node jobs (preferred) or with RSM (previously worked for us) with 24.1<\/p>\n

We have modified the ansoftrsmservice.cfg to contain<\/p>\n

        $begin 'Scheduler'
                'SchedulerName'='generic'
                'ConfigString'='{\"Proxy\":\"slurm\"}'
        $end 'Scheduler'<\/pre>\n
and set multiple relevant environment variables, e.g. <\/p>\n
ANSYS_HOME=\/hpc\/m3\/apps\/ansys\/24R1\/AnsysEM\/v241\/Linux64
export PATH=${ANSYS_HOME}:$PATH
export ANSOFT_DEBUG_LOG=$HOME\/ansys_debug.log
export ANSYSEM_ROOT241=${ANSYS_HOME}
export ANSYSEM_COMMON_PREFIX=${ANSYS_HOME}\/common
export ANSYSEM_TASKS_PER_NODE=$( echo ${SLURM_TASKS_PER_NODE} | cut -f 1 -d \\()
export ANSYS_EM_EXEC_DIR=${ANSYS_HOME}
export ANSYSEM_GENERIC_MPI_WRAPPER=${ANSYS_HOME}\/schedulers\/scripts\/utils\/slurm_srun_wrapper.sh
export ANS_NODEPCHECK=1
export ANSYSEM_GENERIC_EXEC_PATH=${ANSYS_HOME}\/common\/mono\/Linux64\/bin:${ANSYS_HOME}\/common\/IronPython
export ANSYS_EM_GENERIC_COMMON_TEMP=\/tmp<\/pre>\n
Our batch options look like:<\/p>\n
$begin 'Config'
'HFSS\/NumCoresPerDistributedTask'=4
'HFSS 3D Layout Design\/NumCoresPerDistributedTask'=4
'HFSS\/HPCLicenseType'='Pool'
'HPCLicenseType'='Pool'
'HFSS\/SolveAdaptiveOnly'=0
'HFSS\/MPIVendor'='Intel'
'HFSS 3D Layout Design\/MPIVendor'='Intel'
'Maxwell 2D\/MPIVendor'='Intel'
'Maxwell 3D\/MPIVendor'='Intel'
'Q3D Extractor\/MPIVendor'='Intel'
'Icepak\/MPIVendor'='Intel'
'HFSS\/RemoteSpawnCommand'='scheduler'
'HFSS 3D Layout Design\/RemoteSpawnCommand'='scheduler'
'Maxwell 3D\/RemoteSpawnCommand'='scheduler'
'Maxwell 2D\/RemoteSpawnCommand'='scheduler'
'Q3D Extractor\/RemoteSpawnCommand'='scheduler'
'Icepak\/RemoteSpawnCommand'='scheduler'
'Desktop\/Settings\/ProjectOptions\/ProductImprovementOptStatus'='0'
'Desktop\/Settings\/ProjectOptions\/AnsysEMPreferredSubnetAddress'='10.215.24.0\/21'
$end 'Config'<\/pre>\n
It appears that the slurm_srun_wrapper<\/em> is not setting the host(s) correctly, for example, this is from a log generated:<\/p>\n
(02:14:48 PM May 24, 2024) Command = \/hpc\/m3\/apps\/ansys\/24R1\/AnsysEM\/v241\/Linux64\/schedulers\/scripts\/utils\/slurm_srun_wrapper.sh
(02:14:48 PM May 24, 2024) All args = --nodelist c001.infiniband.cluster,c002.infiniband.cluster -N 2 -n 2 --input none --external-launcher \/hpc\/m3\/apps\/ansys\/24R1\/AnsysEM\/v241\/Linux64\/common\/fluent_mpi\/multiport\/mpi\/lnamd64\/intel\/bin\/pmi_proxy --control-port c001.cm.cluster:36271 --pmi-connect alltoall --pmi-aggregate -s 0 --rmk user --launcher slurm --launcher-exec \/hpc\/m3\/apps\/ansys\/24R1\/AnsysEM\/v241\/Linux64\/schedulers\/scripts\/utils\/slurm_srun_wrapper.sh --launcher-exec-args --external-launcher --demux poll --pgid 0 --enable-stdin 1 --retries 10 --control-code 1142573241 --usize -2 --proxy-id -1<\/pre>\n
and it sets host=$3<\/em>, which is “-N”. We assume that something is not being passed correctly, but the end result is that this generates a malformed srun command that kills our slurm daemons. Changing it to host=$2 in this case seems to help as it actually gets a hostname, but it doesn’t appear to communicate between nodes.<\/p>\n
We’ve also been unable to get a multinode job to work with RSM like we’ve been able previously (We did something very similar to https:\/\/www.chpc.utah.edu\/documentation\/software\/scripts\/run_ansysedt.slr). Note, we also see “errors” in the RSM logs saying we have multiple network interfaces (which is true) and to see the docs on how to set the appropriate network interface. We have been unable to find that documentation for the RSM that is bundled with the Electronics Desktop — we can only find the documation for the full RSM which says to set it with rsmutil, but that does not exist in the Electronics Desktop installation (e.g. https:\/\/ansyshelp.ansys.com\/account\/secured?returnurl=\/Views\/Secured\/corp\/v241\/en\/wb_rsm\/wb_rsm_mult_nic.html)  <\/p>\n","protected":false},"template":"","class_list":["post-369236","topic","type-topic","status-closed","hentry","topic-tag-ansys-hfss","topic-tag-batch-hpc","topic-tag-hpc-cluster-1","topic-tag-mpi-with-slurm","topic-tag-multiple-nodes-1","topic-tag-slurm"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t