{"id":303369,"date":"2023-08-25T14:45:13","date_gmt":"2023-08-25T14:45:13","guid":{"rendered":"\/forum\/forums\/reply\/303369\/"},"modified":"2023-08-25T14:45:13","modified_gmt":"2023-08-25T14:45:13","slug":"303369","status":"publish","type":"reply","link":"https:\/\/innovationspace.ansys.com\/forum\/forums\/reply\/303369\/","title":{"rendered":"Reply To: Distribute HFSS tasks on multiple nodes"},"content":{"rendered":"

<p>Hi Ziqi,<br><br>What is the OS version:   cat \/etc\/*release<br>What is the SLURM version:  sinfo -V<br><br>Note that AnsysEM2021R2 was the first release to officially support SLURM. There were a few bugs\/workarounds needed.  I would consider migrating to a current version AEDT.<br><br>Here are my notes regarding AEDT2021R2 on SLURM<\/strong><br><br>If you get a case with AEDT2021R2 and SLURM 20.x – run through this first.<\/p><div> <\/div><div>Important,  customer must be running AEDT 2021R2 and SLURM 20.x for the following steps<\/div><div>– You would want to gather the following to confirm scheduler version and network info<\/div><div>$ cat \/etc\/*release<\/div><div>$ sinfo -V<\/div><div>$ ifconfig<\/div><div> <\/div><div> <\/div><div>Enabling tight integration change – this requires editing slurm_srun_wrapper.sh and setting batchoption ‘RemoteSpawncommand'<\/div><div> <\/div><div>1.. (TFS447753)  edit the slurm_srun_wrapper.sh  <\/div><div>archive\/copy …\/AnsysEM21.2\/Linux64\/schedulers\/scripts\/utils\/slurm_srun_wrapper.sh to slurm_srun_wrapper.sh.ORIG<\/div><div> <\/div><div>edit …\/AnsysEM21.2\/Linux64\/schedulers\/scripts\/utils\/slurm_srun_wrapper.sh and insert the following at line 28: host=$(echo “${host}” | cut -d’.’ -f1)<\/div><div>ex:<\/div><div>if [[ -n “$ANSYSEM_SLURM_JOB_ID” ]]<\/div><div>then<\/div><div> export SLURM_JOB_ID=”${ANSYSEM_SLURM_JOB_ID}”<\/div><div> echo “set SLURM_JOB_ID=${SLURM_JOB_ID}” >> “$DEBUG_FILE”<\/div><div>fi<\/div><div>=> # tfs447753<\/div><div>=> host=$(echo “${host}” | cut -d’.’ -f1)<\/div><div>verStr=scontrol --version<\/code><\/div><div> <\/div><div> <\/div><div>2.. (TFS554680)  If SLURM version 20.0-20.10 – Generic SLURM scheduler integration does not check for SLURM minor version<\/div><div>     srun ‘–overlap’ option was introduced in SLURM VERSION 20.11<\/div><div>     slurm_srun_wrapper.sh only checks if the version is >= 20 and attempts to apply the srun ‘–overlap’ option<\/div><div>edit …\/AnsysEM21.2\/Linux64\/schedulers\/scripts\/utils\/slurm_srun_wrapper.sh and comment out the appropriate srun and if\/else\/fi lines at bottom of file<\/div><div> <\/div><div># if [ “${ver[0]}” -ge 20 ];then # SLURM ver >= 20.**.**<\/div><div>#     echo “srun –overcommit –overlap –export=ALL -n 1 -N 1 –cpu-bind=none –mem-per-cpu=0 -w ${host} $@” >> “$DEBUG_FILE”<\/div><div>#     srun –overcommit –overlap –export=ALL -n 1 -N 1 –cpu-bind=none –mem-per-cpu=0 -w $host “$@”<\/div><div># else<\/div><div>    echo “srun –overcommit –export=ALL -n 1 -N 1 –cpu-bind=none –mem-per-cpu=0 -w ${host} $@” >> “$DEBUG_FILE”<\/div><div>    srun –overcommit –export=ALL -n 1 -N 1 –cpu-bind=none –mem-per-cpu=0 -w $host “$@”<\/div><div># fi<\/div><div> <\/div><div>This should enable customer to run manual jobs for adaptive meshing, and then auto jobs for frequency sweeps. <\/div><div> <\/div><div> <\/div><div> <\/div><div>3.. After making the slurm_srun_wrapper.sh file change, run the following to set defaults to tight integration   (will create\/modify  \/<installpath>\/AnsysEM21.2\/Linux64\/config\/default.XML)<\/div><div> <\/div><div>as root or installation owner<\/div><div>cd \/opt\/AnsysEM\/AnsysEM21.2\/Linux64<\/div><div>.\/UpdateRegistry -set -ProductName ElectronicsDesktop2021.2 -RegistryKey “Desktop\/Settings\/ProjectOptions\/ProductImprovementOptStatus” -RegistryValue 0 -RegistryLevel install<\/div><div>.\/UpdateRegistry -set -ProductName ElectronicsDesktop2021.2 -RegistryKey “HFSS\/MPIVendor” -RegistryValue “Intel” -RegistryLevel install<\/div><div>.\/UpdateRegistry -set -ProductName ElectronicsDesktop2021.2 -RegistryKey “HFSS 3D Layout Design\/MPIVendor” -RegistryValue “Intel” -RegistryLevel install<\/div><div>.\/UpdateRegistry -set -ProductName ElectronicsDesktop2021.2 -RegistryKey “Maxwell 2D\/MPIVendor” -RegistryValue “Intel” -RegistryLevel install<\/div><div>.\/UpdateRegistry -set -ProductName ElectronicsDesktop2021.2 -RegistryKey “Maxwell 3D\/MPIVendor” -RegistryValue “Intel” -RegistryLevel install<\/div><div>.\/UpdateRegistry -set -ProductName ElectronicsDesktop2021.2 -RegistryKey “Q3D Extractor\/MPIVendor” -RegistryValue “Intel” -RegistryLevel install<\/div><div>.\/UpdateRegistry -set -ProductName ElectronicsDesktop2021.2 -RegistryKey “Icepak\/MPIVendor” -RegistryValue “Intel” -RegistryLevel install<\/div><div>.\/UpdateRegistry -set -ProductName ElectronicsDesktop2021.2 -RegistryKey “HFSS\/RemoteSpawnCommand” -RegistryValue “scheduler” -RegistryLevel install<\/div><div>.\/UpdateRegistry -set -ProductName ElectronicsDesktop2021.2 -RegistryKey “HFSS 3D Layout Design\/RemoteSpawnCommand” -RegistryValue “scheduler” -RegistryLevel install<\/div><div>.\/UpdateRegistry -set -ProductName ElectronicsDesktop2021.2 -RegistryKey “Maxwell 3D\/RemoteSpawnCommand” -RegistryValue “scheduler” -RegistryLevel install<\/div><div>.\/UpdateRegistry -set -ProductName ElectronicsDesktop2021.2 -RegistryKey “Maxwell 2D\/RemoteSpawnCommand” -RegistryValue “scheduler” -RegistryLevel install<\/div><div>.\/UpdateRegistry -set -ProductName ElectronicsDesktop2021.2 -RegistryKey “Q3D Extractor\/RemoteSpawnCommand” -RegistryValue “scheduler” -RegistryLevel install<\/div><div>.\/UpdateRegistry -set -ProductName ElectronicsDesktop2021.2 -RegistryKey “Icepak\/RemoteSpawnCommand” -RegistryValue “scheduler” -RegistryLevel install<\/div><div>.\/UpdateRegistry -set -ProductName ElectronicsDesktop2021.2 -RegistryKey “Desktop\/Settings\/ProjectOptions\/ProductImprovementOptStatus” -RegistryValue 0 -RegistryLevel install<\/div><div># .\/UpdateRegistry -set -ProductName ElectronicsDesktop2021.2 -RegistryKey “Desktop\/Settings\/ProjectOptions\/AnsysEMPreferredSubnetAddress” -RegistryValue “192.168.1.0\/24” -RegistryLevel install<\/div><div> <\/div><div> <\/div><div> <\/div><div>4.. An example batch script:<\/div><div> <\/div><div>Create %HOME\/anstest\/job.sh with the following contents (correct highlighted):<\/div><div> <\/div><div>#!\/bin\/bash<\/div><div>#SBATCH -N 3        # allocate 3 nodes<\/div><div>#SBATCH -n 12       # 12 tasks total<\/div><div>##SBATCH –exclusive    # no other jobs on the nodes while job is running<\/div><div>#SBATCH -J AnsysEMTest   # sensible name for the job<\/div><div> <\/div><div>#Set job folder, scratch folder, project, and design (Design is optional)<\/div><div>JobFolder=$(pwd)<\/div><div>ProjName=OptimTee-DiscreteSweep-FineMesh.aedt<\/div><div>DsnName=”TeeModel:Nominal”<\/div><div> <\/div><div># Executable path and SLURM custom integration variables<\/div><div>AppFolder=\/opt\/AnsysEM\/AnsysEM21.2\/Linux64<\/div><div> <\/div><div># setup environment and srun<\/div><div>export ANSYSEM_GENERIC_MPI_WRAPPER=${AppFolder}\/schedulers\/scripts\/utils\/slurm_srun_wrapper.sh<\/div><div>export ANSYSEM_COMMON_PREFIX=${AppFolder}\/common<\/div><div>export ANSYSEM_TASKS_PER_NODE=${SLURM_TASKS_PER_NODE}  <\/div><div> <\/div><div># setup srun<\/div><div>srun_cmd=”srun –overcommit –export=ALL -n 1 –cpu-bind=none –mem-per-cpu=0 –overlap “<\/div><div># note: srun ‘–overlap’ option was introduced in SLURM VERSION 20.11″<\/div><div> <\/div><div> <\/div><div># MPI timeout set to 30min default for cloud suggest lower to 120 or 240 seconds for onprem<\/div><div>export MPI_TIMEOUT_SECONDS=120<\/div><div> <\/div><div># System networking environment variables – HPC system dependent should not be user edits!<\/div><div># export ANSOFT_MPI_INTERCONNECT=ib<\/div><div># export ANSOFT_MPI_INTERCONNECT_VARIANT=ofed<\/div><div> <\/div><div># Skip dependency check<\/div><div># export ANS_NODEPCHECK=1<\/div><div> <\/div><div># Autocompute total cores from node allocation<\/div><div>CoreCount=$((SLURM_JOB_NUM_NODES * SLURM_CPUS_ON_NODE))<\/div><div> <\/div><div># Run Job<\/div><div>${srun_cmd} ${AppFolder}\/ansysedt -ng -monitor -waitforlicense -useelectronicsppe=1 -distributed -auto -machinelist numcores=$CoreCount -batchoptions “” -batchsolve ${DsnName} ${JobFolder}\/${ProjName}<\/div><div> <\/div><div> <\/div><div> <\/div><div>Then run it:<\/div><div>$ dos2unix $HOME\/anstest\/job.sh<\/div><div>$ chmod +x  $HOME\/anstest\/job.sh<\/div><div>$ sbatch $HOME\/anstest\/job.sh<\/div><div> <\/div><div>When complete, send resulting:<\/div><div>$HOME\/anstest\/OptimTee-DiscreteSweep-FineMesh.aedt.batchinfo\/*.log<\/div><div> <\/div><div> <\/div><div>5. If customer using Windows to Linux submission<\/div><div>Ansoftrsmservice cannot run as root.<\/div><div>– make sure to set ANSYSEM_GENERIC_EXEC_PATH before starting ansoftrsmservice<\/div><div>–  In ansoftrsmservice.cfg, the content should be:<\/div><div>$begin ‘Scheduler'<\/div><div>   ‘SchedulerName’=’generic'<\/div><div>   ‘ConfigString’='{“Proxy”:”slurm”, “Data”:””}'<\/div><div>$end ‘Scheduler'<\/div><p> <\/p><p><br><br><br><br><\/p><\/p>\n","protected":false},"template":"","class_list":["post-303369","reply","type-reply","status-publish","hentry"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t