{"id":347091,"date":"2024-01-29T17:26:39","date_gmt":"2024-01-29T17:26:39","guid":{"rendered":"\/forum\/forums\/topic\/ansysedt-slurm-tight-intelmpi-integration-issue\/"},"modified":"2024-02-12T22:42:02","modified_gmt":"2024-02-12T22:42:02","slug":"ansysedt-slurm-tight-intelmpi-integration-issue","status":"closed","type":"topic","link":"https:\/\/innovationspace.ansys.com\/forum\/forums\/topic\/ansysedt-slurm-tight-intelmpi-integration-issue\/","title":{"rendered":"AnsysEDT SLURM tight IntelMPI integration issue"},"content":{"rendered":"<p>Hello all,<\/p>\n<p>&nbsp;<\/p>\n<p>I have been trying to run HFSS simulations on the HPC available at my institution. AnsysEDT 2023R2 has recently been installed on Rocky Linux 8.8.<\/p>\n<p>&nbsp;<\/p>\n<p>Running jobs on an exclusive node obtained through salloc work fine. As in I can salloc &#8211;nodes=1 &#8211;exclusive, ssh in, launch AnsysEDT and run both manual configuration and auto configuration jobs with no problem after changing MPI Version to 2021. The job distributes and runs super well.<\/p>\n<p>Issues arrise when trying to utilize the SLURM integration available.<\/p>\n<p>When running a job with auto settings and auto setup, say across 2 nodes. The multiple hf3d processes launched on each node only get pinned to 1 CPU per node. This has been verified by enabling the debug options and inspecting the log_mpirun.fl_xxxxxxxxx.log file. This happens both with the version of IntelMPI bundles with AnsysEDT and the one on the cluster (changed with $INTELMPI_ROOT var). I have set the $AnsTempDir and the tempdirectory batch option to locations accessible by all nodes. I have tried various different $ANSOFT_MPI_INTERCONNECT options and $ANSOFT_MPI_INTERCONNECT_VARIANT too.<\/p>\n<p>When running a job with manual settings and the manu setup, again across 2 nodes. The pre-processing completes successfully, and in a distributed fashion, however, upon solving the first frequency for adaptive meshing the process exits with a message to contact customer support. I have tracked the issue down to be a SIGSEGV 11 from the MUMPS driver called by hf3d through the log files. Again happens with all possible variations as mentioned above.<\/p>\n<p>HPCLicenseType is pool<\/p>\n<p>tempdirectory is set to something reasonable<\/p>\n<p>HFSS\/MPIVendor intel<\/p>\n<p>HFSS\/MPIVersion 2021<\/p>\n<p>HFSS\/RemoteSpawnCommand scheduler<\/p>\n<p>&nbsp;<\/p>\n<p>Any help would be great! Thank you.<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"template":"","class_list":["post-347091","topic","type-topic","status-closed","hentry","topic-tag-intel-mpi-1","topic-tag-mpi-with-slurm","topic-tag-slurm"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 4.9.10 - aioseo.com -->\n\t<meta name=\"description\" content=\"Hello all, I have been trying to run HFSS simulations on the HPC available at my institution. AnsysEDT 2023R2 has recently been installed on Rocky Linux 8.8. Running jobs on an exclusive node obtained through salloc work fine. As in I can salloc --nodes=1 --exclusive, ssh in, launch AnsysEDT and run both manual configuration and auto configuration\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<link rel=\"canonical\" href=\"https:\/\/innovationspace.ansys.com\/forum\/forums\/topic\/ansysedt-slurm-tight-intelmpi-integration-issue\/\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 4.9.10\" \/>\n\t\t<meta property=\"og:locale\" content=\"en_US\" \/>\n\t\t<meta property=\"og:site_name\" content=\"Ansys Learning Forum | Ansys Innovation Space\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"AnsysEDT SLURM tight IntelMPI integration issue | Ansys Learning Forum\" \/>\n\t\t<meta property=\"og:description\" content=\"Hello all, I have been trying to run HFSS simulations on the HPC available at my institution. AnsysEDT 2023R2 has recently been installed on Rocky Linux 8.8. Running jobs on an exclusive node obtained through salloc work fine. As in I can salloc --nodes=1 --exclusive, ssh in, launch AnsysEDT and run both manual configuration and auto configuration\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/innovationspace.ansys.com\/forum\/forums\/topic\/ansysedt-slurm-tight-intelmpi-integration-issue\/\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2024-01-29T17:26:39+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2024-02-12T22:42:02+00:00\" \/>\n\t\t<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n\t\t<meta name=\"twitter:title\" content=\"AnsysEDT SLURM tight IntelMPI integration issue | Ansys Learning Forum\" \/>\n\t\t<meta name=\"twitter:description\" content=\"Hello all, I have been trying to run HFSS simulations on the HPC available at my institution. AnsysEDT 2023R2 has recently been installed on Rocky Linux 8.8. Running jobs on an exclusive node obtained through salloc work fine. As in I can salloc --nodes=1 --exclusive, ssh in, launch AnsysEDT and run both manual configuration and auto configuration\" \/>\n\t\t<script type=\"application\/ld+json\" class=\"aioseo-schema\">\n\t\t\t{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/innovationspace.ansys.com\\\/forum\\\/forums\\\/topic\\\/ansysedt-slurm-tight-intelmpi-integration-issue\\\/#breadcrumblist\",\"itemListElement\":[{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/innovationspace.ansys.com\\\/forum#listItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/innovationspace.ansys.com\\\/forum\",\"nextItem\":{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/innovationspace.ansys.com\\\/forum\\\/topics\\\/#listItem\",\"name\":\"Topics\"}},{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/innovationspace.ansys.com\\\/forum\\\/topics\\\/#listItem\",\"position\":2,\"name\":\"Topics\",\"item\":\"https:\\\/\\\/innovationspace.ansys.com\\\/forum\\\/topics\\\/\",\"nextItem\":{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/innovationspace.ansys.com\\\/forum\\\/forums\\\/topic-tag\\\/mpi-with-slurm\\\/#listItem\",\"name\":\"mpi-with-slurm\"},\"previousItem\":{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/innovationspace.ansys.com\\\/forum#listItem\",\"name\":\"Home\"}},{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/innovationspace.ansys.com\\\/forum\\\/forums\\\/topic-tag\\\/mpi-with-slurm\\\/#listItem\",\"position\":3,\"name\":\"mpi-with-slurm\",\"item\":\"https:\\\/\\\/innovationspace.ansys.com\\\/forum\\\/forums\\\/topic-tag\\\/mpi-with-slurm\\\/\",\"nextItem\":{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/innovationspace.ansys.com\\\/forum\\\/forums\\\/topic\\\/ansysedt-slurm-tight-intelmpi-integration-issue\\\/#listItem\",\"name\":\"AnsysEDT SLURM tight IntelMPI integration issue\"},\"previousItem\":{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/innovationspace.ansys.com\\\/forum\\\/topics\\\/#listItem\",\"name\":\"Topics\"}},{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/innovationspace.ansys.com\\\/forum\\\/forums\\\/topic\\\/ansysedt-slurm-tight-intelmpi-integration-issue\\\/#listItem\",\"position\":4,\"name\":\"AnsysEDT SLURM tight IntelMPI integration issue\",\"previousItem\":{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/innovationspace.ansys.com\\\/forum\\\/forums\\\/topic-tag\\\/mpi-with-slurm\\\/#listItem\",\"name\":\"mpi-with-slurm\"}}]},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/innovationspace.ansys.com\\\/forum\\\/#organization\",\"name\":\"Ansys Learning Forum\",\"description\":\"Ansys Innovation Space\",\"url\":\"https:\\\/\\\/innovationspace.ansys.com\\\/forum\\\/\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/innovationspace.ansys.com\\\/forum\\\/forums\\\/topic\\\/ansysedt-slurm-tight-intelmpi-integration-issue\\\/#webpage\",\"url\":\"https:\\\/\\\/innovationspace.ansys.com\\\/forum\\\/forums\\\/topic\\\/ansysedt-slurm-tight-intelmpi-integration-issue\\\/\",\"name\":\"AnsysEDT SLURM tight IntelMPI integration issue | Ansys Learning Forum\",\"description\":\"Hello all, I have been trying to run HFSS simulations on the HPC available at my institution. AnsysEDT 2023R2 has recently been installed on Rocky Linux 8.8. Running jobs on an exclusive node obtained through salloc work fine. As in I can salloc --nodes=1 --exclusive, ssh in, launch AnsysEDT and run both manual configuration and auto configuration\",\"inLanguage\":\"en-US\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/innovationspace.ansys.com\\\/forum\\\/#website\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/innovationspace.ansys.com\\\/forum\\\/forums\\\/topic\\\/ansysedt-slurm-tight-intelmpi-integration-issue\\\/#breadcrumblist\"},\"datePublished\":\"2024-01-29T17:26:39+00:00\",\"dateModified\":\"2024-02-12T22:42:02+00:00\"},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/innovationspace.ansys.com\\\/forum\\\/#website\",\"url\":\"https:\\\/\\\/innovationspace.ansys.com\\\/forum\\\/\",\"name\":\"Ansys Learning Forum\",\"description\":\"Ansys Innovation Space\",\"inLanguage\":\"en-US\",\"publisher\":{\"@id\":\"https:\\\/\\\/innovationspace.ansys.com\\\/forum\\\/#organization\"}}]}\n\t\t<\/script>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"AnsysEDT SLURM tight IntelMPI integration issue | Ansys Learning Forum","description":"Hello all, I have been trying to run HFSS simulations on the HPC available at my institution. AnsysEDT 2023R2 has recently been installed on Rocky Linux 8.8. Running jobs on an exclusive node obtained through salloc work fine. As in I can salloc --nodes=1 --exclusive, ssh in, launch AnsysEDT and run both manual configuration and auto configuration","canonical_url":"https:\/\/innovationspace.ansys.com\/forum\/forums\/topic\/ansysedt-slurm-tight-intelmpi-integration-issue\/","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"BreadcrumbList","@id":"https:\/\/innovationspace.ansys.com\/forum\/forums\/topic\/ansysedt-slurm-tight-intelmpi-integration-issue\/#breadcrumblist","itemListElement":[{"@type":"ListItem","@id":"https:\/\/innovationspace.ansys.com\/forum#listItem","position":1,"name":"Home","item":"https:\/\/innovationspace.ansys.com\/forum","nextItem":{"@type":"ListItem","@id":"https:\/\/innovationspace.ansys.com\/forum\/topics\/#listItem","name":"Topics"}},{"@type":"ListItem","@id":"https:\/\/innovationspace.ansys.com\/forum\/topics\/#listItem","position":2,"name":"Topics","item":"https:\/\/innovationspace.ansys.com\/forum\/topics\/","nextItem":{"@type":"ListItem","@id":"https:\/\/innovationspace.ansys.com\/forum\/forums\/topic-tag\/mpi-with-slurm\/#listItem","name":"mpi-with-slurm"},"previousItem":{"@type":"ListItem","@id":"https:\/\/innovationspace.ansys.com\/forum#listItem","name":"Home"}},{"@type":"ListItem","@id":"https:\/\/innovationspace.ansys.com\/forum\/forums\/topic-tag\/mpi-with-slurm\/#listItem","position":3,"name":"mpi-with-slurm","item":"https:\/\/innovationspace.ansys.com\/forum\/forums\/topic-tag\/mpi-with-slurm\/","nextItem":{"@type":"ListItem","@id":"https:\/\/innovationspace.ansys.com\/forum\/forums\/topic\/ansysedt-slurm-tight-intelmpi-integration-issue\/#listItem","name":"AnsysEDT SLURM tight IntelMPI integration issue"},"previousItem":{"@type":"ListItem","@id":"https:\/\/innovationspace.ansys.com\/forum\/topics\/#listItem","name":"Topics"}},{"@type":"ListItem","@id":"https:\/\/innovationspace.ansys.com\/forum\/forums\/topic\/ansysedt-slurm-tight-intelmpi-integration-issue\/#listItem","position":4,"name":"AnsysEDT SLURM tight IntelMPI integration issue","previousItem":{"@type":"ListItem","@id":"https:\/\/innovationspace.ansys.com\/forum\/forums\/topic-tag\/mpi-with-slurm\/#listItem","name":"mpi-with-slurm"}}]},{"@type":"Organization","@id":"https:\/\/innovationspace.ansys.com\/forum\/#organization","name":"Ansys Learning Forum","description":"Ansys Innovation Space","url":"https:\/\/innovationspace.ansys.com\/forum\/"},{"@type":"WebPage","@id":"https:\/\/innovationspace.ansys.com\/forum\/forums\/topic\/ansysedt-slurm-tight-intelmpi-integration-issue\/#webpage","url":"https:\/\/innovationspace.ansys.com\/forum\/forums\/topic\/ansysedt-slurm-tight-intelmpi-integration-issue\/","name":"AnsysEDT SLURM tight IntelMPI integration issue | Ansys Learning Forum","description":"Hello all, I have been trying to run HFSS simulations on the HPC available at my institution. AnsysEDT 2023R2 has recently been installed on Rocky Linux 8.8. Running jobs on an exclusive node obtained through salloc work fine. As in I can salloc --nodes=1 --exclusive, ssh in, launch AnsysEDT and run both manual configuration and auto configuration","inLanguage":"en-US","isPartOf":{"@id":"https:\/\/innovationspace.ansys.com\/forum\/#website"},"breadcrumb":{"@id":"https:\/\/innovationspace.ansys.com\/forum\/forums\/topic\/ansysedt-slurm-tight-intelmpi-integration-issue\/#breadcrumblist"},"datePublished":"2024-01-29T17:26:39+00:00","dateModified":"2024-02-12T22:42:02+00:00"},{"@type":"WebSite","@id":"https:\/\/innovationspace.ansys.com\/forum\/#website","url":"https:\/\/innovationspace.ansys.com\/forum\/","name":"Ansys Learning Forum","description":"Ansys Innovation Space","inLanguage":"en-US","publisher":{"@id":"https:\/\/innovationspace.ansys.com\/forum\/#organization"}}]},"og:locale":"en_US","og:site_name":"Ansys Learning Forum | Ansys Innovation Space","og:type":"article","og:title":"AnsysEDT SLURM tight IntelMPI integration issue | Ansys Learning Forum","og:description":"Hello all, I have been trying to run HFSS simulations on the HPC available at my institution. AnsysEDT 2023R2 has recently been installed on Rocky Linux 8.8. Running jobs on an exclusive node obtained through salloc work fine. As in I can salloc --nodes=1 --exclusive, ssh in, launch AnsysEDT and run both manual configuration and auto configuration","og:url":"https:\/\/innovationspace.ansys.com\/forum\/forums\/topic\/ansysedt-slurm-tight-intelmpi-integration-issue\/","article:published_time":"2024-01-29T17:26:39+00:00","article:modified_time":"2024-02-12T22:42:02+00:00","twitter:card":"summary_large_image","twitter:title":"AnsysEDT SLURM tight IntelMPI integration issue | Ansys Learning Forum","twitter:description":"Hello all, I have been trying to run HFSS simulations on the HPC available at my institution. AnsysEDT 2023R2 has recently been installed on Rocky Linux 8.8. Running jobs on an exclusive node obtained through salloc work fine. As in I can salloc --nodes=1 --exclusive, ssh in, launch AnsysEDT and run both manual configuration and auto configuration"},"aioseo_meta_data":{"post_id":"347091","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"content","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":true,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":"default","schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"limit_modified_date":false,"created":"2024-10-28 17:02:40","updated":"2024-10-28 17:02:40","ai":null,"breadcrumb_settings":null,"seo_analyzer_scan_date":null},"aioseo_breadcrumb":"<div class=\"aioseo-breadcrumbs\"><span class=\"aioseo-breadcrumb\">\n\t\t\t<a href=\"https:\/\/innovationspace.ansys.com\/forum\" title=\"Home\">Home<\/a>\n\t\t<\/span><span class=\"aioseo-breadcrumb-separator\">&raquo;<\/span><span class=\"aioseo-breadcrumb\">\n\t\t\t<a href=\"https:\/\/innovationspace.ansys.com\/forum\/topics\/\" title=\"Topics\">Topics<\/a>\n\t\t<\/span><span class=\"aioseo-breadcrumb-separator\">&raquo;<\/span><span class=\"aioseo-breadcrumb\">\n\t\t\t<a href=\"https:\/\/innovationspace.ansys.com\/forum\/forums\/topic-tag\/mpi-with-slurm\/\" title=\"mpi-with-slurm\">mpi-with-slurm<\/a>\n\t\t<\/span><span class=\"aioseo-breadcrumb-separator\">&raquo;<\/span><span class=\"aioseo-breadcrumb\">\n\t\t\tAnsysEDT SLURM tight IntelMPI integration issue\n\t\t<\/span><\/div>","aioseo_breadcrumb_json":[{"label":"Home","link":"https:\/\/innovationspace.ansys.com\/forum"},{"label":"Topics","link":"https:\/\/innovationspace.ansys.com\/forum\/topics\/"},{"label":"mpi-with-slurm","link":"https:\/\/innovationspace.ansys.com\/forum\/forums\/topic-tag\/mpi-with-slurm\/"},{"label":"AnsysEDT SLURM tight IntelMPI integration issue","link":"https:\/\/innovationspace.ansys.com\/forum\/forums\/topic\/ansysedt-slurm-tight-intelmpi-integration-issue\/"}],"acf":[],"custom_fields":[{"0":{"_bbp_subscription":["213550","2937"],"_bbp_author_ip":["23.206.193.59"]," _bbp_last_reply_id":["0"]," _bbp_likes_count":["0"],"_btv_view_count":["1016"],"_bbp_topic_status":["answered"],"_edit_lock":["1706626134:109055"],"_bbp_topic_id":["347091"],"_bbp_forum_id":["27793"],"_bbp_engagement":["2937","213550"],"_bbp_voice_count":["2"],"_bbp_reply_count":["3"],"_bbp_last_reply_id":["349404"],"_bbp_last_active_id":["349404"],"_bbp_last_active_time":["2024-02-06 15:28:31"]},"test":"krrs87durham-ac-uk"}],"_links":{"self":[{"href":"https:\/\/innovationspace.ansys.com\/forum\/wp-json\/wp\/v2\/topics\/347091","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/innovationspace.ansys.com\/forum\/wp-json\/wp\/v2\/topics"}],"about":[{"href":"https:\/\/innovationspace.ansys.com\/forum\/wp-json\/wp\/v2\/types\/topic"}],"version-history":[{"count":1,"href":"https:\/\/innovationspace.ansys.com\/forum\/wp-json\/wp\/v2\/topics\/347091\/revisions"}],"predecessor-version":[{"id":350913,"href":"https:\/\/innovationspace.ansys.com\/forum\/wp-json\/wp\/v2\/topics\/347091\/revisions\/350913"}],"wp:attachment":[{"href":"https:\/\/innovationspace.ansys.com\/forum\/wp-json\/wp\/v2\/media?parent=347091"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}