{"id":432685,"date":"2025-05-28T00:44:08","date_gmt":"2025-05-28T00:44:08","guid":{"rendered":"https:\/\/innovationspace.ansys.com\/forum\/forums\/topic\/inverse-design-with-fdtd-and-python-api-in-a-headless-linux-hpc-system\/"},"modified":"2025-05-28T00:44:08","modified_gmt":"2025-05-28T00:44:08","slug":"inverse-design-with-fdtd-and-python-api-in-a-headless-linux-hpc-system","status":"publish","type":"topic","link":"https:\/\/innovationspace.ansys.com\/forum\/forums\/topic\/inverse-design-with-fdtd-and-python-api-in-a-headless-linux-hpc-system\/","title":{"rendered":"Inverse design with FDTD and Python API in a headless Linux HPC system"},"content":{"rendered":"<p>&lt;p&gt;Hello. I am doing inverse design with FDTD and the Python API. On my local machine I use the GUI to launch the python script SWDM.py and everything runs fine, albeit very slowly. I would like to speed up the optimization process by running it on a high-performance computing (HPC) system. I have Lumerical v242 installed on a Rocky Linux HPC system (without a GUI) called Lonestar6. I am able to run a single simulation file SWDM.fsp utilizing the Slurm job script SWDM_one_sim.txt.&lt;\/p&gt;&lt;p&gt;Now, I have tried to create a Slurm job script SWDM_optimize.txt which calls out the Python script SWDM.py. However, when I try to run it, it seems that some of the settings declared in the job script are not recognized during execution of the Python script. As I am new to HPC, surely I am missing something. I would like to know if you have any ideas or suggestions to make this work? The files SWDM_one_sim.txt, SWDM_optimize.txt, SWDM.py, and SWDM_error.log are provided below. Thanks.&lt;\/p&gt;&lt;p&gt;&lt;br&gt;<strong>SWDM_one_sim.txt<\/strong>&lt;\/p&gt;&lt;p&gt;#!\/bin\/bash&lt;br&gt;#SBATCH -J SWDM &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;# Job name&lt;br&gt;#SBATCH -o SWDM_out.log &nbsp; &nbsp; &nbsp;# Standard output log&lt;br&gt;#SBATCH -e SWDM_err.log &nbsp; &nbsp; &nbsp;# Standard error log&lt;br&gt;#SBATCH -p normal &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;# Queue\/partition name&lt;br&gt;#SBATCH -N 1 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; # Number of nodes&lt;br&gt;#SBATCH -n 1 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; # Total number of tasks&lt;br&gt;#SBATCH &#8211;cpus-per-task=8 &nbsp; &nbsp;# Number of cores\/threads (should match -t flag)&lt;br&gt;#SBATCH &#8211;mem=1024 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; # Memory (in MB)&lt;br&gt;#SBATCH -t 00:10:00 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;# Runtime (hh:mm:ss)&lt;br&gt;#SBATCH -A PHY25016 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;# Lonestar6 allocation&lt;\/p&gt;&lt;p&gt;# Load modules&lt;br&gt;module reset&lt;br&gt;module load intel\/19.1.1&lt;br&gt;module load impi\/19.0.9&lt;\/p&gt;&lt;p&gt;# Set path to libglut.so.3 (from user gda)&lt;br&gt;export LD_LIBRARY_PATH=\/home1\/01249\/gda\/test\/glut\/usr\/lib64:$LD_LIBRARY_PATH&lt;\/p&gt;&lt;p&gt;# Set Ansys licensing environment variable&lt;br&gt;export ANSYSLMD_LICENSE_FILE=XXXX&lt;\/p&gt;&lt;p&gt;# Set Ansys Lumerical API and bundled Python3 path&lt;br&gt;export PATH=\/work\/10565\/jmidkiff\/opt\/lumerical\/v242\/bin:\/work\/10565\/jmidkiff\/opt\/lumerical\/v242\/python\/bin:$PATH&lt;br&gt;export PYTHONPATH=\/work\/10565\/jmidkiff\/opt\/lumerical\/v242\/api\/python:$PYTHONPATH&lt;\/p&gt;&lt;p&gt;set -e&lt;\/p&gt;&lt;p&gt;# Run a single FDTD simulation&lt;br&gt;fdtd-engine-impi-lcl -t 8 SWDM.fsp&lt;\/p&gt;&lt;p&gt;&nbsp;&lt;\/p&gt;&lt;p&gt;&lt;br&gt;<strong>SWDM_optimize.txt<\/strong>&lt;\/p&gt;&lt;p&gt;#!\/bin\/bash&lt;br&gt;#SBATCH -J SWDM &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;# Job name&lt;br&gt;#SBATCH -o SWDM_out.log &nbsp; &nbsp; &nbsp;# Standard output log&lt;br&gt;#SBATCH -e SWDM_err.log &nbsp; &nbsp; &nbsp;# Standard error log&lt;br&gt;#SBATCH -p normal &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;# Queue\/partition name&lt;br&gt;#SBATCH -N 1 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; # Number of nodes&lt;br&gt;#SBATCH -n 4 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; # Total number of tasks&lt;br&gt;#SBATCH &#8211;cpus-per-task=8 &nbsp; &nbsp;# Number of cores\/threads (should match -t flag)&lt;br&gt;#SBATCH &#8211;mem=1024 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; # Memory (in MB)&lt;br&gt;#SBATCH -t 00:10:00 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;# Runtime (hh:mm:ss)&lt;br&gt;#SBATCH -A PHY25016 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;# Lonestar6 allocation&lt;\/p&gt;&lt;p&gt;# Load any necessary modules&lt;br&gt;module reset&lt;br&gt;module load intel\/19.1.1&lt;br&gt;module load impi\/19.0.9&lt;\/p&gt;&lt;p&gt;# Set path to libglut.so.3 (from user gda)&lt;br&gt;export LD_LIBRARY_PATH=\/home1\/01249\/gda\/test\/glut\/usr\/lib64:$LD_LIBRARY_PATH&lt;\/p&gt;&lt;p&gt;# Set Ansys licensing environment variable&lt;br&gt;export ANSYSLMD_LICENSE_FILE=XXXX&lt;\/p&gt;&lt;p&gt;# Set Ansys Lumerical API and bundled Python3 path&lt;br&gt;export PATH=\/work\/10565\/jmidkiff\/opt\/lumerical\/v242\/bin:\/work\/10565\/jmidkiff\/opt\/lumerical\/v242\/python\/bin:$PATH&lt;br&gt;export PYTHONPATH=\/work\/10565\/jmidkiff\/opt\/lumerical\/v242\/api\/python:$PYTHONPATH&lt;\/p&gt;&lt;p&gt;# Set hidden CAD (Required for Lumerical script, Python API, LumOpt)&lt;br&gt;export QT_QPA_PLATFORM=offscreen&lt;\/p&gt;&lt;p&gt;set -e&lt;\/p&gt;&lt;p&gt;# Run Python optimization script&lt;br&gt;python3 SWDM.py&lt;\/p&gt;&lt;p&gt;&nbsp;&lt;\/p&gt;&lt;p&gt;&lt;br&gt;<strong>SWDM.py<\/strong>&lt;\/p&gt;&lt;p&gt;######## IMPORTS ########&lt;br&gt;# General purpose imports&lt;br&gt;import os&lt;br&gt;import sys&lt;br&gt;import numpy as np&lt;\/p&gt;&lt;p&gt;# Optimization specific imports&lt;br&gt;from lumopt import CONFIG&lt;br&gt;from lumopt.geometries.topology import TopologyOptimization2D&lt;br&gt;from lumopt.utilities.load_lumerical_scripts import load_from_lsf&lt;br&gt;from lumopt.figures_of_merit.modematch import ModeMatch&lt;br&gt;from lumopt.optimization import Optimization&lt;br&gt;from lumopt.optimizers.generic_optimizers import ScipyOptimizers&lt;br&gt;from lumopt.utilities.wavelengths import Wavelengths&lt;\/p&gt;&lt;p&gt;######## DEFINE BASE SIMULATION ########&lt;\/p&gt;&lt;p&gt;def runSim(params, eps_bg, eps_wg, x_pos, y_pos, size_x, size_y, filter_R, working_dir, beta):&lt;\/p&gt;&lt;p&gt;&nbsp; &nbsp; ######## DEFINE GEOMETRY ########&lt;br&gt;&nbsp; &nbsp; geometry = TopologyOptimization2D(params=params, eps_min=eps_bg, eps_max=eps_wg, x=x_pos, y=y_pos, z=0, filter_R=filter_R, beta=beta)&lt;br&gt;&nbsp;&lt;br&gt;&nbsp; &nbsp; ######## DEFINE FIGURE OF MERIT FOR EACH OUTPUT WAVEGUIDE ########&lt;br&gt;&nbsp; &nbsp; fom1 = ModeMatch(monitor_name = &#8216;fom_1&#8217;, mode_number = &#8216;Fundamental TE mode&#8217;, direction = &#8216;Forward&#8217;, norm_p = 2, target_fom=1)&lt;br&gt;&nbsp; &nbsp; fom2 = ModeMatch(monitor_name = &#8216;fom_2&#8217;, mode_number = &#8216;Fundamental TE mode&#8217;, direction = &#8216;Forward&#8217;, norm_p = 2, target_fom=1)&lt;\/p&gt;&lt;p&gt;&nbsp; &nbsp; ######## DEFINE OPTIMIZATION ALGORITHM ########&lt;br&gt;&nbsp; &nbsp; optimizer = ScipyOptimizers(max_iter=400, method=&#8217;L-BFGS-B&#8217;, pgtol=1e-6, ftol=1e-5, scale_initial_gradient_to=0.25)&lt;\/p&gt;&lt;p&gt;&nbsp; &nbsp; ######## DEFINE SETUP SCRIPT AND INDIVIDUAL OPTIMIZERS ########&lt;br&gt;&nbsp; &nbsp; script = load_from_lsf(&#8216;SWDM.lsf&#8217;)&lt;\/p&gt;&lt;p&gt;&nbsp; &nbsp; wavelengths1 = Wavelengths(start = 834e-9, stop = 848e-9, points = 6)&lt;br&gt;&nbsp; &nbsp; opt1 = Optimization(base_script=script, wavelengths = wavelengths1, fom=fom1, geometry=geometry, optimizer=optimizer, use_deps=False, hide_fdtd_cad=True, plot_history=False, store_all_simulations=False, save_global_index=False)&lt;br&gt;&nbsp; &nbsp; wavelengths2 = Wavelengths(start = 860e-9, stop = 938e-9, points = 7)&lt;br&gt;&nbsp; &nbsp; opt2 = Optimization(base_script=script, wavelengths = wavelengths2, fom=fom2, geometry=geometry, optimizer=optimizer, use_deps=False, hide_fdtd_cad=True, plot_history=False, store_all_simulations=False, save_global_index=False)&lt;\/p&gt;&lt;p&gt;&nbsp; &nbsp; ######## PUT EVERYTHING TOGETHER AND RUN ########&lt;br&gt;&nbsp; &nbsp; opt = opt1+opt2&lt;br&gt;&nbsp; &nbsp; opt.continuation_max_iter = 30 #&lt; How many iterations per binarization step (default is 20)&lt;br&gt;&nbsp; &nbsp; opt.run(working_dir = working_dir)&lt;\/p&gt;&lt;p&gt;if __name__ == &#8216;__main__&#8217;:&lt;br&gt;&nbsp; &nbsp; size_x = 9000&lt;br&gt;&nbsp; &nbsp; size_y = 4000&lt;br&gt;&nbsp; &nbsp; filter_R = 200e-9&lt;\/p&gt;&lt;p&gt;&nbsp; &nbsp; x_points=int(size_x\/20)+1&lt;br&gt;&nbsp; &nbsp; y_points=int(size_y\/20)+1&lt;br&gt;&nbsp; &nbsp; eps_wg = 1.74**2&lt;br&gt;&nbsp; &nbsp; eps_bg = 1.45**2&lt;br&gt;&nbsp; &nbsp; x_pos = np.linspace(-size_x\/2*1e-9,size_x\/2*1e-9,x_points)&lt;br&gt;&nbsp; &nbsp; y_pos = np.linspace(-size_y\/2*1e-9,size_y\/2*1e-9,y_points)&lt;\/p&gt;&lt;p&gt;&nbsp; &nbsp; ## Initial condition:&lt;br&gt;&nbsp; &nbsp; params = 0.5*np.ones((x_points,y_points)) &nbsp; &nbsp; #&lt; Start with the domain filled with (eps_wg+eps_bg)\/2&lt;br&gt;&nbsp; &nbsp; beta = 1&lt;\/p&gt;&lt;p&gt;&nbsp; &nbsp; working_dir = &#8216;SWDM_x{:04d}_y{:04d}_f{:04d}&#8217;.format(size_x,size_y,int(filter_R*1e9))&lt;br&gt;&nbsp; &nbsp; runSim(params, eps_bg, eps_wg, x_pos, y_pos, size_x*1e-9, size_y*1e-9, filter_R, working_dir=working_dir, beta=beta)&lt;\/p&gt;&lt;p&gt;&nbsp;&lt;\/p&gt;&lt;p&gt;&lt;br&gt;<strong>SWDM_error.log<\/strong>&lt;\/p&gt;&lt;p&gt;Traceback (most recent call last):&lt;br&gt;&nbsp; File &#8220;\/work\/10565\/jmidkiff\/SWDM.py&#8221;, line 78, in &lt;module&gt;&lt;br&gt;&nbsp; &nbsp; runSim(params, eps_bg, eps_wg, x_pos, y_pos, size_x*1e-9, size_y*1e-9, filter_R, working_dir=working_dir, beta=beta)&lt;br&gt;&nbsp; File &#8220;\/work\/10565\/jmidkiff\/SWDM.py&#8221;, line 50, in runSim&lt;br&gt;&nbsp; &nbsp; opt.run(working_dir = working_dir)&lt;br&gt;&nbsp; File &#8220;\/work\/10565\/jmidkiff\/opt\/lumerical\/v242\/api\/python\/lumopt\/optimization.py&#8221;, line 471, in run&lt;br&gt;&nbsp; &nbsp; self.initialize(working_dir=working_dir)&lt;br&gt;&nbsp; File &#8220;\/work\/10565\/jmidkiff\/opt\/lumerical\/v242\/api\/python\/lumopt\/optimization.py&#8221;, line 119, in initialize&lt;br&gt;&nbsp; &nbsp; self.one_forward = check_one_forward_sim(self.optimizations[0])&lt;br&gt;&nbsp; File &#8220;\/work\/10565\/jmidkiff\/opt\/lumerical\/v242\/api\/python\/lumopt\/optimization.py&#8221;, line 82, in check_one_forward_sim&lt;br&gt;&nbsp; &nbsp; co_opt.sim = Simulation(self.workingDir, co_opt.use_var_fdtd, co_opt.hide_fdtd_cad)&lt;br&gt;&nbsp; File &#8220;\/work\/10565\/jmidkiff\/opt\/lumerical\/v242\/api\/python\/lumopt\/utilities\/simulation.py&#8221;, line 22, in __init__&lt;br&gt;&nbsp; &nbsp; self.fdtd = lumapi.MODE(hide = hide_fdtd_cad) if use_var_fdtd else lumapi.FDTD(hide = hide_fdtd_cad)&lt;br&gt;&nbsp; File &#8220;\/work\/10565\/jmidkiff\/opt\/lumerical\/v242\/api\/python\/lumapi.py&#8221;, line 1541, in __init__&lt;br&gt;&nbsp; &nbsp; super(FDTD, self).__init__(&#8216;fdtd&#8217;, filename, key, hide, serverArgs, remoteArgs, **kwargs)&lt;br&gt;&nbsp; File &#8220;\/work\/10565\/jmidkiff\/opt\/lumerical\/v242\/api\/python\/lumapi.py&#8221;, line 1196, in __init__&lt;br&gt;&nbsp; &nbsp; handle = self.__open__(iapi, product, key, hide, serverArgs, remoteArgs)&lt;br&gt;&nbsp; File &#8220;\/work\/10565\/jmidkiff\/opt\/lumerical\/v242\/api\/python\/lumapi.py&#8221;, line 1415, in __open__&lt;br&gt;&nbsp; &nbsp; raise LumApiError(error)&lt;br&gt;lumapi.LumApiError: &#8216;appOpen error: \\n Failed to start messaging, check licenses&#8230;\/work\/10565\/jmidkiff\/opt\/lumerical\/v242\/bin\/fdtd-solutions-app: error while loading shared libraries: libglut.so.3: cannot open shared object file: No such file or directory\\n'&lt;br&gt;Exception ignored in: &lt;function Simulation.__del__ at 0x1533afd720d0&gt;&lt;br&gt;Traceback (most recent call last):&lt;br&gt;&nbsp; File &#8220;\/work\/10565\/jmidkiff\/opt\/lumerical\/v242\/api\/python\/lumopt\/utilities\/simulation.py&#8221;, line 67, in __del__&lt;br&gt;&nbsp; &nbsp; self.fdtd.close()&lt;br&gt;AttributeError: &#8216;Simulation&#8217; object has no attribute &#8216;fdtd'&lt;\/p&gt;&lt;p&gt;&nbsp;&lt;\/p&gt;&lt;p&gt;&lt;br&gt;<strong>SWDM_out.log<\/strong>&lt;\/p&gt;&lt;p&gt;CONFIGURATION FILE {&#8216;root&#8217;: &#8216;\/work\/10565\/jmidkiff\/opt\/lumerical\/v242\/api\/python&#8217;, &#8216;lumapi&#8217;: &#8216;\/work\/10565\/jmidkiff\/opt\/lumerical\/v242\/api\/python&#8217;}&lt;br&gt;Initializing super optimization&lt;br&gt;Checking for one forward simulation : &nbsp; &nbsp;&lt;\/p&gt;<\/p>\n","protected":false},"template":"","class_list":["post-432685","topic","type-topic","status-publish","hentry","topic-tag-hpc","topic-tag-inverse-design-2","topic-tag-linux","topic-tag-LumericalFDTD-1","topic-tag-lumopt-1","topic-tag-python","topic-tag-slurm"],"aioseo_notices":[],"acf":[],"custom_fields":[{"0":{"_bbp_forum_id":["27833"],"_bbp_topic_id":["432685"],"_bbp_subscription":["261859","30281"],"_bbp_author_ip":["2603:8080:19f0:1630:4875:2bfb:d7bb:4650"],"_bbp_last_reply_id":["432770"],"_bbp_last_active_id":["432770"],"_bbp_last_active_time":["2025-05-28 16:58:46"],"_bbp_reply_count":["1"],"_bbp_reply_count_hidden":["0"],"_bbp_voice_count":["2"],"_bbp_engagement":["261859","30281"],"_btv_view_count":["817"],"_bbp_topic_status":["unanswered"],"_edit_lock":["1748428614:473055"]},"test":"jmidkiffutexas-edu"}],"_links":{"self":[{"href":"https:\/\/innovationspace.ansys.com\/forum\/wp-json\/wp\/v2\/topics\/432685","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/innovationspace.ansys.com\/forum\/wp-json\/wp\/v2\/topics"}],"about":[{"href":"https:\/\/innovationspace.ansys.com\/forum\/wp-json\/wp\/v2\/types\/topic"}],"version-history":[{"count":0,"href":"https:\/\/innovationspace.ansys.com\/forum\/wp-json\/wp\/v2\/topics\/432685\/revisions"}],"wp:attachment":[{"href":"https:\/\/innovationspace.ansys.com\/forum\/wp-json\/wp\/v2\/media?parent=432685"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}