Learning Forum

[jmidkiff]

Hello. I am doing inverse design with FDTD and the Python API. On my local machine I use the GUI to launch the python script SWDM.py and everything runs fine, albeit very slowly. I would like to speed up the optimization process by running it on a high-performance computing (HPC) system. I have Lumerical v242 installed on a Rocky Linux HPC system (without a GUI) called Lonestar6. I am able to run a single simulation file SWDM.fsp utilizing the Slurm job script SWDM_one_sim.txt.

Now, I have tried to create a Slurm job script SWDM_optimize.txt which calls out the Python script SWDM.py. However, when I try to run it, it seems that some of the settings declared in the job script are not recognized during execution of the Python script. As I am new to HPC, surely I am missing something. I would like to know if you have any ideas or suggestions to make this work? The files SWDM_one_sim.txt, SWDM_optimize.txt, SWDM.py, and SWDM_error.log are provided below. Thanks.

SWDM_one_sim.txt

#!/bin/bash
#SBATCH -J SWDM              # Job name
#SBATCH -o SWDM_out.log      # Standard output log
#SBATCH -e SWDM_err.log      # Standard error log
#SBATCH -p normal            # Queue/partition name
#SBATCH -N 1                 # Number of nodes
#SBATCH -n 1                 # Total number of tasks
#SBATCH --cpus-per-task=8    # Number of cores/threads (should match -t flag)
#SBATCH --mem=1024           # Memory (in MB)
#SBATCH -t 00:10:00          # Runtime (hh:mm:ss)
#SBATCH -A PHY25016          # Lonestar6 allocation

# Load modules
module reset
module load intel/19.1.1
module load impi/19.0.9

# Set path to libglut.so.3 (from user gda)
export LD_LIBRARY_PATH=/home1/01249/gda/test/glut/usr/lib64:$LD_LIBRARY_PATH

# Set Ansys licensing environment variable
export ANSYSLMD_LICENSE_FILE=XXXX

# Set Ansys Lumerical API and bundled Python3 path
export PATH=/work/10565/jmidkiff/opt/lumerical/v242/bin:/work/10565/jmidkiff/opt/lumerical/v242/python/bin:$PATH
export PYTHONPATH=/work/10565/jmidkiff/opt/lumerical/v242/api/python:$PYTHONPATH

set -e

# Run a single FDTD simulation
fdtd-engine-impi-lcl -t 8 SWDM.fsp

 

SWDM_optimize.txt

#!/bin/bash
#SBATCH -J SWDM              # Job name
#SBATCH -o SWDM_out.log      # Standard output log
#SBATCH -e SWDM_err.log      # Standard error log
#SBATCH -p normal            # Queue/partition name
#SBATCH -N 1                 # Number of nodes
#SBATCH -n 4                 # Total number of tasks
#SBATCH --cpus-per-task=8    # Number of cores/threads (should match -t flag)
#SBATCH --mem=1024           # Memory (in MB)
#SBATCH -t 00:10:00          # Runtime (hh:mm:ss)
#SBATCH -A PHY25016          # Lonestar6 allocation

# Load any necessary modules
module reset
module load intel/19.1.1
module load impi/19.0.9

# Set path to libglut.so.3 (from user gda)
export LD_LIBRARY_PATH=/home1/01249/gda/test/glut/usr/lib64:$LD_LIBRARY_PATH

# Set Ansys licensing environment variable
export ANSYSLMD_LICENSE_FILE=XXXX

# Set Ansys Lumerical API and bundled Python3 path
export PATH=/work/10565/jmidkiff/opt/lumerical/v242/bin:/work/10565/jmidkiff/opt/lumerical/v242/python/bin:$PATH
export PYTHONPATH=/work/10565/jmidkiff/opt/lumerical/v242/api/python:$PYTHONPATH

# Set hidden CAD (Required for Lumerical script, Python API, LumOpt)
export QT_QPA_PLATFORM=offscreen

set -e

# Run Python optimization script
python3 SWDM.py

 

SWDM.py

######## IMPORTS ########
# General purpose imports
import os
import sys
import numpy as np

# Optimization specific imports
from lumopt import CONFIG
from lumopt.geometries.topology import TopologyOptimization2D
from lumopt.utilities.load_lumerical_scripts import load_from_lsf
from lumopt.figures_of_merit.modematch import ModeMatch
from lumopt.optimization import Optimization
from lumopt.optimizers.generic_optimizers import ScipyOptimizers
from lumopt.utilities.wavelengths import Wavelengths

######## DEFINE BASE SIMULATION ########

def runSim(params, eps_bg, eps_wg, x_pos, y_pos, size_x, size_y, filter_R, working_dir, beta):

    ######## DEFINE GEOMETRY ########
    geometry = TopologyOptimization2D(params=params, eps_min=eps_bg, eps_max=eps_wg, x=x_pos, y=y_pos, z=0, filter_R=filter_R, beta=beta)
 
    ######## DEFINE FIGURE OF MERIT FOR EACH OUTPUT WAVEGUIDE ########
    fom1 = ModeMatch(monitor_name = 'fom_1', mode_number = 'Fundamental TE mode', direction = 'Forward', norm_p = 2, target_fom=1)
    fom2 = ModeMatch(monitor_name = 'fom_2', mode_number = 'Fundamental TE mode', direction = 'Forward', norm_p = 2, target_fom=1)

    ######## DEFINE OPTIMIZATION ALGORITHM ########
    optimizer = ScipyOptimizers(max_iter=400, method='L-BFGS-B', pgtol=1e-6, ftol=1e-5, scale_initial_gradient_to=0.25)

    ######## DEFINE SETUP SCRIPT AND INDIVIDUAL OPTIMIZERS ########
    script = load_from_lsf('SWDM.lsf')

    wavelengths1 = Wavelengths(start = 834e-9, stop = 848e-9, points = 6)
    opt1 = Optimization(base_script=script, wavelengths = wavelengths1, fom=fom1, geometry=geometry, optimizer=optimizer, use_deps=False, hide_fdtd_cad=True, plot_history=False, store_all_simulations=False, save_global_index=False)
    wavelengths2 = Wavelengths(start = 860e-9, stop = 938e-9, points = 7)
    opt2 = Optimization(base_script=script, wavelengths = wavelengths2, fom=fom2, geometry=geometry, optimizer=optimizer, use_deps=False, hide_fdtd_cad=True, plot_history=False, store_all_simulations=False, save_global_index=False)

    ######## PUT EVERYTHING TOGETHER AND RUN ########
    opt = opt1+opt2
    opt.continuation_max_iter = 30 #< How many iterations per binarization step (default is 20)
    opt.run(working_dir = working_dir)

if __name__ == '__main__':
    size_x = 9000
    size_y = 4000
    filter_R = 200e-9

    x_points=int(size_x/20)+1
    y_points=int(size_y/20)+1
    eps_wg = 1.74**2
    eps_bg = 1.45**2
    x_pos = np.linspace(-size_x/2*1e-9,size_x/2*1e-9,x_points)
    y_pos = np.linspace(-size_y/2*1e-9,size_y/2*1e-9,y_points)

    ## Initial condition:
    params = 0.5*np.ones((x_points,y_points))     #< Start with the domain filled with (eps_wg+eps_bg)/2
    beta = 1

    working_dir = 'SWDM_x{:04d}_y{:04d}_f{:04d}'.format(size_x,size_y,int(filter_R*1e9))
    runSim(params, eps_bg, eps_wg, x_pos, y_pos, size_x*1e-9, size_y*1e-9, filter_R, working_dir=working_dir, beta=beta)

 

SWDM_error.log

Traceback (most recent call last):
  File "/work/10565/jmidkiff/SWDM.py", line 78, in
    runSim(params, eps_bg, eps_wg, x_pos, y_pos, size_x*1e-9, size_y*1e-9, filter_R, working_dir=working_dir, beta=beta)
  File "/work/10565/jmidkiff/SWDM.py", line 50, in runSim
    opt.run(working_dir = working_dir)
  File "/work/10565/jmidkiff/opt/lumerical/v242/api/python/lumopt/optimization.py", line 471, in run
    self.initialize(working_dir=working_dir)
  File "/work/10565/jmidkiff/opt/lumerical/v242/api/python/lumopt/optimization.py", line 119, in initialize
    self.one_forward = check_one_forward_sim(self.optimizations[0])
  File "/work/10565/jmidkiff/opt/lumerical/v242/api/python/lumopt/optimization.py", line 82, in check_one_forward_sim
    co_opt.sim = Simulation(self.workingDir, co_opt.use_var_fdtd, co_opt.hide_fdtd_cad)
  File "/work/10565/jmidkiff/opt/lumerical/v242/api/python/lumopt/utilities/simulation.py", line 22, in __init__
    self.fdtd = lumapi.MODE(hide = hide_fdtd_cad) if use_var_fdtd else lumapi.FDTD(hide = hide_fdtd_cad)
  File "/work/10565/jmidkiff/opt/lumerical/v242/api/python/lumapi.py", line 1541, in __init__
    super(FDTD, self).__init__('fdtd', filename, key, hide, serverArgs, remoteArgs, **kwargs)
  File "/work/10565/jmidkiff/opt/lumerical/v242/api/python/lumapi.py", line 1196, in __init__
    handle = self.__open__(iapi, product, key, hide, serverArgs, remoteArgs)
  File "/work/10565/jmidkiff/opt/lumerical/v242/api/python/lumapi.py", line 1415, in __open__
    raise LumApiError(error)
lumapi.LumApiError: 'appOpen error: \n Failed to start messaging, check licenses.../work/10565/jmidkiff/opt/lumerical/v242/bin/fdtd-solutions-app: error while loading shared libraries: libglut.so.3: cannot open shared object file: No such file or directory\n'
Exception ignored in:
Traceback (most recent call last):
  File "/work/10565/jmidkiff/opt/lumerical/v242/api/python/lumopt/utilities/simulation.py", line 67, in __del__
    self.fdtd.close()
AttributeError: 'Simulation' object has no attribute 'fdtd'

 

SWDM_out.log

CONFIGURATION FILE {'root': '/work/10565/jmidkiff/opt/lumerical/v242/api/python', 'lumapi': '/work/10565/jmidkiff/opt/lumerical/v242/api/python'}
Initializing super optimization
Checking for one forward simulation :    

[Taylor Robertson]

Hello,

So in the first case you are running the FDTD engine on a file, which is straightforward on a headless cluster. 

With the python api in the second case you need to use lumerical gui which can be a bit more complicated. Make sure you set these environment variables correctly.

Running-CAD-jobs-on-headless-Linux-systems

From what you shared I think the issue is something I have seen before, and it appears to a dependecies issue.

lumapi.LumApiError: 'appOpen error: \n Failed to start messaging, check licenses.../work/10565/jmidkiff/opt/lumerical/v242/bin/fdtd-solutions-app: error while loading shared libraries: libglut.so.3:

See this page for how to resolve it. The issue happens when installing without root access or not using the default install process using our script 'install.sh'. This lumerical install.sh script with root access will check for dependencies and download and install them together with Lumerical.

Best Regards,