Learning Forum

[marcin.lelit]

I use recent version of Lumerical Suite on Windows. When running training simulation set up accordingly to FDTD100 tutorial video, simulation crashes at approx. 15%, with an error: "Process number 0 unable to locate checkpoint file." Can anyone point me to the solution of the problem? Log file attached.

training simulation 1_p0.log

[Lito]

Array, Is the simulation file saved in a location that you have write access? Try to save the simulation example into your user account/Downloads or Desktop folder.
Copy and paste the log results, we are unable to download files from ALF.

[marcin.lelit]

.

Dear Lito, thank you for taking interest in my question. I tried various directories, I have admin privileges. Interrupted simmulation generates movie file and log file in simulation file directory. Results are always the same, as described. Log results below and prtscr of job details:

Ansys Lumerical 2021 R1.2 FDTD Solver Version 8.25.2621 (Windows 64bit)

expires (month/day/year): Permanent

using config file: Ansys Licensing set with ANSYSLMD_LICENSE_FILE

license host: 1055@10.0.3.13

current time: Fri Mar 26 09:14:20 2021

Running on host: 49244_CEZ

host available memory (GB): 60.266922

All processes are communicating and ready to proceed to simulation...

Running fsp file: C:Program FilesLumericalv211infdtd-engine-msmpi.exe -t 1 -remote -resume C:/Users/m.lelit/Documents/Lumerical training/training simulation 1.fsp

This simulation has already been run, will run again.

The program terminated due to an error:Unable to locate any file matching 'C:Usersm.lelitDocumentsLumerical training raining simulation 1_*_p0.fsp.ckpt'

Error: Process number 0 unable to locate checkpoint file.

.

[Lito]

.

lelma87, try to disable checkpoint on the simulation.

Can you try and run the mwp.fsp file from this example? Does the issue happen to this example, mwp.fsp?

.

[marcin.lelit]

Array, thank ou for this suggestion. This worked but for some reason the other way around: at first, chceckpoint option was already disabled and error occured. However, when I checked both "checkpoint during simulation" and "checkpoint at schutoff", checkpoint files were generated and simmulation completed cuccessfuly.
mwp.fsp works fine regardless of checkpoint options - simmulation completed cuccessfuly with and without checkpoints.

[Lito]

Array, Glad to hear that this has been resolved.

[d08941008]

.

Hi, I recently got the similar issue on the linux cluster with limited walltime of the queue. Is there any way to enable the checkpoint feature over terminal via fdtd-engine-impi-lcl command?

.

[Lito]

.

RyanChung

Starting a simulation job with checkpoint:

• Enable checkpoint in FDTD.

• Run the simulation as you would without the -resume option.
• This will create the checkpoint files based on the number of processes and checkpoint period.
• Ensure that the checkpoint files are created and not deleted when the simulation terminates.

To resume the simulation based on the last checkpoint:

• Do not change or edit the simulation.fsp file.
• Ensure that the checkpoint files are found together with your simulation.fsp file.
• Run the simulation with the -resume option. (do not use the "-resume" option if this is the first time you are running the simulation and you are not "resuming" a simulation job)

./fdtd-engine-  -resume  -fullinfo  simulationfile.fsp

Best,

Lito

.

[d08941008]

.

Thanks. I’ll enable it via CAD on the local PC before uploading it onto the CAD-free cluster.

.

[Lito]

.

RyanChung

You are welcome.

Best,

Lito

.

[qiu shi]

i also meet this problem,have you solved this problem?if you have solved it,i am very thankful if you can contact me,here is my eamil:asumirioyuzukarei@gmail.com