Learning Forum

[amitrotem2]

Hi,
I'm running FDTD simulation using GPU. The problem is that when exceeding the GPU dedicated memory windows 11 freezes and crash.
Is that an expected behavior? Shouldn't only Lumerical crash? Or just the simulation engine?

[Amrita Pati]

Hi Amit,

In my experience so far, I have seen that the Job Manager shows an error message in red font. And if you check the log or right click on the message to view more details, it will tell you the GPU memory required to run the simulation, as opposed to the available GPU memory. But let me check internally and see if what you observe is also a likelihood.

Regards,
Amrita

[Lito]

@Amit Rotem, 

Please send a screenshot of the job manager - error details. Also copy and paste the contents of the "simulationfilename_p0.log" here. The log file is created when you run the "simulationfilename.fsp". 

[amitrotem2]

I've updated my GPU driver, so no I can control (via the GPU driver) whether or not exceeding the GPU dedicated memory will fallback on RAM.

If I choose not to fallback on RAM, job manager returns an "out of memory error", which is great!

But if I do want to fallback on RAM, now the job manager gets to "running" state, the GPU memory is full, but it seems to not actually compute anything, in the sense that the GPU and CPU "Utilization" (from windows task manager) is low just like when idling.
Annoyed by this, I click "don't save & quit", and that is when windows freezes and crash.

Log file;

start loading CUDA query DLL...
load CUDA query DLL successfully.
GPU streaming multiprocessors(SMs): 26
Warning: your license expires in 0 days!
Ansys Lumerical 2023 R2.3 FDTD Solver Version 8.30.3578 (Windows 64bit)
 expires (month/day/year): 2/20/2024
using config file: Ansys Licensing set with ANSYSLMD_LICENSE_FILE
license host: ----@----------
current time: Tue Feb 20 11:12:26 2024
Running on host: -----
host available memory (GB): 126.208737
All processes are communicating and ready to proceed to simulation...
Running fsp file: C:\Program Files\Lumerical\v232\bin\fdtd-engine.exe -gpu -t 32 -remote -shared-solve-license QU5TWVNfTElDRU5TSU5HX0NPTlRFWFRfREFUQT08QU5TWVNfTElDRU5TSU5HX0NPTlRFWFRfREFUQT48SUQ+MWFmOTJlMWYtN2I4MC00MTQxLWE4OWYtZmE1NjAzM2NmMDdkPC9JRD48VFlQRT5BUFBfREVGSU5FRDwvVFlQRT48TkFNRT5sdW1fc3RhbmRhcmRfc29sdmVfc2hhcmU8L05BTUU+PEhBTkRMRVI+NDk4NjdATHVtZUtpbGxlcjwvSEFORExFUj48L0FOU1lTX0xJQ0VOU0lOR19DT05URVhUX0RBVEE+Cg== C:/------/Lumerical code/crash_test.fsp
Starting meshing
Simulation size in gridpoints: 1 x 408 x 220
Refined 0 mesh cells (0%)
Completed meshing!
Meshing complete in 0 seconds of CPU time
Starting meshing
Simulation size in gridpoints: 1 x 408 x 220
Refined 0 mesh cells (0%)
Completed meshing!
Meshing complete in 0.015625 seconds of CPU time
Starting meshing
Simulation size in gridpoints: 1 x 408 x 220
Refined 0 mesh cells (0%)
Completed meshing!
Meshing complete in 0.015625 seconds of CPU time
Starting meshing
Simulation size in gridpoints: 1 x 408 x 220
Refined 0 mesh cells (0%)
Completed meshing!
Meshing complete in 0.03125 seconds of CPU time
Starting meshing
Simulation size in gridpoints: 1 x 408 x 220
Refined 0 mesh cells (0%)
Completed meshing!
Meshing complete in 0 seconds of CPU time
Starting meshing
Simulation size in gridpoints: 1 x 408 x 220
Refined 0 mesh cells (0%)
Completed meshing!
Meshing complete in 0 seconds of CPU time
left is a duplicate key.
bottom is a duplicate key.
z1 is a duplicate key.
z2 is a duplicate key.
Nx is a duplicate key.
Ny is a duplicate key.
Nz is a duplicate key.
Starting meshing
Simulation size in gridpoints: 1 x 408 x 220
Refined 0 mesh cells (0%)
Completed meshing!
Meshing complete in 0.015625 seconds of CPU time
Starting meshing
Simulation size in gridpoints: 1 x 408 x 220
Refined 0 mesh cells (0%)
Completed meshing!
Meshing complete in 0.015625 seconds of CPU time
Starting meshing
Simulation size in gridpoints: 1 x 408 x 220
Refined 0 mesh cells (0%)
Completed meshing!
Meshing complete in 0.015625 seconds of CPU time
left is a duplicate key.
bottom is a duplicate key.
z1 is a duplicate key.
z2 is a duplicate key.
Nx is a duplicate key.
Ny is a duplicate key.
Nz is a duplicate key.
number of processors is 1
processors are distributed as: 1x1x1
Running 3D Simulation
Starting meshing
Simulation size in gridpoints: 1030 x 646 x 212
Refined 0 mesh cells (0%)
Completed meshing!
Meshing complete in 239.719 seconds of CPU time
Starting meshing
Simulation size in gridpoints: 1030 x 646 x 212
Refined 0 mesh cells (0%)
Completed meshing!
Meshing complete in 136.812 seconds of CPU time
Starting meshing
Simulation size in gridpoints: 1030 x 646 x 212
Refined 0 mesh cells (0%)
Completed meshing!
Meshing complete in 126.078 seconds of CPU time
finished material properties
Meshing Time: 90.0455 seconds, Init Electrical Material Time: 0 seconds, Init Magnetic Material Time: 0
Using selected optimization level: 2
Selecting 3d kernel for splitFieldDirection=0 and optLevel=2
-----------------------------------------
Beginning initialization of 3D Simulation
Starting to add sources to simulation
Adding 3D Mode source propagating in X
Starting to add boundary conditions to simulation
Adding 3D PML BC
Adding 3D PML BC
Adding 3D PML BC
Adding 3D PML BC
Adding 3D PML BC
Adding 3D PML BC
Starting to add monitors to Simulation
Adding analysis monitor
Adding 3D DFT monitor
Adding 3D DFT monitor
Finished initialization of 3D Simulation
-----------------------------------------
All processes are communicating and ready to proceed to simulation...
Starting 897 total iterations
Estimated memory use on the GPU (preliminary): 14.29 GiB

Loading simulation data...

Estimated memory use on the GPU (precise): 

Fields:   9.53 GiB

Coeffs:   1.59 GiB

Monitors: 0.02 GiB

Sources:  0.00 GiB

Estimated memory use on the CPU (precise): 

Fields:   0.00 GiB

Coeffs:   3.15 GiB

Monitors: 0.00 GiB

Sources:  0.00 GiB

 

[Lito]

@Amit Rotem,

I am not sure if falling back between GPU SDRAM and system RAM is supported. Either it runs using the GPU & its SDRAM or the system's RAM when running with CPU. 

What is the memory requirement for your simulation? Is the required memory more than your GPU SDRAM? 
If you have more system RAM than what the simulation requires, better run this with CPU instead of GPU. 
>> Running FDTD simulations from the design environment – Ansys Optics 
 

[amitrotem2]

How can we make sure if fallback is supported? If not, this would be a nice feature to have in the future release.

As far as I understand it, running on GPU with shared memory (fallback on system RAM) could still run faster compared to running solely on CPU.

[Lito]

Currently, the support for GPU processing is limited to the GPU and its resources and will not be making use of other system resources. We do not know if this will be supported in the future.

You can submit this as an Idea/feature request in the Ideas Exchange (IX).