Learning Forum

[yu272]

Hi all,

I am doing a two-way FSI simulation with Transient structral and Fluent via System coupling. I run the simulation on the university cluster with 30 - 40 processors.

Recently, I found when I restart the simulation, the mechanical part calculation becomes very slow. I also met the case where I cannot properly restart the FSI due to mechanical simulation as it says it cannot find all files required for restart (.Rnn etc), where I used the exact same amount of processors I used for the first run. 

I found the problem is caused by the following notes from output in my first run, where I requested 35 processors:

ANSYS did not use all the processors I requested, therefore when I tried to restart, it cannot find past files for those extra processors not used.

Once I re-specify the number of processors, simulation continues. However, mechanical simulation becomes very slow while fluent part is fine.

Previously, I did not meet the case where ANSYS did not use all processors, and restarted simulation was fine. Anyone can help me with this? How to let ANSYS use all processors requested?

Sincerely,

Eric

[Gary Stofan]

The Ansys APDL program has a Hybrid Parallel mode where for example:

32 cores requested might automatically choose 8 DMP x 4 SMP.

You can disable this auto Hybrid Parallel mode by adding -nt 1 to the APDL Command Line. 

In this way, 32 cores requested will choose 32 DMP

 

[yu272]

Hi Gary,

I cannot access the image you uploaded. Could you upload it again?

Currently, I call structural simulation via System coupling, so my job submit file only specifies a number of node for system coupling and it automatically assign the same number for structrual, fluent, and system coupling. I attached a screenshot of my submit file:

and the corresponding generated output file that calls mechanical is below:

Is there anyway I can pre-set it in workbench before I export the system coupling setting?

Thanks!

Eric

[Lane Jancy]

For instance, in the Hybrid Parallel mode of the Ansys APDL program:

If you specify 32 cores, the system may opt for 8 DMP x 4 io games SMP.

By including -nt 1 in the APDL Command Line, you have the option to disable this auto Hybrid Parallel mode. 

This method, 32 DMPs will be selected when 32 cores are needed.

[Gary Stofan]

Hybrid parallel does not apply in this case.

This issue has to do with Domain Decompsiton and efficiency.

The very high load balance ratio 5.633 triggers the solver to use only 32 cores instead of 40
as 40 would not be efficient for your problem. Load balance 1.0 is theortically perfect. In practice, 2.0 or less is reasonable. 
The APDL runs is likely very slow due to this high imbalance. Contact elements and how they are scoped causes this imbalance. 

What to do?
 - Set for 32 cores and allow it to run.
 - Or, try fewer cores to see if the load balance improves.
 - Or, Rescope contact pairs. 

 

[yu272]

Hi Gary,

Thanks for the suggestion. You are right, my simulation has highly imbalanced contact problems. However, for the same setup, I also tried to run with 32 processors. The simulation will assign 28 processors instead, seems like it never use all the processors. I will try to optimize the region of contact and keep you updated.

Sincerely,

Eric