Learning Forum

[yu272]

Hi all, 

I am doing a system coupling FSI simulation (Transient Mechanical + Fluent) on cluster. Due to the limit available time of computational resources, each job can run up to 3 days. Therefore, I need to restart my simulation after the it is killed by the cluster. 

However, I met problems when restarting. For example, I run 10 steps, and I used following python code to resubmit my job from step 6, where restart file for step 6 is available:

# load current steps
Open(CouplingStep = 6)

# Solve the co-simulation. 
Solve()

I found that, if I successfully finished previous simulation. That is, if previous simulation ends successfully. I can restart with any restart step I want. On the other hand, if my previous run got killed by the cluster, such as only finished 9 steps. I am unable to restart from any steps even the restart files are available. The reason is System Coupling cannot successfully detect all participants, with following output:

Awaiting connections from coupling participants...Abort(100) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 100) - process 0

Elapsed time: 5 min. Still awaiting connection from: MAPDL-1. If you feel the participant(s) should already have connected, check participant logs and/or abort the run.
Elapsed time: 10 min. Still awaiting connection from: MAPDL-1. If you feel the participant(s) should already have connected, check participant logs and/or abort the run.

My question is: what might cause this issue? the main difference between above two cases are some temporary files generated since previous simulation is not done, such as sysSolve.lock, which I removed before restarting. I really need to restart my simulation properly and genuinely hope anyone offer a solution.

 

Thank you!

Eric

 

 

[Rahul]

Hi

You can restart a completed or interrupted coupled analysis from either the final restart point or an intermediate restart point. Based on the restart point selected, System Coupling automatically finds all the necessary restart files (both for itself and for all participants) and then restarts the run from the designated point in the analysis.

Restarts from an intermediate restart point are possible with any analysis for which intermediate start points have been generated. An aborted run can be restarted from an intermediate restart point that was completed before the analysis was aborted.

Note:  some participants may not allow restarts. For example: AEDT participants do not support restarts from intermediate restart points or for solutions where the coupled analysis is transient. and Mechanical participants do not support restarts for steady-state thermal and transient thermal analyses.

Please refer System coupling user manual for more information: Restarting a Coupled Analysis (ansys.com)

 

[yu272]

Hi RM,

Thank you for your suggestions! I screened MAPDL files and found that this error may be caused by the file.lock file that should be removed. Now I can restart system coupling simulation successfully. However, sometimes I met a problem that the system cannot detect required files to restart the simulation, the message is:

Participant MAPDL-1 encountered a fatal error. Message: During the          |
| multiframe restart process, the .RDB file was found but the .Rnnn files     |
| were not found.  Both files are required to perform the multiframe restart.

However, I am pretty sure the required files (file0.r001, file1.r20 etc) are correctly presented in the Mechanical folder. The exact same setup works in my other simulations. Do you have any thoughts on this?

Sinerely,

Eric

 

[Rahul]

Hello Eric,

Yes, lock file will not allow the restarts and should be removed.

If you change the number of cores between the two simulations (while restarting), you will get the error message "During the multiframe restart process, the .RDB file was found but the .Rnnn files were not found".

To avoid this ensure that you are using same number of cores as earlier run.

[yu272]

Thanks RM! It works!

[yu272]

Hi all,

Sorry that I have encountered new problems. Somehow when I tried to restart the above FSI simulations recently, it does not work again. I have the same error message: 

During the multiframe restart process, the .RDB file was found but the  
 .Rnnn files were not found. 

However, I am sure that I used same numbers of cores for as the earlier submission. This setup worked well before but suddenly I got this error again. Do you have any suggestion for the cause?

 

Besides, I also have another question about particle tracking using DPM model in Fluent, which gives a xml file for particle tracking that can be read by CFD POST. How do I keep writing particle tracking data into this file after restarting instead of overwriting?

Sincerely,

Eric