Learning Forum

[Alexandre Mongrain]

Hi,

I'm currently making a large model to that simulates a lot of contacts in-between beams (a cable analysis), and I have been successfully able to simulate small cables. However, when I scale my model (more wires and longer cables), I reach a point where ANSYS just crashes without any error. All contacts are separated in pairs of likely-to contact beam regions, so I have >100k real sets in my model.

I have a workstation with 512Gb of RAM, and I get crashes with models that requires more than 200Gb of for in-core mode. The crash almost always happen during the first iteration or just before the it, and on rare cases, after 5-6 iterations. I also tried to force the out-of-core memory mode, but it crashes also, at the same place.

Here is an example of the last lines of a log file that crashed : 

DISTRIBUTED SPARSE MATRIX DIRECT SOLVER.
  Number of equations =    19535375,    Maximum wavefront =    243

  Process memory allocated for solver              =    11.275 GB
  Process memory required for in-core solution     =    66.652 GB
  Process memory required for out-of-core solution =    10.907 GB

  Total memory allocated for solver                =    86.199 GB
  Total memory required for in-core solution       =   463.498 GB
  Total memory required for out-of-core solution   =    83.486 GB

 *** WARNING ***                         CP =    2165.953   TIME= 21:18:22
 The Distributed Sparse Matrix Solver is currently running in the        
 out-of-core memory mode.  This memory mode may provide significantly    
 worse performance compared to the in-core memory mode, depending on     
 the amount of available system memory and I/O speed.  Please monitor    
 the solver performance to ensure that the large amount of I/O to the    
 solver files does not create a bottleneck for performance.    

For reference, this analysis was using the distributed sparse matrix solver, with 32 cores (CPU is a Ryzen Threadripper PRO 5975WX). 

I also tested the shared memory mode and running with and without the GUI and there is no difference.

I also, rarely, get the following error, but not in a reproductible way :

*** ERROR ***                           CP =    3186.766   TIME= 13:57:02
An error has occurred in the Sparse Matrix Solver while factoring the   
matrix.  Error code = 154.  Please send the data leading to this        
operation to your technical support provider, as this will allow        
ANSYS, Inc to improve the program. 

Any thoughts?

Thanks !

[mrife]

Alexandre

Can you post the actual error reported in the "crash" that goes along with the first posted image (the standard memory allocation information screen shot).  I've seen similar issues on similar AMD CPUs so try this; in the BIOS turn off Simultaneous Multithreading then redo the solve...what happens?

[Alexandre Mongrain]

I tried again after turning off the Simultaneous Multithreading option, it did not change anything. It crashed at the first iteration with no error.

[Alexandre Mongrain]

Thanks for your answer,

There is absolutely no error message when ANSYS crashes using the GUI, the log file ends with the memory allocation information with no accompanying error. Sometimes, some information from a first completed iteration is also written to the log file, and it just hangs for an hour or two before closing itself. When I start my job from a command line, I get the following on the console when it crashes, but I think it is only a generic message :

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 0 PID 19180 RUNNING AT MEC053695
=   EXIT STATUS -1 (ffffffff)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 1 PID 12748 RUNNING AT MEC053695
=   EXIT STATUS -1073741819 (c0000005)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 2 PID 13072 RUNNING AT MEC053695
=   EXIT STATUS -1 (ffffffff)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 3 PID 18524 RUNNING AT MEC053695
=   EXIT STATUS -1 (ffffffff)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 4 PID 12372 RUNNING AT MEC053695
=   EXIT STATUS -1 (ffffffff)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 5 PID 18528 RUNNING AT MEC053695
=   EXIT STATUS -1 (ffffffff)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 6 PID 19092 RUNNING AT MEC053695
=   EXIT STATUS -1 (ffffffff)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 7 PID 18168 RUNNING AT MEC053695
=   EXIT STATUS -1 (ffffffff)
===================================================================================

I will try to change the BIOS setting tomorrow and re-test, and I will come back with the results.

[mrife]

What is the OS and version used.  Are you submitting to a job scheduler and if so do you have the scheduler log for the job?

[Alexandre Mongrain]

I'm using a Windows workstation with ANSYS 2022R2, I'm submitting my job directly with a command prompt. 
I get a log with the -o that ends, when the process crash, either with the memory informations or with some information from the first iteration.
Other than that, I get the "BAD TERMINATION" as in my previous reply inside the command prompt.

[mrife]

Alexandre - are you an Ansys customer?  Or are using a Academic license?  We are getting to the point of needing to get the input file (as stated in that Error message you posted).  However before that is there any CPU or RAM over-clocking set on the system?  Can you try another application to stress-test the system, like Prime95? 

https://en.wikipedia.org/wiki/Prime95

https://www.mersenne.org/download/#stresstest

 

[Alexandre Mongrain]

I use a academic research license, I'm using ANSYS for my masters project. The CPU and RAM clocks are factory default.

I will try to stress-test the system with Prime95. I'm not sure that it may be a hardware problem, though. I have successfully ran different, big models (not cables) on this machine. I suspect it may have to do with having a very large number of realsets (for each contact pair), or something like this.

I can share an input file, just to check if it runs successfully on another machine. Can I send it to you directly ? However, I really don't expect my model to converge as it is.

[mrife]

Alexandre - which institution do you attend?

[Alexandre Mongrain]

I attend ÉTS school in Québec, Canada

[mrife]

Alexandre - which department?  

[Alexandre Mongrain]

Mechanical engineering department

[mrife]

Alexandre - I reached out to the ASC (Ansys Support Coordinator) for that department asking them to forward you an email.  If they do forward the email please reply to me directly.

Mike