The memory allocation in Mechanical APDL is dynamic so usually if the solver is running optimally out-of-core it is because it could not allocate enough to run in-core. So the first step would be to review the solve.out for the analysis and see just what the memory numbers were. There might not have been enough RAM to solve in-core. However user can manually control the memory mode using BCSOPTION command.