Fluids

Fluids

Topics relate to Fluent, CFX, Turbogrid and more

How to resolve MPI deadlock? (Forte)

    • _aryaman
      Subscriber

      I'm trying to run a non-combustion IC Engine simulation in Ansys Forte. The simulation runs perfectly with experimental settings (temp, pressure, velocity etc.) but the results vary a little.


      As I try to tweak a little with the crevice model factors the simulation starts crashing giving the following errors::

      Temperature overflow during fluxing.
      Try reducing the convection time step control tolerance and restarting.
      


      Error: Run aborting because the timestep is being driven too small.
      The cause of this is: "Halved (recycling)"
          dt = 7.11500E-13 [s]
           t = 1.69980E-02 [s]
         cycle =    537
         crank =   76.49 [deg]
      Error: Consider adjusting the timestep control tolerances according to the cause of the timestep being too small.
      


      To resolve this I decided to alter convection factor in time step settings, but again the 'timestep being driven too small error came'. To this I decreased the starin rate but then it got crashed with this error::


      Error: Temperature overflow during fluxing.
       Try reducing the convection time step control tolerance and restarting.
       There seems to be a deadlock in MPI communication for this job.
       FORTE will attempt to kill all the active processes at this time.
       Otherwise the deadlock will simply hang all the processes forever.
       Please send this message, FORTE.log file and .ftsim case file to Ansys support.
      CGNS solution file closed successfully
      


      Could anyone please help how I can resolve these errors?

    • RK
      Ansys Employee
      Hello,
      Can you please refine the mesh and try again?
    • _aryaman
      Subscriber
      when I try to refine the mesh I get the following error ::
      forrtl: severe (174): SIGSEGV, segmentation fault occurred
      ImagePCRoutineLineSource
      libifcoremt.so.500002B94C664D216for__signal_handlUnknownUnknown
      libpthread-2.17.s00002B94C740B5E0UnknownUnknownUnknown
      libauto_mesh.so00002B94DAE07B80_ZNSt8_Rb_treeI12UnknownUnknown
      libauto_mesh.so00002B94DAE07B89_ZNSt8_Rb_treeI12UnknownUnknown
      libauto_mesh.so00002B94DAE07B89_ZNSt8_Rb_treeI12UnknownUnknown
      libauto_mesh.so00002B94DAE07B89_ZNSt8_Rb_treeI12UnknownUnknown
      libauto_mesh.so00002B94DAE07AF1_ZN11ElementDataDUnknownUnknown
      libauto_mesh.so00002B94DADF5330_ZN17VertexDataBuUnknownUnknown
      libauto_mesh.so00002B94DACDF39D_ZN20OctreeMeshCoUnknownUnknown
      libauto_mesh.so00002B94DAC4C290_ZN6Domain6updateUnknownUnknown
      libauto_mesh.so00002B94DAE1A077_ZN15AutoMeshAPI_UnknownUnknown
      libauto_mesh.so00002B94DAE19C44api_createmeshUnknownUnknown
      libamg_api.so00002B94CF2BCF4Aamg_createmeshUnknownUnknown
      forte0000000000895376UnknownUnknownUnknown
      forte0000000000B65D5DUnknownUnknownUnknown
      forte0000000000B58556UnknownUnknownUnknown
      libiomp5.so00002B94D069EC53__kmp_invoke_micrUnknownUnknown
      libiomp5.so00002B94D066E357UnknownUnknownUnknown
      libiomp5.so00002B94D066F413__kmp_fork_callUnknownUnknown
      libiomp5.so00002B94D0645E2A__kmpc_fork_callUnknownUnknown
      forte0000000000B56A9AUnknownUnknownUnknown
      forte00000000004D965EUnknownUnknownUnknown
      libc-2.17.so00002B94D0C1AC05__libc_start_mainUnknownUnknown
      forte00000000004D953DUnknownUnknownUnknown

      For resolving this I've tried to increase the CPU cores to 16 and using 4GB space for the simulation, but still it crashes.

      I don't know how but at one case the simulation ran sucessfully, altough still my model was not getting validated so now when I'm trying to vary the parameters to get the correct results I'm recieving the following error in each and every simulation::

      Error: on rank0 reached unexpected location x in bcmombal for vertex11011 with ifirst =11011, momflagX(i4) =-1, vrflux(0,i4) =4, rimtab(i4) =23, cell_size(imtab(i4)) =128, r1tab(i4) =11014, i1tab(i4) =11014, vrflux(0,r1tab(i4)) =4, rimtab(r1tab(i4)) =11011, cell_size(imtab(r1tab(i4))) =128

      Could you please tell what can I do here, I'm really stuck here for quite some time now.

      Thankyou very much.
    • jerom26
      Subscriber
      and
      Have you found a solution for the error on rank?
      I have the same issue half way my simulation.
      Error: on rank1 reached unexpected location x in bcmombal for vertex63352 with ifirst =52349, momflagX(i4) =-1, vrflux(0,i4) =4, rimtab(i4) =1820, cell_size(imtab(i4)) =16, r1tab(i4) =221086, i1tab(i4) =221092, vrflux(0,r1tab(i4)) =4, rimtab(r1tab(i4)) =63352, cell_size(imtab(r1tab(i4))) =32
      Error: on rank1 reached unexpected location x in bcmombal for vertex221089 with ifirst =52349, momflagX(i4) =-1, vrflux(0,i4) =4, rimtab(i4) =1822, cell_size(imtab(i4)) =16, r1tab(i4) =221090, i1tab(i4) =221092, vrflux(0,r1tab(i4)) =4, rimtab(r1tab(i4)) =221089, cell_size(imtab(r1tab(i4))) =32
      Error: on rank1 reached unexpected location x in bcmombal for vertex221103 with ifirst =52349, momflagX(i4) =-1, vrflux(0,i4) =4, rimtab(i4) =1834, cell_size(imtab(i4)) =16, r1tab(i4) =221104, i1tab(i4) =221107, vrflux(0,r1tab(i4)) =4, rimtab(r1tab(i4)) =221103, cell_size(imtab(r1tab(i4))) =32
      I can't find anything what might be the error..

      Greetings Jerom Mestrum
    • Shyam Prasad V Atri
      Subscriber

      I am getting this error and unable to resolve this i could not find anything on the forum. Can you please advise how to fix it.
    • Shyam Prasad V Atri
      Subscriber
      I am trying everything possible to run The explicit dynamics model on HPC and I am really not able to understand how to get this to work.

      Ansys documentation says do this
      -genv ANSYS_EXD_MPI_TYPE intelmpi -host [machinenameA] -np 1 "/usr/ansys_inc/v221/autodyn/bin/autodyn221" -i ./admodel_0.ad -c 0
      -genv ANSYS_EXD_MPI_TYPE intelmpi -host [machinenameB] -np [number_of_processes] "/usr/ansys_inc/v221/autodyn/bin/adworker221"
      But when i do this on Mobaex term I keep getting error
      \par \cf0\highlight0 [s@login01 MECH]$ -genv ANSYS_EXD_MPI_TYPE openmpi -host [login01] -np 5 autodyn191 -i ./admodel_0.ad -c 0\cf1\highlight2
      \par \cf0\highlight0 -genv: No match.\cf1\highlight2
      \par \cf0\highlight0 [s@login01 MECH]$ ANSYS_EXD_MPI_TYPE openmpi -host [login01] -np 5 autodyn191 -i ./admodel_0.ad -c 0\cf1\highlight2
      \par \cf0\highlight0 ANSYS_EXD_MPI_TYPE: No match.\cf1\highlight2
      \par \cf0\highlight0 [s@login01 MECH]$ -genv ANSYS_EXD_MPI_TYPE intelmpi -host [login01] -np 5 autodyn191 -i ./admodel_0.ad -c 0\cf1\highlight2
      \par \cf0\highlight0 -genv: No match.\cf1\highlight2
      \par \cf0\highlight0 [s@login01 MECH]$ -genv ANSYS_EXD_MPI_TYPE intelmpi -host [login01] -np 1 autodyn191 -i ./admodel_0.ad -c 0\cf1\highlight2
      \par \cf0\highlight0 -genv: No match.\cf1\highlight2
      \par \cf0\highlight0 [s@login01 MECH]$ -x ANSYS_EXD_MPI_TYPE opempi -host [login01]:nn -np 1 autodyn191 -i ./admodel_0.ad -c 0\cf1\highlight2
      \par \cf0\highlight0 -x: No match.\cf1\highlight2
      \par \cf0\highlight0 [s@login01 MECH]$\cf1\highlight2

      Please tell me how to run explicit dynamics on HPC i am not able to find any tutorial about that anywhere.

    • marc3llo
      Subscriber
      I have the exact same error (C:\Program Files\ANSYS Inc\v212\aisol\bin\winx64\fort.38) Is there a solution?

    • Shyam Prasad V Atri
      Subscriber
      check your firewall/network settings (it might be blocking ansys) or ansys installation folder permissions in your c drive to give full access permissions. it should solve the problem.
Viewing 7 reply threads
  • The topic ‘How to resolve MPI deadlock? (Forte)’ is closed to new replies.