Learning Forum

[Psyne]

Hello,

I am currently running transient simulations on a 128-core processor and I specify to fluent that I only want 8 cores out of the 128. (I have an academic and HPC licensing)

And I discovered to my amazement that although I had specified that I only wanted 8 cores, fluent uses all the resource at its disposal and generates 260 threads, which is hyper-threading. Only 8 threads are active during the computation phase, but when the time-step is beeing passed, the 252 remaining threads are activated and most of them have a 100% utilization rate.

This raises a problem: I can't run several simulations at the same time because each one will impact the others due to these threads.

But one greater problem I'm facing with, is that this 'time-step phase' is very long on this 128-core processor. It represents always about 10 seconds or more (which is not negligible on thousands of time-steps), when on a smaller computer with a 6-core processor (and 15 threads generated) it just represents less than 2 seconds. Thus, I wonder whether having 260 threads is not the cause of the problem compared to the smaller pc which has only 15 threads.

Hence my initial question : Is is possible to specify to fluent the number of threads generated in addition to CPU and GPU numbers ?

I thank you in advance for your interest in my problem. Have a good day !

[Psyne]

NB : I've tried to use 'The Thread Control Dialog Box' and its associated commands but it turns out that fluent does not seem to take into account the restrictions we impose on it and in fact uses all the available resources.

[Rob]

Launching on 8 cores should only generate 8 parallel tasks. Please post a screen grab of the Fluent launcher.

[Psyne]

Hello Thank you for your answer I don't run fluent through the launcher but only with commands lines where I specify 8 CPUs. In the journal run with the simulation I specify the number of threads.
In the attached pictures, you can see that effectively 8 threads are running with the simulation (picture 1).
When I look closer at the child-threads of one of the 8 threads launched by fluent (here the n┬░4117664), I see that each of the 8 parallel threads gives birth to 257 child-threads (picture 2) which are activated when passing time-steps (picture 3).
Thank you in advance, have a good day !

[Amine Ben Hadj Ali]

More details about system and how you are starting and if you are using DPM Model.

[Psyne]

My system is a 2*64-core processor (AMD Rome), with 8 Nvidia A100 and I have enough HPC licences to run simulations.
I run the following command to run a simulation :
/fluent/v212/fluent/bin/fluent 2ddp -t8 -gpgpu=1 -g -i
As for the DPM model, I don't need it so I don't use it.

[MangeshANSYS]

When running with gpgpu, try running with only one core i.e. -t1

[Amine Ben Hadj Ali]

Do you see the same number of children processes without GPGPU?
Regarding my DPM question: As I do not know what you are doing for that reason I have asked.

[Psyne]

Yes I tried with and without GPGPU, I have not observed any change. I see the same number of children processes.

[Amine Ben Hadj Ali]

Okay. Thanks. I will need to think more about this and check with my colleagues.

[Amine Ben Hadj Ali]

How are you getting the child processes? Which OS are you using? Are you starting using any job scheduler?

[Psyne]

,I am using CentOs 7 and I am running simulations on a cluster with a job scheduler : slurm.
To get child processes, I use this command : top -H -p #thread_number

[MangeshANSYS]

 

running under Slurm – what is the final command line submitted to sbatch? if running a script then what is the final commandline created by the submit script?

 

are there any SLURM job related variables in the environment which may cause the solver to perceive there are more cores available ?

 

[Rob]

Can you also check if those processes are actually from that job, as opposed to defunct processes from earlier problems. Is your userID really abenete+ as I'd expect LINUX to not like the + sign in the path.

[Psyne]

To determine which processes are running on the cluster I run this command :
ps -ef| grep fluent_mpi.21.2 | grep -v -e mpiexec -e' /bin/sh'
The result was indicated in the 1st picture : the 8 processes linked to my simulation.
The processes (in the pictures 2 & 3) are from that job because the command : only returns the processes linked to the process 4117664, which corresponds to one of the processes launched by my job. There could be defunct processes from earlier problems but if it were the case, they would have been spotted by the previous command, not this one which only deals with child processes of a given process.
About my UserID, it's not abenete+, linux automatically adds a "+" when there is not enough place to print the whole name ! So, there is no influence with my userID, don't worry about that !

[Amine Ben Hadj Ali]

Let me check and get back to you.

[Psyne]

Finally while you were looking for solutions, I tested new things on the linux side and finally found a command that worked. Thanks to it, I was able to control the number of fluent threads and thus reduce the simulation time.
I'll give you what I did, maybe it will be useful for other people in my situation! Before starting the fluent simulation, I type this command:
(4 for the number of desired threads)
With this command, I can restrict fluent.
Thank you for your time and research on my problem.

[Psyne]

Finally while you were looking for solutions, I tested new things on the linux side and finally found a command that worked. Thanks to it, I was able to control the number of fluent threads and thus reduce the simulation time.
I'll give you what I did, maybe it will be useful for other people in my situation! Before starting the fluent simulation, I type this command:
(4 for the number of desired threads)
Thank you for your time and research on my problem.

[Amine Ben Hadj Ali]

Our observation on our cluster is inline with what we would expect, i.e., 3 threads per process. In your case that was more than normal :). I wanted to ask you today to modify that command.

[Amine Ben Hadj Ali]

Can you please, after you you reset that command you are now using, share with us the output of this (with your username):

pstree -u $USER

[Psyne]

Here are the screenshots you asked me. The 1st image is with and the 2nd without.

[Amine Ben Hadj Ali]

Is not really clear. I will write an email.