Learning Forum

[Roberto Scipioni]

Hi guys,

 

Here at University of Bath we have an Azure Cluster (Nimbus) with Ansys vs 231 232

We have been asked to be able to submit jobs from a Windows Laptop after the user installed the Ansys Client RSM.

 

Upon attempting to connect we got the error (in attached)

 

We would need help in troubleshooting the error:

 

 

Could it be ....

 

1) Port/Firewall

2) Issues with versions of Ansys

3) Ansys server RSM not set properly on the HPC

4) Ansys client RSM not set properly

 

Please advice

 

Kind Regards

Dr Roberto Scipioni

University of Bath, UK

[Divya Shree Surendran]

Hi Dr. Roberto,

Inorder to submit jobs to a HPC cluster environment from a Windows or Linux client in the same network, please ensure to install and run RSM Launcher service for the same ANSYS version on the headnode of the HPC cluster.

The RSM launcher usually runs on the port 9212 for ANSYS version 2021 R2

Please ensure the Firewall on the HPC side allows communication from and to the client on that port.

Reference: https://ansyshelp.ansys.com/account/secured?returnurl=/Views/Secured/corp/v212/en/wb_rsm/rsm_inst_config.html

[Roberto Scipioni]

Hi,

 

Can you clarify the meaning of the statement:

"This does not apply if SSH is being used for clients-HPC communication, or if the remote HPC resource is a Cloud"

from the mention link you referenced?

In fact, the remote HPC resource IS a Cloud.

 

At University of Bath, users connect to the Head node via ssh (possibly using a VPN if not on Campus).

From the Head node they submit jobs, the CycleCloud orchestrator then will schedule the jobs to run on instances as you go.

 

Please let us know if this is even possible

 

Kind regards

 

Roberto

 

 

[Divya Shree Surendran]

Hi Dr. Roberto,

You should be able to use the Option under HPC resource that says Use SSH or custom communication and mention the useraccount name

[Roberto Scipioni]

If I understand correctly:

When using the RSM Client, we should be using HPC Resource ===>>> Use SSH ...  ===>>> useraccount

Also if I understand correctly; the RSM launcher should be started by the user on the Scheduler head node so to have its own session?

Please advice

Regards

Roberto

[Divya Shree Surendran]

Hi Dr. Roberto,

Selecting the Use SSH option will not require you to start the RSM launcher service on the Headnode, the job submission happens via SSH

[Roberto Scipioni]

Regarding the port 9212 does it need to be open inbound   (Client to HPC)  outbound (HPC to Client) or both?

On Azure CycleCloud Resource Groups we have independent NSG rules that may need to be updated.

Regards

Roberto

[Roberto Scipioni]

so then I only need port 22 (ssh) which is already open :-)

R.

[Divya Shree Surendran]

The 9212 port is only required to be open if RSM launcher service is running, which is not required in your case.

have you tried the Use SSH option and still seeing the error?

[Roberto Scipioni]

We will need to have session with the user and try again. Please keep the Ticket/Discussion open.

Will keep you posted.

Many thanks so far

[Divya Shree Surendran]

Sure Dr. Roberto, please feel free to let us know :)

[Roberto Scipioni]

Hi,

 

We had a session with the user using the ssh protocol.

The main error we got is one regarding the definition of an environment variable for the KEYPATH and a related Python error.

See attached figures.

 

Does that mean that we need to specify where the private ssh key is?

 

 

 

Does this mean that Ansys is looking for some files that are not found?

 

Please advice

 

Kind regards

Roberto

 

 

 

 

 

[Divya Shree Surendran]

Hi Dr. Roberto,

Please install putty on the user machine and check for SSH connection with the Azure cloud HPC headnode.

Can you try RSM Internal File transfer mechanism for the File management tab under RSM Configuration?

[Roberto Scipioni]

Hi,

Why would we need Putty?

Ansys is able to connect via ssh right? Please elaborate

"RSM Internal File transfer ...." we will try next time I guess ....

Kind regards

Roberto

 

 

 

 

[Roberto Scipioni]

The user was able to connect to the HPC using a standard Putty ssh connection using a standard password.

Regards

Roberto

[Divya Shree Surendran]

Hi Dr.Roberto,

Putty is to check the SSH connection for the credentials. RSM internal file transfer will be the easiest way here, please do give it a try.

[Roberto Scipioni]

Hi,

Sure, we will try it.

Thanks.

Kind regards

Roberto

[Roberto Scipioni]

Hi,

We may not have that option. I reckon this the window where the setting should be?

Please advice

Regards

R.

[Divya Shree Surendran]

Hi Dr.Roberto,

How are you plannin to transfer the project files into the staging directory mentioned above?

Is that staging director mapped to the Windows clients via a Samba or CIFS share?

[Roberto Scipioni]

If by staging directories you mean files/directories of the files that need to be present on the Remote HPC as well as the local machine, at the moment we do not mount

shared drives (we use NFS on the Azure HPC) on Windows machines. Indeed that would require a samba export from the HPC.

Are you saying that we need to have a drive/volume/directory accessible both from the local and the remote HPC machine?

Please advice

Roberto

[Divya Shree Surendran]

Hi Dr.Roberto,

You either need a Samba share that is accessible by both the Windows client and the HPC environment or you need to install Filezilla like secure copy client on Windows to transfer the files to the RSM staging directory path /shared/home/cy466

[Roberto Scipioni]

Hi,

Thanks.

Will continue tomorrow :-)

Kind regards

Roberto

[Divya Shree Surendran]

Thank you Dr.Roberto :)

[Roberto Scipioni]

Ok.

Using Filezilla. Can you elaborate on this?

How would you use it with the RSM? How would you tell Ansys to use it?

Would it not also WinSCP work?

Please advice

Roberto

[Roberto Scipioni]

...or you just mean to copy the required Ansys files to /shared/home/cy466  (or whatever is the chosen directory) Remote HPC for the submission to work?

Please advice

Roberto

[Divya Shree Surendran]

Hi Dr. Roberto, WinSCP or Filezilla are secure copy clients to copy your files via SSH - SCP into the RSM Staging directory on the cluster. This is outside RSM and hence you can say External File transfer for File management and use the clients for File copy.

If that is not preferred you can setup a Samba or CIFS share that enables your Windows clients to be mapped with the Staging directory

[Roberto Scipioni]

OK, I will inform the user and see what happens .....

Thanks

Roberto

[Roberto Scipioni]

Do we need to test the connection to avoid this error?

What is the meaning of KEYPATH?

You did not answer this before. Is Ansys looking for an ssh key pair or?

Please advice

Roberto

[Roberto Scipioni]

Hi,

So apparently we DO have the RSM internal transfer mechanism :-) (apologies)

How would we go about using it?

Does Ansys use an ssh channel or?

Please advice

Roberto

 

[Divya Shree Surendran]

Dr. Roberto,

RSM Internal file transfer can be used only if you have the RSM Launcher service running on the headnode of the cluster.

In your case, you mentioned you cant run the service, so go for No file transfer and use FIlezilla or WinSCP to transport your files OR use a samba share

Please stop confusing between the options

The KEYPATH error is due to External file transfer setting that requires SSH private key path saved - dont use this.

[Roberto Scipioni]

Hey.... I am just trying to understand here....

 

1) RSM internal transfer requires that the RSM launcher service running on the headnode:

    Fine:

           1a) As a standard user?

           1b) As a root?

           1c) What is the binary/utilities in a Linux environment that launches it?

                 I was looking for it and I could not find it.

I never said it cannot be used just that we need to understand how it works and whether it can be used.

 

2) Why is it that we cannot use the SSH private key path? Please elaborate. Does it require a special Windows environment variable/

 

3) Do you reckon the easiest option would be to just copy the available files to the remote HPC directory?

 

Thank you for your help

Regards

Roberto

 

 

 

 

 

 

 

[Divya Shree Surendran]

Dr.Roberto,

1. RSM launcher service is needed to be installed and runnin as a daemon on the headnode under root. If that is not possible you can  manually launch RSM launcher under the username. Refer to this link for the instructions, binaries, etc https://ansyshelp.ansys.com/account/secured?returnurl=/Views/Secured/corp/v241/en/wb_rsm/rsm_ch_config_rsm.html
2. Yes you need a KEYPATH variable pointing to the SSH private key setup for SSH between your Windows client and headnode of cluster - this is required only for External file transfer File management - not recommended
3. Easiest is go for No file transfer under File management and use Filezilla or Winscp to have your files pushed to the cluster staging path 

[Roberto Scipioni]

Point 3. You mean that the user should transfer the files before using RSM ? so not related to starting RSM at all right?

Regards

Roberto

[Divya Shree Surendran]

Dr. Roberto, yes

[Roberto Scipioni]

I imagine for the first option to work port 9212 needs to open and accessible right?

Please confirm

Regards

Roberto

[Divya Shree Surendran]

Dr.Roberto, yes

[Roberto Scipioni]

Is there a piece of documentaion that clearly states the usage of those ports?

I cant find any

Many thanks

Roberto

[Divya Shree Surendran]

Dr.Roberto,

You can refer to this link, but it gives a FAQ based suggestion for the port 9212 being used for RSM launcher. We do not have other documentation on the ports for RSM

https://ansyshelp.ansys.com/account/secured?returnurl=/Views/Secured/corp/v212/en/wb_rsm/rsm_trouble_rsm.html

[Roberto Scipioni]

Thanks.

We have contacted the Network team to have the port accessible.

As soon as it is done I will ask the user to test the connection (after the RSM launcher is started on the HPC)

Please keep the ticket open.

I will keep you informed

Kind regards

Roberto

[Roberto Scipioni]

Hi,

The Network team has indicated that the port has become available, however, we got the same error:

I think this may be related to credentials validation. Can you guide use on how the credentials should be loaded and/or cached?

Please advice

Roberto

[Divya Shree Surendran]

Hello Dr.Roberto,

Is the RSM launcher service running on nimbus.hpc.bath.ac.uk for 2021 R2? If so, are there multiple network IP addresses on the machine?

[Roberto Scipioni]

Yes I started 

rsmlauncher start

for the user cy466

It says it is running as a normal process.

 

Roberto

[Roberto Scipioni]

Hi,

 

Do we need a special config for users and/or to add credentials?

Regards

[Roberto Scipioni]

Perhaps the usahe of arccredentials ?

Regards

[Roberto Scipioni]

We have two login nodes users can connect to with two different IP adresses.

Would that represent a problem?

Please advice

Roberto

[Divya Shree Surendran]

Hi Dr. Roberto, no need for arccredentials - its only used if the scheduler is ARC

In your case its SLURM

Its the multiple IP address thats the issue.

Please edit Ans.RSM.Appsettings.config under the Install path of ANSYS on the cluster or you can follow this link to set RSM to point to the specific IP address of the headnode that can be reached by the clients.

https://ansyshelp.ansys.com/account/secured?returnurl=/Views/Secured/corp/v212/en/wb_rsm/wb_rsm_mult_nic.html

[Roberto Scipioni]

OK, thanks

Will try and let you know ...

Have a nice weekend

Kind regards

Roberto

[Roberto Scipioni]

Hi,

 

I have added both the login nodes IPs to the config using the rsmutils command.

I will ask the user to try again

Kind regards

Roberto

[Roberto Scipioni]

Hi there,

 

Just to make sure do we need to use this option?

Please advice?

Kind regards

Roberto

[Roberto Scipioni]

also ...it looks like now we need a way to see the drive mapped to the Windows Laptop.

This is an extract of the conversation with the Customer.

 

Any idea how to help?

Kind regards

 

Roberto

 

[Roberto Scipioni]

so the question would be:

1) Does the RSM Client need to access the /shared/home/cy466 directory on the remote host?

    I would guess so.

2) Does /shared/home/cy466 need to be "mounted/readable" on the local Windows machine?

 

Presumambly the user needs to have access to his remote home directory?

Please advice

 

Kind regards

Roberto

 

 

[Roberto Scipioni]

Hi,

 

Did you receive my last messag?

Any comment?

Please advice

Regards

Roberto

[Roberto Scipioni]

Dear Sir or Madam,

 

Upon using ssh we got this error again:

 

Any advice?

I am looking forward to hearing from you

 

Kind regards

Roberto

[Roberto Scipioni]

Hi there,

by reverting to using internal RSM and specifying the proper IP we got the error:

Any advice?

The RSM server was started on the HPC as a standard user cy466

Kind regards

Roberto

 

 

[Roberto Scipioni]

Dear Sir or Madam,

 

Is it related to the UserProxy ports that need to be available?

rsmutils appsettings set UserProxy PortRange

Please advice

Kind regards

Roberto

 

[Divya Shree Surendran]

Hi Roberto,

Sure, please ensure that the RSM Launcher service port 9212, User proxy port range are open on the Firewall after fixing the port range using the command that you mentioned.

[Roberto Scipioni]

We got the same error.

Frankly it is becoming ridicolous. We are spending so much time on this:

1) We have started the RSM on the remote machine as cy466

2) We have opened ports 9212,9213,9214

3) Specify the ports for UserProxy and SocketTransfer as 9213 and 9214 respectively.

The user has tried connecting and it did not work. Why do you create products that are so difificult to use/set ?

Can we get some help here?

 

Kind regards

Roberto

 

 

 

[Roberto Scipioni]

Now we got the error:

Any idea what this means?

Kind regards

Roberto

[Roberto Scipioni]

Dear Sir or Madam,

 

Would it be possible to organise a Team call with one of your Engineers so that we can have an online live session where we show what the user is trying to do?

Please advice

 

Kind regards

Roberto

[Roberto Scipioni]

Hi,

 

Did you my previous message? Can you please reply?

Does a range like 9213-9213 work or does it need at least two values like 9213-9214?

"PortRange" value="9213-9214"

Please advise

Kind regards

Roberto

[Roberto Scipioni]

Incidentally I can see there was a similar case that was NOT followed through!

/forum/forums/topic/rsm-failed-to-verify-my-credintials-when-connecting-to-the-cluster/

Please advise

Kind regards

Roberto

[Divya Shree Surendran]

Hi Roberto, Looks like the Userproxy ports are still being blocked, please ensure the port range set on the Ans.Rsm.AppSettings.config file for Userproxy is open on the Firewall for communication from client to the headnode

[Roberto Scipioni]

The networking team said that the ports are open, however, Ansys is not able to use them...

Any idea?

Kind regards

Roberto

[Roberto Scipioni]

Dear (by the way have you got a name ?)

 

We need to have a proper talk with engineers. Ansys is not able to accept the TCP ports which are open on the Remote machine.

Is there any debug/logs we could access to troubleshoot this?

This is becoming urgent!

please advice

Roberto

[Divya Shree Surendran]

Hi Roberto,

Please share your email and I can reach out to you via ANSYS email

[Roberto Scipioni]

roberto.scipioni@redoakconsulting.co.uk

Many thanks

Roberto

[Roberto Scipioni]

??????