Learning Forum

[edu_ogando]

Hello,

   We have an academic floating license with 20 TDFT (old Lumerical) tokens for cluster computing. In our cluster, we have a Slurm workload manager.

   Sometimes, our cluster jobs appear duplicated in the license server, therefore we can use only halve of licenses, for example (lmstat output):

  mschnell atlas-142 atlas-142 148513 (v2021.0811) (lic13.adm.ehu.es/1055 2201), start Thu 10/21 6:17

  mschnell atlas-142 atlas-142 148513 (v2021.0811) (lic13.adm.ehu.es/1055 507), start Thu 10/21 6:27

  mschnell atlas-145 atlas-145 115758 (v2021.0811) (lic13.adm.ehu.es/1055 1850), start Thu 10/21 6:27

  mschnell atlas-145 atlas-145 115758 (v2021.0811) (lic13.adm.ehu.es/1055 624), start Thu 10/21 6:27

   There are only two real jobs in the cluster, but each of them uses two licenses. We attach the logs files of these jobs. It seems that they have lost connection with the license server or that it rebooted. The license served is managed by the IT central services of the university, we do not have direct access to it.

   Thank you for your attention.

Best regards,

Edu.

ansyscl.atlas-145.115758.90973.log

[Lito]

When the client machine loses connection with the license server while it is using the license, the license will not be "returned/checkin" to the license manager. The license manager sees the license is still "in use/checkout" by the machine from the previous connection/session. Once the machine is reconnected - this is now a new and different connection/session and a license checkout request will now occupy another/different license since the previous one is still "in use/checkout".
Try to: 1) Resolve the connection issue between the client machine and the license server. 2) Wait for 90 minutes before the license manager will automatically "release/checkin" the license if it does not receive a "checkpoint/sync" from the machine that checkout the license. This is the minimum "inactivity timeout" to "release/checkin" license.
Best Lito

[edu_ogando]

[edu_ogando]

Hello Thank you very much for explaining to me how it works. Regarding the checkpoint time, it would be OK if the licensing issue would be fixed, 90 minutes after the server reconnects. But, we believe the problem took more than 90 minutes to resolve. In fact, from the logs of our jobs, we would say that the checkpoints are realized every 6 hours:
(lines with "checkpoint" word filtered)
2021/10/19 17:45:00CHECKPOINT1/1/1/1
2021/10/19 17:45:00CHECKPOINT1/1/1/1
2021/10/19 23:45:00CHECKPOINT1/1/1/1
2021/10/19 23:45:00CHECKPOINT1/1/1/1
2021/10/20 05:45:00CHECKPOINT1/1/1/1
2021/10/20 05:45:00CHECKPOINT1/1/1/1
2021/10/20 11:45:00CHECKPOINT1/1/1/1
2021/10/20 11:45:00CHECKPOINT1/1/1/1
2021/10/20 17:45:00CHECKPOINT1/1/1/1
2021/10/20 17:45:00CHECKPOINT1/1/1/1
2021/10/20 23:45:00CHECKPOINT1/1/1/1
2021/10/20 23:45:03CHECKPOINT1/1/1/1
2021/10/21 05:45:00CHECKPOINT1/1/1/1
2021/10/21 05:45:00CHECKPOINT1/1/1/1
2021/10/21 11:45:00CHECKPOINT1/1/1/1
2021/10/21 11:45:02CHECKPOINT
Nevertheless, we do not know the exact meaning of these checkpoint messages in the logs.
Thank you very much.

[Lito]

, From where is this logs with the checkpoint? Is this from the client machine running Lumerical or the license server?

[edu_ogando]

Hello The log is from the client machine running Lumerical. The server is in the corporative computers of the university and, therefore, we do not have access to its management.
Regards Edu.

[Lito]

Sorry we have 2 "heartbeats".
(1) While the client machine is connected to the license manager is will send heartbeat every 6 hours after the last checkout "if it is still using the license". If the client detects a network issue it will sync and try to reconnect with the license manager. Otherwise, the simulation job might get terminated and will have to be run from the start.
(2) The license manager sends a heartbeat to the client machines 15 minutes after the clients' last checkout. Then 2 hours after - where It checks in the "Lumerical license feature" if there is no response. Sorry, I think the 90 minutes idle timeout is for some other Ansys products.
Hope this helps.
-Lito