Need to find RCA even though .lk file exist Application team wont able to connect via ERP.

1:- We have progress version 11.2 and OS version :-Hp-UX

Yesterday there was an issue with database or server . User not able to connect database ERP CHUI mode and showing black screen. We login to the server and went to the db path where we see .lk file exist .We tried to do Promon and db and it did not work and all looks hang .We tried to stop database manually with -by command but commands did not work. We tried emergency shutdown as well but did not Finally, we reboot the server and remove the .lk file and start Db and it worked. We just want to know what the root cause for this issue was below is log file, Please help us, Please find the below log file attached. Please help us ASAP

Most likely, a login semaphore or USR latch was locked.
promon -F would connect the database (like proshut -F did) and would allow to find out a process that hold the USR latch.
If it was the login semaphore, then OS commands would help.

BTW, 11.7.10 update provides
a. Additional Usr latch protection at logout and login,
b. Additional checking for USR latch at user disconnect
c. Additional information is written to the database lg file should these be encountered

This information is recorded in the following public Articles:

000154587, USR latch remains after user logs out preventing shutdown
Progress Customer Community

000202064, When a PROMON -F session is disconnected the database sometimes becomes connectionless and won't shutdown
Progress Customer Community

Plus, promon since 11.7.10 has some secret (undocumented) feature to debug the issues with a forgotten USR latch.

George Potemkin said:

BTW, 11.7.10 update provides
a. Additional Usr latch protection at logout and login,
b. Additional checking for USR latch at user disconnect
c. Additional information is written to the database lg file should these be encountered

This information is recorded in the following public Articles:

000154587, USR latch remains after user logs out preventing shutdown
Progress Customer Community

000202064, When a PROMON -F session is disconnected the database sometimes becomes connectionless and won't shutdown
Progress Customer Community

Plus, promon since 11.7.10 has some secret (undocumented) feature to debug the issues with a forgotten USR latch.

Click to expand...

Hi George ,

Thanks you for your response on this issue. But I am seeking the root cause that what happened on that day? I tried -by option and even emergency shut down but nothing worked and users wont able to connect with Chui and problem resolved soon after server reboot and db start up after deleting .lk file.

Thanks and Regards
Mike

Can you please check the db log file and let me know something for RCA?

Did you translate that .lg file somehow or other? There are a LOT of messages whose text is strangely worded.

TomBascom said:

Did you translate that .lg file somehow or other? There are a LOT of messages whose text is strangely worded.

Click to expand...

Hi Tom,

Yes I did translation from Spanish to English. Request you to please help in this .

Thanks and regards
Mike

Did you translate that .lg file somehow or other? There are a LOT of messages whose text is strangely worded.

Hi Tom,

Please help me to find out root cause please?

Mike said:

But I am seeking the root cause that what happened on that day?

Click to expand...

Some self-service process tried to log out.
It took the USR latch and freed the slot in Connection Table.
Then something gone wrong - the process hung and did not release the latch. I bet the process was still running while you tried to stop the database.
For such situations watchdog can't release the USR latch lock because 1) the latch owner is not in Connection Table; 2) the lock owner is still alive.
All processes (except promon/proshut started with the -F option) needs to lock USR latch for login/logout. That is why your log fragment does not contain the login/logout messages (except for SHUT 17) - from 03:49:00 to 09:26:04.
Find out the last login/logout message. Is it a logout of self-service process? Then it's a process-troublemaker.

Since 11.7.10 it's possible to release the locked USR latch without db shutdown but Progress does not like this option to be publicly available. That is why I can't share the details.

The issue is extremely rare but our customers had it a few times.

Long time ago I wrote a script (logsem.sh) to read the database login semaphore. It was tested only on Linux and AIX but not on HP-UX. I don't know if HP-UX has the commands to get the requested information but why not? When nobody can't connect a database it's good idea to check what is going on with a login semaphore of this database. There are no Progress utilities to do such things.