Need to find out root cause Please help

[Mike]

Hi Team ,

Progress version :-9.1d
We had the situation that DB went down suddenly and we have checked with users that any abnormal termination from user . They replied no .Than we tried to find out the progress core file or protrace we did not find any core file in server .Seeking for root cause with below db logs . Anybody can help in this please.The abnormal shut down highlighted in Bold black. Please help me.

Wed Apr 12 08:29:09 202308:29:09 Usr 29: Login by phchuang on /dev/pts/tl. (452)08:29:14 Usr 30: Login by top01 on /dev/pts/tG. (452)08:30:21 Usr 31: Login by vxw0813 on /dev/pts/tH. (452)08:30:23 Usr 32: Login by top14 on /dev/pts/tI. (452)08:30:48 Usr 33: Login by lctangc on /dev/pts/tL. (452)08:30:51 Usr 34: Login by cslinc on /dev/pts/tK. (452)08:31:20 Usr 31: Logout by vxw0813 on /dev/pts/tH. (453)08:31:45 Usr 29: Logout by phchuang on /dev/pts/tl. (453)08:33:02 Usr 25: Logout by lnkuo on /dev/pts/tB. (453)08:33:18 Usr 25: Login by phchuang on /dev/pts/tl. (452)08:35:31 Usr 25: Logout by phchuang on /dev/pts/tl. (453)08:35:57 Usr 25: Login by phchuang on /dev/pts/tl. (452)08:36:23 Usr 29: Login by ricewuc on /dev/pts/tP. (452)08:37:45 Usr 25: Logout by phchuang on /dev/pts/tl. (453)08:38:00 Usr 25: Login by yxl0310c on /dev/pts/tS. (452)08:39:14 Usr 31: Login by cchsyuc on /dev/pts/tT. (452)08:40:24 Usr 35: Login by top02 on /dev/pts/tW. (452)08:41:37 Usr 36: Login by cxc1102 on /dev/pts/tY. (452)08:43:07 Usr 37: Login by top01 on /dev/pts/t1. (452)08:43:43 Usr 38: Login by kxm0917c on /dev/pts/t2. (452)08:45:39 Usr 39: Login by yxl0911a on /dev/pts/t6. (452)08:46:21 Usr 40: Login by top02 on /dev/pts/t8. (452)08:50:30 Usr 41: Login by vxw0813 on /dev/pts/tn. (452)08:50:32 Usr 42: Login by top01 on /dev/pts/t9. (452)08:54:39 Usr 19: Logout by cxc1112 on /dev/pts/tm. (453)08:56:23 Usr 39: Logout by yxl0911a on /dev/pts/t6. (453)08:58:13 Usr 19: Login by top01 on /dev/pts/teb. (452)08:58:57 Usr 39: Login by lnkuo on /dev/pts/tB. (452)09:00:01 Usr 43: Login by root on batch. (452)09:00:07 Usr 43: Logout by root on batch. (453)09:02:25 Usr 31: Logout by cchsyuc on /dev/pts/tT. (453)09:04:41 Usr 31: Login by lxl0821a on /dev/pts/tT. (452)09:05:28 Usr 43: Login by yxl0911a on /dev/pts/t6. (452)09:07:28 Usr 44: Login by vxw0813c on /dev/pts/tjb. (452)09:08:07 Usr 45: Login by cxc0508a on /dev/pts/tkb. (452)09:08:46 Usr 46: Login by top02 on /dev/pts/tmb. (452)09:09:50 Usr 47: Login by cxc1112 on /dev/pts/tob. (452)09:10:29 Usr 41: Logout by vxw0813 on /dev/pts/tn. (453)09:10:33 Usr 45: Logout by cxc0508a on /dev/pts/tkb. (453)09:11:07 Usr 41: Login by vxw0813 on /dev/pts/tH. (452)09:11:56 Usr 45: Login by axk1115c on /dev/pts/tqb. (452)09:12:08 Usr 12: HANGUP signal received. (562)09:12:08 Usr 12: Logout by phchuang on /dev/pts/td. (453)09:15:34 Usr 12: Login by top02 on /dev/pts/tdb. (452)09:17:44 Usr 33: HANGUP signal received. (562)09:17:44 Usr 33: Logout by lctangc on /dev/pts/tL. (453)09:20:00 Usr 33: Login by edi61adm on batch. (452)09:20:07 Usr 33: Logout by edi61adm on batch. (453)09:20:46 Usr 33: Login by top01 on /dev/pts/tL. (452)09:21:16 Usr 41: Logout by vxw0813 on /dev/pts/tH. (453)09:22:01 Usr 41: Login by top01 on /dev/pts/tib. (452)09:23:50 Usr 48: Login by yxl0325c on /dev/pts/trb. (452)09:24:28 Usr 48: HANGUP signal received. (562)09:24:28 Usr 48: Logout by yxl0325c on /dev/pts/trb. (453)09:25:06 Usr 48: Login by yxh1029 on /dev/pts/trb. (452)09:30:08 Usr 49: Login by top21 on /dev/pts/ttb. (452)09:30:32 Usr 50: Login by phchuang on /dev/pts/tl. (452)09:31:22 Usr 50: Logout by phchuang on /dev/pts/tl. (453)09:31:26 Usr 50: Login by cxc0508a on /dev/pts/tkb. (452)09:31:58 Usr 51: Login by ricewuc on /dev/pts/tub. (452)09:32:16 Usr 52: Login by cctuc on /dev/pts/tvb. (452)09:32:18 Usr 53: Login by top02 on /dev/pts/twb. (452)09:34:29 Usr 54: Login by vxt0413c on /dev/pts/tyb. (452)09:38:24 Usr 55: Login by top14 on /dev/pts/tzb. (452)09:39:38 Usr 39: Logout by lnkuo on /dev/pts/tB. (453)09:39:44 Usr 39: Login by lnkuo on /dev/pts/tB. (452)09:41:05 Usr 56: Login by sxy0315 on /dev/pts/tCb. (452)09:41:26 Usr 57: Login by top14 on /dev/pts/tFb. (452)09:42:49 Usr 43: Logout by yxl0911a on /dev/pts/t6. (453)09:44:29 Usr 52: HANGUP signal received. (562)09:44:30 Usr 52: Logout by cctuc on /dev/pts/tvb. (453)09:46:00 Usr 43: Login by phchuang on /dev/pts/tl. (452)09:46:05 Usr 52: Login by top02 on /dev/pts/tHb. (452)09:46:08 Usr 49: Logout by top21 on /dev/pts/ttb. (453)09:46:16 Usr 49: Login by cwlinc on /dev/pts/tIb. (452)09:46:23 Usr 43: Logout by phchuang on /dev/pts/tl. (453)09:46:56 Usr 49: Logout by cwlinc on /dev/pts/tIb. (453)09:47:10 Usr 51: Logout by ricewuc on /dev/pts/tub. (453)09:47:22 Usr 43: Login by vxw0813 on /dev/pts/tH. (452)09:47:45 Usr 49: Login by vxw0813 on /dev/pts/ttb. (452)09:47:55 Usr 43: Logout by vxw0813 on /dev/pts/tH. (453)09:48:25 Usr 43: Login by vxw0813 on /dev/pts/tH. (452)09:50:02 Usr 49: Logout by vxw0813 on /dev/pts/ttb. (453)09:50:21 Usr 49: Login by mctaic on /dev/pts/tJb. (452)09:51:27 Usr 43: Logout by vxw0813 on /dev/pts/tH. (453)09:54:34 Usr 44: Logout by vxw0813c on /dev/pts/tjb. (453)09:54:37 Usr 43: Login by vxw0813c on /dev/pts/tjb. (452)09:55:59 Usr 44: Login by top01 on /dev/pts/tLb. (452)09:57:27 Usr 51: Login by tcchangc on /dev/pts/tMb. (452)09:58:21 Usr 58: Login by cslinc on /dev/pts/tNb. (452)10:00:00 Usr 59: Login by root on batch. (452)10:00:07 Usr 59: Logout by root on batch. (453)10:01:44 Usr 51: Logout by tcchangc on /dev/pts/tMb. (453)10:06:26 Usr 49: HANGUP signal received. (562)10:06:26 Usr 49: Logout by mctaic on /dev/pts/tJb. (453)10:07:35 Usr 49: Login by cctuc on /dev/pts/tJb. (452)10:07:38 Usr 15: Logout by kxy0828c on /dev/pts/tg. (453)10:08:27 Usr 15: Login by wiwuc on /dev/pts/tg. (452)10:08:37 Usr 15: Logout by wiwuc on /dev/pts/tg. (453)10:09:58 Usr 15: Login by sxy0315 on /dev/pts/tPb. (452)10:10:26 Usr 15: Logout by sxy0315 on /dev/pts/tPb. (453)10:10:29 Usr 15: Login by yxl0310c on /dev/pts/tg. (452)10:10:47 Usr 31: HANGUP signal received. (562)10:10:47 Usr 31: Logout by lxl0821a on /dev/pts/tT. (453)10:11:08 Usr 31: Login by sxy0315 on /dev/pts/tOb. (452)10:11:55 Usr 31: Logout by sxy0315 on /dev/pts/tOb. (453)10:13:40 Usr 16: HANGUP signal received. (562)10:13:40 Usr 16: Logout by top02 on /dev/pts/th. (453)10:13:54 Usr 47: Logout by cxc1112 on /dev/pts/tob. (453)10:14:01 Usr 54: Logout by vxt0413c on /dev/pts/tyb. (453)10:15:52 Usr 16: Login by yxl0911a on /dev/pts/t6. (452)10:15:58 Usr 16: Logout by yxl0911a on /dev/pts/t6. (453)10:16:54 Usr 16: Login by cwlinc on /dev/pts/tob. (452)10:17:20 Usr 16: Logout by cwlinc on /dev/pts/tob. (453)10:19:22 Usr 16: Login by jshuc on /dev/pts/tyb. (452)10:19:58 Usr 31: Login by yxl0325c on /dev/pts/tOb. (452)10:20:52 Usr 31: Logout by yxl0325c on /dev/pts/tOb. (453)10:21:49 Usr 31: Login by vxw0813 on /dev/pts/tQb. (452)10:23:07 Usr 31: Logout by vxw0813 on /dev/pts/tQb. (453)10:23:30 Usr 31: Login by cxc1112 on /dev/pts/tp. (452)10:26:18 Usr 18: HANGUP signal received. (562)10:26:18 Usr 18: Logout by cctuc on /dev/pts/tk. (453)10:26:22 Usr 20: HANGUP signal received. (562)10:26:22 Usr 20: Logout by top01 on /dev/pts/tr. (453)10:26:29 Usr 21: HANGUP signal received. (562)10:26:29 Usr 21: Logout by yxl0325c on /dev/pts/ts. (453)10:27:17 Usr 18: Login by vxw0813 on /dev/pts/tr. (452)10:27:47 Usr 20: Login by kxy0828c on /dev/pts/ts. (452)10:27:53 Usr 18: Logout by vxw0813 on /dev/pts/tr. (453)10:28:32 Usr 18: Login by top02 on /dev/pts/tSb. (452)10:29:25 Usr 24: HANGUP signal received. (562)10:29:25 Usr 24: Logout by cchenc on /dev/pts/tA. (453)10:31:03 Usr 21: Login by phchuang on /dev/pts/tl. (452)10:36:18 Usr 50: SYSTEM ERROR: latch 16 depth -1. (3715)10:36:18 Usr 50: ** Save file named core for analysis by Progress Software Corporation. (439)10:36:18 APW 5: Stopped. (2520)10:36:18 BROKER 0: Begin ABNORMAL shutdown code 2 (2249)10:36:18 Usr 15: Begin transaction backout. (2252)10:36:18 Usr 31: Begin transaction backout. (2252)10:36:18 Usr 43: Begin transaction backout. (2252)10:36:18 Usr 25: Begin transaction backout. (2252)10:36:18 Usr 27: Begin transaction backout. (2252)10:36:18 Usr 18: Begin transaction backout. (2252)10:36:18 Usr 18: Transaction backout completed. (2253)10:36:18 Usr 39: Logout by lnkuo on /dev/pts/tB. (453)10:36:18 Usr 43: Transaction backout completed. (2253)10:36:18 Usr 25: Transaction backout completed. (2253)10:36:18 Usr 15: Transaction backout completed. (2253)10:36:18 Usr 31: Transaction backout completed. (2253)10:36:18 Usr 27: Transaction backout completed. (2253)10:36:48 BROKER 0: Sending signal 14 to 48 connected user(s). (2261)10:37:18 BROKER 0: Sending signal 2 to 48 connected user(s). (2261)10:37:49 BROKER 0: Resending shutdown request to 48 user(s). (2263)10:41:21 BROKER 0: Sending signal 14 to 48 connected user(s). (2261)10:41:51 BROKER 0: Sending signal 2 to 48 connected user(s). (2261)10:42:21 BROKER 0: Sending signal 15 to 48 connected user(s). (2261)10:42:53 Usr 21: Logout by phchuang on /dev/pts/tl. (453)10:43:23 BROKER 0: Destroyed user 6 pid 8493. (2251)10:43:23 BROKER 0: Destroyed user 7 pid 8433. (2251)10:43:23 BROKER 0: Destroyed user 8 pid 8554. (2251)10:43:23 BROKER 0: Destroyed u

 

[TomBascom]

Progress version :-9.1d

We had the situation that DB went down suddenly and we have checked with users that any abnormal termination from user. They replied "no". Than we tried to find out the progress core file or protrace we did not find any core file in server. Seeking for root cause with below db logs. Anybody can help in this please. The abnormal shut down highlighted in Bold black.

Please help me.

. . .
09:30:32 Usr 50: Login by phchuang on /dev/pts/tl. (452)
09:31:22 Usr 50: Logout by phchuang on /dev/pts/tl. (453)
. . .
09:31:26 Usr 50: Login by cxc0508a on /dev/pts/tkb. (452)
. . .
10:36:18 Usr 50: SYSTEM ERROR: latch 16 depth -1. (3715)
10:36:18 Usr 50: ** Save file named core for analysis by Progress Software Corporation. (439)
10:36:18 BROKER 0: Begin ABNORMAL shutdown code 2 (2249)

As you know, 9.1D is ancient, obsolete and unsupported and, as I recall, you are running it on an equally unsupported OS. Strange errors and unfixable bugs should be expected. You should have upgraded 20 years ago.

The first login/logout of Usr 50 is normal and has no relevance.

The second login of Usr 50 is the session that eventually brought down the database. You might try talking to user "cxc0508a" or looking in their home directory for any potential clues. Files with a time stamp of April 12, 2023 at 10:36:18 would be potentially interesting. You probably won't find anything and the user is probably innocent of wrong doing but if anyone knows anything from a user perspective it would be that user. Whatever happened is not the users fault - the point of asking them would be to try to gather data just in case the problem turns out to be repeatable. It is, for instance, just barely conceivable that "when I run X with parameter Y" the system always crashes. If you can find such a repeatable pattern then you might be able to do something about it - even if "something" is to simply tell people "don't do that".

None of the other logins or log file entries leading up to the crash are relevant.

At 10:36:18 Usr 50 experienced a problem with latch 16.

There is very little information in the kbase about such an error. The only entry that I found says that it happened to someone who had recently migrated from PA-RISC to Itanium and who was, apparently, still using the wrong license (which I take to mean the wrong executables since progress.cfg is platform neutral).

In version 9 latch 16 was probably MTL_LKF, aka the Lock Table Free List. I say "probably" because I don't have 9.1D running and I'm not going to dig it up and boot that image just to check. Also, knowing what the latch is doesn't usually help very much anyway.

So there are two possibilities:

1) With most latch related crashes what really happened was someone (or some script) used "kill -9" (or an equivalent) to terminate the process. When this occurs while a latch is being held there is no way for the process to be properly cleaned up and the database will shutdown as a result.

2) An error occurred that Progress was not able to handle, terminating the session in such a way that the latch was not properly released. Again this will cause the database to shutdown. Such an error would probably be considered a high priority bug if that version of Progress were still supported.