Mike
Moderator
We had a situation where db went down we found db logs below:- We need to find RCA why and which user did this? and how to avoid in future this?
[2026/07/13@06:19:46.015-0500] P-17611 T-140394007668160 I AIMGT 22: (3778) This is after-image file number 213817 since the last AIMAGE BEGIN
[2026/07/13@06:19:51.016-0500] P-17611 T-140394007668160 I AIMGT 22: (13199) After-image extent /ai/prod/ipurch/ipurchprod.a6 has been copied to /backup/prod/aiarch/ipurch/db~prod~ipurch~ipurchprod.20200606.062533.00213816.ipurchprod.a6.
[2026/07/13@06:19:51.017-0500] P-17611 T-140394007668160 I AIMGT 22: (3789) Marked after-image extent /ai/prod/ipurch/ipurchprod.a6 EMPTY.
[2026/07/13@11:32:32.000+0000] P-12422 T-140278625640704 I WSAGENT28: (49) SYSTEM ERROR: Memory violation.
[2026/07/13@11:32:32.000+0000] P-12422 T-140278625640704 I WSAGENT28: (439) ** Save file named core for analysis by Progress Software Corporation.
[2026/07/13@11:32:32.000+0000] P-12422 T-140278625640704 I WSAGENT28: (-----) Generating /apps/ipurchase/work/webspeed/agents/protrace.12422
[2026/07/13@06:33:13.315-0500] P-17605 T-139854071501248 I WDOG 21: (2527) Disconnecting dead user 28.
[2026/07/13@11:33:37.000+0000] P-11762 T-140200292004096 I WSAGENT27: (49) SYSTEM ERROR: Memory violation.
[2026/07/13@11:33:37.000+0000] P-11762 T-140200292004096 I WSAGENT27: (439) ** Save file named core for analysis by Progress Software Corporation.
[2026/07/13@11:33:37.000+0000] P-11762 T-140200292004096 I WSAGENT27: (-----) Generating /apps/ipurchase/work/webspeed/agents/protrace.11762
[2026/07/13@06:33:39.559-0500] P-17338 T-140657448198400 I WSAGENT26: (452) Login by mfg on batch.
[2026/07/13@06:33:39.559-0500] P-17339 T-140297703747840 I WSAGENT28: (452) Login by mfg on batch.
[2026/07/13@06:33:39.559-0500] P-17334 T-140710392152320 I WSAGENT29: (452) Login by mfg on batch.
[2026/07/13@06:33:39.560-0500] P-17326 T-140587959419136 I WSAGENT30: (452) Login by mfg on batch.
[2026/07/13@06:33:39.561-0500] P-17340 T-139962508124416 I WSAGENT31: (452) Login by mfg on batch.
[2026/07/13@06:33:39.562-0500] P-17338 T-140657448198400 I WSAGENT26: (7129) Usr 26 set name to mfg.
[2026/07/13@06:33:39.562-0500] P-17331 T-140515380281600 I WSAGENT32: (452) Login by mfg on batch.
[2026/07/13@06:33:39.563-0500] P-17322 T-139780878946560 I WSAGENT33: (452) Login by mfg on batch.
[2026/07/13@06:33:39.564-0500] P-17339 T-140297703747840 I WSAGENT28: (7129) Usr 28 set name to mfg.
[2026/07/13@06:33:39.564-0500] P-17326 T-140587959419136 I WSAGENT30: (7129) Usr 30 set name to mfg.
[2026/07/13@06:33:39.565-0500] P-17340 T-139962508124416 I WSAGENT31: (7129) Usr 31 set name to mfg.
[2026/07/13@06:33:39.566-0500] P-17322 T-139780878946560 I WSAGENT33: (7129) Usr 33 set name to mfg.
[2026/07/13@06:33:39.572-0500] P-17345 T-139808252387584 I WSAGENT34: (452) Login by mfg on batch.
[2026/07/13@06:33:39.573-0500] P-17334 T-140710392152320 I WSAGENT29: (7129) Usr 29 set name to mfg.
[2026/07/13@06:33:39.576-0500] P-17345 T-139808252387584 I WSAGENT34: (7129) Usr 34 set name to mfg.
[2026/07/13@06:33:39.586-0500] P-17331 T-140515380281600 I WSAGENT32: (7129) Usr 32 set name to mfg.
[2026/07/13@06:33:39.593-0500] P-17351 T-140309285871872 I WSAGENT35: (452) Login by mfg on batch.
[2026/07/13@06:33:39.599-0500] P-17351 T-140309285871872 I WSAGENT35: (7129) Usr 35 set name to mfg.
[2026/07/13@06:33:39.835-0500] P-17326 T-140587959419136 I WSAGENT30: (-----) Received RECONNECT from WTB
[2026/07/13@06:33:39.835-0500] P-17331 T-140515380281600 I WSAGENT32: (-----) Received RECONNECT from WTB
[2026/07/13@06:33:39.837-0500] P-17340 T-139962508124416 I WSAGENT31: (-----) Received RECONNECT from WTB
[2026/07/13@06:33:39.838-0500] P-17338 T-140657448198400 I WSAGENT26: (-----) Received RECONNECT from WTB
[2026/07/13@06:33:39.841-0500] P-17334 T-140710392152320 I WSAGENT29: (-----) Received RECONNECT from WTB
[2026/07/13@06:33:39.842-0500] P-17339 T-140297703747840 I WSAGENT28: (-----) Received RECONNECT from WTB
[2026/07/13@06:33:39.845-0500] P-17322 T-139780878946560 I WSAGENT33: (-----) Received RECONNECT from WTB
[2026/07/13@06:33:39.849-0500] P-17345 T-139808252387584 I WSAGENT34: (-----) Received RECONNECT from WTB
[2026/07/13@06:33:39.894-0500] P-17351 T-140309285871872 I WSAGENT35: (-----) Received RECONNECT from WTB
[2026/07/13@06:33:40.563-0500] P-17354 T-140209112129792 I WSAGENT36: (452) Login by mfg on batch.
[2026/07/13@06:33:40.566-0500] P-17354 T-140209112129792 I WSAGENT36: (7129) Usr 36 set name to mfg.
[2026/07/13@06:33:40.790-0500] P-17354 T-140209112129792 I WSAGENT36: (-----) Received RECONNECT from WTB
[2026/07/13@06:34:23.321-0500] P-17605 T-139854071501248 I WDOG 21: (2527) Disconnecting dead user 27.
[2026/07/13@06:34:51.106-0500] P-17611 T-140394007668160 I AIMGT 22: (3777) Switched to ai extent /ai/prod/ipurch/ipurchprod.a2.
[2026/07/13@06:34:51.106-0500] P-17611 T-140394007668160 I AIMGT 22: (3778) This is after-image file number 213818 since the last AIMAGE BEGIN
[2026/07/13@06:34:56.114-0500] P-17611 T-140394007668160 I AIMGT 22: (13199) After-image extent /ai/prod/ipurch/ipurchprod.a1 has been copied to /backup/prod/aiarch/ipurch/db~prod~ipurch~ipurchprod.20200606.062533.00213817.ipurchprod.a1.
[2026/07/13@06:34:56.115-0500] P-17611 T-140394007668160 I AIMGT 22: (3789) Marked after-image extent /ai/prod/ipurch/ipurchprod.a1 EMPTY.
Mon Jul 13 06:35:46 2026
[2026/07/13@06:35:46.082-0500] P-17618 T-140465072349632 I APW 0: (2519) Disconnected.
[2026/07/13@06:35:46.082-0500] P-17618 T-140465072349632 I APW 0: (-----) user #: 0, usrinuse #: 0, (srvctl:srvctl) 0:0, latchId: 18
[2026/07/13@06:35:46.082-0500] P-17618 T-140465072349632 F APW 0: (-----) lock stack corrupt expected 18 was 0 sp 0
[2026/07/13@06:35:46.083-0500] P-17618 T-140465072349632 I APW 0: (439) ** Save file named core for analysis by Progress Software Corporation.
[2026/07/13@06:35:46.094-0500] P-17615 T-139779634287040 I BIW 0: (2520) Stopped.
[2026/07/13@06:35:46.103-0500] P-17612 T-139810855203264 I AIW 0: (-----) user #: 0, usrinuse #: 0, (srvctl:srvctl) 0:0, latchId: 14, bot: 1
[2026/07/13@06:35:46.104-0500] P-17612 T-139810855203264 F AIW 0: (-----) lock stack underflow 0
[2026/07/13@06:35:46.104-0500] P-17612 T-139810855203264 I AIW 0: (439) ** Save file named core for analysis by Progress Software Corporation.
[2026/07/13@06:35:46.119-0500] P-17611 T-140394007668160 I AIMGT 0: (2519) Disconnected.
[2026/07/13@11:35:46.000+0000] P-17354 T-140209112129792 I WSAGENT 0: (49) SYSTEM ERROR: Memory violation.
[2026/07/13@11:35:46.000+0000] P-17354 T-140209112129792 I WSAGENT 0: (439) ** Save file named core for analysis by Progress Software Corporation.
[2026/07/13@11:35:46.000+0000] P-17354 T-140209112129792 I WSAGENT 0: (-----) Generating /apps/ipurchase/work/webspeed/agents/protrace.17354
[2026/07/13@06:35:46.576-0500] P-17600 T-140099396777792 I BROKER 0: (15192) The database will complete shutdown within approximately 60 seconds.
[2026/07/13@06:35:46.577-0500] P-17600 T-140099396777792 I BROKER 0: (2249) Begin ABNORMAL shutdown code 2
[2026/07/13@06:35:53.324-0500] P-17605 T-139854071501248 I WDOG 0: (2519) Disconnected.
Mon Jul 13 11:36:53 2026
[2026/07/13@11:36:53.000+0000] P-17338 T-140657448198400 I WSAGENT 0: (49) SYSTEM ERROR: Memory violation.
[2026/07/13@11:36:53.000+0000] P-17338 T-140657448198400 I WSAGENT 0: (439) ** Save file named core for analysis by Progress Software Corporation.
[2026/07/13@11:36:53.000+0000] P-17338 T-140657448198400 I WSAGENT 0: (-----) Generating /apps/ipurchase/work/webspeed/agents/protrace.17338
[2026/07/13@06:19:46.015-0500] P-17611 T-140394007668160 I AIMGT 22: (3778) This is after-image file number 213817 since the last AIMAGE BEGIN
[2026/07/13@06:19:51.016-0500] P-17611 T-140394007668160 I AIMGT 22: (13199) After-image extent /ai/prod/ipurch/ipurchprod.a6 has been copied to /backup/prod/aiarch/ipurch/db~prod~ipurch~ipurchprod.20200606.062533.00213816.ipurchprod.a6.
[2026/07/13@06:19:51.017-0500] P-17611 T-140394007668160 I AIMGT 22: (3789) Marked after-image extent /ai/prod/ipurch/ipurchprod.a6 EMPTY.
[2026/07/13@11:32:32.000+0000] P-12422 T-140278625640704 I WSAGENT28: (49) SYSTEM ERROR: Memory violation.
[2026/07/13@11:32:32.000+0000] P-12422 T-140278625640704 I WSAGENT28: (439) ** Save file named core for analysis by Progress Software Corporation.
[2026/07/13@11:32:32.000+0000] P-12422 T-140278625640704 I WSAGENT28: (-----) Generating /apps/ipurchase/work/webspeed/agents/protrace.12422
[2026/07/13@06:33:13.315-0500] P-17605 T-139854071501248 I WDOG 21: (2527) Disconnecting dead user 28.
[2026/07/13@11:33:37.000+0000] P-11762 T-140200292004096 I WSAGENT27: (49) SYSTEM ERROR: Memory violation.
[2026/07/13@11:33:37.000+0000] P-11762 T-140200292004096 I WSAGENT27: (439) ** Save file named core for analysis by Progress Software Corporation.
[2026/07/13@11:33:37.000+0000] P-11762 T-140200292004096 I WSAGENT27: (-----) Generating /apps/ipurchase/work/webspeed/agents/protrace.11762
[2026/07/13@06:33:39.559-0500] P-17338 T-140657448198400 I WSAGENT26: (452) Login by mfg on batch.
[2026/07/13@06:33:39.559-0500] P-17339 T-140297703747840 I WSAGENT28: (452) Login by mfg on batch.
[2026/07/13@06:33:39.559-0500] P-17334 T-140710392152320 I WSAGENT29: (452) Login by mfg on batch.
[2026/07/13@06:33:39.560-0500] P-17326 T-140587959419136 I WSAGENT30: (452) Login by mfg on batch.
[2026/07/13@06:33:39.561-0500] P-17340 T-139962508124416 I WSAGENT31: (452) Login by mfg on batch.
[2026/07/13@06:33:39.562-0500] P-17338 T-140657448198400 I WSAGENT26: (7129) Usr 26 set name to mfg.
[2026/07/13@06:33:39.562-0500] P-17331 T-140515380281600 I WSAGENT32: (452) Login by mfg on batch.
[2026/07/13@06:33:39.563-0500] P-17322 T-139780878946560 I WSAGENT33: (452) Login by mfg on batch.
[2026/07/13@06:33:39.564-0500] P-17339 T-140297703747840 I WSAGENT28: (7129) Usr 28 set name to mfg.
[2026/07/13@06:33:39.564-0500] P-17326 T-140587959419136 I WSAGENT30: (7129) Usr 30 set name to mfg.
[2026/07/13@06:33:39.565-0500] P-17340 T-139962508124416 I WSAGENT31: (7129) Usr 31 set name to mfg.
[2026/07/13@06:33:39.566-0500] P-17322 T-139780878946560 I WSAGENT33: (7129) Usr 33 set name to mfg.
[2026/07/13@06:33:39.572-0500] P-17345 T-139808252387584 I WSAGENT34: (452) Login by mfg on batch.
[2026/07/13@06:33:39.573-0500] P-17334 T-140710392152320 I WSAGENT29: (7129) Usr 29 set name to mfg.
[2026/07/13@06:33:39.576-0500] P-17345 T-139808252387584 I WSAGENT34: (7129) Usr 34 set name to mfg.
[2026/07/13@06:33:39.586-0500] P-17331 T-140515380281600 I WSAGENT32: (7129) Usr 32 set name to mfg.
[2026/07/13@06:33:39.593-0500] P-17351 T-140309285871872 I WSAGENT35: (452) Login by mfg on batch.
[2026/07/13@06:33:39.599-0500] P-17351 T-140309285871872 I WSAGENT35: (7129) Usr 35 set name to mfg.
[2026/07/13@06:33:39.835-0500] P-17326 T-140587959419136 I WSAGENT30: (-----) Received RECONNECT from WTB
[2026/07/13@06:33:39.835-0500] P-17331 T-140515380281600 I WSAGENT32: (-----) Received RECONNECT from WTB
[2026/07/13@06:33:39.837-0500] P-17340 T-139962508124416 I WSAGENT31: (-----) Received RECONNECT from WTB
[2026/07/13@06:33:39.838-0500] P-17338 T-140657448198400 I WSAGENT26: (-----) Received RECONNECT from WTB
[2026/07/13@06:33:39.841-0500] P-17334 T-140710392152320 I WSAGENT29: (-----) Received RECONNECT from WTB
[2026/07/13@06:33:39.842-0500] P-17339 T-140297703747840 I WSAGENT28: (-----) Received RECONNECT from WTB
[2026/07/13@06:33:39.845-0500] P-17322 T-139780878946560 I WSAGENT33: (-----) Received RECONNECT from WTB
[2026/07/13@06:33:39.849-0500] P-17345 T-139808252387584 I WSAGENT34: (-----) Received RECONNECT from WTB
[2026/07/13@06:33:39.894-0500] P-17351 T-140309285871872 I WSAGENT35: (-----) Received RECONNECT from WTB
[2026/07/13@06:33:40.563-0500] P-17354 T-140209112129792 I WSAGENT36: (452) Login by mfg on batch.
[2026/07/13@06:33:40.566-0500] P-17354 T-140209112129792 I WSAGENT36: (7129) Usr 36 set name to mfg.
[2026/07/13@06:33:40.790-0500] P-17354 T-140209112129792 I WSAGENT36: (-----) Received RECONNECT from WTB
[2026/07/13@06:34:23.321-0500] P-17605 T-139854071501248 I WDOG 21: (2527) Disconnecting dead user 27.
[2026/07/13@06:34:51.106-0500] P-17611 T-140394007668160 I AIMGT 22: (3777) Switched to ai extent /ai/prod/ipurch/ipurchprod.a2.
[2026/07/13@06:34:51.106-0500] P-17611 T-140394007668160 I AIMGT 22: (3778) This is after-image file number 213818 since the last AIMAGE BEGIN
[2026/07/13@06:34:56.114-0500] P-17611 T-140394007668160 I AIMGT 22: (13199) After-image extent /ai/prod/ipurch/ipurchprod.a1 has been copied to /backup/prod/aiarch/ipurch/db~prod~ipurch~ipurchprod.20200606.062533.00213817.ipurchprod.a1.
[2026/07/13@06:34:56.115-0500] P-17611 T-140394007668160 I AIMGT 22: (3789) Marked after-image extent /ai/prod/ipurch/ipurchprod.a1 EMPTY.
Mon Jul 13 06:35:46 2026
[2026/07/13@06:35:46.082-0500] P-17618 T-140465072349632 I APW 0: (2519) Disconnected.
[2026/07/13@06:35:46.082-0500] P-17618 T-140465072349632 I APW 0: (-----) user #: 0, usrinuse #: 0, (srvctl:srvctl) 0:0, latchId: 18
[2026/07/13@06:35:46.082-0500] P-17618 T-140465072349632 F APW 0: (-----) lock stack corrupt expected 18 was 0 sp 0
[2026/07/13@06:35:46.083-0500] P-17618 T-140465072349632 I APW 0: (439) ** Save file named core for analysis by Progress Software Corporation.
[2026/07/13@06:35:46.094-0500] P-17615 T-139779634287040 I BIW 0: (2520) Stopped.
[2026/07/13@06:35:46.103-0500] P-17612 T-139810855203264 I AIW 0: (-----) user #: 0, usrinuse #: 0, (srvctl:srvctl) 0:0, latchId: 14, bot: 1
[2026/07/13@06:35:46.104-0500] P-17612 T-139810855203264 F AIW 0: (-----) lock stack underflow 0
[2026/07/13@06:35:46.104-0500] P-17612 T-139810855203264 I AIW 0: (439) ** Save file named core for analysis by Progress Software Corporation.
[2026/07/13@06:35:46.119-0500] P-17611 T-140394007668160 I AIMGT 0: (2519) Disconnected.
[2026/07/13@11:35:46.000+0000] P-17354 T-140209112129792 I WSAGENT 0: (49) SYSTEM ERROR: Memory violation.
[2026/07/13@11:35:46.000+0000] P-17354 T-140209112129792 I WSAGENT 0: (439) ** Save file named core for analysis by Progress Software Corporation.
[2026/07/13@11:35:46.000+0000] P-17354 T-140209112129792 I WSAGENT 0: (-----) Generating /apps/ipurchase/work/webspeed/agents/protrace.17354
[2026/07/13@06:35:46.576-0500] P-17600 T-140099396777792 I BROKER 0: (15192) The database will complete shutdown within approximately 60 seconds.
[2026/07/13@06:35:46.577-0500] P-17600 T-140099396777792 I BROKER 0: (2249) Begin ABNORMAL shutdown code 2
[2026/07/13@06:35:53.324-0500] P-17605 T-139854071501248 I WDOG 0: (2519) Disconnected.
Mon Jul 13 11:36:53 2026
[2026/07/13@11:36:53.000+0000] P-17338 T-140657448198400 I WSAGENT 0: (49) SYSTEM ERROR: Memory violation.
[2026/07/13@11:36:53.000+0000] P-17338 T-140657448198400 I WSAGENT 0: (439) ** Save file named core for analysis by Progress Software Corporation.
[2026/07/13@11:36:53.000+0000] P-17338 T-140657448198400 I WSAGENT 0: (-----) Generating /apps/ipurchase/work/webspeed/agents/protrace.17338