Error Request for Progress Review – ipurch DB Abnormal Shutdown with WebSpeed Memory Violation and APW/AIW Lock Stack Errors

The probability that the cause is rcode corruption is, IMHO, very low. what is IMHO very low?Are these rcode developed by applicaiton team or default?
 
You're running Webspeed. It looks like these are shared memory Webspeed clients. The symptoms you are seeing are looking incredibly similar to ones I was experiencing.

The cause of mine was a program that was tidying up handles for persistent procedures. It walked the widget tree and was finding handles for persistent procedures it didn't own and was deleting them. This caused symptoms VERY similar to the ones you are using. The solution is to upgrade to the latest service pack. I'm not sure how many more times we should need to say this. It's essential. It's not just recommended. The other part of this is to make sure that programs that create persistent procedures also delete them. This is a code issue. The FINALLY statement can be used to clear up things in all cases, including error states (which was what caused our issues).

Slight aside: Are you using PDF Include by any chance? We are and that was what caused the errors - the PDF Include process wasn't being cleared up when a program errored.
 
Mike said:
This issue is occurring only with this database. We have around 12 databases running on the same server with the same OpenEdge version, and they are not experiencing any similar issue.
Click to expand...
Does each client connect to all these databases?
Is ipurchprod the first database the clients connect to?
 
Mike said:
We had a situation where db went down we found db logs below:- We need to find RCA why and which user did this? and how to avoid in future this?
[2026/07/13@06:19:46.015-0500] P-17611 T-140394007668160 I AIMGT 22: (3778) This is after-image file number 213817 since the last AIMAGE BEGIN
[2026/07/13@06:19:51.016-0500] P-17611 T-140394007668160 I AIMGT 22: (13199) After-image extent /ai/prod/ipurch/ipurchprod.a6 has been copied to /backup/prod/aiarch/ipurch/db~prod~ipurch~ipurchprod.20200606.062533.00213816.ipurchprod.a6.
[2026/07/13@06:19:51.017-0500] P-17611 T-140394007668160 I AIMGT 22: (3789) Marked after-image extent /ai/prod/ipurch/ipurchprod.a6 EMPTY.
[2026/07/13@11:32:32.000+0000] P-12422 T-140278625640704 I WSAGENT28: (49) SYSTEM ERROR: Memory violation.
[2026/07/13@11:32:32.000+0000] P-12422 T-140278625640704 I WSAGENT28: (439) ** Save file named core for analysis by Progress Software Corporation.
[2026/07/13@11:32:32.000+0000] P-12422 T-140278625640704 I WSAGENT28: (-----) Generating /apps/ipurchase/work/webspeed/agents/protrace.12422
[2026/07/13@06:33:13.315-0500] P-17605 T-139854071501248 I WDOG 21: (2527) Disconnecting dead user 28.
[2026/07/13@11:33:37.000+0000] P-11762 T-140200292004096 I WSAGENT27: (49) SYSTEM ERROR: Memory violation.
[2026/07/13@11:33:37.000+0000] P-11762 T-140200292004096 I WSAGENT27: (439) ** Save file named core for analysis by Progress Software Corporation.
[2026/07/13@11:33:37.000+0000] P-11762 T-140200292004096 I WSAGENT27: (-----) Generating /apps/ipurchase/work/webspeed/agents/protrace.11762
[2026/07/13@06:33:39.559-0500] P-17338 T-140657448198400 I WSAGENT26: (452) Login by mfg on batch.
[2026/07/13@06:33:39.559-0500] P-17339 T-140297703747840 I WSAGENT28: (452) Login by mfg on batch.
[2026/07/13@06:33:39.559-0500] P-17334 T-140710392152320 I WSAGENT29: (452) Login by mfg on batch.
[2026/07/13@06:33:39.560-0500] P-17326 T-140587959419136 I WSAGENT30: (452) Login by mfg on batch.
[2026/07/13@06:33:39.561-0500] P-17340 T-139962508124416 I WSAGENT31: (452) Login by mfg on batch.
[2026/07/13@06:33:39.562-0500] P-17338 T-140657448198400 I WSAGENT26: (7129) Usr 26 set name to mfg.
[2026/07/13@06:33:39.562-0500] P-17331 T-140515380281600 I WSAGENT32: (452) Login by mfg on batch.
[2026/07/13@06:33:39.563-0500] P-17322 T-139780878946560 I WSAGENT33: (452) Login by mfg on batch.
[2026/07/13@06:33:39.564-0500] P-17339 T-140297703747840 I WSAGENT28: (7129) Usr 28 set name to mfg.
[2026/07/13@06:33:39.564-0500] P-17326 T-140587959419136 I WSAGENT30: (7129) Usr 30 set name to mfg.
[2026/07/13@06:33:39.565-0500] P-17340 T-139962508124416 I WSAGENT31: (7129) Usr 31 set name to mfg.
[2026/07/13@06:33:39.566-0500] P-17322 T-139780878946560 I WSAGENT33: (7129) Usr 33 set name to mfg.
[2026/07/13@06:33:39.572-0500] P-17345 T-139808252387584 I WSAGENT34: (452) Login by mfg on batch.
[2026/07/13@06:33:39.573-0500] P-17334 T-140710392152320 I WSAGENT29: (7129) Usr 29 set name to mfg.
[2026/07/13@06:33:39.576-0500] P-17345 T-139808252387584 I WSAGENT34: (7129) Usr 34 set name to mfg.
[2026/07/13@06:33:39.586-0500] P-17331 T-140515380281600 I WSAGENT32: (7129) Usr 32 set name to mfg.
[2026/07/13@06:33:39.593-0500] P-17351 T-140309285871872 I WSAGENT35: (452) Login by mfg on batch.
[2026/07/13@06:33:39.599-0500] P-17351 T-140309285871872 I WSAGENT35: (7129) Usr 35 set name to mfg.
[2026/07/13@06:33:39.835-0500] P-17326 T-140587959419136 I WSAGENT30: (-----) Received RECONNECT from WTB
[2026/07/13@06:33:39.835-0500] P-17331 T-140515380281600 I WSAGENT32: (-----) Received RECONNECT from WTB
[2026/07/13@06:33:39.837-0500] P-17340 T-139962508124416 I WSAGENT31: (-----) Received RECONNECT from WTB
[2026/07/13@06:33:39.838-0500] P-17338 T-140657448198400 I WSAGENT26: (-----) Received RECONNECT from WTB
[2026/07/13@06:33:39.841-0500] P-17334 T-140710392152320 I WSAGENT29: (-----) Received RECONNECT from WTB
[2026/07/13@06:33:39.842-0500] P-17339 T-140297703747840 I WSAGENT28: (-----) Received RECONNECT from WTB
[2026/07/13@06:33:39.845-0500] P-17322 T-139780878946560 I WSAGENT33: (-----) Received RECONNECT from WTB
[2026/07/13@06:33:39.849-0500] P-17345 T-139808252387584 I WSAGENT34: (-----) Received RECONNECT from WTB
[2026/07/13@06:33:39.894-0500] P-17351 T-140309285871872 I WSAGENT35: (-----) Received RECONNECT from WTB
[2026/07/13@06:33:40.563-0500] P-17354 T-140209112129792 I WSAGENT36: (452) Login by mfg on batch.
[2026/07/13@06:33:40.566-0500] P-17354 T-140209112129792 I WSAGENT36: (7129) Usr 36 set name to mfg.
[2026/07/13@06:33:40.790-0500] P-17354 T-140209112129792 I WSAGENT36: (-----) Received RECONNECT from WTB
[2026/07/13@06:34:23.321-0500] P-17605 T-139854071501248 I WDOG 21: (2527) Disconnecting dead user 27.
[2026/07/13@06:34:51.106-0500] P-17611 T-140394007668160 I AIMGT 22: (3777) Switched to ai extent /ai/prod/ipurch/ipurchprod.a2.
[2026/07/13@06:34:51.106-0500] P-17611 T-140394007668160 I AIMGT 22: (3778) This is after-image file number 213818 since the last AIMAGE BEGIN
[2026/07/13@06:34:56.114-0500] P-17611 T-140394007668160 I AIMGT 22: (13199) After-image extent /ai/prod/ipurch/ipurchprod.a1 has been copied to /backup/prod/aiarch/ipurch/db~prod~ipurch~ipurchprod.20200606.062533.00213817.ipurchprod.a1.
[2026/07/13@06:34:56.115-0500] P-17611 T-140394007668160 I AIMGT 22: (3789) Marked after-image extent /ai/prod/ipurch/ipurchprod.a1 EMPTY.

Mon Jul 13 06:35:46 2026
[2026/07/13@06:35:46.082-0500] P-17618 T-140465072349632 I APW 0: (2519) Disconnected.
[2026/07/13@06:35:46.082-0500] P-17618 T-140465072349632 I APW 0: (-----) user #: 0, usrinuse #: 0, (srvctl:srvctl) 0:0, latchId: 18
[2026/07/13@06:35:46.082-0500] P-17618 T-140465072349632 F APW 0: (-----) lock stack corrupt expected 18 was 0 sp 0
[2026/07/13@06:35:46.083-0500] P-17618 T-140465072349632 I APW 0: (439) ** Save file named core for analysis by Progress Software Corporation.
[2026/07/13@06:35:46.094-0500] P-17615 T-139779634287040 I BIW 0: (2520) Stopped.
[2026/07/13@06:35:46.103-0500] P-17612 T-139810855203264 I AIW 0: (-----) user #: 0, usrinuse #: 0, (srvctl:srvctl) 0:0, latchId: 14, bot: 1
[2026/07/13@06:35:46.104-0500] P-17612 T-139810855203264 F AIW 0: (-----) lock stack underflow 0
[2026/07/13@06:35:46.104-0500] P-17612 T-139810855203264 I AIW 0: (439) ** Save file named core for analysis by Progress Software Corporation.
[2026/07/13@06:35:46.119-0500] P-17611 T-140394007668160 I AIMGT 0: (2519) Disconnected.
[2026/07/13@11:35:46.000+0000] P-17354 T-140209112129792 I WSAGENT 0: (49) SYSTEM ERROR: Memory violation.
[2026/07/13@11:35:46.000+0000] P-17354 T-140209112129792 I WSAGENT 0: (439) ** Save file named core for analysis by Progress Software Corporation.
[2026/07/13@11:35:46.000+0000] P-17354 T-140209112129792 I WSAGENT 0: (-----) Generating /apps/ipurchase/work/webspeed/agents/protrace.17354
[2026/07/13@06:35:46.576-0500] P-17600 T-140099396777792 I BROKER 0: (15192) The database will complete shutdown within approximately 60 seconds.
[2026/07/13@06:35:46.577-0500] P-17600 T-140099396777792 I BROKER 0: (2249) Begin ABNORMAL shutdown code 2
[2026/07/13@06:35:53.324-0500] P-17605 T-139854071501248 I WDOG 0: (2519) Disconnected.

Mon Jul 13 11:36:53 2026
[2026/07/13@11:36:53.000+0000] P-17338 T-140657448198400 I WSAGENT 0: (49) SYSTEM ERROR: Memory violation.
[2026/07/13@11:36:53.000+0000] P-17338 T-140657448198400 I WSAGENT 0: (439) ** Save file named core for analysis by Progress Software Corporation.
[2026/07/13@11:36:53.000+0000] P-17338 T-140657448198400 I WSAGENT 0: (-----) Generating /apps/ipurchase/work/webspeed/agents/protrace.17338
Click to expand...
The combination of SYSTEM ERROR: Memory violation (49), abnormal database shutdown, and APW/AIW lock stack errors usually indicates that a process crashed while holding shared memory or buffer latches, forcing OpenEdge to shut the database down to protect data integrity. I would first check the generated protrace/core files, verify the OpenEdge version (older releases contain several memory-violation fixes), and review any OS events, antivirus activity, Java updates, or third-party processes running at the same time. If this is reproducible, I'd recommend opening a Progress support case with the protrace, database log, and startup parameters for root cause analysis.
 
You must log in or register to reply here.

Similar threads

Mike
  • Locked
Error Request for Progress Review – ipurch DB Abnormal Shutdown with WebSpeed Memory Violation and APW/AIW Lock Stack Errors
Replies
0
Views
359
Mike
Mike
Back
Top