Error Request for Progress Review – ipurch DB Abnormal Shutdown with WebSpeed Memory Violation and APW/AIW Lock Stack Errors

Mike

Mike

Moderator
We had a situation where db went down we found db logs below:- We need to find RCA why and which user did this? and how to avoid in future this?
[2026/07/13@06:19:46.015-0500] P-17611 T-140394007668160 I AIMGT 22: (3778) This is after-image file number 213817 since the last AIMAGE BEGIN
[2026/07/13@06:19:51.016-0500] P-17611 T-140394007668160 I AIMGT 22: (13199) After-image extent /ai/prod/ipurch/ipurchprod.a6 has been copied to /backup/prod/aiarch/ipurch/db~prod~ipurch~ipurchprod.20200606.062533.00213816.ipurchprod.a6.
[2026/07/13@06:19:51.017-0500] P-17611 T-140394007668160 I AIMGT 22: (3789) Marked after-image extent /ai/prod/ipurch/ipurchprod.a6 EMPTY.
[2026/07/13@11:32:32.000+0000] P-12422 T-140278625640704 I WSAGENT28: (49) SYSTEM ERROR: Memory violation.
[2026/07/13@11:32:32.000+0000] P-12422 T-140278625640704 I WSAGENT28: (439) ** Save file named core for analysis by Progress Software Corporation.
[2026/07/13@11:32:32.000+0000] P-12422 T-140278625640704 I WSAGENT28: (-----) Generating /apps/ipurchase/work/webspeed/agents/protrace.12422
[2026/07/13@06:33:13.315-0500] P-17605 T-139854071501248 I WDOG 21: (2527) Disconnecting dead user 28.
[2026/07/13@11:33:37.000+0000] P-11762 T-140200292004096 I WSAGENT27: (49) SYSTEM ERROR: Memory violation.
[2026/07/13@11:33:37.000+0000] P-11762 T-140200292004096 I WSAGENT27: (439) ** Save file named core for analysis by Progress Software Corporation.
[2026/07/13@11:33:37.000+0000] P-11762 T-140200292004096 I WSAGENT27: (-----) Generating /apps/ipurchase/work/webspeed/agents/protrace.11762
[2026/07/13@06:33:39.559-0500] P-17338 T-140657448198400 I WSAGENT26: (452) Login by mfg on batch.
[2026/07/13@06:33:39.559-0500] P-17339 T-140297703747840 I WSAGENT28: (452) Login by mfg on batch.
[2026/07/13@06:33:39.559-0500] P-17334 T-140710392152320 I WSAGENT29: (452) Login by mfg on batch.
[2026/07/13@06:33:39.560-0500] P-17326 T-140587959419136 I WSAGENT30: (452) Login by mfg on batch.
[2026/07/13@06:33:39.561-0500] P-17340 T-139962508124416 I WSAGENT31: (452) Login by mfg on batch.
[2026/07/13@06:33:39.562-0500] P-17338 T-140657448198400 I WSAGENT26: (7129) Usr 26 set name to mfg.
[2026/07/13@06:33:39.562-0500] P-17331 T-140515380281600 I WSAGENT32: (452) Login by mfg on batch.
[2026/07/13@06:33:39.563-0500] P-17322 T-139780878946560 I WSAGENT33: (452) Login by mfg on batch.
[2026/07/13@06:33:39.564-0500] P-17339 T-140297703747840 I WSAGENT28: (7129) Usr 28 set name to mfg.
[2026/07/13@06:33:39.564-0500] P-17326 T-140587959419136 I WSAGENT30: (7129) Usr 30 set name to mfg.
[2026/07/13@06:33:39.565-0500] P-17340 T-139962508124416 I WSAGENT31: (7129) Usr 31 set name to mfg.
[2026/07/13@06:33:39.566-0500] P-17322 T-139780878946560 I WSAGENT33: (7129) Usr 33 set name to mfg.
[2026/07/13@06:33:39.572-0500] P-17345 T-139808252387584 I WSAGENT34: (452) Login by mfg on batch.
[2026/07/13@06:33:39.573-0500] P-17334 T-140710392152320 I WSAGENT29: (7129) Usr 29 set name to mfg.
[2026/07/13@06:33:39.576-0500] P-17345 T-139808252387584 I WSAGENT34: (7129) Usr 34 set name to mfg.
[2026/07/13@06:33:39.586-0500] P-17331 T-140515380281600 I WSAGENT32: (7129) Usr 32 set name to mfg.
[2026/07/13@06:33:39.593-0500] P-17351 T-140309285871872 I WSAGENT35: (452) Login by mfg on batch.
[2026/07/13@06:33:39.599-0500] P-17351 T-140309285871872 I WSAGENT35: (7129) Usr 35 set name to mfg.
[2026/07/13@06:33:39.835-0500] P-17326 T-140587959419136 I WSAGENT30: (-----) Received RECONNECT from WTB
[2026/07/13@06:33:39.835-0500] P-17331 T-140515380281600 I WSAGENT32: (-----) Received RECONNECT from WTB
[2026/07/13@06:33:39.837-0500] P-17340 T-139962508124416 I WSAGENT31: (-----) Received RECONNECT from WTB
[2026/07/13@06:33:39.838-0500] P-17338 T-140657448198400 I WSAGENT26: (-----) Received RECONNECT from WTB
[2026/07/13@06:33:39.841-0500] P-17334 T-140710392152320 I WSAGENT29: (-----) Received RECONNECT from WTB
[2026/07/13@06:33:39.842-0500] P-17339 T-140297703747840 I WSAGENT28: (-----) Received RECONNECT from WTB
[2026/07/13@06:33:39.845-0500] P-17322 T-139780878946560 I WSAGENT33: (-----) Received RECONNECT from WTB
[2026/07/13@06:33:39.849-0500] P-17345 T-139808252387584 I WSAGENT34: (-----) Received RECONNECT from WTB
[2026/07/13@06:33:39.894-0500] P-17351 T-140309285871872 I WSAGENT35: (-----) Received RECONNECT from WTB
[2026/07/13@06:33:40.563-0500] P-17354 T-140209112129792 I WSAGENT36: (452) Login by mfg on batch.
[2026/07/13@06:33:40.566-0500] P-17354 T-140209112129792 I WSAGENT36: (7129) Usr 36 set name to mfg.
[2026/07/13@06:33:40.790-0500] P-17354 T-140209112129792 I WSAGENT36: (-----) Received RECONNECT from WTB
[2026/07/13@06:34:23.321-0500] P-17605 T-139854071501248 I WDOG 21: (2527) Disconnecting dead user 27.
[2026/07/13@06:34:51.106-0500] P-17611 T-140394007668160 I AIMGT 22: (3777) Switched to ai extent /ai/prod/ipurch/ipurchprod.a2.
[2026/07/13@06:34:51.106-0500] P-17611 T-140394007668160 I AIMGT 22: (3778) This is after-image file number 213818 since the last AIMAGE BEGIN
[2026/07/13@06:34:56.114-0500] P-17611 T-140394007668160 I AIMGT 22: (13199) After-image extent /ai/prod/ipurch/ipurchprod.a1 has been copied to /backup/prod/aiarch/ipurch/db~prod~ipurch~ipurchprod.20200606.062533.00213817.ipurchprod.a1.
[2026/07/13@06:34:56.115-0500] P-17611 T-140394007668160 I AIMGT 22: (3789) Marked after-image extent /ai/prod/ipurch/ipurchprod.a1 EMPTY.

Mon Jul 13 06:35:46 2026
[2026/07/13@06:35:46.082-0500] P-17618 T-140465072349632 I APW 0: (2519) Disconnected.
[2026/07/13@06:35:46.082-0500] P-17618 T-140465072349632 I APW 0: (-----) user #: 0, usrinuse #: 0, (srvctl:srvctl) 0:0, latchId: 18
[2026/07/13@06:35:46.082-0500] P-17618 T-140465072349632 F APW 0: (-----) lock stack corrupt expected 18 was 0 sp 0
[2026/07/13@06:35:46.083-0500] P-17618 T-140465072349632 I APW 0: (439) ** Save file named core for analysis by Progress Software Corporation.
[2026/07/13@06:35:46.094-0500] P-17615 T-139779634287040 I BIW 0: (2520) Stopped.
[2026/07/13@06:35:46.103-0500] P-17612 T-139810855203264 I AIW 0: (-----) user #: 0, usrinuse #: 0, (srvctl:srvctl) 0:0, latchId: 14, bot: 1
[2026/07/13@06:35:46.104-0500] P-17612 T-139810855203264 F AIW 0: (-----) lock stack underflow 0
[2026/07/13@06:35:46.104-0500] P-17612 T-139810855203264 I AIW 0: (439) ** Save file named core for analysis by Progress Software Corporation.
[2026/07/13@06:35:46.119-0500] P-17611 T-140394007668160 I AIMGT 0: (2519) Disconnected.
[2026/07/13@11:35:46.000+0000] P-17354 T-140209112129792 I WSAGENT 0: (49) SYSTEM ERROR: Memory violation.
[2026/07/13@11:35:46.000+0000] P-17354 T-140209112129792 I WSAGENT 0: (439) ** Save file named core for analysis by Progress Software Corporation.
[2026/07/13@11:35:46.000+0000] P-17354 T-140209112129792 I WSAGENT 0: (-----) Generating /apps/ipurchase/work/webspeed/agents/protrace.17354
[2026/07/13@06:35:46.576-0500] P-17600 T-140099396777792 I BROKER 0: (15192) The database will complete shutdown within approximately 60 seconds.
[2026/07/13@06:35:46.577-0500] P-17600 T-140099396777792 I BROKER 0: (2249) Begin ABNORMAL shutdown code 2
[2026/07/13@06:35:53.324-0500] P-17605 T-139854071501248 I WDOG 0: (2519) Disconnected.

Mon Jul 13 11:36:53 2026
[2026/07/13@11:36:53.000+0000] P-17338 T-140657448198400 I WSAGENT 0: (49) SYSTEM ERROR: Memory violation.
[2026/07/13@11:36:53.000+0000] P-17338 T-140657448198400 I WSAGENT 0: (439) ** Save file named core for analysis by Progress Software Corporation.
[2026/07/13@11:36:53.000+0000] P-17338 T-140657448198400 I WSAGENT 0: (-----) Generating /apps/ipurchase/work/webspeed/agents/protrace.17338
 
Did you check the protrace files?

/apps/ipurchase/work/webspeed/agents/protrace.12422
/apps/ipurchase/work/webspeed/agents/protrace.11762
/apps/ipurchase/work/webspeed/agents/protrace.17354
/apps/ipurchase/work/webspeed/agents/protrace.17338

> [2026/07/13@06:35:46.082-0500] P-17618 T-140465072349632 I APW 0: (-----) user #: 0, usrinuse #: 0, (srvctl:srvctl) 0:0, latchId: 18

latchId: 18 = BFP - online backup queue latch.
Was the online backup running at that time?
 
Code: 
[2026/07/13@06:19:46.015-0500] P-17611 T-140394007668160 I AIMGT 22: (3778) This is after-image file number 213817 since the last AIMAGE BEGIN
[2026/07/13@06:19:51.016-0500] P-17611 T-140394007668160 I AIMGT 22: (13199) After-image extent /ai/prod/ipurch/ipurchprod.a6 has been copied to /backup/prod/aiarch/ipurch/db~prod~ipurch~ipurchprod.20200606.062533.00213816.ipurchprod.a6.
[snip]
[2026/07/13@06:32:32.000-0500] First Memory violation.
[snip]
[2026/07/13@06:34:51.106-0500] P-17611 T-140394007668160 I AIMGT 22: (3777) Switched to ai extent /ai/prod/ipurch/ipurchprod.a2.
[2026/07/13@06:34:51.106-0500] P-17611 T-140394007668160 I AIMGT 22: (3778) This is after-image file number 213818 since the last AIMAGE BEGIN
[2026/07/13@06:34:56.114-0500] P-17611 T-140394007668160 I AIMGT 22: (13199) After-image extent /ai/prod/ipurch/ipurchprod.a1 has been copied to /backup/prod/aiarch/ipurch/db~prod~ipurch~ipurchprod.20200606.062533.00213817.ipurchprod.a1.

You're switching ai files every 15 minutes. The first memory violation occurred 2 minutes before the current ai file was archived. What are the latest timestamps in its scan?
 
Hi George,

Yes, I checked protrace.12422. It was a WebSpeed _progres -web agent for ipurchprod_ws. The stack shows:

parse_form_encoded
WebCgiInit
webRunDispatcher
wvRunDispatcher

ABL stack points to:

src/web/objects/web-disp.p at line 775
tptstart-new.p at line 905

I also found in ipurchprod_ws.server.log that PID 12422 received many invalid/blank-token requests just before the crash, with scanner-like paths such as wp-admin, index.php, api, ReportServer, geoserver, and Admin/login.php.

At 06:32:32, the same PID crashed with:

SYSTEM ERROR: Memory violation (49)

and disconnected from DB user number 28.

Regarding online backup: I checked the backup location and the backup does not appear to have been running at the DB crash time around 06:35:46. Our daily backup appears to start around 18:00 server time. Evidence from /backup/proddb:

-rw-r--r-- 1 mfg qad 2266 Jul 13 18:00 ipurchprod.st
-rw-r--r-- 1 mfg qad 7027225671 Jul 13 18:01 ipurchprod.bkup.gz
-rw-r--r-- 1 mfg qad 8781157376 Jul 13 18:01 ipurchprod.bkup

Current server time check:

Tue 14 Jul 2026 04:20:49 AM CDT

So based on these timestamps, the backup activity was around 18:00, not around 06:35. I will still check DB log/job logs for BACKUP/probkup entries around 06:30-06:36 to confirm.

For the AI scan, I decompressed AI 213817 and 213818 to /tmp:

/tmp/ai_213817.a1
/tmp/ai_213818.a2

But rfutil scan against the live DB failed with:

** The database /db/prod/ipurch/ipurchprod is in use in multi-user mode. (276)

So I cannot scan using the live production DB while it is online.

Current finding so far: the first clear failure appears to be WebSpeed agent PID 12422 receiving repeated invalid/blank-token requests and then crashing with memory violation at 06:32:32. After multiple WebSpeed memory violations, APW/AIW later reported lock stack corrupt/underflow and broker initiated abnormal shutdown code 2.

Thanks.
 
Mike said:
** The database /db/prod/ipurch/ipurchprod is in use in multi-user mode. (276)

So I cannot scan using the live production DB while it is online.
Click to expand...
Code: 
prodb empty empty
rfutil empty -C aimage truncate -aiblocksize <your-db-ai-blocksize>
rfutil empty -C aimage scan verbose -a <ai-name> > scan.txt

> APW/AIW later reported lock stack corrupt/underflow and broker initiated abnormal shutdown code 2.

The structures in shared memory were already corrupted before these errors. All processed had "forgot" their numbers - the part of (or the entire?) Connection Table was zeroed out..
 
You didn't mention your OpenEdge version. Error 49 indicates a Progress bug. Are you on the latest update for your release?
 
OpenEdge Release 11.7.9 build

OS: Linux Debian

Hi George / Rob,

I created a dummy database in /tmp and scanned the copied AI files successfully.

Production DB AI block size:

AI block size: 16384

Dummy DB steps used:

cd /tmp/aiscan_ipurch
prodb aiscan empty
rfutil aiscan -C aimage truncate -aiblocksize 16

Then scanned the copied AI files:

rfutil aiscan -C aimage scan verbose -a /tmp/ai_213817.a1 > /tmp/ai_213817_scan_verbose.out 2>&1
rfutil aiscan -C aimage scan verbose -a /tmp/ai_213818.a2 > /tmp/ai_213818_scan_verbose.out 2>&1

AI 213817 scan result:

Latest timestamp seen in the tail output:

Mon Jul 13 06:34:08 2026

Summary:

12195 notes were processed.
0 in-flight transactions.
211 transactions were started.
211 transactions were completed.
At the end of the .ai file, 0 transactions were still active.

AI 213818 scan result:

Latest timestamp seen in the tail output:

Mon Jul 13 07:25:01 2026

Summary:

178 notes were processed.
0 in-flight transactions.
25 transactions were started.
25 transactions were completed.
At the end of the .ai file, 0 transactions were still active.

So from the AI scan output, both AI files completed cleanly with 0 in-flight / 0 active transactions at the end.

Regarding online backup, it does not appear to have been running at the crash time around 06:35:46. Backup files show activity around 18:00/18:01 server time, not around 06:35.

Current finding so far:

The first clear failure was WebSpeed agent PID 12422 receiving repeated invalid/blank-token requests and then crashing with SYSTEM ERROR: Memory violation (49) at 06:32:32. After multiple WebSpeed memory violations, APW/AIW later reported lock stack corrupt/underflow and the broker initiated abnormal shutdown code 2.

Thanks.
 
The latest update to release 11.7 is 11.7.22. So you are missing 13 updates worth of bugs fixes. This is also the last update, since release 11.7 is retired since April 2025.

I strongly suggest that you test and implement 11.7.22.
 
You hit a Progress bug in a shared memory client, some data structures in database shared memory were corrupted, and various processes hit fatal errors and stopped. The database performed an abnormal shutdown to protect data integrity. That might be as much root cause as you ever get.

And it is much more likely to happen to you again if you don't patch your system. 11.7.9 is over five and a half years old.
 
Mike said:
AI 213817 scan result:

Latest timestamp seen in the tail output:

Mon Jul 13 06:34:08 2026
Click to expand...
In other words, transaction activity continued after the memory violation errors. Latches 14 and 18 were not permanently locked. Hence the corresponding error messages are not part of the root cause.
 
George Potemkin said:
In other words, transaction activity continued after the memory violation errors. Latches 14 and 18 were not permanently locked. Hence the corresponding error messages are not part of the root cause.
Click to expand...
Hi George ,

So what was the cause ? which user was responsible? and how to avoid in future?

Thanks
Mike
 
Mike said:
So what was the cause ? which user was responsible? and how to avoid in future?
Click to expand...
It's very difficult to answer these questions for memory violation errors. Strictly speaking, it's not an error, but a consequence of an error. The bug caused an access to an invalid address. As a result, the code received or modified invalid data. This will continue until the invalid address goes beyond the process's address space and then Progress issues the error message. The same bug may sometimes crash a database but othertimes it does not.

To find the root cause of the memory violation error, you need to meticulously collect data on all instances of this error and look for common patterns. It's like a hunting for a maniac.

And I completely agree with Rob Fitzpatrick: the best thing to do in this situation is to install the latest service pack.
 
As Rob has said, error 49 (almost) always points to a Progress bug. I know that there are a number of bugs that cause error 49 in 11.7 that have been fixed in later patch versions. I know because I raised them myself. As Rob says, you'll probably not get much more than this in terms of root cause analysis. Patching to 11.7.22 and planning the migration to 12.x is the solution.
 
Hi All,

Thank you very much for your kind support and valuable guidance. Your inputs helped me understand the issue much more clearly, especially around the WebSpeed memory violation, shared memory corruption, and why APW/AIW errors were likely secondary symptoms. We will use this information for our internal RCA and future prevention plan, including monitoring and patch/upgrade discussion.

Thanks again to everyone for your help.
 
George Potemkin said:
It's very difficult to answer these questions for memory violation errors. Strictly speaking, it's not an error, but a consequence of an error. The bug caused an access to an invalid address. As a result, the code received or modified invalid data. This will continue until the invalid address goes beyond the process's address space and then Progress issues the error message. The same bug may sometimes crash a database but othertimes it does not.

To find the root cause of the memory violation error, you need to meticulously collect data on all instances of this error and look for common patterns. It's like a hunting for a maniac.

And I completely agree with Rob Fitzpatrick: the best thing to do in this situation is to install the latest service pack.
Click to expand...
HI George and team,

Apologies for disturbing you again.

The database went down once more, and we need your assistance to identify the root cause, observed repeated OpenEdge memory violation errors along with database shared memory and WebSpeed-related issues.

As previously discussed, memory violation errors are difficult to analyze because they may be a consequence of another underlying issue. In this case, the database became unavailable again, and we found multiple OpenEdge database and WebSpeed-related errors in the logs.

Below are the sanitized details.

Issue Summary​

The database became unavailable again. Around the time of the issue, the database log showed multiple WebSpeed agent reconnect activities followed by OpenEdge internal errors.

The critical errors observed were:
2026/07/18@07:45:05.717-0500] P-6230 T-140052327674112 I WSAGENT28: (7129) Usr 28 set name to mfg.
[2026/07/18@07:45:05.719-0500] P-6228 T-139890157302016 I WSAGENT30: (452) Login by mfg on batch.
[2026/07/18@07:45:05.719-0500] P-6229 T-140326935539968 I WSAGENT29: (7129) Usr 29 set name to mfg.
[2026/07/18@07:45:05.722-0500] P-6228 T-139890157302016 I WSAGENT30: (7129) Usr 30 set name to mfg.
[2026/07/18@07:45:05.822-0500] P-6239 T-140594687189248 I WSAGENT31: (452) Login by mfg on batch.
[2026/07/18@07:45:05.822-0500] P-6245 T-139840863944960 I WSAGENT32: (452) Login by mfg on batch.
[2026/07/18@07:45:05.825-0500] P-6239 T-140594687189248 I WSAGENT31: (7129) Usr 31 set name to mfg.
[2026/07/18@07:45:05.826-0500] P-6246 T-139705763115264 I WSAGENT33: (452) Login by mfg on batch.
[2026/07/18@07:45:05.826-0500] P-6245 T-139840863944960 I WSAGENT32: (7129) Usr 32 set name to mfg.
[2026/07/18@07:45:05.829-0500] P-6244 T-140393048535296 I WSAGENT34: (452) Login by mfg on batch.
[2026/07/18@07:45:05.829-0500] P-6246 T-139705763115264 I WSAGENT33: (7129) Usr 33 set name to mfg.
[2026/07/18@07:45:05.835-0500] P-6244 T-140393048535296 I WSAGENT34: (7129) Usr 34 set name to mfg.
[2026/07/18@07:45:06.021-0500] P-6228 T-139890157302016 I WSAGENT30: (-----) Received RECONNECT from WTB
[2026/07/18@07:45:06.042-0500] P-6230 T-140052327674112 I WSAGENT28: (7129) Usr 28 set name to iss.
[2026/07/18@07:45:06.117-0500] P-6239 T-140594687189248 I WSAGENT31: (-----) Received RECONNECT from WTB
[2026/07/18@07:45:06.140-0500] P-6244 T-140393048535296 I WSAGENT34: (-----) Received RECONNECT from WTB
[2026/07/18@07:45:06.691-0500] P-6251 T-139917056815360 I WSAGENT35: (452) Login by mfg on batch.
[2026/07/18@07:45:06.694-0500] P-6251 T-139917056815360 I WSAGENT35: (7129) Usr 35 set name to mfg.
[2026/07/18@07:45:06.928-0500] P-6251 T-139917056815360 I WSAGENT35: (-----) Received RECONNECT from WTB
[2026/07/18@07:45:24.688-0500] P-6227 T-140020339867904 I WSAGENT27: (7129) Usr 27 set name to iss.
[2026/07/18@07:45:50.700-0500] P-6246 T-139705763115264 I WSAGENT33: (7129) Usr 33 set name to iss.
[2026/07/18@07:45:51.671-0500] P-6229 T-140326935539968 I WSAGENT29: (7129) Usr 29 set name to iss.
[2026/07/18@07:46:06.829-0500] P-6245 T-139840863944960 I WSAGENT32: (7129) Usr 32 set name to iss.

Sat Jul 18 07:46:28 2026
[2026/07/18@07:46:28.611-0500] P-9536 T-140146538906048 I APW 0: (2519) Disconnected.
[2026/07/18@07:46:28.611-0500] P-9536 T-140146538906048 I APW 0: (-----) user #: 0, usrinuse #: 0, (srvctl:srvctl) 0:0, latchId: 18
[2026/07/18@07:46:28.611-0500] P-9536 T-140146538906048 F APW 0: (-----) lock stack corrupt expected 18 was 0 sp 0
[2026/07/18@07:46:28.611-0500] P-9536 T-140146538906048 I APW 0: (439) ** Save file named core for analysis by Progress Software Corporation.
[2026/07/18@07:46:28.621-0500] P-9530 T-140464060596672 I AIW 0: (-----) user #: 0, usrinuse #: 0, (srvctl:srvctl) 0:0, latchId: 14, bot: 1
[2026/07/18@07:46:28.621-0500] P-9530 T-140464060596672 F AIW 0: (-----) lock stack underflow 0
[2026/07/18@07:46:28.621-0500] P-9530 T-140464060596672 I AIW 0: (439) ** Save file named core for analysis by Progress Software Corporation.
[2026/07/18@07:46:28.632-0500] P-9533 T-140272366674368 I BIW 0: (2520) Stopped.
[2026/07/18@12:46:29.000+0000] P-6251 T-139917056815360 I WSAGENT 0: (49) SYSTEM ERROR: Memory violation.

[2026/07/18@12:46:29.000+0000] P-6227 T-140020339867904 I WSAGENT 0: (49) SYSTEM ERROR: Memory violation.
[2026/07/18@12:46:29.000+0000] P-6230 T-140052327674112 I WSAGENT 0: (49) SYSTEM ERROR: Memory violation.
Sat Jul 18 12:46:29 2026
[2026/07/18@12:46:29.000+0000] P-6229 T-140326935539968 I WSAGENT 0: (49) SYSTEM ERROR: Memory violation.
[2026/07/18@12:46:29.000+0000] P-6251 T-139917056815360 I WSAGENT 0: (439) ** Save file named core for analysis by Progress Software Corporation.
[2026/07/18@12:46:29.000+0000] P-6227 T-140020339867904 I WSAGENT 0: (439) ** Save file named core for analysis by Progress Software Corporation.
[2026/07/18@12:46:29.000+0000] P-6229 T-140326935539968 I WSAGENT 0: (439) ** Save file named core for analysis by Progress Software Corporation.
[2026/07/18@12:46:29.000+0000] P-6230 T-140052327674112 I WSAGENT 0: (439) ** Save file named core for analysis by Progress Software Corporation.

Sat Jul 18 12:46:29 2026
[2026/07/18@12:46:29.000+0000] P-6251 T-139917056815360 I WSAGENT 0: (-----) Generating /apps/ipurchase/work/webspeed/agents/protrace.6230
[2026/07/18@12:46:29.000+0000] P-6227 T-140020339867904 I WSAGENT 0: (-----) Generating /apps/ipurchase/work/webspeed/agents/protrace.6227
[2026/07/18@12:46:29.000+0000] P-6229 T-140326935539968 I WSAGENT 0: (-----) Generating /apps/ipurchase/work/webspeed/agents/protrace.6229
[2026/07/18@12:46:29.000+0000] P-6244 T-140393048535296 I WSAGENT 0: (49) SYSTEM ERROR: Memory violation.
[2026/07/18@12:46:29.000+0000] P-6244 T-140393048535296 I WSAGENT 0: (439) ** Save file named core for analysis by Progress Software Corporation.
[2026/07/18@12:46:29.000+0000] P-6239 T-140594687189248 I WSAGENT 0: (49) SYSTEM ERROR: Memory violation.
[2026/07/18@12:46:29.000+0000] P-6239 T-140594687189248 I WSAGENT 0: (439) ** Save file named core for analysis by Progress Software Corporation.
[2026/07/18@12:46:29.000+0000] P-6244 T-140393048535296 I WSAGENT 0: (-----) Generating /apps/ipurchase/work/webspeed/agents/protrace.6244
[2026/07/18@12:46:29.000+0000] P-6239 T-140594687189248 I WSAGENT 0: (-----) Generating /apps/ipurchase/work/webspeed/agents/protrace.6239
[2026/07/18@07:46:31.238-0500] P-9529 T-139654202111424 I AIMGT 0: (2519) Disconnected.
[2026/07/18@07:46:33.176-0500] P-9523 T-140439709556992 F WSAGENT 0: (7993) procomm: fatal return code -1218 from dsmTransaction.
[2026/07/18@07:46:33.177-0500] P-9523 T-140439709556992 I WSAGENT 0: (439) ** Save file named core for analysis by Progress Software Corporation.
[2026/07/18@07:46:38.478-0500] P-9521 T-139646453510592 I WDOG 0: (2519) Disconnected.

Sun Jul 19 08:46:57 2026
[2026/07/19@08:46:57.000+0000] P-9517 T-139785533794112 I BROKER 0: (3694) SIGTERM received.
[2026/07/19@03:46:57.550-0500] P-9517 T-139785533794112 I BROKER 0: (15193) The normal shutdown of the database will continue for 10 Min 0 Sec if required.
[2026/07/19@03:46:57.551-0500] P-9517 T-139785533794112 I BROKER 0: (2248) Begin normal shutdown





lock stack corrupt expected 18 was 0 sp 0

lock stack underflow 0
3
SYSTEM ERROR: Memory violation

procomm: fatal return code -1218 from dsmTransaction
Show more lines
After this, database utilities failed with shared memory-related errors, and WebSpeed agents were also unable to connect successfully.

The shared memory does not have the correct MAGIC number (1179)


Show more lines

Assistance Required​

Could you please review this issue and help us confirm the root cause?
We need your guidance on the below points:

1:-What may have caused the repeated memory violation?
  1. Do the lock stack corrupt and lock stack underflow messages indicate an OpenEdge engine defect or memory corruption issue?
  2. Could the WebSpeed agent reconnect activity have triggered this issue?
  3. Does fatal return code -1218 from dsmTransaction indicate a transaction manager issue caused by the memory violation?
  4. Is this a known issue in OpenEdge 11.7 SP09?
  5. Is upgrading to the latest service pack/hotfix recommended?
We can provide the full database log, WebSpeed broker log, protrace files, and core files through the approved secure channel if required.

Regards,
Mike
 
Mike said:
Do the lock stack corrupt and lock stack underflow messages indicate an OpenEdge engine defect or memory corruption issue?
Click to expand...

When db will be up and running again check promon / Activity: Latch Counts
Does BFP latch (latchId 18) activity is zero for intervals when backup is not running?

Mike said:
Is upgrading to the latest service pack/hotfix recommended?
Click to expand...
It's HIGHLY recommended

Code: 
[2026/07/18@12:46:29.000+0000] P-6251 T-139917056815360 I WSAGENT 0: (49) SYSTEM ERROR: Memory violation.
[2026/07/18@12:46:29.000+0000] P-6227 T-140020339867904 I WSAGENT 0: (49) SYSTEM ERROR: Memory violation.
[2026/07/18@12:46:29.000+0000] P-6230 T-140052327674112 I WSAGENT 0: (49) SYSTEM ERROR: Memory violation.
[2026/07/18@12:46:29.000+0000] P-6229 T-140326935539968 I WSAGENT 0: (49) SYSTEM ERROR: Memory violation.
[2026/07/18@12:46:29.000+0000] P-6244 T-140393048535296 I WSAGENT 0: (49) SYSTEM ERROR: Memory violation.
[2026/07/18@12:46:29.000+0000] P-6239 T-140594687189248 I WSAGENT 0: (49) SYSTEM ERROR: Memory violation.

Why six db clients got the errors exactly at the same time? Obviously because they read the same data from shared memory and data became corrupted.

What data they were reading? Protrace files can give a hint. Do the WS agents create the client log files? You can find more details about their activity before crash.

Mike said:
Could the WebSpeed agent reconnect activity have triggered this issue?
Click to expand...

Reconnect request forces the WS agents to re-read some data in shared memory. But there are the reconnect messages only from 4 agents. And only 3 of them issued the memory violation error. Another 3 agents issued the memory violation error without the reconnect requests.

Mike said:
Does fatal return code -1218 from dsmTransaction indicate a transaction manager issue caused by the memory violation?
Click to expand...

No. The error can be displayed in instances of a client crash.

What does dsmTransaction error 7993 with fatal return code -1218 mean?​

Progress Customer Community

community.progress.com community.progress.com
 
Hi George,
Thank you for the detailed analysis. I reviewed the logs and protrace files and found that six WebSpeed agents (PIDs 6227, 6229, 6230, 6239, 6244, and 6251) all reported "SYSTEM ERROR: Memory violation" at the exact same timestamp (12:46:29) and generated protrace files simultaneously. I compared all six protrace files and they show the identical stack trace (parse_form_encoded -&gt; WebCgiInit -&gt; webRunDispatcher) and the identical ABL stack trace pointing to src/web/objects/web-disp.p line 775 and tptstart-new.p line 905. Since all affected agents were connected to the same WebSpeed broker and database and failed in the same execution path at the same time, this appears more consistent with a shared memory corruption/OpenEdge runtime issue than six independent client failures. After the database was recovered, PROMON showed no lock waits, semaphore latch waits remained at 0, and we did not observe any obvious backup-related or latch contention issues.
 
Example:
Code: 
07/01/26        Activity: Latch Counts
15:51:15        07/01/26 15:41 to 07/01/26 15:51 (10 min 1 sec)
                ----- Locks -----        ------ Busy ------        Naps        ---------- Spins -----------    ----- Nap Max -----
   Owner        Total        /Sec        /Sec           Pct        /Sec        /Sec       /Lock       /Busy       Total   HWM
LKF  --      11547033       19213           0           0.0           0           0           0           0           0     0
BFP  --             0           0           0           0.0           0           0           0           0           0     0
BHT  --     805279089     1339898           0           0.0           0           0           0           0           0     0

Does your database show no BFP latch locks?
 
George i did promon and 5 option and found this. Can you help me which option i need to chose?
0 Servers, 1 Users (1 Local, 0 Remote, 0 Batch),1 Apws

RETURN - repeat, U - continue uninterrupted, Q - quit:

Activity - Sampled at 07/19/26 09:43 for 0:00:23.

Event Total Per Sec Event Total Per Sec
Commits 0 0.0 Undos 0 0.0
Record Updates 0 0.0 Record Reads 85 3.7
Record Creates 0 0.0 Record Deletes 0 0.0
DB Writes 0 0.0 DB Reads 0 0.0
BI Writes 0 0.0 BI Reads 0 0.0
AI Writes 0 0.0
Record Locks 0 0.0 Record Waits 0 0.0
Checkpoints 0 0.0 Buffs Flushed 0 0.0

Rec Lock Waits 0 % BI Buf Waits 0 % AI Buf Waits 0 %
Writes by APW 0 % Writes by BIW 0 % Writes by AIW 0 %
Buffer Hits 100 % Primary Hits 100 % Alternate Hits 0 %
DB Size 9257 MB BI Size 499 MB AI Size 0 K
FR chain 1272 blocks RM chain 2 blocks
Shared Memory 1093M Segments 1

0 Servers, 1 Users (1 Local, 0 Remote, 0 Batch),1 Apws

RETURN - repeat, U - continue uninterrupted, Q - quit:

RETURN - repeat, U - continue uninterrupted, Q - quit: q


OpenEdge MONITOR Release 11

Database: /db/prod/ipurch/ipurchprod

1. User Control
2. Locking and Waiting Statistics
3. Block Access
4. Record Locking Table
5. Activity
6. Shared Resources
7. Database Status
8. Shut Down Database
9. Currently Connected Tenants

R&D. Advanced options
T. 2PC Transactions Control
L. Resolve 2PC Limbo Transactions
C. 2PC Coordinator Information

J. Resolve JTA Transactions

M. Modify Defaults
Q. Quit
 
You must log in or register to reply here.

Similar threads

Mike
  • Locked
Error Request for Progress Review – ipurch DB Abnormal Shutdown with WebSpeed Memory Violation and APW/AIW Lock Stack Errors
Replies
0
Views
231
Mike
Mike
Back
Top