System Memory Violation

[Jack@dba]

Hi,

Yesterday we faced issue database got down. But we Re-started database it worked fine.
But we seen this error in all corresponding database also but only one database got shutdown.
We found this errors during database shutdown.

Can i know during which situation we will see these errors in log file.

19:12:19 Usr 44: Corrupt block detected when attempting to release a buffer. (4232)
19:12:19 SQLSRV2 6: SYSTEM ERROR: Memory violation. (49)
19:12:19 Usr 44: bmReleaseBuffer: Error occurred in area 6, block number: 8304696, extent:
/pro/prod/crn.d1. (10560)
19:12:19 Usr 44: Writing block 8304696 to log file. Please save and send the log file to
Progress Software Corp. for investigation. (10561)
19:12:19 Usr 44: SYSTEM DEBUG: Database buffer block
19:12:19 Usr 44: pbktbl = 0x30490db4
19:12:19 Usr 44: pbktbl->qself = 0x58490db4
19:12:19 Usr 44: XBKBUF(pbktbl->qself) = 0x30490db4
19:12:19 Usr 44: pbktbl->bt_qbuf = 0x62e30fb4
19:12:19 Usr 44: XBKBUF(pbktbl->bt_qbuf) = 0x42e30fb4
19:12:19 Usr 44: pbkbuf = 0x42e30fb4
19:12:19 Usr 44: Block dbkey = 0x1fae0e40 bt_offset = 0x0
19:12:19 Usr 44: File: /pro/prd/crn.d64
19:12:19 Usr 44: 0000: 0000 0000 0000 05d9 5849 0e64 62e3 0fb4
19:12:19 Usr 44: 0010: 0000 0001 0100 002c 0000 4cae 001b 0000
19:12:19 Usr 44: 0020: 1fae 0e40 0000 0006 5fff f878 0000 0000
19:12:19 Usr 44: 0030: 0000 0000 370f 0000 ffff ffff 0000 0000
19:12:19 Usr 44: 0040: 0000 0000 0000 0000 0000 0000 58c2 4c44
19:12:19 Usr 44: 0050: 58b0 2f14 0000 0000 0000 0000 0000 0000
19:12:19 Usr 44: 0060: 0000 0000 0000 0000 0001 0100 0000 0000
19:12:19 Usr 44: 0070: 0163 0000 0000 0000 0000 0000 0000 0000
19:12:19 Usr 44: 0080: 0000 0000 0000 0000 0000 0000 0000 0000
19:12:19 Usr 44: 0090: 0000 0000 0000 0000 0000 0000 0000 0000
19:12:19 Usr 44: 00a0: 5849 0db4 0000 0000 0000 0000 0000 0000
19:12:19 Usr 44: Block buffer dump follows, from address 0x42e30fb4:

And after two hours we are not able to connect to application but database is up and running.When we are trying to connect to application we received below error.

The server or the system has no more resources. Please contact Progress x
x Technical Support. (748) x
x
Kindly help me what went wrong during that time.

Progress version : 91.e
Unix AI 5.3 version.

Thanks
Jack

 

[Cringer]

Have you rebooted the server? I assume from what you're saying this is a multi-database environment. I suspect that the one database crashed and users got thrown out of the application, but their connections to the other DB(s) didn't clear down properly meaning they are still showing as connected there. Restarting all the databases, or simply rebooting the machine should resolve this problem.

 

[Jack@dba]

Thanks Cringer for quick update but whether this issue will come under database or Network issues.

How to overcome this issue.

Jack

 

[TomBascom]

48 hours later and you are still having the problem? Your users are exceptionally patient.

As Cringer suggests, restarting all of the databases or rebooting will indeed usually clear these things up. Have you tried that?

The original SQL92 failure is unusual. I would suspect a network issue. You could try opening a ticket with tech support but Progress version 9 is ancient, obsolete and unsupported. AIX 5.3 is not much better. You really need to invest in some infrastructure updates.

"Out of resources" is a more routine problem. It means that you are out of some resource. Often it means that you have too many users connected. Frequently this happens after some sessions become disconnected but not completely logged out -- which is why restarting the db (or rebooting) will clear it up.

 

[Jack@dba]

Hi Tom,

You made by day with Joy.

Small confirmation i try to refresh from once main database to test database r but receiving below error for my windows 2003 server.


09:09:57 BROKER 0: Minimum Port for Auto Servers (-minport): 3000. (5648)
09:09:57 BROKER 0: Maximum Port for Auto Servers (-maxport): 5000. (5649)
09:09:57 BROKER 0: This broker supports both 4GL and SQL server groups. (8865)
09:09:57 BROKER 0: Created shared memory with segment_id: 11599872 (9336)
09:11:48 SRV 1: Started on port 3017 using TCP, pid 2644. (5646)
09:11:49 SRV 1: Login usernum 43, userid cyap, on KULPC274. (742)
09:11:57 SRV 1: SYSTEM ERROR: bkioRead: Bad file descriptor was used during Read, fd 620, len 8192, offset 9056, file e:\db\crp91e\crp91e.d59. (9446)
09:11:57 SRV 1: SYSTEM ERROR: read wrong dbkey at offset 74186752 in file e:\db\crp91e\crp91e.d59
found 43454976, expected 280236416, retrying. area 6 (9445)
09:11:58 SRV 1: SYSTEM ERROR: bkioRead: Bad file descriptor was used during Read, fd 620, len 8192, offset 9056, file e:\db\crp91e\crp91e.d59. (9446)
09:11:59 SRV 1: SYSTEM ERROR: bkioRead: Bad file descriptor was used during Read, fd 460, len 8192, offset 46443, file e:\db\crp91e\crp91e.d19. (9446)
09:11:59 SRV 1: SYSTEM ERROR: read wrong dbkey at offset 380461056 in file e:\db\crp91e\crp91e.d19
found 223552, expected 89380416, retrying. area 6 (9445)
09:12:01 SRV 1: SYSTEM ERROR: bkioRead: Bad file descriptor was used during Read, fd 516, len 8192, offset 7480, file e:\db\crp91e\crp91e.d33. (9446)
09:12:01 SRV 1: SYSTEM ERROR: read wrong dbkey at offset 61276160 in file e:\db\crp91e\crp91e.d33
found 271282816, expected 154093056, retrying. area 6 (9445)
09:12:02 SRV 1: SYSTEM ERROR: bkioRead: Bad file descriptor was used during Read, fd 488, len 8192, offset 51832, file f:\db\crp91e\crp91e.d26. (9446)
09:12:02 SRV 1: SYSTEM ERROR: read wrong dbkey at offset 424607744 in file f:\db\crp91e\crp91e.d26
found 101674560, expected 123328448, retrying. area 6 (9445)
09:12:09 SRV 1: Logout usernum 43, userid , on KULPC274. (739)
10:08:11 SRV 1: Login usernum 43, userid cyap, on KULPC274. (742)
10:18:29 SRV 1: Logout usernum 43, userid , on KULPC274. (739)
11:19:57 SRV 1: Login usernum 43, userid cyap, on KULPC283. (742)
11:20:19 SRV 1: Logout usernum 43, userid , on KULPC283. (739)
11:59:23 SRV 1: Login usernum 43, userid btoh, on KULLT409. (742)
12:00:48 SRV 1: Usernum 43 terminated abnormally. (794)
12:00:48 SRV 1: Logout usernum 43, userid , on KULLT409. (739)
13:01:54 SRV 1: Login usernum 43, userid rtaib, on KULPC519. (742)
13:37:17 SRV 1: Login usernum 42, userid btoh, on KULLT409. (742)
13:59:00 SRV 1: Logout usernum 42, userid , on KULLT409. (739)
15:47:58 SRV 1: SYSTEM ERROR: bkioRead: Bad file descriptor was used during Read, fd 404, len 8192, offset 28738, file e:\db\crp91e\crp91e.d5. (9446)
15:47:58 SRV 1: SYSTEM ERROR: read wrong dbkey at offset 235421696 in file e:\db\crp91e\crp91e.d5
found 17081280, expected 21041024, retrying. area 6 (9445)
15:47:59 SRV 1: SYSTEM ERROR: bkioRead: Bad file descriptor was used during Read, fd 404, len 8192, offset 28738, file e:\db\crp91e\crp91e.d5. (9446)
15:48:00 SRV 1: SYSTEM ERROR: bkioRead: Bad file descriptor was used during Read, fd 404, len 8192, offset 28738, file e:\db\crp91e\crp91e.d5. (9446)
15:48:01 SRV 1: SYSTEM ERROR: bkioRead: Bad file descriptor was used during Read, fd 404, len 8192, offset 28738, file e:\db\crp91e\crp91e.d5. (9446)
15:48:02 SRV 1: SYSTEM ERROR: bkioRead: Bad file descriptor was used during Read, fd 404, len 8192, offset 28738, file e:\db\crp91e\crp91e.d5. (9446)
15:48:04 SRV 1: SYSTEM ERROR: bkioRead: Bad file descriptor was used during Read, fd 408, len 8192, offset 12320, file f:\db\crp91e\crp91e.d6. (9446)
15:48:04 SRV 1: SYSTEM ERROR: read wrong dbkey at offset 100925440 in file f:\db\crp91e\crp91e.d6
found 20400384, expected 24790720, retrying. area 6 (9445)
15:48:05 SRV 1: SYSTEM ERROR: bkioRead: Bad file descriptor was used during Read, fd 408, len 8192, offset 12320, file f:\db\crp91e\crp91e.d6. (9446)
15:48:06 SRV 1: SYSTEM ERROR: bkioRead: Bad file descriptor was used during Read, fd 408, len 8192, offset 12320, file f:\db\crp91e\crp91e.d6. (9446)
15:48:07 SRV 1: SYSTEM ERROR: bkioRead: Bad file descriptor was used during Read, fd 408, len 8192, offset 12320, file f:\db\crp91e\crp91e.d6. (9446)
15:48:08 SRV 1: SYSTEM ERROR: bkioRead: Bad file descriptor was used during Read, fd 408, len 8192, offset 12320, file f:\db\crp91e\crp91e.d6. (9446)
15:48:09 SRV 1: SYSTEM ERROR: bkioRead: Bad file descriptor was used during Read, fd 408, len 8192, offset 12320, file f:\db\crp91e\crp91e.d6. (9446)
15:48:11 SRV 1: SYSTEM ERROR: bkioRead: Bad file descriptor was used during Read, fd 408, len 8192, offset 33640, file f:\db\crp91e\crp91e.d6. (9446)
15:48:11 SRV 1: SYSTEM ERROR: read wrong dbkey at offset 275578880 in file f:\db\crp91e\crp91e.d6
found 20701632, expected 26155200, retrying. area 6 (9445)
15:48:12 SRV 1: SYSTEM ERROR: bkioRead: Bad file descriptor was used during Read, fd 408, len 8192, offset 33640, file f:\db\crp91e\crp91e.d6. (9446)
15:48:13 SRV 1: SYSTEM ERROR: bkioRead: Bad file descriptor was used during Read, fd 408, len 8192, offset 33640, file f:\db\crp91e\crp91e.d6. (9446)
15:48:14 SRV 1: SYSTEM ERROR: bkioRead: Bad file descriptor was used during Read, fd 408, len 8192, offset 33640, file f:\db\crp91e\crp91e.d6. (9446)
15:48:15 SRV 1: SYSTEM ERROR: bkioRead: Bad file descriptor was used during Read, fd 408, len 8192, offset 33640, file f:\db\crp91e\crp91e.d6. (9446)
15:48:16 SRV 1: SYSTEM ERROR: bkioRead: Bad file descriptor was used during Read, fd 408, len 8192, offset 33640, file f:\db\crp91e\crp91e.d6. (9446)
15:48:17 SRV 1: SYSTEM ERROR: bkioRead: Bad file descriptor was used during Read, fd 408, len 8192, offset 33640, file f:\db\crp91e\crp91e.d6. (9446)
15:48:23 SRV 1: SYSTEM ERROR: bkioRead: Bad file descriptor was used during Read, fd 416, len 8192, offset 50026, file f:\db\crp91e\crp91e.d8. (9446)
15:48:23 SRV 1: SYSTEM ERROR: read wrong dbkey at offset 409812992 in file f:\db\crp91e\crp91e.d8
found 16017984, expected 36804800, retrying. area 6 (9445)
15:48:25 SRV 1: SYSTEM ERROR: bkioRead: Bad file descriptor was used during Read, fd 420, len 8192, offset 12575, file e:\db\crp91e\crp91e.d9. (9446)
15:48:25 SRV 1: SYSTEM ERROR: read wrong dbkey at offset 103014400 in file e:\db\crp91e\crp91e.d9
found 35735680, expected 39208384, retrying. area 6 (9445)
15:48:26 SRV 1: SYSTEM ERROR: bkioRead: Bad file descriptor was used during Read, fd 420, len 8192, offset 12575, file e:\db\crp91e\crp91e.d9. (9446)
15:48:31 SRV 1: SYSTEM ERROR: bkioRead: Bad file descriptor was used during Read, fd 428, len 8192, offset 50373, file e:\db\crp91e\crp91e.d11. (9446)
15:48:31 SRV 1: SYSTEM ERROR: read wrong dbkey at offset 412655616 in file e:\db\crp91e\crp91e.d11
found 47638016, expected 51228352, retrying. area 6 (9445)
15:48:35 SRV 1: SYSTEM ERROR: bkioRead: Bad file descriptor was used during Read, fd 436, len 8192, offset 62747, file e:\db\crp91e\crp91e.d13. (9446)
15:48:35 SRV 1: SYSTEM ERROR: read wrong dbkey at offset 514023424 in file e:\db\crp91e\crp91e.d13
found 57918976, expected 61621184, retrying. area 6 (9445)
15:48:39 SRV 1: SYSTEM ERROR: bkioRead: Bad file descriptor was used during Read, fd 452, len 8192, offset 4769, file e:\db\crp91e\crp91e.d17. (9446)
15:48:39 SRV 1: SYSTEM ERROR: read wrong dbkey at offset 39067648 in file e:\db\crp91e\crp91e.d17
found 72851392, expected 77112384, retrying. area 6 (9445)
15:48:52 SRV 1: SYSTEM ERROR: bkioRead: Bad file descriptor was used during Read, fd 472, len 8192, offset 45108, file f:\db\crp91e\crp91e.d22. (9446)
15:48:52 SRV 1: SYSTEM ERROR: read wrong dbkey at offset 369524736 in file f:\db\crp91e\crp91e.d22
found 99631616, expected 103696320, retrying. area 6 (9445)
15:49:01 SRV 1: SYSTEM ERROR: bkioRead: Bad file descriptor was used during Read, fd 484, len 8192, offset 33143, file e:\db\crp91e\crp91e.d25. (9446)
15:49:01 SRV 1: SYSTEM ERROR: read wrong dbkey at offset 271507456 in file e:\db\crp91e\crp91e.d25
found 69277376, expected 117331904, retrying. area 6 (9445)
15:49:14 SRV 1: SYSTEM ERROR: bkioRead: Bad file descriptor was used during Read, fd 504, len 8192, offset 21472, file f:\db\crp91e\crp91e.d30. (9446)
15:49:14 SRV 1: SYSTEM ERROR: read wrong dbkey at offset 175898624 in file f:\db\crp91e\crp91e.d30
found 138849152, expected 140587200, retrying. area 6 (9445)
15:49:24 SRV 1: SYSTEM ERROR: bkioRead: Bad file descriptor was used during Read, fd 520, len 8192, offset 133, file f:\db\crp91e\crp91e.d34. (9446)

 

[Jack@dba]

How to refresh the database.

But I have checked and found this is not related to Space issue we have good enough space.

 

[TomBascom]

You have either completely changed topics and completely confused me or you are trying to restore an AIX db to Windows. If you are trying to restore an AIX db to Windows you have discovered that it simply does not work. Which is the expected result.

 

[TomBascom]

If you are changing the topic you should open a new thread and try to clearly state what it is that you need.

 

[TomBascom]

You should also attempt to actually answer clarifying questions that people ask you. Ignoring those questions i.e. "Have you rebooted the server?" is rude and does not encourage people to keep trying to help.

 

[Jack@dba]

Hi Tom,

Kindly aplogise for that I will open a new thread.

FYI... Yes we restarted databases all hung user process got cleared from server database and application running fine.

 

[Jack@dba]

48 hours later and you are still having the problem? Your users are exceptionally patient.

As Cringer suggests, restarting all of the databases or rebooting will indeed usually clear these things up. Have you tried that?

The original SQL92 failure is unusual. I would suspect a network issue. You could try opening a ticket with tech support but Progress version 9 is ancient, obsolete and unsupported. AIX 5.3 is not much better. You really need to invest in some infrastructure updates.

"Out of resources" is a more routine problem. It means that you are out of some resource. Often it means that you have too many users connected. Frequently this happens after some sessions become disconnected but not completely logged out -- which is why restarting the db (or rebooting) will clear it up.

Hi Tom

To overcome this shared memory issue

1) Is there any way we can monitor from database log files.

2) Please advise how we can find shared memory issue before database crash

 

[TomBascom]

I do not see any prior references to a "shared memory issue".

Is this yet a 3rd issue in the same thread?

 

[Jack@dba]

I do not see any prior references to a "shared memory issue".

Is this yet a 3rd issue in the same thread?

No Tom, I am asking for first issue resolution for that.

In future how we can avoid memory violation errors and how to monitor from database level.

It's urgent plz update ASAP.

 

[TomBascom]

"memory violation" has nothing to do with shared memory.

There is no general cure-all for memory violations. They occur for many reasons. Very few of those reasons (I cannot actually think of *any*) are things that you can control. Most of them are due to factors outside of Progress (like faulty hardware) or bugs in Progress. The initial error in your first post _might_ have been faulty RAM. Or it might have been something else -- you would need to work with tech support to dig deeper but given that you are on version 9 *and* on an ancient, obsolete and unsupported HW platform I wouldn't hold my breath.

Monitoring is easy. Implement ProTop. (The paid versions of ProTop include alerts for messages in the .lg file.) Even though v9 is ancient and obsolete we can implement ProTop on v9 -- we do charge extra for that though.

 

[RealHeavyDude]

Really like the "we can implement ProTop on V9 -- we do charge extra for that though." statemtent.

 

[TomBascom]

We don't charge nearly *enough* extra it is a significant PITA to support v9.