QUICK Help : Shared memory utilization

#1
Hi team,

our database went down due to memory violation error.I checked in promon > 1> Status: Shared Memory Segments
It shows all memory utilized.
I tried to kill Seg id with ipcrm -m id to free it but still it is showing utilized.

what action i need to take to resolve this issue.If reboot the server whether it will resolve this issue.


04/09/18 Status: Shared Memory Segments
19:30:12

Seg Id Size Used Free

1 393265 134217728 134217724 4
2 393248 134217728 134217724 4
3 393264 134217728 134217716 12
4 393247 134217728 134217716 12

5 393228 134217728 120246044 13971684
6 393263 25251840 396 25251444

Enter <return>, R, P, T, or X (? for help):

19:08:39 SQLSRV2 6: SYSTEM ERROR: Memory violation. (49)
19:08:47 SQLSRV2 3: SYSTEM ERROR: Memory violation. (49)
19:08:50 SQLSRV2 5: SYSTEM ERROR: Memory violation. (49)
19:08:57 SQLSRV2 4: SYSTEM ERROR: Memory violation. (49)
19:08:59 SRV 35: Corrupt block detected when attempting to release a buffer. (4232)
19:08:59 SRV 35: bmReleaseBuffer: Error occurred in area 6, block number: 7171557, extent: /progdata/prd/slcdb1/prdslc.d1. (10560)
19:08:59 SRV 35: Writing block 7171557 to log file. Please save and send the log file to Progress Software Corp. for investigation. (10561)
19:08:59 SRV 35: SYSTEM DEBUG: Database buffer block
PROGRESS Version 9.1E as of Tue Oct 12 17:19:22 EDT 2004

PROGRESS SHARED MEMORY STATUS
ID ShMemVer Seg# InUse Database
131072 - - - (not PROGRESS)
131073 - - - (not PROGRESS)

===================
Code:
IPC status from /dev/mem as of Mon Apr  9 20:04:16 EDT 2018
T        ID     KEY        MODE       OWNER    GROUP  CREATOR   CGROUP NATTCH     SEGSZ  CPID  LPID   ATIME    DTIME    CTIME
Shared Memory:
m    131072 0x7800002e --rw-rw-rw-     root   system     root   system      1  33554432 35356 627454 19:57:48 19:57:48  2:04:49
m    131073 0x78000028 --rw-rw-rw-     root   system     root   system      1  33554432 35356 627454 19:57:48 19:57:48  2:04:49
m    131074 0x0d05332f --rw-rw----     root   system     root   system     99      1440  5964 504076 20:04:11 20:04:11  2:04:04
m    262147 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     41 134217728 131470 49074 19:00:01 19:00:01  4:13:46
m    262148 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     41 134217728 131470 49074 19:00:01 19:00:01  4:13:46
m    262149 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     52   6246400 233220 59968 17:00:02 17:00:02  4:13:46
m    262150 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     15 134217728 107732 486232 19:00:01 19:00:01  4:13:46
m    262151 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress      4 128491520 53206 63498  4:14:00  4:14:00  4:13:46
m    262152 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     44 134217728 217122 318206 20:00:00 20:00:06  4:13:47
m    262153 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress      5   6672384 174530 636462 12:13:05 12:13:05  4:13:47
m    262154 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     37 134217728 163492 397392 20:00:01 20:00:11  4:13:47
m    262155 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     37 134217728 163492 397392 20:00:01 20:00:11  4:13:47
m    524300 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     13 134217728 538100 618496 20:02:00 20:03:55 19:51:31
m    262157 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     15 134217728 107732 486232 19:00:01 19:00:01  4:13:46
m    262158 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     15 134217728 107732 486232 19:00:01 19:00:01  4:13:46
m    262159 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     15 134217728 107732 486232 19:00:01 19:00:01  4:13:46
m    262160 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     41 134217728 131470 49074 19:00:01 19:00:01  4:13:46
m    262161 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     44 134217728 217122 318206 20:00:00 20:00:06  4:13:47
m    262162 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     44 134217728 217122 318206 20:00:00 20:00:06  4:13:47
m    262163 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     44  26300416 217122 318206 20:00:00 20:00:06  4:13:47
m    262164 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     56   7151616 62802 674100 19:00:02 19:00:02  4:13:47
m    262165 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress      6  43663360 28982 103000 19:13:25 19:13:25  4:13:47
m    262166 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     33 134217728 97290 707660 19:00:02 19:00:02  4:13:47
m    262167 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     37 134217728 163492 397392 20:00:01 20:00:11  4:13:47
m    262168 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     33  38039552 97290 707660 19:00:02 19:00:02  4:13:47
m    262169 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     33 134217728 97290 707660 19:00:02 19:00:02  4:13:47
m    262170 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress      4   2891776 63068 656402 17:00:02 17:00:02  4:13:47
m    262171 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress      6  41148416 46898 535354 19:35:00 19:35:06  4:13:47
m    262172 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress      9 134217728 239146 482782 19:00:02 19:00:02  4:13:46
m    262173 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress      9  37187584 239146 482782 19:00:02 19:00:02  4:13:46
m    262174 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     37 126652416 163492 397392 20:00:01 20:00:11  4:13:47
m    524319 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     13 134217728 538100 618496 20:02:00 20:03:55 19:51:31
m    524320 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     13 134217728 538100 618496 20:02:00 20:03:55 19:51:31
m    262177 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     15 126713856 107732 486232 19:00:01 19:00:01  4:13:46
m    262178 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress      9 134217728 239146 482782 19:00:02 19:00:02  4:13:46
m    262179 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress      9 134217728 239146 482782 19:00:02 19:00:02  4:13:46
m    262180 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress      1   3678208 65614 65614  4:13:46 no-entry  4:13:46
m    262181 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     41 134217728 131470 49074 19:00:01 19:00:01  4:13:46
m    262182 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     41  37175296 131470 49074 19:00:01 19:00:01  4:13:46
m    262183 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     52   6307840 162514 539690 17:00:03 17:00:03  4:13:46
m    262184 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress      1   2830336 33556 680714 17:00:03 17:00:03  4:13:46
m    262185 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress      1   2830336 115754 115754  4:13:46 no-entry  4:13:46
m    262186 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress      1   2830336 118642 118642  4:13:46 no-entry  4:13:46
m    262187 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress      1   2830336 187674 651600 17:00:03 17:00:03  4:13:46
m    262188 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     33 134217728 97290 707660 19:00:02 19:00:02  4:13:47
m    262189 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     37 134217728 163492 397392 20:00:01 20:00:11  4:13:47
m    262190 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress      9 134217728 239146 482782 19:00:02 19:00:02  4:13:46
m    524335 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     13 134217728 538100 618496 20:02:00 20:03:55 19:51:31
m    524336 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     13 134217728 538100 618496 20:02:00 20:03:55 19:51:31
m    524337 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     13  25251840 538100 618496 20:02:00 20:03:55 19:51:31
m    262194 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     52   6426624 145860 378458 17:00:02 17:00:02  4:13:46
m    262195 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     33 134217728 97290 707660 19:00:02 19:00:02  4:13:47
 
Last edited by a moderator:

Rob Fitzpatrick

ProgressTalk.com Sponsor
#2
When posting text from logs or screens, please put it within CODE tags, with the TEXT qualifier. This will make the data you are asking us to interpret mmuch more legible.

Example:
[_code=text_]
...some text...
[_/code_]

Don't type the underscores. I have put them in so the above won't be interpreted as actual tags.
 

Rob Fitzpatrick

ProgressTalk.com Sponsor
#4
our database went down due to memory violation error.I checked in promon > 1> Status: Shared Memory Segments
If the database was down, how were you in promon looking at the database state?

It shows all memory utilized.
No, it shows six shared memory segments, four fully used, one partially used, one not used. But I don't think this is relevant and it is not clear if the state we are seeing is the state of the database pre-crash or the state of the database after it came back up. Please clarify.

I tried to kill Seg id with ipcrm -m id to free it but still it is showing utilized.
I don't see why you are trying to delete shared memory segments at the OS level. The shared memory segment IDs shown in your promon output don't match any of the segment IDs shown in the ipcs command. There is nothing to delete. The segments shown in ipcs seem to belong to other database instances on that machine.
 
#5
Database went down and quckily restated database.

I found in 9.1e doc during this issue we need to release shared segment to free up the memory using ipcrm -m id
 

Rob Fitzpatrick

ProgressTalk.com Sponsor
#6
Corrupt block detected when attempting to release a buffer. (4232)
bmReleaseBuffer: Error occurred in area 6, block number: 7171557, extent: /progdata/prd/slcdb1/prdslc.d1. (10560)
Writing block 7171557 to log file. Please save and send the log file to Progress Software Corp. for investigation. (10561)
This is the area for concern. From ProKB, this is the explanation for a (4232) error:
The Progress DBMS posted a read request for a database block with <dbkey2> but the block header contains <dbkey>. This error can be caused by any of the following: broken disks or controllers, broken memory, corrupted filesystems, filesystem bugs (e.g. not flushing inodes) disk driver bugs, operating system bugs or Progress bugs. This error may indicate that your database is corrupted. If you are unable to restart your database, you should contact Progress technical support to get help determining the extent of the corruption. If your database is corrupted you will have to restore it from a backup.
I suggest you follow your escalation procedure, e.g. if this is a vendor application, open a case with the vendor. If you are a Progress direct customer, open a case with Progress Tech Support.
 
#8
We don't have vendor supportand also we checked with vendor they are out of support for 9.1e version.

Can I reboot the server to fix this issue
 

Rob Fitzpatrick

ProgressTalk.com Sponsor
#9
The reason I ask about the number of databases is that your ipcs -m output suggests there are a lot of shared memory segments. But I don't know whether they belong to running databases or they were left behind by crashed databases. You may be able to find those numbers in your database logs. Look for (9336) messages, just after database startup. E.g.:
Code:
16:28:56 BROKER  0: Created shared memory with segment_id: 524290 (9336)
If you compare the segment IDs shown by ipcs with the ones in those log messages, taking into account whether those database instances are still running, you should be able to determine which segments *should* still appear at the OS level.
 
#11
HI Rob,

as per your suggestion i verified in database log file


4:13:56 BROKER 0: Created shared memory with segment_id: 262191 (9336)
04:13:56 BROKER 0: Created shared memory with segment_id: 262156 (9336)
04:13:56 BROKER 0: Created shared memory with segment_id: 262175 (9336)
04:13:56 BROKER 0: Created shared memory with segment_id: 262192 (9336)
04:13:56 BROKER 0: Created shared memory with segment_id: 262176 (9336)
04:13:56 BROKER 0: Created shared memory with segment_id: 262193 (9336)

But i not able to find these id in ipcs server level

ipcs -m | grep 262192
ipcs -m | grep 262176
ipcs -m | grep 262193
 
Top