Question Db hanged issue with user unable to access application .

[Ravi09]

Hi All ,

Good morning!
Greetings!

We had an issue with database the database was hung state ( No user able to access database and not even any transaction chui mpro not working)
Db was in hung state all the activities and transaction were freeze due to less shared memory. We tried to shut down Db with normal Unconditional shutdown process and disconnect some hanged processes .But nothing was working. Finally we did the last option to have the Database in Emergency shutdown to release hanged processes. Emergency shutdown is the preferred method of forcing a shutdown of a "hung" database. A "hung" database is a database that cannot be shutdown normally (Unconditional Shutdown ).Database went down successfully with Emergency shutdown. Afterwards we tried to start database but it was showing No more available shared-memory (1138),database not have enough memory to start the database broker. After some time we tried to start database again and it started successfully. It might be possibilities that shared memory got release .And after we called to Unix team to check shared memory on server so the database will run smoothly. Unix team checked and confirmed that Server has free memory available now. Unix team justify they have enough memory.

We are seeking any RCA coz we did not find anything in log file of database. Why shared memory suddenly exhausted?

Progress version :- 9.1d its 32 bit
-B value is 60000
Server version:- HP-UX

What preventive measurement we can take so that in future this issue will not happens again? Please suggest.



Thanks and regards
Ravi

 

[TomBascom]

32 bit HPUX has a limit of something like 11 shared memory segments across all users of shared memory and the total amount of shared memory allowed is something really small. Around 1.5GB or less if I recall correctly.

You mention “Db was in hung state all the activities and transaction were freeze due to less shared memory.” How do you know that it was in that state for that reason? Were there some specific error messages that indicated that? If so… please share them.

From the HPUX side of things if you have indications that you cannot start a db due to insufficient shared memory you can ask the OS what shared memory is being allocated by whom with the “ipcs” command. You can then tie that back to running processes to get an understanding of where it is being used.

Obviously it is too late now to go back and look but if it happens again you could get some insight that way.

 

[Ravi09]

Thank you so much for valuable suggestions. Really appreciate it.
How we know the progress version is 32bit or 64 bit any specific info?
What preventive measurement we can take so that in future this issue will not happens again? Please suggest.?

Thanks and Regards
Ravi

 

[Ravi09]

32 bit HPUX has a limit of something like 11 shared memory segments across all users of shared memory and the total amount of shared memory allowed is something really small. Around 1.5GB or less if I recall correctly.

You mention “Db was in hung state all the activities and transaction were freeze due to less shared memory.” How do you know that it was in that state for that reason? Were there some specific error messages that indicated that? If so… please share them.

From the HPUX side of things if you have indications that you cannot start a db due to insufficient shared memory you can ask the OS what shared memory is being allocated by whom with the “ipcs” command. You can then tie that back to running processes to get an understanding of where it is being used.

Obviously it is too late now to go back and look but if it happens again you could get some insight that way.

+ adding this is ipcs result
How can we find which is having more shared memory?


ID ShMemVer Seg# InUse Database
0 - - - (not PROGRESS)
1 - - - (not PROGRESS)
2 - - - (not PROGRESS)
32771 - - - (not PROGRESS)
4 - - - (not PROGRESS)
163848 - - - (not PROGRESS)
10 - - - (not PROGRESS)
11 9126 0 Yes /db/nux.db
12 9126 1 - /db/nux.db
13 9126 2 - /db/nux.db
14 9126 3 - /db/nux.db
15 9126 0 Yes /db/nux.db
16 9126 0 Yes /db/admnux.db
17 9126 0 Yes /db/barnux.db
18 9126 0 Yes /db/hngx.db
19 9126 0 Yes /db/hng.db
20 9126 1 - /db/hng.db
21 9126 2 - /db/hng.db
22 9126 3 - /db/hng.db
23 9126 4 - /db/hng.db
24 9126 0 Yes /db/admhng.db
25 9126 0 Yes /db/taan.db
26 9126 1 - /db/taan.db
27 9126 2 - /db/taan.db
28 9126 3 - /db/taan.db
29 9126 4 - /db/taan.db
30 9126 5 - /db/taan.db
31 9126 0 Yes /db/taanx.db
32 9126 0 Yes /db/admtaan.db
33 9126 0 Yes /db/addtaan.db
37 - - - (not PROGRESS)

How we know the progress version is 32bit or 64 bit any specific info?
What preventive measurement we can take so that in future this issue will not happens again? Please suggest.?

Thanks and Regards
Ravi