Checkpoint Flushes
- Thread starter VageeshA
- Start date
Please find the details:
I am checking this information on the PROMON Checkpoint screen.
There are more than 15000 Flushes per checkpoint.
We are using Progress 9.1E04.
As per our admin team there is no strain on disk subsystem
part of sar -d output during the issue is pasted below:
01:15:05 device %busy avque r+w/s blks/s avwait avserv
01:15:09 md0 0 0.0 0 0 0.0 0.0
md1 0 0.0 0 0 0.0 0.0
md3 15 0.9 33 971 0.0 26.6
md4 0 0.0 0 0 0.0 0.0
md5 0 0.0 0 8 4.0 6.4
md10 0 0.0 0 0 0.0 0.0
md11 0 0.0 0 0 0.0 0.0
md13 13 0.6 33 995 0.0 19.8
md14 0 0.0 0 0 0.0 0.0
md15 0 0.0 0 8 0.0 6.4
md20 0 0.0 0 0 0.0 0.0
md21 0 0.0 0 0 0.0 0.0
md23 13 0.4 33 971 0.0 13.1
md24 0 0.0 0 0 0.0 0.0
md25 0 0.0 0 8 0.0 4.5
nfs1 0 0.0 0 0 0.0 0.0
nfs2 0 0.0 0 0 0.0 0.0
nfs40 0 0.0 0 0 0.0 0.0
sd0 0 0.0 0 0 0.0 0.0
sd0,a 0 0.0 0 0 0.0 0.0
sd0,b 0 0.0 0 0 0.0 0.0
sd0,c 0 0.0 0 0 0.0 0.0
sd0,d 0 0.0 0 0 0.0 0.0
sd0,g 0 0.0 0 0 0.0 0.0
sd1 14 0.4 34 990 0.0 12.7
sd1,a 0 0.0 0 0 0.0 0.0
sd1,b 0 0.0 0 0 0.0 0.0
sd1,c 0 0.0 0 0 0.0 0.0
sd1,d 13 0.4 33 971 0.0 13.0
sd1,e 0 0.0 0 0 0.0 0.0
sd1,f 0 0.0 0 8 0.0 4.4
sd1,h 0 0.0 0 0 0.0 9.1
sd2 14 0.7 34 1014 0.0 19.1
sd2,a 0 0.0 0 0 0.0 0.0
sd2,b 0 0.0 0 0 0.0 0.0
sd2,c 0 0.0 0 0 0.0 0.0
sd2,d 13 0.6 33 995 0.0 19.7
sd2,e 0 0.0 0 0 0.0 0.0
sd2,f 0 0.0 0 8 0.0 6.3
Thank you,
Vageesh
I am checking this information on the PROMON Checkpoint screen.
There are more than 15000 Flushes per checkpoint.
We are using Progress 9.1E04.
As per our admin team there is no strain on disk subsystem
part of sar -d output during the issue is pasted below:
01:15:05 device %busy avque r+w/s blks/s avwait avserv
01:15:09 md0 0 0.0 0 0 0.0 0.0
md1 0 0.0 0 0 0.0 0.0
md3 15 0.9 33 971 0.0 26.6
md4 0 0.0 0 0 0.0 0.0
md5 0 0.0 0 8 4.0 6.4
md10 0 0.0 0 0 0.0 0.0
md11 0 0.0 0 0 0.0 0.0
md13 13 0.6 33 995 0.0 19.8
md14 0 0.0 0 0 0.0 0.0
md15 0 0.0 0 8 0.0 6.4
md20 0 0.0 0 0 0.0 0.0
md21 0 0.0 0 0 0.0 0.0
md23 13 0.4 33 971 0.0 13.1
md24 0 0.0 0 0 0.0 0.0
md25 0 0.0 0 8 0.0 4.5
nfs1 0 0.0 0 0 0.0 0.0
nfs2 0 0.0 0 0 0.0 0.0
nfs40 0 0.0 0 0 0.0 0.0
sd0 0 0.0 0 0 0.0 0.0
sd0,a 0 0.0 0 0 0.0 0.0
sd0,b 0 0.0 0 0 0.0 0.0
sd0,c 0 0.0 0 0 0.0 0.0
sd0,d 0 0.0 0 0 0.0 0.0
sd0,g 0 0.0 0 0 0.0 0.0
sd1 14 0.4 34 990 0.0 12.7
sd1,a 0 0.0 0 0 0.0 0.0
sd1,b 0 0.0 0 0 0.0 0.0
sd1,c 0 0.0 0 0 0.0 0.0
sd1,d 13 0.4 33 971 0.0 13.0
sd1,e 0 0.0 0 0 0.0 0.0
sd1,f 0 0.0 0 8 0.0 4.4
sd1,h 0 0.0 0 0 0.0 9.1
sd2 14 0.7 34 1014 0.0 19.1
sd2,a 0 0.0 0 0 0.0 0.0
sd2,b 0 0.0 0 0 0.0 0.0
sd2,c 0 0.0 0 0 0.0 0.0
sd2,d 13 0.6 33 995 0.0 19.7
sd2,e 0 0.0 0 0 0.0 0.0
sd2,f 0 0.0 0 8 0.0 6.3
Thank you,
Vageesh
TomBascom
Curmudgeon
9.1E04 is ancient, obsolete and almost unsupported. You should upgrade.
Please show the details regarding flushes at checkpoints. The PROMON checkpoint screen would be appropriate. It would also be interesting to see the summary screen (sampled during a busy period -- not the initial "since db startup" metrics).
This is apparently some sort of UNIX. Which particular OS?
Ok, sar, doesn't show any obvious issues. None the less... tell us about the disk subsystem -- is it a SAN? Or plain old disks? Any sort of RAID?
DB startup parameters would be of interest. The first part of the .lg file when you start the db will tell us what we need to know.
Please show the details regarding flushes at checkpoints. The PROMON checkpoint screen would be appropriate. It would also be interesting to see the summary screen (sampled during a busy period -- not the initial "since db startup" metrics).
This is apparently some sort of UNIX. Which particular OS?
Ok, sar, doesn't show any obvious issues. None the less... tell us about the disk subsystem -- is it a SAN? Or plain old disks? Any sort of RAID?
DB startup parameters would be of interest. The first part of the .lg file when you start the db will tell us what we need to know.
Hello Tom,
Please find the details:
As per our developers we have some application dependency for upgrade to OE.
We are using Solaris 10, Databases are hosted on SAN with RAID 10
Screen caps of Checkpoint & Summary screen (tried to get it as close as possible to ckpt) are pasted below. Kindly excuse me for not attaching these as a document.
We find this issue during EOD processing and Todays process had only 25% of User load:
03/13/10 Checkpoints
01:12:17
Ckpt ------ Database Writes ------
No. Time Len Freq Dirty CPT Q Scan APW Q Flushes
1844 01:11:36 41 0 38848 0 1325 5550 0
1843 01:07:35 241 241 32119 0 0 29768 3175
1842 01:03:54 221 221 26800 8220 1699 19464 0
1841 00:59:58 236 236 19235 0 276 25829 776
1840 00:56:01 237 237 16602 0 248 30458 704
1839 00:51:05 286 296 17217 286 749 34399 0
1838 00:47:02 243 243 15376 356 2162 27675 190
1837 00:42:45 248 257 16043 376 2172 28430 0
--------------------------------------------------------------------------------------------
03/13/10 Activity: Summary
01:11:40 03/13/10 01:11 to 03/13/10 01:11 (10 sec)
Event Total Per Sec |Event Total Per Sec
Commits 1123 112.3 |DB Reads 4671 467.1
Undos 1 0.1 |DB Writes 6899 689.9
Record Reads 152184 15218.4 |BI Reads 0 0.0
Record Updates 672 67.2 |BI Writes 139 13.9
Record Creates 578 57.8 |AI Writes 79 7.9
Record Deletes 5 0.5 |Checkpoints 0 0.0
Record Locks 10621 1062.1 |Flushed at chkpt 796 79.6
Record Waits 0 0.0
Rec Lock Waits 0 % BI Buf Waits 0 % AI Buf Waits 0 %
Writes by APW 88 % Writes by BIW 49 % Writes by AIW 96 %
DB Size: 332 GB BI Size: 11 GB AI Size: 18 MB
Empty blocks:13764887 Free blocks: 8232 RM chain: 281564
Buffer Hits 99 % Active trans: 42
640 Servers, 229 Users (52 Local, 177 Remote, 51 Batch), 9 Apws
=======================================================
03/13/10 Checkpoints
01:33:21
Ckpt ------ Database Writes ------
No. Time Len Freq Dirty CPT Q Scan APW Q Flushes
1850 01:33:12 9 0 21302 0 151 3149 0
1849 01:29:47 205 205 18441 0 2313 25017 462
1848 01:26:10 209 217 32841 2081 1301 40714 0
1847 01:22:55 185 195 37430 26550 585 11738 0
1846 01:19:16 209 219 41420 41897 2190 2 0
1845 01:15:39 217 217 43539 15204 1769 23404 3987
1844 01:11:36 243 243 38848 3385 1957 27867 5939
1843 01:07:35 241 241 32119 0 0 29768 3175
--------------------------------------------------------------------------------------------
03/13/10 Activity: Summary
01:33:20 03/13/10 01:33 to 03/13/10 01:33 (10 sec)
Event Total Per Sec |Event Total Per Sec
Commits 2969 296.9 |DB Reads 9447 944.7
Undos 2 0.2 |DB Writes 3590 359.0
Record Reads 657813 65781.3 |BI Reads 4 0.4
Record Updates 2414 241.4 |BI Writes 319 31.9
Record Creates 780 78.0 |AI Writes 174 17.4
Record Deletes 2 0.2 |Checkpoints 1 0.1
Record Locks 19670 1967.0 |Flushed at chkpt 448 44.8
Record Waits 0 0.0
Rec Lock Waits 0 % BI Buf Waits 0 % AI Buf Waits 0 %
Writes by APW 87 % Writes by BIW 37 % Writes by AIW 97 %
DB Size: 332 GB BI Size: 11 GB AI Size: 58 MB
Empty blocks:13760309 Free blocks: 8232 RM chain: 281513
Buffer Hits 99 % Active trans: 43
640 Servers, 232 Users (52 Local, 180 Remote, 51 Batch), 9 Apws
=============================================================
Database Startup Parameters:
01:27:10 WDOG 655: Started. (2518)
01:27:11 BROKER 0: Progress OpenEdge Release 9.1E on SOLARIS. (4234)
01:27:11 BROKER 0: Server started by progress on batch. (4281)
01:27:11 BROKER 0: Started using pid: 19681. (6574)
01:27:11 BROKER 0: Physical Database Name (-db): /emc/reale/prod/amos/data/amos. (4235)
01:27:11 BROKER 0: Database Type (-dt): PROGRESS. (4236)
01:27:11 BROKER 0: Force Access (-F): Not Enabled. (4237)
01:27:11 BROKER 0: Direct I/O (-directio): Enabled. (4238)
01:27:11 BROKER 0: Number of Database Buffers (-B): 250000. (4239)
01:27:11 BROKER 0: Maximum private buffers per user (-Bpmax): 64. (9422)
01:27:11 BROKER 0: Excess Shared Memory Size (-Mxs): 16970. (4240)
01:27:11 BROKER 0: The shared memory segment is locked in memory. (10013)
01:27:11 BROKER 0: Current Size of Lock Table (-L): 200000. (4241)
01:27:11 BROKER 0: Hash Table Entries (-hash): 70289. (4242)
01:27:11 BROKER 0: Current Spin Lock Tries (-spin): 50000. (4243)
01:27:11 BROKER 0: Number of Semaphore Sets (-semsets): 30. (6526)
01:27:11 BROKER 0: Crash Recovery (-i): Enabled. (4244)
01:27:11 BROKER 0: Database Blocksize (-blocksize): 8192. (6573)
01:27:11 BROKER 0: Delay of Before-Image Flush (-Mf): 3. (4245)
01:27:11 BROKER 0: Before-Image File I/O (-r -R): Reliable. (4247)
01:27:11 BROKER 0: Before-Image Truncate Interval (-G): 0. (4249)
01:27:11 BROKER 0: Before-Image Cluster Size: 50331648. (4250)
01:27:11 BROKER 0: Before-Image Block Size: 16384. (4251)
01:27:11 BROKER 0: Number of Before-Image Buffers (-bibufs): 100. (4252)
01:27:11 BROKER 0: BI File Threshold size (-bithold): 0.0 Bytes. (9238)
01:27:11 BROKER 0: BI File Threshold Stall (-bistall): Enabled. (6551)
01:27:11 BROKER 0: After-Image Stall (-aistall): Enabled. (4254)
01:27:11 BROKER 0: After-Image Block Size: 16384. (4255)
01:27:11 BROKER 0: Number of After-Image Buffers (-aibufs): 100. (4256)
01:27:11 BROKER 0: Storage object cache size (-omsize): 1024 (8527)
01:27:11 BROKER 0: Maximum Number of Clients Per Server (-Ma): 3. (4257)
01:27:11 BROKER 0: Maximum Number of Servers (-Mn): 641. (4258)
01:27:11 BROKER 0: Minimum Clients Per Server (-Mi): 1. (4259)
01:27:11 BROKER 0: Maximum Number of Users (-n): 2001. (4260)
01:27:11 BROKER 0: Host Name (-H): nosxprrsnp01. (4261)
01:27:11 BROKER 0: Service Name (-S): amos_prod_odbc. (4262)
01:27:11 BROKER 0: Network Type (-N): tcp. (4263)
01:27:11 BROKER 0: Character Set (-cpinternal): ISO8859-15. (4264)
01:27:11 BROKER 0: Parameter File: /emc/reale/prod/amos/data/amos.pf. (4282)
01:27:11 BROKER 0: Minimum Port for Auto Servers (-minport): 2000. (5648)
01:27:11 BROKER 0: Maximum Port for Auto Servers (-maxport): 20000. (5649)
Many Thanks,
Vageesh
Please find the details:
As per our developers we have some application dependency for upgrade to OE.
We are using Solaris 10, Databases are hosted on SAN with RAID 10
Screen caps of Checkpoint & Summary screen (tried to get it as close as possible to ckpt) are pasted below. Kindly excuse me for not attaching these as a document.
We find this issue during EOD processing and Todays process had only 25% of User load:
03/13/10 Checkpoints
01:12:17
Ckpt ------ Database Writes ------
No. Time Len Freq Dirty CPT Q Scan APW Q Flushes
1844 01:11:36 41 0 38848 0 1325 5550 0
1843 01:07:35 241 241 32119 0 0 29768 3175
1842 01:03:54 221 221 26800 8220 1699 19464 0
1841 00:59:58 236 236 19235 0 276 25829 776
1840 00:56:01 237 237 16602 0 248 30458 704
1839 00:51:05 286 296 17217 286 749 34399 0
1838 00:47:02 243 243 15376 356 2162 27675 190
1837 00:42:45 248 257 16043 376 2172 28430 0
--------------------------------------------------------------------------------------------
03/13/10 Activity: Summary
01:11:40 03/13/10 01:11 to 03/13/10 01:11 (10 sec)
Event Total Per Sec |Event Total Per Sec
Commits 1123 112.3 |DB Reads 4671 467.1
Undos 1 0.1 |DB Writes 6899 689.9
Record Reads 152184 15218.4 |BI Reads 0 0.0
Record Updates 672 67.2 |BI Writes 139 13.9
Record Creates 578 57.8 |AI Writes 79 7.9
Record Deletes 5 0.5 |Checkpoints 0 0.0
Record Locks 10621 1062.1 |Flushed at chkpt 796 79.6
Record Waits 0 0.0
Rec Lock Waits 0 % BI Buf Waits 0 % AI Buf Waits 0 %
Writes by APW 88 % Writes by BIW 49 % Writes by AIW 96 %
DB Size: 332 GB BI Size: 11 GB AI Size: 18 MB
Empty blocks:13764887 Free blocks: 8232 RM chain: 281564
Buffer Hits 99 % Active trans: 42
640 Servers, 229 Users (52 Local, 177 Remote, 51 Batch), 9 Apws
=======================================================
03/13/10 Checkpoints
01:33:21
Ckpt ------ Database Writes ------
No. Time Len Freq Dirty CPT Q Scan APW Q Flushes
1850 01:33:12 9 0 21302 0 151 3149 0
1849 01:29:47 205 205 18441 0 2313 25017 462
1848 01:26:10 209 217 32841 2081 1301 40714 0
1847 01:22:55 185 195 37430 26550 585 11738 0
1846 01:19:16 209 219 41420 41897 2190 2 0
1845 01:15:39 217 217 43539 15204 1769 23404 3987
1844 01:11:36 243 243 38848 3385 1957 27867 5939
1843 01:07:35 241 241 32119 0 0 29768 3175
--------------------------------------------------------------------------------------------
03/13/10 Activity: Summary
01:33:20 03/13/10 01:33 to 03/13/10 01:33 (10 sec)
Event Total Per Sec |Event Total Per Sec
Commits 2969 296.9 |DB Reads 9447 944.7
Undos 2 0.2 |DB Writes 3590 359.0
Record Reads 657813 65781.3 |BI Reads 4 0.4
Record Updates 2414 241.4 |BI Writes 319 31.9
Record Creates 780 78.0 |AI Writes 174 17.4
Record Deletes 2 0.2 |Checkpoints 1 0.1
Record Locks 19670 1967.0 |Flushed at chkpt 448 44.8
Record Waits 0 0.0
Rec Lock Waits 0 % BI Buf Waits 0 % AI Buf Waits 0 %
Writes by APW 87 % Writes by BIW 37 % Writes by AIW 97 %
DB Size: 332 GB BI Size: 11 GB AI Size: 58 MB
Empty blocks:13760309 Free blocks: 8232 RM chain: 281513
Buffer Hits 99 % Active trans: 43
640 Servers, 232 Users (52 Local, 180 Remote, 51 Batch), 9 Apws
=============================================================
Database Startup Parameters:
01:27:10 WDOG 655: Started. (2518)
01:27:11 BROKER 0: Progress OpenEdge Release 9.1E on SOLARIS. (4234)
01:27:11 BROKER 0: Server started by progress on batch. (4281)
01:27:11 BROKER 0: Started using pid: 19681. (6574)
01:27:11 BROKER 0: Physical Database Name (-db): /emc/reale/prod/amos/data/amos. (4235)
01:27:11 BROKER 0: Database Type (-dt): PROGRESS. (4236)
01:27:11 BROKER 0: Force Access (-F): Not Enabled. (4237)
01:27:11 BROKER 0: Direct I/O (-directio): Enabled. (4238)
01:27:11 BROKER 0: Number of Database Buffers (-B): 250000. (4239)
01:27:11 BROKER 0: Maximum private buffers per user (-Bpmax): 64. (9422)
01:27:11 BROKER 0: Excess Shared Memory Size (-Mxs): 16970. (4240)
01:27:11 BROKER 0: The shared memory segment is locked in memory. (10013)
01:27:11 BROKER 0: Current Size of Lock Table (-L): 200000. (4241)
01:27:11 BROKER 0: Hash Table Entries (-hash): 70289. (4242)
01:27:11 BROKER 0: Current Spin Lock Tries (-spin): 50000. (4243)
01:27:11 BROKER 0: Number of Semaphore Sets (-semsets): 30. (6526)
01:27:11 BROKER 0: Crash Recovery (-i): Enabled. (4244)
01:27:11 BROKER 0: Database Blocksize (-blocksize): 8192. (6573)
01:27:11 BROKER 0: Delay of Before-Image Flush (-Mf): 3. (4245)
01:27:11 BROKER 0: Before-Image File I/O (-r -R): Reliable. (4247)
01:27:11 BROKER 0: Before-Image Truncate Interval (-G): 0. (4249)
01:27:11 BROKER 0: Before-Image Cluster Size: 50331648. (4250)
01:27:11 BROKER 0: Before-Image Block Size: 16384. (4251)
01:27:11 BROKER 0: Number of Before-Image Buffers (-bibufs): 100. (4252)
01:27:11 BROKER 0: BI File Threshold size (-bithold): 0.0 Bytes. (9238)
01:27:11 BROKER 0: BI File Threshold Stall (-bistall): Enabled. (6551)
01:27:11 BROKER 0: After-Image Stall (-aistall): Enabled. (4254)
01:27:11 BROKER 0: After-Image Block Size: 16384. (4255)
01:27:11 BROKER 0: Number of After-Image Buffers (-aibufs): 100. (4256)
01:27:11 BROKER 0: Storage object cache size (-omsize): 1024 (8527)
01:27:11 BROKER 0: Maximum Number of Clients Per Server (-Ma): 3. (4257)
01:27:11 BROKER 0: Maximum Number of Servers (-Mn): 641. (4258)
01:27:11 BROKER 0: Minimum Clients Per Server (-Mi): 1. (4259)
01:27:11 BROKER 0: Maximum Number of Users (-n): 2001. (4260)
01:27:11 BROKER 0: Host Name (-H): nosxprrsnp01. (4261)
01:27:11 BROKER 0: Service Name (-S): amos_prod_odbc. (4262)
01:27:11 BROKER 0: Network Type (-N): tcp. (4263)
01:27:11 BROKER 0: Character Set (-cpinternal): ISO8859-15. (4264)
01:27:11 BROKER 0: Parameter File: /emc/reale/prod/amos/data/amos.pf. (4282)
01:27:11 BROKER 0: Minimum Port for Auto Servers (-minport): 2000. (5648)
01:27:11 BROKER 0: Maximum Port for Auto Servers (-maxport): 20000. (5649)
Many Thanks,
Vageesh
TomBascom
Curmudgeon
Like what? Have they actually tried it and found a specific problem? If they have what is that specific problem? (I'll bet you that they cannot tell you...)
If you can compile code you can upgrade.
Upgrade does not mean that you have to start using all of the new coding techniques and features -- all of the old stuff continues to work the same as always.
This sort of stuff is much easier to read if you enclose it between [ C O D E ] tags. Like so:
Code:
We find this issue during EOD processing and Todays process had only 25% of User load:
03/13/10 Checkpoints
01:12:17
Ckpt ------ Database Writes ------
No. Time Len Freq Dirty CPT Q Scan APW Q Flushes
1844 01:11:36 41 0 38848 0 1325 5550 0
1843 01:07:35 241 241 32119 0 0 29768 3175
1842 01:03:54 221 221 26800 8220 1699 19464 0
1841 00:59:58 236 236 19235 0 276 25829 776
1840 00:56:01 237 237 16602 0 248 30458 704
1839 00:51:05 286 296 17217 286 749 34399 0
1838 00:47:02 243 243 15376 356 2162 27675 190
1837 00:42:45 248 257 16043 376 2172 28430 0
--------------------------------------------------------------------------------------------
03/13/10 Activity: Summary
01:11:40 03/13/10 01:11 to 03/13/10 01:11 (10 sec)
Event Total Per Sec |Event Total Per Sec
Commits 1123 112.3 |DB Reads 4671 467.1
Undos 1 0.1 |DB Writes 6899 689.9
Record Reads 152184 15218.4 |BI Reads 0 0.0
Record Updates 672 67.2 |BI Writes 139 13.9
Record Creates 578 57.8 |AI Writes 79 7.9
Record Deletes 5 0.5 |Checkpoints 0 0.0
Record Locks 10621 1062.1 |Flushed at chkpt 796 79.6
Record Waits 0 0.0
Rec Lock Waits 0 % BI Buf Waits 0 % AI Buf Waits 0 %
Writes by APW 88 % Writes by BIW 49 % Writes by AIW 96 %
DB Size: 332 GB BI Size: 11 GB AI Size: 18 MB
Empty blocks:13764887 Free blocks: 8232 RM chain: 281564
Buffer Hits 99 % Active trans: 42
640 Servers, 229 Users (52 Local, 177 Remote, 51 Batch), 9 Apws
=======================================================
03/13/10 Checkpoints
01:33:21
Ckpt ------ Database Writes ------
No. Time Len Freq Dirty CPT Q Scan APW Q Flushes
1850 01:33:12 9 0 21302 0 151 3149 0
1849 01:29:47 205 205 18441 0 2313 25017 462
1848 01:26:10 209 217 32841 2081 1301 40714 0
1847 01:22:55 185 195 37430 26550 585 11738 0
1846 01:19:16 209 219 41420 41897 2190 2 0
1845 01:15:39 217 217 43539 15204 1769 23404 3987
1844 01:11:36 243 243 38848 3385 1957 27867 5939
1843 01:07:35 241 241 32119 0 0 29768 3175
--------------------------------------------------------------------------------------------
03/13/10 Activity: Summary
01:33:20 03/13/10 01:33 to 03/13/10 01:33 (10 sec)
Event Total Per Sec |Event Total Per Sec
Commits 2969 296.9 |DB Reads 9447 944.7
Undos 2 0.2 |DB Writes 3590 359.0
Record Reads 657813 65781.3 |BI Reads 4 0.4
Record Updates 2414 241.4 |BI Writes 319 31.9
Record Creates 780 78.0 |AI Writes 174 17.4
Record Deletes 2 0.2 |Checkpoints 1 0.1
Record Locks 19670 1967.0 |Flushed at chkpt 448 44.8
Record Waits 0 0.0
Rec Lock Waits 0 % BI Buf Waits 0 % AI Buf Waits 0 %
Writes by APW 87 % Writes by BIW 37 % Writes by AIW 97 %
DB Size: 332 GB BI Size: 11 GB AI Size: 58 MB
Empty blocks:13760309 Free blocks: 8232 RM chain: 281513
Buffer Hits 99 % Active trans: 43
640 Servers, 232 Users (52 Local, 180 Remote, 51 Batch), 9 Apws
=============================================================
Database Startup Parameters:
01:27:10 WDOG 655: Started. (2518)
01:27:11 BROKER 0: Progress OpenEdge Release 9.1E on SOLARIS. (4234)
01:27:11 BROKER 0: Server started by progress on batch. (4281)
01:27:11 BROKER 0: Started using pid: 19681. (6574)
01:27:11 BROKER 0: Physical Database Name (-db): /emc/reale/prod/amos/data/amos. (4235)
01:27:11 BROKER 0: Database Type (-dt): PROGRESS. (4236)
01:27:11 BROKER 0: Force Access (-F): Not Enabled. (4237)
01:27:11 BROKER 0: Direct I/O (-directio): Enabled. (4238)
01:27:11 BROKER 0: Number of Database Buffers (-B): 250000. (4239)
01:27:11 BROKER 0: Maximum private buffers per user (-Bpmax): 64. (9422)
01:27:11 BROKER 0: Excess Shared Memory Size (-Mxs): 16970. (4240)
01:27:11 BROKER 0: The shared memory segment is locked in memory. (10013)
01:27:11 BROKER 0: Current Size of Lock Table (-L): 200000. (4241)
01:27:11 BROKER 0: Hash Table Entries (-hash): 70289. (4242)
01:27:11 BROKER 0: Current Spin Lock Tries (-spin): 50000. (4243)
01:27:11 BROKER 0: Number of Semaphore Sets (-semsets): 30. (6526)
01:27:11 BROKER 0: Crash Recovery (-i): Enabled. (4244)
01:27:11 BROKER 0: Database Blocksize (-blocksize): 8192. (6573)
01:27:11 BROKER 0: Delay of Before-Image Flush (-Mf): 3. (4245)
01:27:11 BROKER 0: Before-Image File I/O (-r -R): Reliable. (4247)
01:27:11 BROKER 0: Before-Image Truncate Interval (-G): 0. (4249)
01:27:11 BROKER 0: Before-Image Cluster Size: 50331648. (4250)
01:27:11 BROKER 0: Before-Image Block Size: 16384. (4251)
01:27:11 BROKER 0: Number of Before-Image Buffers (-bibufs): 100. (4252)
01:27:11 BROKER 0: BI File Threshold size (-bithold): 0.0 Bytes. (9238)
01:27:11 BROKER 0: BI File Threshold Stall (-bistall): Enabled. (6551)
01:27:11 BROKER 0: After-Image Stall (-aistall): Enabled. (4254)
01:27:11 BROKER 0: After-Image Block Size: 16384. (4255)
01:27:11 BROKER 0: Number of After-Image Buffers (-aibufs): 100. (4256)
01:27:11 BROKER 0: Storage object cache size (-omsize): 1024 (8527)
01:27:11 BROKER 0: Maximum Number of Clients Per Server (-Ma): 3. (4257)
01:27:11 BROKER 0: Maximum Number of Servers (-Mn): 641. (4258)
01:27:11 BROKER 0: Minimum Clients Per Server (-Mi): 1. (4259)
01:27:11 BROKER 0: Maximum Number of Users (-n): 2001. (4260)
01:27:11 BROKER 0: Host Name (-H): nosxprrsnp01. (4261)
01:27:11 BROKER 0: Service Name (-S): amos_prod_odbc. (4262)
01:27:11 BROKER 0: Network Type (-N): tcp. (4263)
01:27:11 BROKER 0: Character Set (-cpinternal): ISO8859-15. (4264)
01:27:11 BROKER 0: Parameter File: /emc/reale/prod/amos/data/amos.pf. (4282)
01:27:11 BROKER 0: Minimum Port for Auto Servers (-minport): 2000. (5648)
01:27:11 BROKER 0: Maximum Port for Auto Servers (-maxport): 20000. (5649)TomBascom
Curmudgeon
Now that it's a bit more readable...
Are you sure that that SAR data you sent is from a busy period? There is a lot of IO being shown here that doesn't seem to have made it's way into that data.
How sure are you that the SAN is RAID10? I hate to say it but sys admins don't always know what they're talking about. Sometimes they just tell you what they think you want to hear.
You have an 11GB bi file? Why? Is your TRX scope a problem in the code?
Do you have bi buffer waits?
You say that the issue occurs during EOD processing. Is it a real issue (something isn't performing the way that you would like) or is it just that you see this in the metrics and have focused on it for that reason?
Are you sure that that SAR data you sent is from a busy period? There is a lot of IO being shown here that doesn't seem to have made it's way into that data.
How sure are you that the SAN is RAID10? I hate to say it but sys admins don't always know what they're talking about. Sometimes they just tell you what they think you want to hear.
You have an 11GB bi file? Why? Is your TRX scope a problem in the code?
Do you have bi buffer waits?
You say that the issue occurs during EOD processing. Is it a real issue (something isn't performing the way that you would like) or is it just that you see this in the metrics and have focused on it for that reason?
TomBascom
Curmudgeon
One last thing...
You have a large, complex system that is the key application in a multi-billion dollar enterprise. You might get some ideas here but we lack the full context. You really ought to get qualified professional help (see signature line for ideas on that) for this sort of problem as it is likely that it is not just a simple tuning tweak. In fact it is very likely that your issues span the database, sysadmin & programming domains and need to be addressed from a cross-discipline point of view.
You have a large, complex system that is the key application in a multi-billion dollar enterprise. You might get some ideas here but we lack the full context. You really ought to get qualified professional help (see signature line for ideas on that) for this sort of problem as it is likely that it is not just a simple tuning tweak. In fact it is very likely that your issues span the database, sysadmin & programming domains and need to be addressed from a cross-discipline point of view.
Hello Tom,
The issue looks related to Busy Disks. Please find the complete sar -d output.
Thanks,
Vageesh
The issue looks related to Busy Disks. Please find the complete sar -d output.
Code:
01:15:15 device %busy avque r+w/s blks/s avwait avserv
md3 32 2.4 83 2712 0.0 28.7
md13 29 1.3 83 2712 0.0 15.6
md23 28 1.3 83 2712 0.0 15.7
sd1 28 1.3 84 2718 0.0 15.6
sd1,d 28 1.3 84 2712 0.0 15.7
sd2 29 1.3 84 2718 0.0 15.5
sd2,d 29 1.3 84 2712 0.0 15.5
ssd300 22 0.4 5 108 0.0 80.7
ssd300,c 22 0.4 5 108 0.0 80.7
ssd301 26 0.4 5 99 0.0 77.6
ssd301,c 26 0.4 5 99 0.0 77.6
ssd302 2 0.0 17 528 0.0 1.2
ssd302,c 2 0.0 17 528 0.0 1.2
ssd303 1 0.0 13 425 0.0 1.1
ssd303,c 1 0.0 13 425 0.0 1.1
ssd304 3 0.0 26 825 0.0 1.2
ssd304,c 3 0.0 26 825 0.0 1.2
ssd305 3 0.0 21 663 0.0 1.3
ssd305,c 3 0.0 21 663 0.0 1.3
ssd306 59 1.2 33 866 0.0 37.5
ssd306,c 59 1.2 33 866 0.0 37.5
ssd307 73 1.5 44 1039 0.0 33.7
ssd307,c 73 1.5 44 1039 0.0 33.7
ssd308 95 8.8 135 3764 0.0 65.3
ssd308,c 95 8.8 135 3764 0.0 65.3
ssd309 97 9.6 148 3802 0.0 65.2
ssd309,c 97 9.6 148 3802 0.0 65.2
ssd310 11 0.2 5 92 0.0 33.3
ssd310,c 11 0.2 5 92 0.0 33.3
ssd311 15 0.2 7 132 0.0 33.5
ssd311,c 15 0.2 7 132 0.0 33.5
ssd312 59 1.2 37 594 0.0 31.1
ssd312,c 59 1.2 37 594 0.0 31.1
ssd313 67 1.3 50 807 0.0 26.2
ssd313,c 67 1.3 50 807 0.0 26.2
ssd314 76 9.9 144 4242 0.0 69.1
ssd314,c 76 9.9 144 4242 0.0 69.1
ssd315 73 8.3 117 4097 0.0 71.2
ssd315,c 73 8.3 117 4097 0.0 71.2
ssd318 20 0.4 7 158 0.0 52.1
ssd318,c 20 0.4 7 158 0.0 52.1
ssd319 14 0.3 7 147 0.0 42.1
ssd319,c 14 0.3 7 147 0.0 42.1
ssd320 29 0.8 22 421 0.0 36.4
ssd320,c 29 0.8 22 421 0.0 36.4
ssd321 48 1.0 27 498 0.0 36.2
ssd321,c 48 1.0 27 498 0.0 36.2
ssd322 2 0.1 5 105 0.0 14.0
ssd322,c 2 0.1 5 105 0.0 14.0
ssd323 1 0.0 3 54 0.0 13.5
ssd323,c 1 0.0 3 54 0.0 13.5
ssd324 2 0.1 9 179 0.0 14.6
ssd324,c 2 0.1 9 179 0.0 14.6
ssd325 3 0.1 10 198 0.0 12.9
ssd325,c 3 0.1 10 198 0.0 12.9
ssd326 18 0.4 6 123 0.0 57.6
ssd326,c 18 0.4 6 123 0.0 57.6
ssd327 2 0.0 17 542 0.0 1.1
ssd327,c 2 0.0 17 542 0.0 1.1
ssd328 3 0.0 24 763 0.0 1.2
ssd328,c 3 0.0 24 763 0.0 1.2
ssd329 68 1.4 36 883 0.0 39.1
ssd329,c 68 1.4 36 883 0.0 39.1
ssd331 14 0.2 7 113 0.0 31.6
ssd331,c 14 0.2 7 113 0.0 31.6
ssd332 59 1.0 37 586 0.0 27.5
ssd332,c 59 1.0 37 586 0.0 27.5
ssd333 73 9.1 122 4074 0.0 74.8
ssd333,c 73 9.1 122 4074 0.0 74.8
ssd335 29 0.5 9 193 0.0 55.5
ssd335,c 29 0.5 9 193 0.0 55.5
ssd336 44 0.9 30 576 0.0 30.4
ssd336,c 44 0.9 30 576 0.0 30.4
ssd337 2 0.1 6 104 0.0 13.7
ssd337,c 2 0.1 6 104 0.0 13.7
ssd338 2 0.1 9 189 0.0 15.4
ssd338,c 2 0.1 9 189 0.0 15.4
ssd339 25 0.4 7 117 0.0 63.5
ssd339,c 25 0.4 7 117 0.0 63.5
ssd340 2 0.0 16 513 0.0 1.1
ssd340,c 2 0.0 16 513 0.0 1.1
ssd341 3 0.0 25 802 0.0 1.3
ssd341,c 3 0.0 25 802 0.0 1.3
ssd342 59 1.3 37 864 0.0 35.9
ssd342,c 59 1.3 37 864 0.0 35.9
ssd344 15 0.3 7 122 0.0 38.6
ssd344,c 15 0.3 7 122 0.0 38.6
ssd345 63 1.2 44 697 0.0 28.0
ssd345,c 63 1.2 44 697 0.0 28.0
ssd346 76 8.7 115 3821 0.0 75.5
ssd346,c 76 8.7 115 3821 0.0 75.5
ssd348 24 0.4 9 172 0.0 50.9
ssd348,c 24 0.4 9 172 0.0 50.9
ssd349 39 0.8 24 484 0.0 32.3
ssd349,c 39 0.8 24 484 0.0 32.3
ssd350 2 0.1 4 95 0.0 13.5
ssd350,c 2 0.1 4 95 0.0 13.5
ssd351 4 0.1 10 223 0.0 12.3
ssd351,c 4 0.1 10 223 0.0 12.3
md3 38 1.9 85 2802 0.0 22.1
md13 32 1.0 85 2802 0.0 11.9
md23 32 1.1 85 2802 0.0 13.0
sd1 32 1.1 85 2802 0.0 13.0
sd1,d 32 1.1 85 2802 0.0 13.0
sd2 32 1.0 85 2802 0.0 11.8
sd2,d 32 1.0 85 2802 0.0 11.8
ssd300 15 0.6 12 355 0.0 52.5
ssd300,c 15 0.6 12 355 0.0 52.5
ssd301 13 0.7 11 315 0.0 59.4
ssd301,c 13 0.7 11 315 0.0 59.4
ssd302 1 0.0 9 303 0.0 1.1
ssd302,c 1 0.0 9 303 0.0 1.1
ssd303 1 0.0 9 291 0.0 1.1
ssd303,c 1 0.0 9 291 0.0 1.1
ssd304 2 0.0 13 401 0.0 2.1
ssd304,c 2 0.0 13 401 0.0 2.1
ssd305 2 0.0 14 455 0.0 1.4
ssd305,c 2 0.0 14 455 0.0 1.4
ssd306 47 1.0 27 618 0.0 35.8
ssd306,c 47 1.0 27 618 0.0 35.8
ssd307 33 0.5 23 581 0.0 23.0
ssd307,c 33 0.5 23 581 0.0 23.0
ssd308 82 21.6 199 4117 0.0 108.2
ssd308,c 82 21.6 199 4117 0.0 108.2
ssd309 76 20.4 167 3383 0.0 122.1
ssd309,c 76 20.4 167 3383 0.0 122.1
ssd310 7 0.1 4 67 0.0 23.9
ssd310,c 7 0.1 4 67 0.0 23.9
ssd311 5 0.1 3 58 0.0 21.3
ssd311,c 5 0.1 3 58 0.0 21.3
ssd313 51 1.6 55 1419 0.0 28.4
ssd313,c 51 1.6 55 1419 0.0 28.4
ssd314 92 16.8 200 6511 0.0 84.0
ssd314,c 92 16.8 200 6511 0.0 84.0
ssd315 95 15.8 194 6827 0.0 81.7
ssd315,c 95 15.8 194 6827 0.0 81.7
ssd318 16 0.2 4 84 0.0 60.6
ssd318,c 16 0.2 4 84 0.0 60.6
ssd319 16 0.2 2 58 0.0 86.1
ssd319,c 16 0.2 2 58 0.0 86.1
ssd320 43 0.8 20 328 0.0 42.6
ssd320,c 43 0.8 20 328 0.0 42.6
ssd321 61 1.0 20 327 0.0 52.8
ssd321,c 61 1.0 20 327 0.0 52.8
ssd322 2 0.1 9 149 0.0 15.3
ssd322,c 2 0.1 9 149 0.0 15.3
ssd323 2 0.2 11 198 0.0 13.7
ssd323,c 2 0.2 11 198 0.0 13.7
ssd324 2 0.1 9 178 0.0 11.1
ssd324,c 2 0.1 9 178 0.0 11.1
ssd325 2 0.1 9 194 0.0 11.5
ssd325,c 2 0.1 9 194 0.0 11.5
ssd326 17 0.6 11 413 0.0 55.2
ssd326,c 17 0.6 11 413 0.0 55.2
ssd327 1 0.0 13 401 0.0 1.0
ssd327,c 1 0.0 13 401 0.0 1.0
ssd329 53 1.1 33 748 0.0 32.7
ssd329,c 53 1.1 33 748 0.0 32.7
ssd330 76 15.9 189 3879 0.0 84.3
ssd330,c 76 15.9 189 3879 0.0 84.3
ssd331 9 0.1 6 91 0.0 26.9
ssd331,c 9 0.1 6 91 0.0 26.9
ssd332 66 2.9 67 1513 0.0 43.4
ssd332,c 66 2.9 67 1513 0.0 43.4
ssd333 97 17.2 209 7053 0.0 82.3
ssd333,c 97 17.2 209 7053 0.0 82.3
ssd335 20 0.3 5 107 0.0 62.9
ssd335,c 20 0.3 5 107 0.0 62.9
ssd336 53 0.8 27 436 0.0 31.4
ssd336,c 53 0.8 27 436 0.0 31.4
ssd337 3 0.1 10 189 0.0 14.4
ssd337,c 3 0.1 10 189 0.0 14.4
ssd338 2 0.1 10 189 0.0 11.0
ssd338,c 2 0.1 10 189 0.0 11.0
ssd339 16 0.7 13 367 0.0 51.5
ssd339,c 16 0.7 13 367 0.0 51.5
ssd340 1 0.0 9 285 0.0 1.0
ssd340,c 1 0.0 9 285 0.0 1.0
ssd342 47 0.8 30 733 0.0 24.8
ssd342,c 47 0.8 30 733 0.0 24.8
ssd343 83 19.8 210 4338 0.0 94.4
ssd343,c 83 19.8 210 4338 0.0 94.4
ssd344 8 0.1 5 84 0.0 26.6
ssd344,c 8 0.1 5 84 0.0 26.6
ssd345 61 2.3 68 1338 0.0 34.7
ssd345,c 61 2.3 68 1338 0.0 34.7
ssd346 99 17.5 216 7431 0.0 81.1
ssd346,c 99 17.5 216 7431 0.0 81.1
ssd348 23 0.3 4 79 0.0 76.9
ssd348,c 23 0.3 4 79 0.0 76.9
ssd349 59 1.0 24 398 0.0 42.6
ssd349,c 59 1.0 24 398 0.0 42.6
ssd350 2 0.1 8 132 0.0 11.9
ssd350,c 2 0.1 8 132 0.0 11.9
ssd351 2 0.1 9 186 0.0 10.6
ssd351,c 2 0.1 9 186 0.0 10.6Thanks,
Vageesh
Similar threads
- Locked
- Jessica Kent
- Progress Blog
- Locked
- Yana Vasileva
- Progress Blog
- Locked
- Vyom Srivastava
- Progress Blog
