Hello Tom,
Please find the details:
As per our developers we have some application dependency for upgrade to OE.
We are using Solaris 10, Databases are hosted on SAN with RAID 10
Screen caps of Checkpoint & Summary screen (tried to get it as close as possible to ckpt) are pasted below. Kindly excuse me for not attaching these as a document.
We find this issue during EOD processing and Todays process had only 25% of User load:
03/13/10 Checkpoints
01:12:17
Ckpt ------ Database Writes ------
No. Time Len Freq Dirty CPT Q Scan APW Q Flushes
1844 01:11:36 41 0 38848 0 1325 5550 0
1843 01:07:35 241 241 32119 0 0 29768 3175
1842 01:03:54 221 221 26800 8220 1699 19464 0
1841 00:59:58 236 236 19235 0 276 25829 776
1840 00:56:01 237 237 16602 0 248 30458 704
1839 00:51:05 286 296 17217 286 749 34399 0
1838 00:47:02 243 243 15376 356 2162 27675 190
1837 00:42:45 248 257 16043 376 2172 28430 0
--------------------------------------------------------------------------------------------
03/13/10 Activity: Summary
01:11:40 03/13/10 01:11 to 03/13/10 01:11 (10 sec)
Event Total Per Sec |Event Total Per Sec
Commits 1123 112.3 |DB Reads 4671 467.1
Undos 1 0.1 |DB Writes 6899 689.9
Record Reads 152184 15218.4 |BI Reads 0 0.0
Record Updates 672 67.2 |BI Writes 139 13.9
Record Creates 578 57.8 |AI Writes 79 7.9
Record Deletes 5 0.5 |Checkpoints 0 0.0
Record Locks 10621 1062.1 |Flushed at chkpt 796 79.6
Record Waits 0 0.0
Rec Lock Waits 0 % BI Buf Waits 0 % AI Buf Waits 0 %
Writes by APW 88 % Writes by BIW 49 % Writes by AIW 96 %
DB Size: 332 GB BI Size: 11 GB AI Size: 18 MB
Empty blocks:13764887 Free blocks: 8232 RM chain: 281564
Buffer Hits 99 % Active trans: 42
640 Servers, 229 Users (52 Local, 177 Remote, 51 Batch), 9 Apws
=======================================================
03/13/10 Checkpoints
01:33:21
Ckpt ------ Database Writes ------
No. Time Len Freq Dirty CPT Q Scan APW Q Flushes
1850 01:33:12 9 0 21302 0 151 3149 0
1849 01:29:47 205 205 18441 0 2313 25017 462
1848 01:26:10 209 217 32841 2081 1301 40714 0
1847 01:22:55 185 195 37430 26550 585 11738 0
1846 01:19:16 209 219 41420 41897 2190 2 0
1845 01:15:39 217 217 43539 15204 1769 23404 3987
1844 01:11:36 243 243 38848 3385 1957 27867 5939
1843 01:07:35 241 241 32119 0 0 29768 3175
--------------------------------------------------------------------------------------------
03/13/10 Activity: Summary
01:33:20 03/13/10 01:33 to 03/13/10 01:33 (10 sec)
Event Total Per Sec |Event Total Per Sec
Commits 2969 296.9 |DB Reads 9447 944.7
Undos 2 0.2 |DB Writes 3590 359.0
Record Reads 657813 65781.3 |BI Reads 4 0.4
Record Updates 2414 241.4 |BI Writes 319 31.9
Record Creates 780 78.0 |AI Writes 174 17.4
Record Deletes 2 0.2 |Checkpoints 1 0.1
Record Locks 19670 1967.0 |Flushed at chkpt 448 44.8
Record Waits 0 0.0
Rec Lock Waits 0 % BI Buf Waits 0 % AI Buf Waits 0 %
Writes by APW 87 % Writes by BIW 37 % Writes by AIW 97 %
DB Size: 332 GB BI Size: 11 GB AI Size: 58 MB
Empty blocks:13760309 Free blocks: 8232 RM chain: 281513
Buffer Hits 99 % Active trans: 43
640 Servers, 232 Users (52 Local, 180 Remote, 51 Batch), 9 Apws
=============================================================
Database Startup Parameters:
01:27:10 WDOG 655: Started. (2518)
01:27:11 BROKER 0: Progress OpenEdge Release 9.1E on SOLARIS. (4234)
01:27:11 BROKER 0: Server started by progress on batch. (4281)
01:27:11 BROKER 0: Started using pid: 19681. (6574)
01:27:11 BROKER 0: Physical Database Name (-db): /emc/reale/prod/amos/data/amos. (4235)
01:27:11 BROKER 0: Database Type (-dt): PROGRESS. (4236)
01:27:11 BROKER 0: Force Access (-F): Not Enabled. (4237)
01:27:11 BROKER 0: Direct I/O (-directio): Enabled. (4238)
01:27:11 BROKER 0: Number of Database Buffers (-B): 250000. (4239)
01:27:11 BROKER 0: Maximum private buffers per user (-Bpmax): 64. (9422)
01:27:11 BROKER 0: Excess Shared Memory Size (-Mxs): 16970. (4240)
01:27:11 BROKER 0: The shared memory segment is locked in memory. (10013)
01:27:11 BROKER 0: Current Size of Lock Table (-L): 200000. (4241)
01:27:11 BROKER 0: Hash Table Entries (-hash): 70289. (4242)
01:27:11 BROKER 0: Current Spin Lock Tries (-spin): 50000. (4243)
01:27:11 BROKER 0: Number of Semaphore Sets (-semsets): 30. (6526)
01:27:11 BROKER 0: Crash Recovery (-i): Enabled. (4244)
01:27:11 BROKER 0: Database Blocksize (-blocksize): 8192. (6573)
01:27:11 BROKER 0: Delay of Before-Image Flush (-Mf): 3. (4245)
01:27:11 BROKER 0: Before-Image File I/O (-r -R): Reliable. (4247)
01:27:11 BROKER 0: Before-Image Truncate Interval (-G): 0. (4249)
01:27:11 BROKER 0: Before-Image Cluster Size: 50331648. (4250)
01:27:11 BROKER 0: Before-Image Block Size: 16384. (4251)
01:27:11 BROKER 0: Number of Before-Image Buffers (-bibufs): 100. (4252)
01:27:11 BROKER 0: BI File Threshold size (-bithold): 0.0 Bytes. (9238)
01:27:11 BROKER 0: BI File Threshold Stall (-bistall): Enabled. (6551)
01:27:11 BROKER 0: After-Image Stall (-aistall): Enabled. (4254)
01:27:11 BROKER 0: After-Image Block Size: 16384. (4255)
01:27:11 BROKER 0: Number of After-Image Buffers (-aibufs): 100. (4256)
01:27:11 BROKER 0: Storage object cache size (-omsize): 1024 (8527)
01:27:11 BROKER 0: Maximum Number of Clients Per Server (-Ma): 3. (4257)
01:27:11 BROKER 0: Maximum Number of Servers (-Mn): 641. (4258)
01:27:11 BROKER 0: Minimum Clients Per Server (-Mi): 1. (4259)
01:27:11 BROKER 0: Maximum Number of Users (-n): 2001. (4260)
01:27:11 BROKER 0: Host Name (-H): nosxprrsnp01. (4261)
01:27:11 BROKER 0: Service Name (-S): amos_prod_odbc. (4262)
01:27:11 BROKER 0: Network Type (-N): tcp. (4263)
01:27:11 BROKER 0: Character Set (-cpinternal): ISO8859-15. (4264)
01:27:11 BROKER 0: Parameter File: /emc/reale/prod/amos/data/amos.pf. (4282)
01:27:11 BROKER 0: Minimum Port for Auto Servers (-minport): 2000. (5648)
01:27:11 BROKER 0: Maximum Port for Auto Servers (-maxport): 20000. (5649)
01:15:15 device %busy avque r+w/s blks/s avwait avserv
md3 32 2.4 83 2712 0.0 28.7
md13 29 1.3 83 2712 0.0 15.6
md23 28 1.3 83 2712 0.0 15.7
sd1 28 1.3 84 2718 0.0 15.6
sd1,d 28 1.3 84 2712 0.0 15.7
sd2 29 1.3 84 2718 0.0 15.5
sd2,d 29 1.3 84 2712 0.0 15.5
ssd300 22 0.4 5 108 0.0 80.7
ssd300,c 22 0.4 5 108 0.0 80.7
ssd301 26 0.4 5 99 0.0 77.6
ssd301,c 26 0.4 5 99 0.0 77.6
ssd302 2 0.0 17 528 0.0 1.2
ssd302,c 2 0.0 17 528 0.0 1.2
ssd303 1 0.0 13 425 0.0 1.1
ssd303,c 1 0.0 13 425 0.0 1.1
ssd304 3 0.0 26 825 0.0 1.2
ssd304,c 3 0.0 26 825 0.0 1.2
ssd305 3 0.0 21 663 0.0 1.3
ssd305,c 3 0.0 21 663 0.0 1.3
ssd306 59 1.2 33 866 0.0 37.5
ssd306,c 59 1.2 33 866 0.0 37.5
ssd307 73 1.5 44 1039 0.0 33.7
ssd307,c 73 1.5 44 1039 0.0 33.7
ssd308 95 8.8 135 3764 0.0 65.3
ssd308,c 95 8.8 135 3764 0.0 65.3
ssd309 97 9.6 148 3802 0.0 65.2
ssd309,c 97 9.6 148 3802 0.0 65.2
ssd310 11 0.2 5 92 0.0 33.3
ssd310,c 11 0.2 5 92 0.0 33.3
ssd311 15 0.2 7 132 0.0 33.5
ssd311,c 15 0.2 7 132 0.0 33.5
ssd312 59 1.2 37 594 0.0 31.1
ssd312,c 59 1.2 37 594 0.0 31.1
ssd313 67 1.3 50 807 0.0 26.2
ssd313,c 67 1.3 50 807 0.0 26.2
ssd314 76 9.9 144 4242 0.0 69.1
ssd314,c 76 9.9 144 4242 0.0 69.1
ssd315 73 8.3 117 4097 0.0 71.2
ssd315,c 73 8.3 117 4097 0.0 71.2
ssd318 20 0.4 7 158 0.0 52.1
ssd318,c 20 0.4 7 158 0.0 52.1
ssd319 14 0.3 7 147 0.0 42.1
ssd319,c 14 0.3 7 147 0.0 42.1
ssd320 29 0.8 22 421 0.0 36.4
ssd320,c 29 0.8 22 421 0.0 36.4
ssd321 48 1.0 27 498 0.0 36.2
ssd321,c 48 1.0 27 498 0.0 36.2
ssd322 2 0.1 5 105 0.0 14.0
ssd322,c 2 0.1 5 105 0.0 14.0
ssd323 1 0.0 3 54 0.0 13.5
ssd323,c 1 0.0 3 54 0.0 13.5
ssd324 2 0.1 9 179 0.0 14.6
ssd324,c 2 0.1 9 179 0.0 14.6
ssd325 3 0.1 10 198 0.0 12.9
ssd325,c 3 0.1 10 198 0.0 12.9
ssd326 18 0.4 6 123 0.0 57.6
ssd326,c 18 0.4 6 123 0.0 57.6
ssd327 2 0.0 17 542 0.0 1.1
ssd327,c 2 0.0 17 542 0.0 1.1
ssd328 3 0.0 24 763 0.0 1.2
ssd328,c 3 0.0 24 763 0.0 1.2
ssd329 68 1.4 36 883 0.0 39.1
ssd329,c 68 1.4 36 883 0.0 39.1
ssd331 14 0.2 7 113 0.0 31.6
ssd331,c 14 0.2 7 113 0.0 31.6
ssd332 59 1.0 37 586 0.0 27.5
ssd332,c 59 1.0 37 586 0.0 27.5
ssd333 73 9.1 122 4074 0.0 74.8
ssd333,c 73 9.1 122 4074 0.0 74.8
ssd335 29 0.5 9 193 0.0 55.5
ssd335,c 29 0.5 9 193 0.0 55.5
ssd336 44 0.9 30 576 0.0 30.4
ssd336,c 44 0.9 30 576 0.0 30.4
ssd337 2 0.1 6 104 0.0 13.7
ssd337,c 2 0.1 6 104 0.0 13.7
ssd338 2 0.1 9 189 0.0 15.4
ssd338,c 2 0.1 9 189 0.0 15.4
ssd339 25 0.4 7 117 0.0 63.5
ssd339,c 25 0.4 7 117 0.0 63.5
ssd340 2 0.0 16 513 0.0 1.1
ssd340,c 2 0.0 16 513 0.0 1.1
ssd341 3 0.0 25 802 0.0 1.3
ssd341,c 3 0.0 25 802 0.0 1.3
ssd342 59 1.3 37 864 0.0 35.9
ssd342,c 59 1.3 37 864 0.0 35.9
ssd344 15 0.3 7 122 0.0 38.6
ssd344,c 15 0.3 7 122 0.0 38.6
ssd345 63 1.2 44 697 0.0 28.0
ssd345,c 63 1.2 44 697 0.0 28.0
ssd346 76 8.7 115 3821 0.0 75.5
ssd346,c 76 8.7 115 3821 0.0 75.5
ssd348 24 0.4 9 172 0.0 50.9
ssd348,c 24 0.4 9 172 0.0 50.9
ssd349 39 0.8 24 484 0.0 32.3
ssd349,c 39 0.8 24 484 0.0 32.3
ssd350 2 0.1 4 95 0.0 13.5
ssd350,c 2 0.1 4 95 0.0 13.5
ssd351 4 0.1 10 223 0.0 12.3
ssd351,c 4 0.1 10 223 0.0 12.3
md3 38 1.9 85 2802 0.0 22.1
md13 32 1.0 85 2802 0.0 11.9
md23 32 1.1 85 2802 0.0 13.0
sd1 32 1.1 85 2802 0.0 13.0
sd1,d 32 1.1 85 2802 0.0 13.0
sd2 32 1.0 85 2802 0.0 11.8
sd2,d 32 1.0 85 2802 0.0 11.8
ssd300 15 0.6 12 355 0.0 52.5
ssd300,c 15 0.6 12 355 0.0 52.5
ssd301 13 0.7 11 315 0.0 59.4
ssd301,c 13 0.7 11 315 0.0 59.4
ssd302 1 0.0 9 303 0.0 1.1
ssd302,c 1 0.0 9 303 0.0 1.1
ssd303 1 0.0 9 291 0.0 1.1
ssd303,c 1 0.0 9 291 0.0 1.1
ssd304 2 0.0 13 401 0.0 2.1
ssd304,c 2 0.0 13 401 0.0 2.1
ssd305 2 0.0 14 455 0.0 1.4
ssd305,c 2 0.0 14 455 0.0 1.4
ssd306 47 1.0 27 618 0.0 35.8
ssd306,c 47 1.0 27 618 0.0 35.8
ssd307 33 0.5 23 581 0.0 23.0
ssd307,c 33 0.5 23 581 0.0 23.0
ssd308 82 21.6 199 4117 0.0 108.2
ssd308,c 82 21.6 199 4117 0.0 108.2
ssd309 76 20.4 167 3383 0.0 122.1
ssd309,c 76 20.4 167 3383 0.0 122.1
ssd310 7 0.1 4 67 0.0 23.9
ssd310,c 7 0.1 4 67 0.0 23.9
ssd311 5 0.1 3 58 0.0 21.3
ssd311,c 5 0.1 3 58 0.0 21.3
ssd313 51 1.6 55 1419 0.0 28.4
ssd313,c 51 1.6 55 1419 0.0 28.4
ssd314 92 16.8 200 6511 0.0 84.0
ssd314,c 92 16.8 200 6511 0.0 84.0
ssd315 95 15.8 194 6827 0.0 81.7
ssd315,c 95 15.8 194 6827 0.0 81.7
ssd318 16 0.2 4 84 0.0 60.6
ssd318,c 16 0.2 4 84 0.0 60.6
ssd319 16 0.2 2 58 0.0 86.1
ssd319,c 16 0.2 2 58 0.0 86.1
ssd320 43 0.8 20 328 0.0 42.6
ssd320,c 43 0.8 20 328 0.0 42.6
ssd321 61 1.0 20 327 0.0 52.8
ssd321,c 61 1.0 20 327 0.0 52.8
ssd322 2 0.1 9 149 0.0 15.3
ssd322,c 2 0.1 9 149 0.0 15.3
ssd323 2 0.2 11 198 0.0 13.7
ssd323,c 2 0.2 11 198 0.0 13.7
ssd324 2 0.1 9 178 0.0 11.1
ssd324,c 2 0.1 9 178 0.0 11.1
ssd325 2 0.1 9 194 0.0 11.5
ssd325,c 2 0.1 9 194 0.0 11.5
ssd326 17 0.6 11 413 0.0 55.2
ssd326,c 17 0.6 11 413 0.0 55.2
ssd327 1 0.0 13 401 0.0 1.0
ssd327,c 1 0.0 13 401 0.0 1.0
ssd329 53 1.1 33 748 0.0 32.7
ssd329,c 53 1.1 33 748 0.0 32.7
ssd330 76 15.9 189 3879 0.0 84.3
ssd330,c 76 15.9 189 3879 0.0 84.3
ssd331 9 0.1 6 91 0.0 26.9
ssd331,c 9 0.1 6 91 0.0 26.9
ssd332 66 2.9 67 1513 0.0 43.4
ssd332,c 66 2.9 67 1513 0.0 43.4
ssd333 97 17.2 209 7053 0.0 82.3
ssd333,c 97 17.2 209 7053 0.0 82.3
ssd335 20 0.3 5 107 0.0 62.9
ssd335,c 20 0.3 5 107 0.0 62.9
ssd336 53 0.8 27 436 0.0 31.4
ssd336,c 53 0.8 27 436 0.0 31.4
ssd337 3 0.1 10 189 0.0 14.4
ssd337,c 3 0.1 10 189 0.0 14.4
ssd338 2 0.1 10 189 0.0 11.0
ssd338,c 2 0.1 10 189 0.0 11.0
ssd339 16 0.7 13 367 0.0 51.5
ssd339,c 16 0.7 13 367 0.0 51.5
ssd340 1 0.0 9 285 0.0 1.0
ssd340,c 1 0.0 9 285 0.0 1.0
ssd342 47 0.8 30 733 0.0 24.8
ssd342,c 47 0.8 30 733 0.0 24.8
ssd343 83 19.8 210 4338 0.0 94.4
ssd343,c 83 19.8 210 4338 0.0 94.4
ssd344 8 0.1 5 84 0.0 26.6
ssd344,c 8 0.1 5 84 0.0 26.6
ssd345 61 2.3 68 1338 0.0 34.7
ssd345,c 61 2.3 68 1338 0.0 34.7
ssd346 99 17.5 216 7431 0.0 81.1
ssd346,c 99 17.5 216 7431 0.0 81.1
ssd348 23 0.3 4 79 0.0 76.9
ssd348,c 23 0.3 4 79 0.0 76.9
ssd349 59 1.0 24 398 0.0 42.6
ssd349,c 59 1.0 24 398 0.0 42.6
ssd350 2 0.1 8 132 0.0 11.9
ssd350,c 2 0.1 8 132 0.0 11.9
ssd351 2 0.1 9 186 0.0 10.6
ssd351,c 2 0.1 9 186 0.0 10.6