G
George Potemkin
Guest
DimitriG4, > My feeling is that the "rm" being executed might not be the "out of the box" "rm". I never saw such things but your guess may be right: rm is a tracked alias for /usr/bin/rm > still strange since the files are small, and why a different number of processes every time? Really the sript is dbmon.sh: community.progress.com/.../25700 The script was used for all (10) databases running on the system. By default dbmon is starting 3 promon sessions and one 4G session for each database. In other words, it has started 40 processes. All processes were started inside 3 sec interval - from 14:05:08 to 14:05:11 (though the script is trying to start them simultaneously). Then dbmon has deleted the list of databases to monitor and the result was: time rm $DbList2 real 0m12.56s user 0m0.57s sys 0m4.65s Then dbmon.sh has started vmstat and iostat. Both commands have started instantly. vmstat 4 6 2016/07/05 14:05:24 procs memory page faults cpu 2016/07/05 14:05:24 r b w avm free re at pi po fr de sr in sy cs us sy id 2016/07/05 14:05:24 19 4 0 11644791 736268 3 0 14 1559 0 0 10 33642 86662 11038 18 7 76 2016/07/05 14:05:28 19 4 0 11644791 734051 0 0 21 311 0 0 0 13606 601601 22498 38 54 8 2016/07/05 14:05:32 19 3 0 11562973 758078 0 0 12 328 0 0 0 14214 594769 23221 37 55 8 2016/07/05 14:05:36 19 3 0 11562973 782987 0 0 6 319 0 0 0 12863 542690 22198 33 51 16 2016/07/05 14:05:40 19 3 0 11562973 794595 0 0 10 266 0 0 0 9009 497981 11478 35 20 45 2016/07/05 14:05:44 3 4 0 11521635 792468 0 0 3 247 0 0 0 6715 220381 6276 26 6 67 Progress processes started by dbmon caused a rather large increase of system time (time spent running kernel code). IMHO, it's not typical for Progress processes. Due to the delay caused by the 'rm' command the vmstat started and finished by 13 seconds after Progress processes. And we can see that the system time returned to a normal value when Progress processes terminated. The same customer, the identical Itanium box, the same dbmon script but the result is different: 2016/06/23 10:23:26 procs memory page faults cpu 2016/06/23 10:23:26 r b w avm free re at pi po fr de sr in sy cs us sy id 2016/06/23 10:23:26 1 0 0 1487739 3634458 0 0 2 14 0 0 0 1257 924006 843 17 1 82 2016/06/23 10:23:30 1 0 0 1487739 3631967 0 0 65 0 0 0 0 1505 85419 1473 8 4 89 2016/06/23 10:23:34 1 0 0 1487739 3631985 0 0 36 0 0 0 0 1311 47381 1111 7 1 92 2016/06/23 10:23:38 1 0 0 1497610 3631646 0 0 24 0 0 0 0 1231 31851 982 6 2 92 2016/06/23 10:23:42 1 0 0 1497610 3631646 0 0 16 0 0 0 0 1218 22970 938 7 1 91 2016/06/23 10:23:46 2 0 0 1501899 3631564 0 0 16 0 0 0 0 1239 21148 982 7 2 92 iostat 4 6 on the first server: 2016/07/05 14:05:24 device bps sps msps 2016/07/05 14:05:24 2016/07/05 14:05:28 disk64 43 4.0 1.0 2016/07/05 14:05:28 disk33 0 0.0 1.0 2016/07/05 14:05:28 disk42 677 5.8 1.0 2016/07/05 14:05:28 disk72 319 44.5 1.0 2016/07/05 14:05:28 disk75 5 1.2 1.0 2016/07/05 14:05:28 disk78 0 0.0 1.0 2016/07/05 14:05:28 disk81 0 0.0 1.0 2016/07/05 14:05:28 disk84 2 0.2 1.0 2016/07/05 14:05:28 disk87 753 25.9 1.0 2016/07/05 14:05:28 disk90 0 0.0 1.0 2016/07/05 14:05:28 disk93 0 0.0 1.0 2016/07/05 14:05:28 disk94 0 0.0 1.0 2016/07/05 14:05:28 disk95 0 0.0 1.0 2016/07/05 14:05:28 disk96 633 135.0 1.0 2016/07/05 14:05:28 disk97 0 0.0 1.0 2016/07/05 14:05:28 disk98 4556 290.7 1.0 2016/07/05 14:05:28 disk99 0 0.0 1.0 2016/07/05 14:05:28 disk102 0 0.0 1.0 2016/07/05 14:05:28 disk107 1097 78.3 1.0 2016/07/05 14:05:28 disk108 17800 391.8 1.0 2016/07/05 14:05:28 disk113 103 11.2 1.0 2016/07/05 14:05:28 disk114 254 31.7 1.0 One database on this server reads a lot from disk: 07/05/16 Activity: Summary 14:05:12 07/05/16 14:05 to 07/05/16 14:05 (4 sec) Event Total Per Sec |Event Total Per Sec Commits 33 8.2 |DB Reads 17263 4315.8 Undos 0 0.0 |DB Writes 14 3.5 Record Reads 307602 76900.5 |BI Reads 0 0.0 Record Updates 8 2.0 |BI Writes 3 0.8 Record Creates 8 2.0 |AI Writes 0 0.0 Record Deletes 0 0.0 |Checkpoints 0 0.0 Record Locks 89 22.2 |Flushed at chkpt 0 0.0 Record Waits 0 0.0 |Active trans 6 Rec Lock Waits 0 % BI Buf Waits 0 % AI Buf Waits 0 % Writes by APW 100 % Writes by BIW 33 % Writes by AIW 0 % DB Size: 1771 GB BI Size: 20 GB AI Size: 0 K Empty blocks: 455 Free blocks: 1860 RM chain: 83 Buffer Hits 97 % Primary Hits 97 % Alternate Hits 0 % 4 Servers, 175 Users (126 Local, 49 Remote, 72 Batch), 2 Apws Summary: Discs are busy by the read operations. Starting 40 Progress processes that are not resource consumimg (like promon) caused a significant increase of CPU system time and caused a huge delay while using rm command but reads/writes operations are not noticeably affected. I do not still understand what is going wrong with this server.
Continue reading...
Continue reading...