Question probkup takes a very long time after automated test run

[aknyazeva]

Hi all,

I have an application that uses Progress 10.2B that I'm running nightly automated regression tests against. There are multiple test projects, and I would like to backup the state of the database after every individual test project using probkup. Our database is tiny - only 2.9GB when probkup is used with the -com switch. When I run probkup manually from cmd, it takes a minute, maybe two. What I'm noticing recently, though, is that if I run one full test project, which can take anywhere from half an hour to two hours depending on which project I'm running, the script that I wrote in at the end that runs probkup takes over half an hour! Like I said, I'm using the -com switch, and I am deleting the previous .bak file created by probkup before probkup runs rather than overwriting it.
This is incredibly inefficient and actually breaks my test run most of the time. Could anyone suggest why this is happening only during the automated test run, not a manual run of probkup, and what I could do to get around it?
Thanks!

 

[TomBascom]

Sounds like the cache in your storage system is saturated.

 

[oli]

Do you work with Windows Vista or Server 2008?
If so, and if by "automated" you mean "via the Task Scheduler (2.0)", keep in mind that tasks are defined by default with a low I/O priority (i.e. 7: BELOW_NORMAL_PRIORITY_CLASS).
You have to export the task definition, edit the generated .xml file, modify the priority (set it between 4 and 6), and then re-import the task.

 

[Cringer]

That's a good point oli.

 

[aknyazeva]

Sounds like the cache in your storage system is saturated.

I'm looking into how the machine in question handles caching. I'll let you know what I find.

Do you work with Windows Vista or Server 2008?
If so, and if by "automated" you mean "via the Task Scheduler (2.0)", keep in mind that tasks are defined by default with a low I/O priority (i.e. 7: BELOW_NORMAL_PRIORITY_CLASS).
You have to export the task definition, edit the generated .xml file, modify the priority (set it between 4 and 6), and then re-import the task.

I am using Windows 7, and I am using Task Scheduler to start up TestExecute to run the automated tests. I noticed that the tasks are all configured for Vista and Server 2008, though. It does seem to happen more often when I schedule the run via Task Scheduler rather than when I start it manually. I'll mess around with the priority, and change the configuration to Windows 7.

 

[aknyazeva]

I changed the configuration of the tasks in Task Scheduler to Windows 7, manually set the priority of each to 4 instead of 7, and increased the number of blocks in the DB buffer (through Progress Explorer) to 196000. Unfortunately, the problem is still happening. Is there something I'm overlooking?

 

[Cringer]

Is there anything pertinent in the database log files?

 

[aknyazeva]

The log file does have some strange stuff in it, now that you mention it.
For the first couple of tests I ran last night (and this is after the changes), the database never stopped properly. The only message I see is "Usernum 24 terminated abnormally" - this is most likely because at the beginning of every new test run, I run a taskkill command on prowin32.exe, because sometimes the process sticks around even after the application has been closed.
After those two tests, in the middle of a third test, I see this message: "WARNING: -l exceeded. Automatically increasing from 200 to 210." It keeps increasing until 1100.
After that happened, the database started receiving shutdown requests from the admin server, like it's supposed to. Before that happened, there were no shutdown requests received.
One of the backups still took well over half an hour, though, and there were no unusual messages there.

 

[cj_brandt]

setup perfmon to monitor disk queues where the database and backups reside. compare the values during the slowdown to when there is normal operations.

 

[Cringer]

The -l messages mean that you have a potential transaction scoping issue in your code. You are locking a lot of records. It may be that this is necessary, but it should be an unusual situation for this to occur. Ideally you want your transactions as small as possible to avoid problems with locked records for other users. This is unlikely to be the cause of your backup issues though as it would be the same whether you run manually or automatically I'd have thought.

 

[TomBascom]

Cringer, small ell and upper-case L are different parameters.

Local Buffer Size (-l)

Use Local Buffer Size (-l) to change the size of the local record buffer in which the AVM stores all variables, work files, and records that are in use at one time for a user.

 

[TomBascom]

I stand by my original comment. Your disks are probably saturated. What sort of NAS are you using?

 

[Cringer]

Cringer, small ell and upper-case L are different parameters.

Oops! Need to read more closely!

 

[aknyazeva]

setup perfmon to monitor disk queues where the database and backups reside. compare the values during the slowdown to when there is normal operations.

In the past few days, I've been trying to replicate the problem with perfmon running, and naturally, because I'm watching it, it runs without a hitch. These tests usually run overnight; I've disabled probkup in the overnight run for now because I'm reluctant to to include a script that often crashes the whole run, but that seems to be the only time it happens! I'm not sure what to make of that.

I stand by my original comment. Your disks are probably saturated. What sort of NAS are you using?

I admit, I'm very unfamiliar with database systems, so I may be totally wrong, but I'm not sure this database is on a NAS at all. Like I said, it's a tiny 3GB database. The whole thing is held on the test machine's HDD.

 

[TomBascom]

So your backup is going to the same disk you're reading from?

Is it a real disk? Or some virtualized thing provisioned by someone else?

 

[aknyazeva]

Yep, same disk. It's a real physical disk.

 

[TomBascom]

You're not giving me any reasons to think that this is anything other than an overwhelmed disk drive. When you write read lots of stuff to the same disk you are writing it too you are pretty much ensuring that you will suffer. For a while Windows will help you out by buffering the data -- but eventually those buffers will be full and you get to see the raw performance of the underlying disk. Which is going to be awful.

You can improve it by adding a second disk and making it the target of your backup. Adding a network share as the target will also help, but not as much as adding a real disk.

 

[aknyazeva]

That makes sense, and it's unfortunate that this is the way it's set up. I'm still curious about why it only seems to happen with scheduled tasks, though, and not when I run the backup manually. In my mind, there shouldn't be a difference in how the backup accesses the disk between the two scenarios.

 

[TomBascom]

Maybe there is more time between tasks when you do it manually?

 

[Rob Fitzpatrick]

If you're able to watch as it runs, use Process Explorer to look at the details of the probkup process (_dbutil.exe) run from the scheduled task. Then the backup again in the shell from a proenv prompt and compare. Process Explorer will show you thread priority, I/O priority, and memory priority, among other things. As oli said, there may be a difference between the two that you need to track down.