Webspeed Performance Issue

santhosh_bala

New Member
We have a web based application running in Web Speed 3.1d with Progress 9.1d as the backend. We made a recent enhancement to one of our reports where few rows and columns were added along with calculation logic. When we try to move this enhancement to production, all our brokers goes down (We have 5 brokers with each broker having 6 agents). We noticed that there were many ‘Progress.exe’ processes running at that point which consumes 100% of CPU usage.

We are unable to simulate this in our development and test environments. We would appreciate if we can get any inputs on this.
 

Casper

ProgressTalk.com Moderator
Staff member
Hi,

saying that it probably has to do something with the changes you made to the report doesn't help you very much I suppose:awink:

I don't know if I can help you but first a few questions:
1- which platform you work (on windows probably regarding the progress.exe)? Java version?
2- why do you use 5 brokers?
3- What changes are made to the report?
4- Where is the database residing (OS plus OS-level)
5- which SP level do you have with Progress?
6- Are messenger and WTA on the same machine?
7- Do you use IIS or Apache?
9- Is there a certain point in the report where this happens?
10- If in your test environment you generate many reports at the same time, what happens?
11-Do you have webspeed configured as stateless?
12- Do you use shared variables?
13- Does the broker hang or do the agents hang?
14- anything in the log files?
15- Are there shared locks involved? Did you check with promon (with -F option perhaps: record locking table)?
Well, sorry for so many questions, but since I don't know much of your configuration I figured that it would make sense to ask some questions first....

And maybe the answers help other people answering as well.....

Regards,

Casper.

Casper
 

santhosh_bala

New Member
Hi Casper,

Please find the answers below. Thanks for your time.

1- which platform you work (on windows probably regarding the progress.exe)? Java version?
***** App Server and Web Server are in Windows and Database in Linux.

2- why do you use 5 brokers?
***** We have configured it that way to have dedicated brokers for certain applications.

3- What changes are made to the report?
***** Along with existing data, we have added few rows and columns of data. Technically, from the same table few more columns were fetched for data. This table will have close to 2 million records and is growing.

4- Where is the database residing (OS plus OS-level)
***** Database is residing in Linux server.

5- which SP level do you have with Progress?
***** What do you mean by this?

6- Are messenger and WTA on the same machine?
***** They are in the same machine.

7- Do you use IIS or Apache?
***** We use IIS.

9- Is there a certain point in the report where this happens?
***** We tried logging the time taken for the report and messages to track the point where the error comes. But unfortunately could not make out anything. A report taken for a specific criteria has gone through well at a point of time, where as the same report has taken triple the time at a different time.(at the time the server went down).

10- If in your test environment you generate many reports at the same time, what happens?
***** In test environment, the report gets over without any issues. If more requests are given than the available agents, then the excess requests are put in queue and processed subsequently.

11-Do you have webspeed configured as stateless?
***** Yes, we have configured the operating mode as Stateless.

12- Do you use shared variables?
***** Yes we used shared variables.

13- Does the broker hang or do the agents hang?
***** Agents hang and at that time, we could see many Progress.exe processes running with 100% CPU usage.

14- anything in the log files?
***** No help with logs.

15- Are there shared locks involved? Did you check with promon (with -F option perhaps: record locking table)?
***** As this is a report, we have looped in with NO-LOCK only.
 

TomBascom

Curmudgeon
What do you mean when you say that the server has gone down? Does the OS crash? Does Progress crash? If so are there specific error messages in the logs?

Or do you just mean that performance gets really bad?
 

Casper

ProgressTalk.com Moderator
Staff member
Hi there,

is there a posibillity for an infinite loop within the changes you made?
Does the report consists of procedures which are run persistent?
-If so:
  • any chnance some programms doen't close proper and hence the agent remains busy. (is it possible to do wtbman -query if the proces hangs?)
  • Did you run a session with -yx (or -yd) to look at stats for the call of these procedures?
Is it possible to reproduce this at the production server? It would be nice if you have a reproducable situation to tackle the problem.....
Also curious what promon says. (INregard to database activity at the momnet the agents ' hang')
Some not to the problem related question, but just curiousity: Is there a special reason you run the Webspeed nameserver and agents remote from the database? I think most time performancewise it's more preferable to have the nameserver and webspeed transaction server at the same machine as where the database resides.

Well, my thoughts for now....

Regards,

Casper
 

santhosh_bala

New Member
Hi Tom,

What I meant by 'server has gone down' was, all the agents gets busy (or hung) for a long time, resulting in webspeed errors for subsequent requests. To put forward simply, the requests are taking considerably longer time than the actual (not at all times).

Hi Casper,

We have checked the code and found that there is no infinite looping in it. In fact, when we logged the kind of selection criteria that takes the agent's time for a longer time, we found that the same selection criteria is not taking the same time at all times. (i.e) running a report for supplier X has taken 15 mts at a point of time and the same report has taken 2 mts at another.

At the moment, we will not be able to have the dB server and the WTS in the same machine.

Thanks for your help.
 

Biga Cubensis

New Member
Hello, I have similar problem and configuration except we have the databases on AIX server.

I read some threads about the timeout problems of webspeed and about the famous "Webwrite failed" errors but I still don't understand how to correct this error. I checked the Progress KB pages and tried the recommendations: set all timeout values high at CGI settings in IIS properties window. I also set AGENT_RETRY in environment and SPIN parameter at command line of the WS server. My report still gives up and I get CGI Timeout error. All timeout parameters (which I found) was set to 30 minutes but the report stops at 4-5 seconds.

The biggest problem is the same: the server slows down seriously (_progres task eats 98% CPU) and only server restart helps.

The structure of my program is the following after POST (I don't copy the whole program because it is enough big):
1. fill a temp-table from a file given in the form (a few seconds)
2. database read corresponding to temp-file (it can work 5-10 minutes) and making the result temp-table
3. output header and output content is excel, download file name set
4. table output from the temp-table (a few minutes) in excel format

Any ideas?
thanks
Biga
 

Casper

ProgressTalk.com Moderator
Staff member
To Santosh:

You say the same report takes sometimes 2 and sometimes 15 minutes to finish with the exact same conditions?
What happens if you run the report form a client and not from webspeed, still much differences in time it takes to finish? Still much cpu time consumed?
Do you have difficulty's with performance in general on that database?
What other processes run on the windows machine?
Did you XREF the code?

To biga:
-spin is a server parameter so has no effect on client....
reports stops after 4-5 seconds?
What if you run the same report outside of webspeed? Does it run?
Does it also consume lots of cpu time?
How do you know the reports stops after 4-5 seconds? You say the program takes more then 10 minutes to finish?? Don't understand what you mean...
Did you XREF the code?

To both:
Long reports shouldn't be made by a webspeed agent. it's much much much (many more muches...) better to run the report as a batch job and make it available to the webspeeduser as soon it's finished.( by URL or e-mail).

Regards,

Casper.
 
Top