Is Progress capable?

[ChrisJRutter]

Hi all, we have been a Progress shop since Version 3 so are well versed in the ways of PSC. We have had a recent request to build a number crunching system that needs to cater for up to a 100,000,000,000 (yes 100 million) transactions per annum. Can't tell you anymore at this stage but let's just assume the records are on the small size with few indexes, etc so quite a compact and efficient structure. We are running OpenEdge10 on a Windows SBS2K8 in a VMWare environment on mirrored SSDs. My question is simple; Having never dealt with this sort of volume of data, is Progress up for the task or should we be looking at a different database engine?

 

[Cringer]

I'm going to move this to the DBA forum...

 

[TomBascom]

100 million per year is 2.7 million per day or 114,155/hour or 3,677 per second.

That is well within the established capabilities of the untuned database on commodity hardware.

You can, of course, do much better with a little tuning or with Big Iron.

 

[cj_brandt]

Yes Progress is capable. OE seems to handle number crunching and other daily db items very well.

Are you going to have a constant steady stream of transactions as Tom calculated above, or are you going to have bursts of activity during batch processing ? So maybe for 4 hours each night you'll need to average 10,000 per second ?

Have you considered what you are going to do with all the data this system produces ? If you are in the planning stages, please give some thought to archiving and purging old data.
Have you considered how critical 24x7 will be ?

These are 2 areas I think Progress is far behind the others.

 

[TomBascom]

As far as archiving history goes... nobody is making any promises but, reading the tea leaves, the OE 11 "mutli-tenancy" feature is a very short step from horizontal partitioning. I would not be at all surprised to see horizontal partitioning available in -- 11.1. A number of large customers are very anxious for it.

Nobody is releasing time tables either. But there seems to be a sense that 11.1 is going to follow 11.0 very quickly.

 

[Rob Fitzpatrick]

Hi all, we have been a Progress shop since Version 3 so are well versed in the ways of PSC. We have had a recent request to build a number crunching system that needs to cater for up to a 100,000,000,000 (yes 100 million) transactions per annum.

I may be slow, but is this 100 million transactions (100 * 1,000,000) or 100 billion transactions ("100,000,000,000", or 100 * 1,000,000,000)?

 

[TomBascom]

Good question... looks like I messed up my math either way

Going with 100,000,000 (one hundred million) the per second equivalent would be:

100,000,000 / 365 = 273,973 per day.
273,973 / 24 = 11,416 per hour.
11,416 / 3600 = 3.17 per second.

Which would be a cakewalk (assuming a steady transaction rate).

Going with 100,000,000,000 we get:

100,000,000,000 / 365 = 273,973,602 per day.
273,973,602 / 24 = 11,415,525 per hour.
11,415,525 / 3600 = 3,171 per second.

So, my intermediate results were whacked but the bottom line still works out. I plead lack of coffee.

It's a good thing that someone is paying attention!

 

[Rob Fitzpatrick]

No worries, I also measure my IQ in molecules of caffeine.

ChrisJRutter:
Given my background, where "transaction" has a different meaning, I'm just curious: were you referring to database transactions, or something at the application level (e.g. financial transactions). From your intro I assume the former, but you know what they say about assumptions...

And to second cj's comment, if you indeed mean 100 billion txns/year, your purging and archiving strategy should be well thought out and tested before you go live (with sign-off from the business side on data retention policies, etc.).