Rounding issue

[mprabu_p]

Hi All,

I am facing the following problem in progress.

When I run the folllwing query in the progress editor

display 900000000 * ( 100 / (100 - 10)) format "9999999999.99".

It given the following results
999999999.99

But if I do the same calculation in excel sheet or calculator
I got the value 1000,000,000. ( I am also expecting this value)

It seems progress storess only 10 digits in decimals value. if value exceeds 10 decimal vaues in gives the problem.

Can anyone please suggest any workaround for this.?

 

[tamhas]

Declare more decimals for a variable and compute into the variable. 10 decimals is the default for a variable and consequently the value used in an in-line computation.

 

[TomBascom]

"decimals 10" is the limit. You cannot store more than 10 places to the right. You could always shift your math by multiplying everything by an appropriate power of 10 and then unbias it afterwards.

Personally I've never felt the urge to do so. Could you enlighten us as to the use case?

 

[Pavan Yadav]

Hey Prabu,
Just try using using statement like
display 900000000 * 100 / (100 - 10) format "9999999999.99".
Instead of :
display 900000000 * ( 100 / (100 - 10)) format "9999999999.99".

You are getting the correct result as per that also. This is my understanding...
Please some seniors, put your views also for this, if i am wrong.

 

[TomBascom]

It depends entirely on what the real issue is. But sure, generally speaking you don't want to divide a large number by a comparatively small number if you can avoid doing so and re-grouping the expression to keep the magnitudes closer is a good idea.