N
Nicholas Womack
Guest
Say that I needed to turn the character variable "0.0000000001" into a decimal. But if I were to write out the following logic:
It produces this result:
So that must not be the solution, and truncating would also just remove the data I'm trying to preserve. Does anyone know how I can preserve the precision of small decimal numbers? It doesn't have to be this crazy case here to the ten-billionth place, but having at least 7 or 8 numbers of precision would help with my issue.
Continue reading...
Code:
define variable tinyChar as character initial "0.0000000001" no-undo.
define variable tinyNum as decimal no-undo.
assign tinyNum = decimal(tinyChar).
display tinyNum.
It produces this result:
Code:
0.00
So that must not be the solution, and truncating would also just remove the data I'm trying to preserve. Does anyone know how I can preserve the precision of small decimal numbers? It doesn't have to be this crazy case here to the ten-billionth place, but having at least 7 or 8 numbers of precision would help with my issue.
Continue reading...