What error code to return for Math.sqrt() overflow condition?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

What error code to return for Math.sqrt() overflow condition?

kraftwerk1611
Hi,

For Math class in OS there is only one error code related to Math.sqrt()

Sys.error(4) for 'can not compute square root of a negative number'

We are also supposed to check for overflow condition

   [(y+2^j)*(y+2^j)] > 0

What error code should we return if this overflow occurs, the same Sys.error(4) ?

Is there a recommended error code for general/unknown errors?

Thanks.

Reply | Threaded
Open this post in threaded view
|

Re: What error code to return for Math.sqrt() overflow condition?

cadet1620
Administrator
Square root can not overflow.  The square root of 32767, the largest integer, is 181.

In the presented algorithm, (y+2^j)^2 can be greater than 32767, which results in integer overflow. For a safe implementation you do need to check if (y+2^j)^2 overflowed into a negative number. If the overflow occurred, treat it as if it was > x.
t = (y+2^j)^2
if (t <= x and t >= 0)  // t < 0 indicates overflow
    y = y+2^j

--Mark