Rounding and Precision Errors

hilss

Active Member
Joined
May 22, 2007
Messages
379
Hello,

In Excel/VBA, I have variables: Q and R. They are both a function of x (complicated formula - but no need to describe it here).
D = Q^3 + R^2

So I have:
x in column A
Q in column B
R in column C
D in column D

At one point, Q^3 so small that D = R^2. I am adding a small number (Q^3) with a big number (R^2).

If I graph D vs x, I see a drop in the graph of D (a discontinuity). I'm wondering if there's a way to make this drop smoother.

D is later used to compute other things. This discontinuity in D affects the remaining results.

The values of Q and R can vary a lot. So it's not like they stay in a certain range.

Thanks,
hilss
 

Excel Facts

Can you AutoAverage in Excel?
There is a drop-down next to the AutoSum symbol. Open the drop-down to choose AVERAGE, COUNT, MAX, or MIN

Forum statistics

Threads
1,223,214
Messages
6,170,771
Members
452,353
Latest member
strainu

We've detected that you are using an adblocker.

We have a great community of people providing Excel help here, but the hosting costs are enormous. You can help keep this site running by allowing ads on MrExcel.com.
Allow Ads at MrExcel

Which adblocker are you using?

Disable AdBlock

Follow these easy steps to disable AdBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the icon in the browser’s toolbar.
2)Click on the "Pause on this site" option.
Go back

Disable AdBlock Plus

Follow these easy steps to disable AdBlock Plus

1)Click on the icon in the browser’s toolbar.
2)Click on the toggle to disable it for "mrexcel.com".
Go back

Disable uBlock Origin

Follow these easy steps to disable uBlock Origin

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back

Disable uBlock

Follow these easy steps to disable uBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back
Back
Top