Rounding and Precision Errors

hilss

Active Member
Joined
May 22, 2007
Messages
379
Hello,

In Excel/VBA, I have variables: Q and R. They are both a function of x (complicated formula - but no need to describe it here).
D = Q^3 + R^2

So I have:
x in column A
Q in column B
R in column C
D in column D

At one point, Q^3 so small that D = R^2. I am adding a small number (Q^3) with a big number (R^2).

If I graph D vs x, I see a drop in the graph of D (a discontinuity). I'm wondering if there's a way to make this drop smoother.

D is later used to compute other things. This discontinuity in D affects the remaining results.

The values of Q and R can vary a lot. So it's not like they stay in a certain range.

Thanks,
hilss
 

Excel Facts

Back into an answer in Excel
Use Data, What-If Analysis, Goal Seek to find the correct input cell value to reach a desired result

Forum statistics

Threads
1,223,903
Messages
6,175,284
Members
452,630
Latest member
OdubiYouth

We've detected that you are using an adblocker.

We have a great community of people providing Excel help here, but the hosting costs are enormous. You can help keep this site running by allowing ads on MrExcel.com.
Allow Ads at MrExcel

Which adblocker are you using?

Disable AdBlock

Follow these easy steps to disable AdBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the icon in the browser’s toolbar.
2)Click on the "Pause on this site" option.
Go back

Disable AdBlock Plus

Follow these easy steps to disable AdBlock Plus

1)Click on the icon in the browser’s toolbar.
2)Click on the toggle to disable it for "mrexcel.com".
Go back

Disable uBlock Origin

Follow these easy steps to disable uBlock Origin

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back

Disable uBlock

Follow these easy steps to disable uBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back
Back
Top