Rounding and Precision Errors

hilss

Active Member
Joined
May 22, 2007
Messages
379
Hello,

In Excel/VBA, I have variables: Q and R. They are both a function of x (complicated formula - but no need to describe it here).
D = Q^3 + R^2

So I have:
x in column A
Q in column B
R in column C
D in column D

At one point, Q^3 so small that D = R^2. I am adding a small number (Q^3) with a big number (R^2).

If I graph D vs x, I see a drop in the graph of D (a discontinuity). I'm wondering if there's a way to make this drop smoother.

D is later used to compute other things. This discontinuity in D affects the remaining results.

The values of Q and R can vary a lot. So it's not like they stay in a certain range.

Thanks,
hilss
 

Excel Facts

Is there a shortcut key for strikethrough?
Ctrl+S is used for Save. Ctrl+5 is used for Strikethrough. Why Ctrl+5? When you use hashmarks to count |||| is 4, strike through to mean 5.

Forum statistics

Threads
1,226,112
Messages
6,189,040
Members
453,521
Latest member
Chris_Hed

We've detected that you are using an adblocker.

We have a great community of people providing Excel help here, but the hosting costs are enormous. You can help keep this site running by allowing ads on MrExcel.com.
Allow Ads at MrExcel

Which adblocker are you using?

Disable AdBlock

Follow these easy steps to disable AdBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the icon in the browser’s toolbar.
2)Click on the "Pause on this site" option.
Go back

Disable AdBlock Plus

Follow these easy steps to disable AdBlock Plus

1)Click on the icon in the browser’s toolbar.
2)Click on the toggle to disable it for "mrexcel.com".
Go back

Disable uBlock Origin

Follow these easy steps to disable uBlock Origin

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back

Disable uBlock

Follow these easy steps to disable uBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back
Back
Top