Although you could round the value-if-true and value-if-false expressions separately, you could simply round the entire IF-expression because it always returns a number, to wit:
=ROUND(IF(AND(ISERROR(AOT_PricePerShare),AOT_Shares>0),-AOT_BasePrincipal,AOT_PricePerShare*AOT_Shares), 2)
OTOH, it seems odd that you will (try to) calculate AOT_PricePerShare*AOT_Shares if ISERROR(AOT_PricePerShare) is true, but AOT_Shares>0 is false (i.e. AOT_Shares<=0). That will result in an Excel error, namely whatever is making ISERROR return TRUE.
So I suspect your formula is faulty to begin with. And depending on how you fix it, rounding the entire IF-expression might no longer be correct.
But perhaps it is as simple as changing the logic to OR(ISERROR(AOT_PricePerShare),AOT_Shares<=0) .