I'm surprised someone told you that you need to do this. At this point you should install 64-bit on any machine capable of it. Do you have any idea what is the technical reason that you have to downgrade? Does the Access program use any VBA? Some VBA is dependent on 32 vs. 64 but this is not very common for the average user. It is better to the fix the VBA than to downgrade your whole Office install. If you have Excel files with VBA that depend on 64-bit you will have the opposite problem after a downgrade, although again, that's not common for the average user.
Other than that you could potentially lose addressability of large address spaces. Office will also require you to downgrade all your apps, not just Access, so you will have to do analysis for impact on everything else.