Gary Oldman says he broke down physically after ex-wife's comments