Supposedly an accusation of trying to smuggle his heir out of the country was levelled at Clarence during his downfall and included in his act of attainder (Clarence also supposedly visited Ireland in 1476/7).
Why? Well it's Clarence we're talking about here, his decision-making processes and mental state, especially after his wife's death, were less than great. Given he believed his wife (and second son?) had been poisoned, general paranoia, and belief the Woodvilles wanted to get rid of him, is it hard to believe he wanted to smuggle his heir to safety?