A man has been accused of drugging and sexually assaulting his former wife with five other men over a 13-year period.