A man accused of sexually assaulting a woman drugged by her husband has told a court he was used as a pawn in a “vile monster’s” game.