Exclusive: ‘We are suffering a punishing Brexit tax every day,’ says former foreign secretary in withering assessment of decision to leave the bloc