Philip Rycroft said the British public was ‘falling out of love’ with Brexit