‘What we do in our next manifesto will be a question for the Labour Party’, a government minister said