The PM said he wants a "factual" and "evidence-based debate" about returning to free movement between Britain and Europe.