Much of the economic cost stems from UK’s departure from the single market – suggesting recent calls for a new customs union with the EU would have little impact