Reform's Zia Yusuf and Tory Chris Philp were speaking against converting a former military site into asylum accommodation.