Open letter criticising invitation to Nigel Farage warns of association with ‘racism and inflammatory rhetoric’