Residents fighting plans to house more than 480 asylum seekers in former university accommodation share their concerns