Residents are fighting plans to convert Pinewood House into 51 temporary short-stay accommodation homes.