The nearly $6 million project will replace the current town hall on Woodrow Street near the town’s police department and courthouse.