A derelict building on High Street is currently being transformed into four one-bedroom flats in a project led by Bolsover District Council and Action Housing and Support.
The project aims to help former offenders find accommodation and live responsibly in society.
Sarah Bryan and Thomas Smith are among a group of villagers concerned about who may soon be living in the properties, which are close to Clowne Nursery and Infant School.
They said: “We feel utterly demoralised.”
“We have worked so hard to be able to purchase our first home.”
“We thought we had managed to find a great location for starting a family.”
Clara Baldwin added: “We’re concerned we could have alcoholics, drug addicts, severely mentally ill people and convicts just out of prison on our doorstep and around our kids.”
When contacted by the Derbyshire Times, Clare McGonigle, chief executive of Action Housing and Support, was unable to categorically state who would be moving into the flats.
She said: “It could be vulnerable people or it could be families, for example.”
“But we are very, very conscious of local people’s concerns,” she added.
A spokesman for Bolsover District Council said: “All the properties are being refurbished to a very high standard.”
“The properties are due to be completed by the end of the year.”
“Anyone wishing to rent one of these flats must be on our housing register where allocation of properties is done in strict accordance with Government guidelines,” they added.