They have launched a petition to express their dismay at the prospect of boundary changes pushing them into other parts of the city.
Penn councillor Celia Hibbert has organised the campaign after proposed changes were agreed by the Local Government Boundary Commission for England.
The changes would see addresses on Coalway Road moved into inner-city Graiseley. Meanwhile, Goldthorn Hill, Waverley Crescent, Marlbrook Drive and Coton Road would be transferred into Blakenhall.
Councillor Hibbert said she did not agree with the decision, arguing that Graiseley and Blakenhall did not have greater community linkage to the addresses than Penn.
She said: “I just feel disheartened as it doesn’t make a difference to what they’re looking to achieve because the average electorate per ward in Wolverhampton is 3,175.
“The changes they are proposing don’t make a significant change to the ward, but will have a significant impact on the residents themselves who live on those streets.
“As a councillor, I have to represent my residents and, clearly, they are not happy about it.”
Councillor Hibbert said she had spoken to residents who said they had bought their houses specifically because of being in Penn and had voiced their displeasure at moving into Graiseley and Blakenhall.
She said: “There has been a lack of communication and awareness of the situation, which is really poor when you consider how big an issue it is.
"A lot of people put in a lot of time to secure their properties or think about what part of town they wanted to live in and I believe Penn is one of those wards people would aspire to live in.
I think we will lose our cultural identity and our boundary and I would like to see it dropped for the moment as I see Merry Hill, Tettenhall Regis and Tettenhall Wightwick staying the same and I want to know why Penn is not."
The Local Government Boundary Commission for England said it welcomes feedback.
A spokesman said: "We are carrying out an electoral review of Wolverhampton, which we are doing because of inequality in Wolverhampton.
"What that means is that some councillors represent more people than in other parts of the authority.
"The stage we're at in Wolverhampton is we are consulting on some proposals and we would encourage local people to give us their views on the proposals as we take local insight very seriously and quite often make changes as a result."
Residents are asked to get in touch by September 6 by emailing email@example.com or online at consultation.lgbce.org.uk/have-your-say/24695