Pat Glass, who has already apologised earlier in the campaign after calling a voter a 'horrible racist', made the remarks as she spoke alongside shadow chancellor John McDonnell, Wolverhampton South West MP Rob Marris and Guardian columnist Owen Jones at Wolverhampton Grand Station.
Taking questions from a 200-strong audience of Labour supporters, Ms Glass was responding to a question about encouraging voters to support the Remain campaign.
She said women and young people were 'key' to a successful Remain vote in the referendum before adding: "Go and speak to your mother, your grandmother. Don't speak to your grandfather, we know the problem are older white men."
The comments were met by muted laughter.
MORE: Latest polls: Remain and Leave both on 44% as Brexit warning issued
Mr McDonnell, who appeared to sit back when Ms Glass made the comment, later tried to laugh off the remark.
He said: "Can I speak up for a couple of older white men? I'll let Jeremy know."
His own comments were met by loud laughter in the hall.
Ms Glass's comment despite polls showing more men and pensioners are snubbing the Leave campaign.
Monday's 'Another Europe' event kicked off a week-long tour by Mr McDonnell which includes visits to Leeds and Leicester before ending with a rally in Cardiff alongside Labour leader Jeremy Corbyn.
During the one-and-a-half hour session, Mr McDonnell warned Labour supporters to ignore voters' anxieties over immigration 'at our peril'.
He also said the so-called 'Project Fear' was doing damage to both sides of the EU debate.
He told the Express & Star he believed traditional Labour voters who may have defected to the Tories were now coming back to the party.
He said: "I think we are on the stomp now putting over the positive arguments about Europe rather than UKIP who are frightening people. I'm concerned about the debate so far has been about Project Fear on both sides and we don't think that is working. When you are on the doorstep most people just want the facts and they want a rational debate.
On that basis I think we are mobilising people to engage in the debate and to support a reformed EU."
He also spoke about 'retrieving' the EU referendum debate from the 'mess' the Conservative Party has found itself in with questions over its leadership.
Last month Ms Glass issued an apology after she was caught on microphone describing a radio phone-in caller in Derbyshire a horrible racist' after he described some Polish neighbours as 'spongers'.
A West Midlands Regional Labour spokesman said: "It was a light-hearted comment and it was received in that way by the audience. Labour are clearly fighting for every single vote and are of course encouraging everyone to vote to remain on June 23."