Valerie Vaz has raised concerns over the proposed site at the Bescot sidings freight yard on the Wednesbury and Walsall border, which would see 600,000 concrete rail sleepers built each year.
The Walsall South Labour MP has questioned the decision making process behind the scheme, warning that it would see up to 35 lorries a day travelling through her constituency.
This would worsen air quality in an area which already suffers from some of the highest levels of air pollution in the country, she said.
The factory has been proposed by Network Rail as a replacement for its site in Washwood Heath, which is being demolished to make way for HS2.
Ms Vaz has written to Douglas Oakervee, chair of the HS2 review, calling for the project to be scrapped.
It is estimated that 4,000 HGV deliveries would be needed during the construction of the factory, with up to 12,816 deliveries made each year once it is operational.
"The factory would operate twenty-four hours a day and the route passes homes, schools and care homes," she said.
“Walsall has some of the highest levels of air pollution in the country owing to its proximity to the M6 and the congestion on its major roads.
"This development would worsen air quality in Walsall and increase congestion. Residents do not wish to see a factory built on the Bescot Railway Sidings."
Ms Vaz also criticised Network Rail's handling of the consultation over the site, accusing the body of being "opaque and obstructive" throughout the process.
In her letter she said that residents had not been informed of the plans until March, while two consultation events in that month failed to provide the necessary information to the public.
Network Rail submitted a planning application for the factory in August after the Bescot site had been selected out of a total of 28 potential plots.
Sandwell Council has said the development "does not accord with the provisions of the development plan in force in the area in which it is situated".
Rail bosses insist the factory, which they say will create 100 jobs, is "perfectly placed" to transport sleepers by rail "right to where they need to be".