The boy was angling near a wooded area off Brierley Hill Road, in Wordsley.
The man is described as being in his 20s and around 6ft tall, with olive skin and black hair. Officers believe he may have been in the area previously.
The incident took place at around 7.30pm on Thursday, June 10.
A spokesman for West Midlands Police said: "We are investigating after a man indecently exposed himself in front of a 12-year-old boy by a canal in Dudley.
"It happened as the youngster was fishing near a wooded area off Brierley Hill Road, Wordsley, at around 7.30pm on June 10.
"The man is described as being in his 20s, around 6ft tall with olive skin, and having black hair which is short on the back and sides and permed on the top.
"He had a moustache, local accent was wearing a black tracksuit with yellow stripes down the arms.
"We believe he may have been in the area previously and are asking anyone with information to contact us."
Anyone with information can call West Midlands Police on 101, quoting incident reference 20/388669/21.
Alternatively, contact Crimestoppers anonymously on 0800 555111 or at crimestoppers-uk.org