The Darlaston light-heavyweight was edged out on a split decision by Cuba’s Arlen Lopez in the gold medal bout at Tokyo 2020.
He told the BBC: "You don't win silver, you lose gold. I'm very disappointed with myself. I didn't perform good.
“I am very disappointed. I feel like a failure. He (Lopez) was very good but you are in this game to win gold, not silver. This feeling will stick with me now. I don't want this feeling again."
Lopez claimed a 4:1 decision on the scorecards after a close contest, the Cuban becoming a double Olympic champion in the process, having claimed the middleweight crown at Rio 2016.
A visibly upset Whittaker immediately pocketed his silver during the medal ceremony and explained: “I don’t want to look like a baby or a spoiled brat but I am so upset that I didn’t win the gold. I feel like I lost gold. I can’t celebrate silver.
“Maybe in a couple of years when I’ve got kids I can look back and say your dad lost a gold medal. But that is all it is.
“I want to put this to the back of my mind because I didn’t achieve what I wanted to achieve. I want to thank God for putting me in this position and everyone in Great Britain for getting behind me. Sorry for making you wake-up and seeing me let you down.”
Whittaker has made headlines for his performances both in and out of the ring in Japan, having previously claimed he wanted to become Mayor of Wolverhamton on his return to the UK.
He said: “The mayor did a little video wishing me good luck and said if I got the gold I could get the chain.
“I didn’t get the gold so I am sure he can keep it for a bit. When I go pro and get a world championship belt we can talk about this again.”