Oliver initially awarded a spot-kick after Trezeguet tumbled under the challenge of Solly March but changed his mind after consulting the pitchside monitor.
That meant March’s second half strike proved to be the winner as the Seagulls won for the first time ever at Villa Park.
Danny Welbeck had earlier put the visitors in front before Ezri Konsa levelled for Villa.
The Seagulls also saw Tariq Lamptey sent-off for a second bookable offence during a chaotic finish.
Villa, once more erratic on their own turf, thought they would have the chance to claim a point when Oliver pointed to the spot after March appeared to get both ball and man when challenging Trezeguet.
But after consulting VAR Christoper Kavanagh, Oliver went to look at the pitchside monitor and promptly changed his mind, much to the fury of home players and their head coach Dean Smith.
Yet whatever the rights or wrongs of Oliver’s decision-making or your view on VAR, Villa ultimately paid the price for a disjointed performance in which they lacked composure in both boxes.
Having started brightly, the hosts were guilty of dire defending which allowed Welbeck a clean run at goal from inside his own half.
Konsa got Villa back on terms early in the second half when he finished Bertrand Traore’s free-kick but the hosts failed to kick on.
To add further to Smith’s concerns was an injury to Ross Barkley which saw the on loan Chelsea midfielder forced off inside the opening five minutes.
Smith unsurprisingly stuck with the team which had won so impressively at Arsenal but within five minutes Barkley’s injury forced a change.
The midfielder was left holding his right hamstring after delivering a free-kick from the left just 50 seconds into the match. After a discussion with the club’s medical staff and a period in which Villa played a man down, Barkley walked down the tunnel and Traore replaced him on the pitch.
Villa were still the brighter team at that point but neither Ollie Watkins or Konsa could make contact with an inviting Traore free-kick delivery.
Yet on 12 minutes the hosts were behind after being completely caught out. Welbeck was the furthest player forward, five yards inside his own half, when he collected Lallana’s pass, setting off on a 40-yard run toward goal before sending a composed, lobbed finish over Emiliano Martinez.
The England international might have had another soon after had he been able to make a better contact with Tariq Lamptey’s cross, after John McGinn had been guilty of giving the ball away in his own half.
Villa needed a spark and Watkins almost provided it, bursting down the left before finding Grealish. The skipper’s cross ran to Trezegurt who, after seeing his first effort blocked by Adam Webster, somehow put wide from seven yards out.
Chances continued to come at both ends of the pitch with the hosts looking dangerous in attack but shoddy at the back.
Traore forced Mat Ryan into a sharp save, before Martinez denied Maupay one-on-one, having raced out of his box, after Villa’s backline had parted like the Red Sea.
Grealish then tested out Ryan with a curling effort but Lallana should have done better than side foot wide after being picked out by the excellent Lamptey.
Villa ramped up the pressure as half-time approached but a leveller proved frustratingly elusive.
Ryan kicked the ball off his line in unorthodox fashion after Mings had met a Grealish free-kick with a looping effort, before brilliant defending from Ben White saw him dive in to deny Watkins a tap-in and deflect a Grealish cross off the post.
Having put in such a big effort to hold out, the manner in which they eventually conceded barely a minute into the second half will have been hugely frustrating for the visitors.
Targett, booked late in the opening period after clashing with Maupay, was brought down on the left, Traore swung in the free-kick and Konsa arrived unmarked to finish past Ryan.
The platform appeared to be there for Villa to push on but they still looked out-of-sorts at the back and it was Brighton who duly went back in front nine minutes later.
Neat build-up play on the edge of the Villa box ended with Pascal Gross delivering a low cross which March despatched first time into the top corner.
Villa’s attacks were lacking the same fizz as the first-half and Smith made a double change with 15 minutes remaining, bringing on Conor Hourihane and Anwar El Ghazi.
But it was Brighton who had the next big chance when March floated in a cross and Maupay headed wide.
Villa looked short on ideas before chaos ensued in the closing stages. Lamptey picked up two quick bookings to leave the visitors a man short before Oliver ensured he would be the man taking the headlines.
12 - GOAL - Danny Welbeck puts the visitors ahead, running from inside his own half before lifting a finish over Emiliano Martinez.
47 - GOAL - Villa level. Traore swings in a free-kick and Ezri Konsa arrives at the far post to finish.
56 - GOAL - Brighton back in front. Solly March finishes superbly from the edge of the box after meeting Pascal Gross’s pull back.
Villa (4-2-3-1): Martinez, Cash, Konsa, Mings, Targett, Luiz (Hourihane 75), McGinn, Trezeguet, Barkley (Traore 5 (El Ghazi 75)), Grealish, Watkins Subs not used: Elmohamady, Taylor, Nakamba, Steer (gk).
Brighton: Ryan, Lamptey, Dunk, White, Webster, March, Gross, Bissouma, Lallana (Veltman HT), Welbeck, Maupay (Burn 80) Subs not used: Connolly, Jahanbakhsh, Alzate, Molumby, Steele (gk).