Birmingham substitute Patrick Roberts snatched an 89th-minute equaliser at St Andrew's to cancel out Lukas Nmecha's 49th-minute opener for Leeds.
With a feisty fourth round tie level at 1-1 after extra time, Leeds won the shoot-out 4-2.
Joel Piroe, Dominic Calvert-Lewin, Brenden Aaronson and Sean Longstaff all converted their penalties for Daniel Farke's team.
Tommy Doyle's effort was saved by Leeds goalkeeper Lucas Perri before Roberts fired over the crossbar.
Battling to avoid relegation from the Premier League, Leeds are one win away from reaching the quarter-finals for the first time since 2002-03.
Wolves also avoided an upset as they won 1-0 at fourth-division Grimsby.
Grimsby stunned Manchester United in the League Cup earlier this season.
But Wolves avoided similar embarrassment thanks to Santiago Bueno's 60th-minute goal from the visitors' first shot on target on a mud-caked pitch at Blundell Park.
Grimsby didn't manage to force Wolves 'keeper Sam Johnstone into a save until Andy Cook's header in the 90th minute.
It was Wolves' first away win since beating Manchester United in April.
Mired in a miserable season that has left them bottom of the Premier League and almost certain to be relegated, Rob Edwards' side can still salvage some pride in the FA Cup.
