At the end of a goalless draw the Matildas lost 7-6 after goalkeeper Barbara saved Alanna Kennedy's spot kick.
The Australians had a glorious chance to win the match when Marta missed Brazil's fifth kick but in reply and with a place in the semis beckoning Katrina Gorry saw her shot saved.
Brazil now meet Sweden in the semi-finals in Rio on Thursday (AEST).
The match itself was a pulsating affair with playmaker Marta's Brazilians having more of the ball but the Matildas stood their ground with goalkeeper Lydia Williams in sparkling form.
The first half provided little goalmouth action as both teams were locked in a fruitless battle for supremacy.
The home side stepped up a gear in the second half and dominated possession but the Matildas defence repelled every Brazilian attack.
The Matildas were unfortunate not to open the scoring when Chloe Logarzo hit the bar with a firm drive five minutes from the end of normal time.
Seconds later Williams rescued Australia with a diving save to thwart Andressa Alves's header and take the match into extra time.
In the later semi-final on Thursday Canada face Germany.
The Canadians beat France 1-0 while the Germans beat China by the same score.
