Australia celebrate their goal
The Danes, needing victory to progress, dominated possession but were caught out on the break when Australia scored the winner on the hour mark.
Denmark lost the ball in the Australian box and Mathew Leckie finished a brilliant counter-attack with a low shot past Kasper Schmeichel.
Australia finish in second place in Group D on six points, only behind world champions France on goal difference.
Denmark, meanwhile, exit Qatar 2022 in disappointing fashion, with just one point from three games, sitting bottom of the group.
Australia will now play Group C winners Argentina on Saturday (19:00 GMT).
Denmark came into the World Cup tipped by many fans and pundits as 'dark horses' after their impressive run to the semi-finals of Euro 2020 and a strong Uefa Nations League campaign in which they beat France home and away.
However, they failed to impress in Qatar and go home having scored only once in three Group D games.
Their attacking struggles were acknowledged by manager Kasper Hjulmand, who made three changes from the defeat by France, all in attacking positions.
Martin Braithwaite, Brentford midfielder Mathias Jensen and Andreas Skov Olsen came in as Mikkel Damsgaard, Victor Nelsson and striker Andreas Cornelius dropped to the bench - but the changes made little difference.
The Danes' best chance came in the first half through Jensen, who forced Australia captain Mat Ryan into an impressive acrobatic save after being played through by a Braithwaite flick.
However, it was telling that Denmark's most threatening attacking outlet was left-back Joakim Maehle, who tested Australia with his probing runs but was not adequately supported by those further forward.
They demanded a penalty in the second half when Kasper Dolberg was hauled down in the area, however he was narrowly offside as the Danish campaign petered out.
Denmark continue their longest winless World Cup run, having failed to earn a victory in any of their past six games in the competition, with four of them draws.
Australia, the lowest-seeded side in Group D, ran the game at their tempo and the largely uneventful encounter suited their needs perfectly, with a draw potentially enough to see them through having started the day second in the group on three points.
However just as the news filtered through of Tunisia taking a shock lead against France, which would have eliminated them on goal difference had they drawn here, Leckie sprang into action.
Dolberg lost the ball in the Australian box and his side were caught out on the break. Melbourne City winger Leckie ran confidently at the Danish defence, and struck left-footed through Maehle's legs past Schmeichel inside the far post.
It ensured they reach the last 16 of the World Cup for the first time since 2006, when a star-studded side featuring Harry Kewell, Tim Cahill and Mark Schwarzer were only stopped by eventual champions Italy.
This team features far fewer big names but, marshalled by captain and former Brighton and Arsenal goalkeeper Ryan, they have recovered from a 4-1 loss to France in their opening game - where they were naively open and picked apart by Kylian Mbappe and co - to grind out the results required.
It was Ryan who guided Australia through the nervy final six minutes of added time, claiming corners with aplomb.
A Cornelius header looped onto the roof of the net in stoppage time but a historic victory was secured.
-- Courtesy of BBC Sport