In power since 2019, Frederiksen, 48, had campaigned on a promise that her tough and tested leadership skills would help the Nordic nation navigate a complex relationship with US President Donald Trump and the European response to Russia's war in Ukraine.
But on Tuesday she emerged bruised both from the left and the right at home, where the cost-of-living crisis has come to the front of voter concerns, observers said.
Frederiksen's Social Democrats, the architects of Denmark's cradle-to-grave welfare state, were seen winning 38 seats in the legislature, the Folketing, compared with 50 four years earlier.
Her chances of staying in for a third term were not gone, observers said, although coalition talks could take weeks.
Frederiksen's left-wing bloc was seen winning 84 seats in parliament, versus 77 for the right-leaning parties, projections by local media based on 100 per cent of votes counted showed.
Many of her left-wing supporters appeared frustrated with an immigration policy they saw as too tough, while some on the right saw her too soft and untrustworthy on economic issues.
"She is between a rock and a hard place because the numbers are bad for her," said Andreas Thyrring, a partner at Ulveman & Borsting public affairs advisory firm.
In Brussels, Frederiksen is widely respected for her clear line on Greenland and for her efforts to ramp up Denmark's defence spending in the wake of the Ukraine conflict. But her negotiating style is seen by some as abrasive and many Danes sought change.
The vote was also being closely watched in Greenland, with many hoping it will be a chance for the territory to leverage Trump's unprecedented desire to wield control over the Arctic island to wrangle concessions from its former colonial power in Copenhagen.
Underscoring the broad backlash against Frederiksen, support for the anti-immigration Danish People's Party, led by Morten Messerschmidt, surged to 9.1 per cent with more than 90 per cent of votes counted by public broadcaster DR, up nearly seven percentage points compared to the last election.
Messerschmidt had campaigned on a pledge to ensure zero net migration of Muslims and to abolish petrol taxes as a measure to ease living costs.
"The fact that the Danish People's Party has now tripled its support clearly shows that Danes are fed up with this and that there are a great many people who want a different direction for Denmark," Messerschmidt said after exit polls were published.
The non-aligned Moderates party of Lars Lokke Rasmussen could hold the key to the next ruling coalition, some observers said, with the outgoing foreign minister calling on Frederiksen to drop her calls for a wealth tax.
"There is no hard-red majority to our left, and no hard-blue majority to our right," Rasmussen said at his party's election-night party in Copenhagen.
Frederiksen proposed the tax - at a modest rate of 0.5 per cent aimed at funding education reform - to rebuild her leftist credentials that had been damaged by a coalition with the centre-right.
She has also overseen one of the toughest approaches to migration in Europe, with refugee status temporary, conditional support and expectations of integration in society.
She also co-led a push by nine European Union countries for easier expulsion of foreign criminals, and earlier in 2026 proposed legislation to increase deportations.
The leader of the Liberal Party, Defence Minister Troels Lund Poulsen, said he was no longer interested in coalition rule with Frederiksen, underscoring complex talks ahead for her.
"The possibility is there, Lars!" Poulsen said in Copenhagen in an apparent nudge to Rasmussen.
