Based on the first 87,000 votes counted of the 270,000 voters eligible, some 51.2 per cent supported the government's proposal to restart negotiations, while 48.8 per cent said no, according to the RUV broadcaster.
A reliable result might not be available until midday on Sunday, election officials said.
Voting ended at 10pm (8am AEST Sunday), kicking off a complex operation to bring ballots from towns and rural polling stations by plane, boat or car.
On Friday, a Gallup poll suggested a narrow majority opposed the government's proposal of reopening negotiations, a slight shift from earlier surveys that pointed to a narrow lead for supporters of renewed talks.
"Yes" campaigners backing talks say joining the bloc could ease high interest rates and offer more security in a world unsettled by the war in Ukraine and US President Donald Trump's threats over another Arctic island, Greenland.
"No" campaigners argue EU membership would threaten the island's sovereignty and fishing waters.
The referendum campaign has fuelled debate about Iceland's place in the world, Prime Minister Kristrun Frostadottir told reporters after casting her ballot in the capital on Saturday.
"This is a great day, we've been waiting for this for years to be able to have a vote," Frostadottir said, adding that her government would continue its work regardless of the outcome.
A "yes" vote would open the way to discussions with Brussels. Iceland would have to hold a second referendum, likely several years later, on whether to join the EU.
Iceland first applied to join the EU in 2009, amid a financial crisis that hit the country's small, exposed economy especially hard. Talks ended in 2013 when a Eurosceptic government shelved the process.
Renewed geopolitical turmoil has brought fresh urgency to an issue Icelanders have debated on and off for decades.
Debates inside Iceland have focused more on interest rates, fishing rights and the cost of living than on security.
Iceland has grappled with persistently high inflation and rates that made mortgages punishingly expensive.
Supporters of a "yes" vote say EU membership, and possibly adoption of the euro, could offer some relief. Iceland's central bank rate stands at 8.00 per cent, against 2.25 per cent at the European Central Bank.
"It would not solve Iceland's structural faults, but it could become a catalyst to a more healthy economy," said Vilhjalmur Hilmarsson, chief economist at Viska, one of Iceland's largest labour unions.
Opposition leader Gudrun Hafsteinsdottir, who has led the "no" campaign, has argued the economic case is overstated.
"These are not problems that Brussels will solve for us. These are problems that Icelandic politicians must solve," she said.
10°C