Rubio said the United States was "very happy" to be facilitating the discussions, while noting that "we understand we're working against decades of history and complexities" that will not be quickly resolved.
Hezbollah opposes the direct talks and was not represented while the Iranian-algined group appeared to step up its fire on northern Israel as the discussions began.
"But we can begin to move forward with a framework where something can happen, something very positive, something very permanent, so that the people of Lebanon can have the kind of future they deserve, and so that the people of Israel can live without fear," Rubio said.
Despite Hezbollah's outright rejection, the talks are a major step for two countries with no diplomatic relations that have been officially at war since Israel's inception in 1948.
The latest round of fighting was sparked by Hezbollah firing rockets into northern Israel on March 2, days after the United States and Israel attacked Iran, Hezbollah's key ally and patron.
Rubio and US ambassador to the United Nations Mike Waltz attended the opening of the session at the State Department that is being led by US ambassador to Lebanon Michel Issa, Israeli ambassador to the US Yechiel Leiter and Lebanese ambassador to the US Nada Hamadeh Moawad.
The Lebanese government hopes the talks will pave the way to an end to the war.
While Iran has set ending the wars in Lebanon and the region as a condition for talks with the US, Lebanon insists on representing itself.
Hezbollah and other critics maintain that Lebanon's government lacks leverage and that it should instead back Iran's position.
Wafiq Safa, a high-ranking member of the militant group's political council, told the Associated Press that it will not abide by any agreements that are made during the talks.
On the day of the talks, incoming fire triggered nonstop drone and rocket alert sirens in Israeli communities near the Lebanese border.
Hezbollah so far on Tuesday has claimed 24 attacks on northern Israel and on Israeli troops in southern Lebanon.
At least 2124 people have been killed in Israeli strikes in Lebanon, the health ministry said, including hundreds of women and children.
More than a million people are displaced.
The deadliest day of the war took place last week, when Israel launched 100 air strikes across Lebanon in 10 minutes, including in the heart of the capital, killing more than 350 people.
The Israeli military has invaded southern Lebanon, a move some Israeli officials have said aims to create a "security zone" from the border to the Litani River, 30km to the north.
Israel's defence minister says hundreds of thousands of people uprooted from southern Lebanon will not be allowed to return home until the area is demilitarised and Israel believes its northern communities are safe.
Hezbollah, though weakened in its last war with Israel that ended in November 2024, still fires drones, rockets and artillery daily into northern Israel and on ground troops inside Lebanon.
Hezbollah enjoys wide influence in Beirut's southern suburbs as well as large swaths of the country's southern and eastern provinces.
Hezbollah-allied politicians hold two cabinet minister positions although the group's ties have soured with Lebanon's top political authorities, who have been critical of Hezbollah's decision to enter the war last month and who have since criminalised the group's military activities in the country.
The talks are the first between Israel and Lebanon since 1993.
Both countries have relied on indirect communication, often brokered by the United States or UNIFIL, the United Nations peacekeeping mission in southern Lebanon.
The Israeli and Lebanese governments were meeting to discuss ways to ensure long-term security on Israel's northern border and support for Lebanon seeking to take control of its territory and political future from Hezbollah, a US State Department official said.
