Israel has called off the planned release of 26 Palestinian prisoners, placing already embattled peace talks in further jeopardy after both sides took steps Washington called "unhelpful".

Israel's chief negotiator with the Palestinians has told them that the planned release cannot go ahead, a source close to the embattled talks told AFP on Thursday.

A frustrated US Secretary of State John Kerry earlier demanded that recalcitrant Israeli and Palestinian leaders demonstrate leadership in the crisis-hit peace process.

Speaking on condition of anonymity, the source said Justice Minister Tzipi Livni told a meeting with her counterparts on Wednesday that the prisoner release could not happen because the Palestinians had resumed a diplomatic campaign meant to challenge Israel more fully in the international arena.

The talks hit a new impasse when Israel failed to free the prisoners as expected at the weekend.

In response, the Palestinians formally requested accession to several international treaties.

The source said Livni told the Palestinians that her government had been seeking to expedite the release when the Palestinians submitted their accession request to UN bodies, and that they had therefore breached their commitments under the terms of peace talks restarted under US auspices last July.

Livni urged the Palestinians to cancel the move and return to talks, the source said.

White House spokesman Jay Carney called Israel's decision a further "challenge" to peace efforts.

"The decision by the Israelis to delay the release of the fourth tranche of prisoners creates challenges," he said.

Shortly after news of the cancellation broke, four rockets fired from the Hamas-ruled Gaza Strip slammed into southern Israel, a military spokesman told AFP.

He said there were no casualties and army radio said all the rockets fell in open countryside.

US officials said that Kerry, who has pursued more than a year of intensive shuttle diplomacy, spoke by phone to Israeli Prime Minister Benjamin Netanyahu on Thursday afternoon, and to Palestinian president Mahmud Abbas later, but they gave no details.

Kerry said in Algiers that Israelis and Palestinians made "progress" in lengthy overnight talks in Jerusalem, also attended by the Americans.

His efforts appeared to be on the brink of collapse this week after Israel announced a fresh wave of settlement tenders and the Palestinians resumed international recognition moves.

Settlements and international recognition are two of the touchiest issues dividing the two sides.

Israeli settlements in the West Bank and east Jerusalem are a source of much bitterness for Palestinians, who want these areas for their long-promised state.

Palestinian moves to join international treaties and organisations are meanwhile seen as a bid to unilaterally further their statehood claim.

Washington described the latest moves as "unhelpful, unilateral actions", but insisted diplomacy still had a chance.

Kerry threw down the gauntlet, telling both sides it was time for compromise at what he called a "critical moment" in the peace process.

"You can facilitate, you can push, you can nudge, but the parties themselves have to make fundamental decisions to compromise," he said.

"The leaders have to lead, and they have to be able to see a moment when it's there."

- Marathon overnight meeting -

He said, however, that progress was made at a meeting between Livni, US special envoy Martin Indyk and Palestinian negotiator Saeb Erakat that lasted until 4:00 am Thursday.

The overnight marathon "focused on the necessity of releasing the prisoners", a Palestinian official told AFP on condition of anonymity, adding that the applications for accession to several international treaties were "irreversible".

Each side accused the other of violating undertakings given when the talks began.

"Israel has a habit of evading agreements and conventions it has signed," Yasser Abed Rabbo, general secretary of the PLO executive committee, told AFP.

"That is why conditions for future negotiations must change radically."

The moves dealt a hammer blow to Kerry's frenetic efforts to broker an extension of the negotiations beyond their original April 29 deadline.

UN Middle East peace envoy Robert Serry confirmed receiving the Palestinian applications, with a spokesman for the secretary general saying they would consider the "appropriate next steps".

The first treaty the Palestinians applied to was the Fourth Geneva Convention, which holds huge symbolic importance as it provides the legal basis of their opposition to Israeli settlements in the occupied territories.

In Israel, there was surprise and anger over the Palestinian move.

"The Palestinians have returned to a diplomatic intifada," one political official told Yediot Aharonot newspaper, using the Arabic word for uprising.

burs-jkb-hmw/scw/jhb