Iran is wrestling with a complex array of historic alliances and enmities as it tries to develop a coherent response to the swift advance of hostile Sunni militants in neighbouring Iraq, said experts.

Despairing of its protege Nuri al-Maliki, Tehran's Shi'ite clerical establishment has sent mixed messages on working with the Iraqi prime minister's other sponsor, the United States, with which it shares a goal of averting the country's break-up.

After decades competing with Washington's Sunni Arab allies for influence, it hopes for relief from U.S. sanctions by cutting a deal on its nuclear programme in the next few weeks and wants to avoid its defence of non-Sunni forces in Baghdad, Syria, Lebanon and elsewhere fuelling a sectarian regional war.

"For Iran always, national interests have priority over religious divides," said a senior official close to Supreme Leader Ayatollah Ali Khamenei. "Our main concern now is to safeguard the Iranian nation's interests."

While steadfast in condemnation of the Sunni militants who vow to massacre Shi'ites as heretics, Iran's leadership has been at pains to stress a desire for coexistence with other sects - now a key element of its criticism of Shi'ite premier Maliki.

"We have so far supported Maliki," the official said. "But as his failure to form an inclusive government has led to a chaos in Iraq, our support will be conditional and limited."

U.S. President Barack Obama has urged Maliki to embrace the Sunni minority that lost influence when U.S. troops overthrew Saddam Hussein - whom Washington had quietly backed against revolutionary Tehran in the long Iran-Iraq war of the 1980s.

Yet Tehran and Washington now seem unable to agree on how to achieve their shared aim of defeating the militants, pacifying moderate Sunnis and stabilising Iraq, even if each has signalled that coordination with the other would help to map out a plan.

The result is that Iran, as much as the United States, is groping for a combined political and military strategy to push back the dash across northern Iraq by the Islamic State of Iraq and the Levant (ISIL), three weeks after it began on June 10.

"Iran's goal in Iraq is to return to the status quo ante, which is a Shi'ite-dominated, ideologically like-minded government whom Iran can rely upon as a junior partner," said Karim Sadjadpour, senior associate at the Carnegie Endowment for International Peace think-tank in Washington.

"But the big question is: does such an individual exist who is palatable to both Iran, Sunni factions, the United States?"

That is not Iran's only dilemma.

Tehran ideally wants to block any expanded role for its U.S. adversary in Iraq; and yet, hungry for U.S. recognition as a regional political force, Tehran has signalled it may welcome a limited, quiet partnership with Washington in defending Baghdad.

Iran initially hoped that by helping Maliki with weapons and intelligence he would be able to contain the crisis. It urged him to form a more inclusive government to bring in Sunnis. But as the crisis has dragged on, doubts about him have grown and analysts and officials say Tehran is considering alternatives.

"We have a few names in mind and discussed our list with our allies in Iraq," said a senior Iranian security official who stressed that any new premier should be a friend of Tehran.

Complicating the picture are occasional public divergences between the positions taken by Iranian officials.

Khamenei, echoed by security hardliners, has accused the West and its Gulf Arab allies of supporting ISIL to overthrow the Baghdad government and restore Sunni political dominance.

Other officials, however, have suggested the crisis in Iraq presents an opportunity to defuse hostility with Washington that has been bad for both countries and has favoured their mutual enemies among Sunni Islamists in al Qaeda and other movements.

"We have common interests and common enemies," said another senior Iranian official, speaking of the United States.

"We face similar security threats in Iraq. Without our cooperation, stability cannot be restored in the Middle East."

President Hassan Rouhani has taken a more conciliatory tone toward the "Great Satan" since his election a year ago, though he remains subordinate to Khamenei in setting foreign policy.

He has suggested cooperating with the United States in Iraq, if Washington tackles "terrorism" in the region - a swipe in part at support from U.S. allies like Saudi Arabia and Qatar to Sunni Islamists fighting Iranian allies in Damascus and Baghdad.

"There is this odd dynamic whereby ... when it comes to Iraq, the U.S. and Iran are allies but not friends, and the U.S. and Saudi Arabia are friends but not allies," said Sadjadpour.

Washington has backed the revolt against Syria's Iranian-backed president Bashar al-Assad but has pulled its punches as Islamists have come to dominate rebel ranks. U.S. officials have been pushing Riyadh to use its influence among fellow Sunnis in Iraq to persuade them to join a coalition government in Baghdad.

There is precedent for U.S.-Iranian military cooperation. Tehran provided intelligence and political support when U.S. forces invaded Taliban-ruled Afghanistan in 2001 - only to hear President George W. Bush declare Iran to be part of an "axis of evil" with Saddam and North Korea just a few months later.

"Both the chaos in Afghanistan and in Iraq could have been avoided had Washington recognised the stabilising role Iran can play if it isn't treated as an outcast," Trita Parsi, president of the National Iranian American Council, wrote recently.

Yet for all the possible benefits to Iran of cooperation with the West, its rulers are also anxious not to be seen as an ally of the United States, and especially not of Washington's protege Israel, against Sunnis who outnumber Shi'ites two to one across the Middle East. More mixed messages can be expected.

"The removal of the Taliban in Afghanistan was in Iran's interests but Khamenei denounced it. The removal of Saddam Hussein was in Iran's interests but Khamenei denounced it," said Sadjadpour at the Carnegie Endowment in Washington.

"Now you can argue that the pushback against ISIL is in Iran's interests, but the leader will denounce it.

"Khamenei is in a challenging position because over three decades ... his philosophy has been to try to partner with Sunni Arabs against the U.S. and Israel.

"Now he is kind of being put in the opposite position, which is ... inimical to the revolutionary ideology that he has been espousing all these years."-Reuters