A U.S.-Iran alliance on Iraq is a tricky issue

Updated at 6:35 p.m.

President Obama has gotten heat from U.S. lawmakers for engaging Iran in talks to curb its nuclear program, but those conversations may have laid the groundwork for cooperation between two longtime nemeses on dealing with the violence in Iraq.

Neither the U.S. nor Iran wants to see the Islamic State of Iraq and Syria (ISIS) continue to gain a foothold in the Middle East. For the U.S., the violent ISIS advance has wiped out nearly all the gains made during the Iraq war. Iran's Shiite Muslim leaders don't want to see Sunni Muslims take over their neighboring state, especially if it means the ouster of Iraq's Shiite Prime Minister, Nouri al-Maliki.

Secretary of State John Kerry said in an interview with Yahoo News Monday that the U.S. might be open to what he described as, "any constructive process that could minimize the violence, hold Iraq together, the integrity of the country, and eliminate the presence of outside terrorist forces that are ripping it apart." As for what kind of process that might entail, Kerry said he "wouldn't rule out anything."

The State Department has since clarified that "anything" means no coordinated military intervention, but rather any discussion that can help strengthen the Iraqi government.

"We believe the focus should be on encouraging Iraqi leaders to govern in a non-sectarian way, and our discussion wouldn't be about cooperating or coordinating on military goals," State Department spokeswoman Jen Psaki told reporters.

The nuclear talks offer a convenient background for these conversations to take place: Deputy Secretary of State William Burns and Iranian Foreign Minister Zarif are both in Vienna this week for the P5+1 nuclear talks. A senior State Department official said that the Iraq issue came up on the margins of the meeting Monday, separate from the nuclear talks.

"These engagements will not include military coordination or strategic determinations about Iraq's future over the heads of the Iraqi people. We will discuss how [ISIS] threatens many countries in the region, including Iran, and the need to support inclusivity in Iraq and refrain from pressing a sectarian agenda," the official said.

Experts on the region say the ongoing nuclear talks make it more likely the U.S. and Iran could have a dialogue on Iraq, but warn that American officials must be exceedingly wary of complexities of engagement on this issue.

"The fact that we've been talking to the Iranians and there are some American officials who have decent rapport with their Iranian counterparts and vice versa is helpful. The fact that Iran has more options than the United States is less helpful from a U.S. perspective because you run the risk of needing the Iranians more than they need you," Jon Alterman, the director of the Middle East program at the Center for Strategic and International Studies and a former state department official told CBS News. "You can work with the Iranians but the Iranians are pretty shrewd analysts of their own interests...the Iranians see themselves as being vastly overmatched by the United States and any shred of advantage they're can salvage, they're going to salvage."

Michael Singh, the managing director of the Washington Institute for Near East Policy and a former senior director for Middle East affairs at the National Security Council in the George W. Bush White House, agreed that the nuclear talks have provided an opening for discussions on Iraq, but said there are a host of potential problems that come with expanding those talks to include regional issues.

"To the extent that you have sectarian tensions at the root of what's happening inside Iraq I think the prospect of Iranian involvement, which will be viewed with great suspicion...is going to inflame those tensions, not calm them," Singh told CBS News. "I think that it also has the prospect of polarizing the conflict through the reactivation of the Shia militia organizations."

Singh also noted that Iran's active engagement in Syria didn't succeed in stopping the rise of ISIS there, so the U.S., "shouldn't overestimate the capabilities of the Revolutionary Guard to actually be helpful."

"Iran has shown itself to be good at stoking insurgencies but hasn't shown any particular ability in combating insurgencies," he said.

The U.S. and Iran have cooperated on regional issues in the past. Iran was supportive of the U.S. invasion of Afghanistan and U.S. diplomats met with their Iranian counterparts to discuss cooperation during the early years of the war.

But even experts who say that engagement with Iran is possible in this situation are still highly skeptical it would benefit U.S. interests.

"We do share an interest with Iran in a semi-stable Iraq. That said, Iran showed little interest in that goal during the 2003 through 2008 period. And now, its means to that end are likely to be very different and perhaps at cross-purposes with our own, since Tehran presumably wishes to empower only Shia, whereas for us, taking that approach would likely make the situation and problem worse," Michael O'Hanlon, a foreign policy expert at the Brookings Institution, told CBS News.

Alterman called the opening for dialogue a "godsend" from the Iranian perspective because it would allow them to frame the civil war in Syria - where they have been supportive of President Bashar al-Assad - as a battle against Sunni extremists, make al-Maliki even more dependent on Iran for military and intelligence support, and present themselves to the world as a force for peace.

Alumni of both the Bush and Obama White Houses have expressed wariness of getting involved with Iran. CBS News National Security Analyst Juan Zarate, who was a counterterrorism advisor to President Bush said the U.S. does not want to stoke a battle between Shia and Sunni Muslims that could extend well beyond Iraq. And Michael Morell, another CBS News contributor and the former deputy director of the CIA during President Obama's tenure, said it was not in the interest of the United States to work with Iran.

"There is a long-term struggle going on in the Middle East between Iran and our allies, Saudi Arabia, the United Arab Emirates and the moderate gulf states, a fight going on for influence, and I don't think we want to give Iran a foothold in Iraq," Morell said. "We need to help the Iraqis, the moderate Gulf States need to help the Iraqis, [and] we need to keep the Iranians out of this."

It would also certainly draw criticism from members of Congress who already oppose the nuclear talks with Iraq. Sen. John McCain, R-Ariz., said in a statement that it would be the "height of folly" to count on Iran as a partner.

"The reality is, U.S. and Iranian interests and goals do not align in Iraq, and greater Iranian intervention would only make the situation dramatically worse. It would inflame sectarian tensions, strengthen the Islamic State of Iraq and Syria, drive more Sunnis into ISIS's ranks, empower the most radical Shia militants, deepen the Iraqi government's dependence on Iran, alienate U.S. allies and partners in the region, and set back the prospects of national reconciliation," McCain said. Instead, he advocated that the U.S. work to minimize greater involvement by taking rapid action to halt ISIS' advance.

But McCain's close Senate ally, Sen. Lindsey Graham, R-S.C., said on "Face the Nation" Sunday that some dialogue with the Iranians was necessary.

"We need a dialogue of some kind with the Iranians, but we also need to put them on notice: don't use this crisis as a way to create a satellite state of Iraq controlled by Iran," he said.