By Martin Indyk, Wall Street Journal: Last week, despite Donald Trump’s repeated pledge to end American involvement in the Middle East’s conflicts, the U.S. was on the brink of another war in the region, this time with Iran. If Iran’s retaliation for the Trump administration’s targeted killing of Tehran’s top commander, Maj. Gen. Qassem Soleimani, had resulted in the deaths of more Americans, Washington was, as Mr. Trump tweeted, “locked and loaded” for all-out confrontation.

Why does the Middle East always seem to suck the U.S. back in? What is it about this troubled region that leaves Washington perpetually caught between the desire to end U.S. military involvement there and the impulse to embark on yet another Middle East war?

As someone who has devoted four decades of his life to the study and practice of U.S. diplomacy in the Middle East, I have been struck by America’s inability over the past two administrations to resolve this dilemma. Previously, presidents of both parties shared a broad understanding of U.S. interests in the region, including a consensus that those interests were vital to the country—worth putting American lives and resources on the line to forge peace and, when necessary, wage war.

Today, however, with U.S. troops still in harm’s way in Iraq and Afghanistan and tensions high over Iran, Americans remain war-weary. Yet we seem incapable of mustering a consensus or pursuing a consistent policy in the Middle East. And there’s a good reason for that, one that’s been hard for many in the American foreign-policy establishment, including me, to accept: Few vital interests of the U.S. continue to be at stake in the Middle East. The challenge now, both politically and diplomatically, is to draw the necessary conclusions from that stark fact.

Mr. Trump, like Barack Obama before him, is discovering just how difficult it is to make this adjustment. Four months ago, he declared that it wasn’t America’s responsibility to defend Saudi Arabia’s oil facilities in the wake of an Iranian attack that reduced Saudi oil production by more than 50% and cut world crude-oil production by 5%. “That was an attack on Saudi Arabia, and that wasn’t an attack on us,” the president said—challenging an imperative that has underpinned U.S. policy since 1945, when President Franklin Roosevelt made a pact with King Abdul Aziz al-Saud to protect the kingdom’s oil.

Yet Mr. Trump subsequently sent some 14,000 more U.S. troops to the Gulf, along with an aircraft carrier strike group that the Pentagon would have vastly preferred to deploy to the South China Sea to deal with the more important 21st-century threat of a rising China.

The difficulty of getting out was also manifest last October, when Secretary of Defense Mark Esper announced that Mr. Trump had ordered the withdrawal of all 1,000 U.S. troops from northern Syria—provoking a howl of bipartisan criticism from Capitol Hill (and consternation in Israel) for Mr. Trump’s abandonment of America’s faithful Syrian-Kurdish allies. Yet American troops remain in Syria today, their mission now recast by Mr. Trump as ostensibly defending Syrian oil fields. The tension between our objectives in Syria and the means we are prepared to commit to achieve them remains unresolved; Secretary of State Mike Pompeo has declared that America’s purpose there is to “expel every last Iranian boot.”

Some would attribute all of this toing and froing to simple incoherence under Mr. Trump, or to the tension between the president’s instinct to disengage from the Middle East and the hard-line impulses of his closest advisers, including Mr. Pompeo, Sen. Lindsey Graham and John Bolton, Mr. Trump’s former national security adviser.

Obama was also determined to end America’s ‘forever wars’ in the Middle East and to avoid new ones.

But similar contradictions could be seen under Mr. Trump’s predecessor. Mr. Obama was also determined to end America’s “forever wars” in the Middle East and to avoid new ones. Yet during the pro-democracy uprisings of the Arab Spring in 2011, he couldn’t resist calling for the overthrow of the regimes in Egypt, Libya and Syria, even though he had scant desire to commit America’s resources to toppling them.

To fulfill his popular campaign promise to end America’s war of choice in Iraq, Mr. Obama withdrew all U.S. forces from the country in 2011. Just three years later, he sent some 5,000 troops back after the jihadists of Islamic State exploited the vacuum to seize swaths of Iraqi territory for its self-styled “caliphate.”

In Syria, Mr. Obama declared in 2012 that the use of chemical weapons by the bloodstained regime of Bashar al-Assad would cross “a red line for us.” But when Mr. Assad used sarin gas to kill more than 1,400 Syrian civilians in a Damascus suburb in August 2013, Mr. Obama balked at planned U.S. strikes. Without congressional backing, he chose not to retaliate and risk embroiling the U.S. in the Syrian civil war. He suffered withering criticism for failing to defend the Syrian people from the Assad regime’s atrocities.

Behind all of this vacillating lies a 21st-century reality: There has been a structural shift in American interests in the Middle East, one that Washington is having a hard time acknowledging.

In the past, the U.S. has had two clear priorities in the Middle East: to keep Gulf oil flowing at reasonable prices and to ensure Israel’s survival. But the U.S. economy no longer relies on imported petroleum. Fracking has turned the U.S. into a net oil and natural-gas exporter. The countries that still depend on the oil flowing from the Gulf are in Europe and Asia.

To be sure, the global economy—and therefore the American economy—would be hurt by a major disruption in oil supplies from the Gulf. But the natural-gas revolution in the U.S., the discovery and development of energy sources elsewhere and the growing substitution of “clean energy” have made markets surprisingly resilient in the face of chaos in the Middle East. The Arab countries that export oil and natural gas are still important to us, but the free flow of their oil is no longer a vital interest—that is, one worth fighting for. Difficult as it might be to get our heads around the idea, China and India need to be protecting the sea lanes between the Gulf and their ports, not the U.S. Navy.

As for Israel, it is still very much in America’s national interest to support the security of the Jewish state, but its survival is no longer in question. Decades of American economic and military largess and close security cooperation have made it possible for Israel to defend itself by itself. We are right to be concerned by Iran’s repeated threats to destroy Israel, but it is today’s nuclear-armed Israel that has the means to crush Iran, not the other way around.

Similarly, in decades past, reconciling Israel with its Arab neighbors was vitally important to regional stability. During the 1973 Yom Kippur War, for example, the Arab oil producers’ embargo quadrupled the price of oil, plunging the U.S. economy into a deep recession. But assiduous U.S. diplomacy, initiated by Secretary of State Henry Kissinger and concluded by President Jimmy Carter, produced a peace treaty that removed Egypt—the most populous and militarily powerful Arab state—from any potential Arab war coalition against Israel. That made it impossible for Israel’s remaining, weaker Arab neighbors to contemplate a return to war.

In 1994, President Bill Clinton brokered a second peace treaty, this time between Israel and Jordan, that helped to secure the Hashemite kingdom and stabilize the Middle East heartland. The more recent disintegration of Iraq and Syria—once led by ferocious foes of Israel—reinforced the impossibility of another wide-scale Arab-Israeli war.

Today, Israel enjoys stronger strategic relations with the leading Sunni Arab states—Saudi Arabia, Egypt and others—than they maintain with one another. And this is despite the lack of any progress in forging peace between the Israelis and the Palestinians.

Arab-Israeli peacemaking has captivated me for my entire professional life. Yet it has been more than 20 years since the last U.S.-brokered Israeli-Palestinian agreement was signed (the Wye River Memorandum of October 1998), and the task is now clearly hopeless.

I know this from heartbreaking personal experience. Six years ago, as Mr. Obama’s peace envoy, I participated in the last direct Israeli-Palestinian negotiations. At the end of that nine-month encounter, the two sides were farther apart on all the core issues than when we started. Nothing since has changed that reality. Mr. Trump’s advisers are now hinting that they will soon release his long-touted plan for a “deal of the century” to end the conflict. Because it is likely to tilt toward Israel on crucial issues, the Palestinians will surely reject it.

A two-state solution to the Palestinian problem is a vital Israeli interest, not a vital American one.

Hard as it is for me to admit it, a two-state solution to the Palestinian problem is not a vital American interest. It is a vital /Israeli /interest if the country wants to survive as a Jewish and democratic state. Because the U.S. is Israel’s friend, we should encourage it to hold open that possibility, down the road, by avoiding West Bank settlement construction or annexation that would make territorial compromise with the Palestinians impossible. But it’s time to end the farce of putting forward American peace plans only to have one or both sides reject them.

If oil and Arab-Israeli peace are no longer vital interests, what about stopping Islamic State? After all, what starts in the Middle East doesn’t stay in the Middle East, as the attacks of 9/11 so agonizingly demonstrated. But since the destruction of Islamic State’s territorial caliphate, the challenge is to deal with the remnants of both that group and al Qaeda. This mopping up operation can be achieved by small numbers of U.S. troops, combined with close cooperation and support for local partners, including the Kurds, Iraq and our associates in the anti-Islamic State coalition.

That leaves Iran’s nuclear program. Preventing a nuclear arms race in the Middle East does remain a vital U.S. interest—the one current case where the U.S. might need to resort to war. But we should be wary of those who would rush to battle stations.

Unlike North Korea or Pakistan, Iran doesn’t have nuclear weapons. U.S. sanctions are choking Iran’s economy, and the regime faces growing internal dissent and regional opposition. Mr. Trump unwisely pulled out of the 2015 Iran nuclear deal, but Iran’s leaders have already expressed a willingness to return to the negotiating table and clearly want to avoid an escalating conflict.

Curbing Iran’s nuclear aspirations and ambitions for regional dominance will require assiduous American diplomacy, not war. Sanctions have given Mr. Trump considerable leverage. He should now signal to Tehran that he is willing to ease sanctions if it reverses its recent violation of its commitments under the nuclear pact. We should also start courting our European allies, rather than disparaging them, and coordinate with Russia and China, who share our objective of stopping Iran from acquiring the bomb.

Diplomatic backing for efforts to end the Yemen war, solidify the de facto truce in Gaza and eventually reconstruct Syria can help to reduce Iran’s ability to meddle in regional conflicts. We should be joining with Israel and the Saudi-led Sunni Arabs in these endeavors rather than just backing their right to defend themselves, which only seems to benefit Iran.

In 1975, the U.S. turned its back on Southeast Asia after the debilitating war in Vietnam. At the time, Mr. Kissinger remarked that America, in its foreign involvements, oscillates between exuberance and exhaustion, between crusading impulses and retreats into self-doubt.

We cannot afford to turn our backs on the Middle East—the cradle of Judaism, Christianity and Islam, the locus of vast oil reserves and the focus of a continuing “great game” of rivalry between aspiring and established powers. The Middle East will continue to capture the imagination of Americans, with our great power, our unique mix of innocence and arrogance, our belief that every problem has a solution and our seemingly insatiable desire to make the region over in our own image.

Yet after the sacrifice of so many American lives, the waste of so much energy and money in quixotic efforts that ended up doing more harm than good, it is time for the U.S. to find a way to escape the costly, demoralizing cycle of crusades and retreats. We need a sustainable Middle East strategy based on a more realistic assessment of our interests. It is time to eschew never-ending wars and grandiose objectives—like pushing Iran out of Syria, overthrowing Iran’s ayatollahs or resolving the Israeli-Palestinian conflict—in favor of more limited goals that can be achieved with more modest means.

First published in Wall Street Journal. Cartoon by Ben Garrison.

Martin Sean Indyk is a diplomat and foreign relations analyst with expertise in the Middle East. In addition to serving in academia and as adviser to political administrations, he was a distinguished fellow in International Diplomacy and later executive vice president at the Brookings Institution in Washington, D.C from 2001-2018. He took leave from the Brookings Institution to serve as the U.S. Special Envoy for Israeli–Palestinian Negotiations from 2013 to 2014. He is currently a distinguished fellow at the Council on Foreign Relations.