On the surface, times could hardly be better for the petroleum industry. Surging demand and a tight global supply have pushed oil prices to record heights, with no downturn in sight. The major oil companies, even while investing at record levels in renewed exploration and development efforts, still have enough cash left over to return huge sums to shareholders. And projects now in development will keep production growing for the next five to eight years.

Yet all is not as it seems: Big Oil confronts its most far-reaching test in decades. The top five companies—Exxon Mobil, BP, Royal Dutch/Shell, ChevronTexaco, and Total—face increasingly tough challenges finding new sources of oil and natural gas to replace existing reserves. Access to the Middle East, which holds half the world's known petroleum reserves, has been difficult—especially for oil—since the 1960s and 1970s, when governments there nationalized the assets of Western oil companies. Many of the world's remaining potential new sources of oil and natural gas are in countries with relatively high political and legal instability, such as Nigeria and Russia, or technically challenging regions such as the Arctic and Asia-Pacific. The complex refineries needed to process the world's vast reserves of heavy, sulphur-laden crude represent a large and risky new investment.

Moreover, competition for untapped energy deposits is fiercer than ever. China and India, anxious to secure sufficient fuel for their growing economies, are investing large sums in oil and natural-gas sources as far afield as Angola and Venezuela. Government-owned national oil companies, such as Malaysia's Petroliam Nasional Berhad (Petronas), are expanding internationally, while small to midsize companies such as Apache and BHP Billiton are flush with profits and scrambling to build their own reserves. In the past, major international oil companies—with their leading-edge technology, unrivaled expertise in managing complex projects, and, last but not least, deep pockets—had a clear edge in negotiations with the national governments in control of energy resources. But those advantages have become less pronounced as resource-rich countries have increased their technical and managerial capabilities, thus strengthening their position at the negotiating table.

All this might amount to no more than a footnote in the long history of the big international oil companies' preeminence, but as more players compete for new reserves, there is a growing risk that Big Oil could face a difficult choice between shrinking volumes and shrinking margins. To keep that from happening, the oil majors will have to take big gambles on hard-to-reach energy sources. More important, they must make tough decisions about how to become more attractive partners to the national governments that control the reserves. The key will be building capabilities that give these companies a clear advantage over their rivals. Four areas provide the biggest opportunities: developing unique technologies, providing a market for difficult-to-exploit energy sources, using diplomatic savoir faire to surmount political obstacles, and addressing the broad social needs of the resource-rich countries where the majors do business.

Big Oil's challenge

Historically, the industry's winners have been large oil companies, which have created at least twice as much shareholder value as midsize and independent ones. As a group, the five major companies represent more than 50 percent of the market capitalization of all publicly traded oil stocks, and during the past 20 years these corporations have generated returns 10 percent higher than the industry average. Even though the major oil companies account for just 15 percent of global energy production, they have the largest acreage and the most infrastructure across a majority of the significant hydrocarbon basins outside of the Middle East. And thanks to their lead in technology and their willingness to take on risk, they have captured the largest and most profitable energy deposits.

Today, however, many of the major companies' oil and natural-gas basins are aging and generate less than they did in their prime. That is particularly true in North America and northwestern Europe, which account for about 60 percent of the majors' current oil and natural-gas production and where more than 50 percent of the reserves have been extracted. In those areas, production costs continue to climb, and every new investment to extend the life of the reservoirs becomes more marginal, as fixed costs are covered by shrinking volumes. In the North Sea, for instance, the average extraction cost for a barrel of oil rose 42 percent from 2000 to 2005. The efforts of oil companies to replenish their reserves through traditional exploration have been increasingly unfruitful (Exhibit 1). Since the late 1990s, when several large finds in deep-water locations raised reserves, the average size of new discoveries around the world has declined threefold, to about 22 million barrels of oil. In 2004, for instance, Shell replaced just 15 to 25 percent of the oil and natural gas it pulled out of the ground, as measured by conservative accounting rules used by the US Securities and Exchange Commission.

Limited access

To maximize returns from investments in exploration and extraction, the big oil companies depend on their ability to secure an equity stake in a country's potential oil and natural-gas reserves, with no limit on the returns they can earn. In exchange, they invest significant amounts of capital, provide expertise, and pay tax on every unit of energy they extract. In recent years, however, many governments have curtailed the role of international oil companies and capped their returns, making these investments less attractive.

State-owned oil companies are expanding rapidly beyond their borders and becoming direct competitors to the oil majors

State-owned oil companies are expanding rapidly beyond their borders and becoming direct competitors to the oil majors (Exhibit 2). State-owned companies typically have less financial discipline than do corporations beholden to the stock market and so often settle for lower returns. Indeed, some have lost large sums of money: by the time the Japan National Oil Corporation was disbanded, in April 2005, it had destroyed an estimated $7 billion in shareholder value during its 40 years of operation. Nonetheless, the growing technical and commercial capabilities of these homegrown companies, which frequently use technology from oil industry service providers such as Halliburton and Schlumberger, provide governments with a viable alternative to working with Big Oil.

In addition, competition from smaller Western oil companies has increased. In the past decade or so, North American producers such as Apache and Talisman Energy have expanded, and they now extract significant amounts of oil from international sources (Exhibit 3). These midsize players have joined the big leagues and are capable of competing with rivals in a number of regions. The mature oil fields of countries such as Iran, Iraq, and Kuwait, where oil production can be handled with mainstream technology, represent an attractive opportunity for these companies.

Volatile prices

Most analysts expect oil prices to remain high over the next few years, partly because the world's excess production capacity has shrunk to three million to five million barrels of crude oil a day (compared with as much as ten million in the 1970s and 1980s). Yet experienced oil companies know that a multibillion-dollar investment calculated to break even at $35 a barrel can quickly become an expensive liability if prices fall to $25 or below. To be safe and to reassure shareholders, most companies are therefore making investments with the assumption that long-term prices will range from $25 to $30 a barrel. Such conservative estimates allowed the five majors to return $184 billion to their shareholders from 2000 to 2004, but this strategy also explains why they lose out to competitors willing to bet that higher prices are here to stay.

Of course, Big Oil has surmounted challenges in the past. BP, Exxon Mobil, and Shell all had to rebuild after the wave of nationalizations in the Middle East and South America. Indeed, BP's 2004 production of four million barrels of oil a day only now equals the level it achieved in 1975. A series of mergers and acquisitions has helped the major companies regain their global scale and scope in operations from exploration to retailing. Further marriages of industry giants are certainly possible. However, such mergers are unlikely to offer a long-term solution, since everyone faces roughly the same challenges in securing potential reserves. And while big companies will continue to buy small and midsize competitors when the price is right, in the long term that strategy likely won't be as successful as it was in the past, since there simply aren't enough takeover targets to sustain those companies with annual revenues of $200 billion or more.

Standing out from the crowd

The big oil companies must find new ways to distinguish themselves from the competition in order to continue thriving. Governments are likely to offer the most attractive terms to partners that possess rare and valuable skills. To avoid competing directly with rivals in auctions for acreage and production licenses, major oil companies must make tough choices instead of simply trying to keep pace with the pack. Four broad areas offer potential opportunities for a company to differentiate itself from its rivals.

Seize the technology high ground

Big Oil's technical leadership has been its main edge. Historically, the ability of the oil majors to develop and field-test customized technologies has allowed them to trump small companies and state-owned enterprises. Over the past 10 to 15 years, however, the majors have let this advantage slip by increasingly relying on technologies developed by oil service companies. Governments now question why they should buy technology from the majors when it can be acquired more cheaply from national oil companies, midsize Western players, and service companies.

In the future, companies that are able to exploit hard-to-reach energy reserves located outside the Middle East will be distinctive. Key areas to focus on include developing efficient extraction equipment that can withstand the stress of year-round ice and designing reliable technology to separate oil, natural gas, and water at the bottom of the sea (or, in the case of wells located far from land, equipment to carry material hundreds of miles to a shore-based separation facility). A company's corporate goals and its geographic portfolio will define its investments in technology over the long haul. In the 1980s, Shell bet on deep-water exploration in the Gulf of Mexico, for example, and became an early leader there by developing the necessary techniques and equipment for extraction.

Major oil companies need to increase their spending on research and development to levels well above the industry average—about 1 percent of revenues—in order to regain their edge. Industrial-equipment companies, for example, spend 4.5 percent of revenues on R&D. More important, the oil majors must manage their technology investments more effectively. A system that uses a series of stage gates to winnow out the duds from a portfolio would allow companies to start with an array of options and to make big investments in only the most promising opportunities. These techniques are widely used in venture capital firms and should be adopted by the oil giants.

They also need to forge closer partnerships with suppliers and small service companies, which have initiated many of the industry's big innovations over the past 20 years. Just as the drug industry has done, Big Oil needs to do more to share the risks and benefits of R&D with innovative companies—for instance, by offering them test beds for emerging technologies. In exchange, the oil company could negotiate exclusive rights to the new technology for a certain period of time.

Provide custom routes to market

In many cases, finding hydrocarbons and extracting them are not enough to create a profitable business. While the global market for light crude oil (which can be easily refined into gasoline and other products) is well developed, that isn't true for the natural gas stranded in the world's remote corners or for heavy, sulfur-laden oil requiring special equipment to extract and refine. Around the world, there are an estimated 700 billion barrels of recoverable heavy oil and 5,800 trillion cubic feet of stranded natural gas—enough to meet US demand for 96 and 260 years, respectively. Countries rich in these resources need a custom-designed route to market in order to generate attractive returns. The oil majors—thanks to their refineries, retail networks, and relationships with major customers in important markets—can offer unique solutions.

Exploiting new sources of natural gas will be one of the industry's big battlegrounds over the next decade. This relatively clean fuel is increasingly used to generate electricity in Europe and North America. The world's proven reserves are in places such as the Middle East and Siberia that are far from customers. Countries need guaranteed markets to justify the multibillion-dollar investments required to build long-distance pipelines or the plants that liquefy natural gas for shipment. The companies that control important facilities (such as regasification plants) or provide access to a large market are attractive partners for these projects. Another opportunity is to invest in the specialized coking¹ and hydrocracking² facilities needed to convert heavy oil (from countries such as Canada and Venezuela) into marketable products. Today there is insufficient capacity to exploit these vast reserves, and companies offering ready-made solutions will be the most attractive partners for governments.

To take advantage of these opportunities, oil companies will need to upgrade their refineries and invest in other forms of infrastructure before securing customers—a break from the past. Big Oil's scale and scope lower the risk of making such infrastructure bets (which can cost as much as $5 billion), since these investments represent a smaller proportion of overall capital spending than they do at smaller companies. And because the majors have a vast number of oil and natural-gas sources around the globe, the likelihood is greater that new facilities will be utilized quickly. Such a strategy carries real risks, but the company that moves first stands to gain handsomely.

Play a diplomatic role

In some cases, the technology and the market are in place, but politics remains a barrier, especially when an oil or natural-gas pipeline crosses national borders or when the project requires long-term commitments from the buyer and the seller. Eastern Siberia's gas reserves will remain undeveloped, for example, until China and Russia can agree on price and delivery terms for the estimated 30-year lives of these reservoirs. This type of cooperation requires a willingness on each side to honor the bargain.

Major oil companies can play a unique role in such projects as a diplomatic go-between and financial guarantor. Their substantial balance sheets allow them to be partners in huge infrastructure projects without undue risk. By taking an equity stake, oil companies can ensure that a deal goes through while they gain access to energy reserves. Their strong relationships with governments mean that they can also facilitate complex, cross-border deals. BP's role as a significant coinvestor in Azerbaijan, for instance, helped seal a pipeline deal with Georgia and Turkey that will allow oil to flow from the Caspian Sea to the Mediterranean in the near future.

In some places, there is growing distrust of Big Oil's political ties with Europe and North America

In some parts of the world, growing distrust of Big Oil's political ties with Europe and North America makes it all the more urgent for companies that want to succeed in the political arena to build small teams of specialists adept at negotiations and shuttle diplomacy. Ex-politicians and former senior civil servants with both the standing and the experience to deal with top government ministers are natural candidates. Chief executives must also become more involved in building long-term relationships with heads of state and other government officials. Whereas in the past CEOs typically showed up just to sign important deals, now they must play aninstrumental role in laying the groundwork.

Make a positive economic impact

Paradoxically, petroleum-rich countries often suffer as a result of their energy wealth. In many cases, the influx of foreign funds raises the value of the local currency. As a result, the country's other exports can't compete on the world market, so that whole sectors of the economy are decimated. Indeed, such problems prompted Juan Pablo Perez Alfonso, a founder of the Organization of Petroleum Exporting Countries (OPEC), to describe oil as "the devil's excrement." Nigeria is often cited as an example of what can go wrong: it remains one of the world's poorest countries, despite having earned some $300 billion from oil exports since the late 1950s.

International oil companies can capture an important competitive advantage and improve their chances of gaining access to important new reserves by demonstrating the broad positive economic impact their presence can have. At best, the record of oil companies in this respect is mixed, and clearly there are limits to what they can achieve. Nonetheless, a key initial step for companies is to develop and train local employees so that they can fill key roles traditionally held by expatriate managers. Companies need to help local employees break through the glass ceiling—a task that is harder than it sounds. Mentoring often fails because many expatriates are poorly equipped, both technically and culturally, to nurture their local colleagues. Companies must first train the mentors and then give them incentives to further the careers of local workers. Organizations must also cope with high turnover, since local employees with management experience at multinational corporations are often highly sought after in developing economies.

The oil majors could also do far more to enhance the positive economic and social impact of their investments. Too often, energy projects create pockets of wealth for a privileged few but fail to lift the economy. One potentially positive measure companies can take would be to help develop the technical and management know-how of local contractors and service companies. Most oil companies outsource low-skilled work such as cleaning and security to local organizations, in part to meet government-imposed hiring quotas. To broaden a country's economic base, oil companies should also nurture local companies that provide higher-skilled services, such as information technology and equipment maintenance. The Japanese carmakers Honda Motor and Toyota Motor assign engineers or other professionals (such as accountants) to local purchasing organizations; these specialists can spend months improving the capabilities of individual suppliers by working on the factory floor and in the front office. Similarly, oil companies can partner with governments and international organizations to improve standards of governance, thus reducing the risk that oil revenues will be wasted by public bodies. Anticorruption efforts—for example, the Extractive Industries Transparency Initiative—offer companies other ways to engage with governments.

In practice, none of these propositions is easy to act upon. Moreover, companies cannot single-handedly solve the deep-rooted problems of poverty and underdevelopment. During the course of a 30- to 50-year relationship between an oil company and a country, however, even partial successes can have a significant impact. Companies that are astute enough to navigate these complex and politically sensitive issues are likely to have an edge over their rivals.

Major international oil companies face an era of unprecedented change and new threats to their long-term viability. To avoid becoming irrelevant, they must expand their capabilities and increase their distinctiveness. Doing so will require bold strategic choices and new investments at a time when profits and the risk of complacency are high. Future winners will become true front-runners in the eyes of their customers: the governments controlling access to the world's hydrocarbons.