A bigger problem than a software update.
On Thursday, Boeing for the first time officially took responsibility for the two crashes of 737 Max jets that got the planes grounded by regulators.
Claiming responsibility was part of an attempt to get the planes approved to fly again. Boeing was trying to say that it now understands why the planes crashes — flawed software — and has a plan in place to replace it with new software that will eliminate the problem and persuade regulators to get the planes off the ground. But then Friday morning, the company announced that it had found a second, unrelated software flaw that it also needs to fix and will somewhat delay the process of getting the planes cleared to fly again.
All of which, of course, raises the question of why such flawed systems were allowed to fly in the first place.
And that story begins nine years ago when Boeing was faced with a major threat to its bottom line, spurring the airline to rush a series of kludges through the certification process — with an underresourced Federal Aviation Administration (FAA) seemingly all too eager to help an American company threatened by a foreign competitor, rather than to ask tough questions about the project.
The specifics of what happened in the regulatory system are still emerging (and despite executives’ assurances, we don’t even really know what happened on the flights yet). But the big picture is coming into view: A major employer faced a major financial threat, and short-term politics and greed won out over the integrity of the regulatory system. It’s a scandal.
The 737 versus 320 rivalry, explained
There are lots of different passenger airplanes on the market, but just two very similar narrow-body planes dominate domestic (or intra-European) travel. One is the European company Airbus’s 320 family, with models called A318, A319, A320, or A321 depending on how long the plane is. These four variants, by design, have identical flight decks, so pilots can be trained to fly them interchangeably.
The 320 family competes with a group of planes that Boeing calls the 737 — there’s a 737-600, a 737-700, a 737-800, and a 737-900 — with higher numbers indicating larger planes. Some of them are also extended-range models that have an ER appended to the name, and, as you would probably guess, they have longer ranges.
Importantly, even though there are many different flavors of 737, they are all in some sense the same plane, just as all the 320 family planes are the same plane. Southwest Airlines, for example, simplifies its overall operations by exclusively flying different 737 variants.
Both the 737 and the 320 come in lots of different flavors, so airlines have plenty of options in terms of what kind of aircraft should fly exactly which route. But because there are only two players in this market, and because their offerings are so fundamentally similar, the competition for this slice of the plane market is both intense and weirdly limited. If one company were to gain a clear technical advantage over the other, it would be a minor catastrophe for the losing company.
And that’s what Boeing thought it was facing.
The A320neo was trouble for Boeing
Jet fuel is a major cost for airlines. With labor costs largely driven by collective bargaining agreements and regulations that require minimum ratios of flight attendants per passenger, fuel is the cost center airlines have the most capacity to do something about. Consequently, improving fuel efficiency has emerged as one of the major bases of competition between airline manufacturers.
If you roll back to 2010, it began to look like Boeing had a real problem in this regard.
Airbus was coming out with an updated version of the A320 family that it called the A320neo, with “neo” meaning “new engine option.” The new engines were going to be more fuel-efficient, with a larger diameter than previous A320 engines, that could nonetheless be mounted on what was basically the same airframe. This was a nontrivial engineering undertaking both in designing the new engines and in figuring out how to make them work with the old airframe, but even though it cost a bunch of money, it basically worked. And it raised the question of whether Boeing would respond.
Initial word was that it wouldn’t. As CBS Moneywatch’s Brett Snyder wrote in December 2010, the basic problem was that you couldn’tslap the new generation of more efficient, larger-diameter engines onto the 737:
One of the issues for Boeing is that it takes more work to put new engines on the 737 than on the A320. The 737 is lower to the ground than the A320, and the new engines have a larger diameter. So while both manufacturers would have to do work, the Boeing guys would have more work to do to jack the airplane up. That will cost more while reducing commonality with the current fleet. As we know from last week, reduced commonality means higher costs for the airlines as well.
Under the circumstances, Boeing’s best option was to just take the hit for a few years and accept that it was going to have to start selling 737s at a discount price while it designed a whole new airplane. That would, of course, be time-consuming and expensive, and during the interim, it would probably lose a bunch of narrow-body sales to Airbus.
The original version of the 737 first flew in 1967, and a decades-old decision about how much height to leave between the wing and the runway left them boxed out of 21st-century engine technology — and there was simply nothing to be done about it.
Unless there was.
Boeing decided to put on the too-big engines anyway
As late as February 2011, Boeing chair and CEO James McNerney was sticking to the plan to design a totally new aircraft.
“We’re not done evaluating this whole situation yet,” he said on an analyst call, “but our current bias is to move to a newer airplane, an all-new airplane, at the end of the decade, beginning of the next decade. It’s our judgment that our customers will wait for us.”
But in August 2011, Boeing announced that it had lined up orders for 496 re-engined Boeing 737 aircraft from five airlines.
It’s not entirely clear what happened, but, reading between the lines, it seems that in talking to its customers Boeing reached the conclusion that airlines would not wait for them. Some critical mass of carriers (American Airlines seems to have been particularly influential) was credible enough in its threat to switch to Airbus equipment that Boeing decided it needed to offer 737 buyers a Boeing solution sooner rather than later.
Committing to putting a new engine that didn’t fit on the plane was the corporate version of the Fyre Festival’s “let’s just do it and be legends, man” moment, and it unsurprisingly wound up leading to a slew of engineering and regulatory problems.
New engines on an old plane
As the industry trade publication Leeham News and Analysis explained earlier in March, Boeing engineers had been working on the concept that became the 737 Max even back when the company’s plan was still not to build it.
In a March 2011 interview with Aircraft Technology, Mike Bair, then the head of 737 product development, said that reengineering was possible.
“There’s been fairly extensive engineering work on it,” he said. “We figured out a way to get a big enough engine under the wing.”
The problem is that an airplane is a big, complicated network of interconnected parts. To get the engine under the 737 wing, engineers had to mount the engine nacelle higher and more forward on the plane. But moving the engine nacelle (and a related change to the nose of the plane) changed the aerodynamics of the plane, such that the plane did not handle properly at a high angle of attack. That, in turn, led to the creation of the Maneuvering Characteristics Augmentation System (MCAS). It fixed the angle-of-attack problem in most situations, but it created new problems in other situations when it made it difficult for pilots to directly control the plane without being overridden by the MCAS.
On Wednesday, Boeing rolled out a software patch that it says corrects the problem, and it hopes to persuade the FAA to agree.
But note that the underlying problem isn’t really software; it’s with the effort to use software to get around a whole host of other problems.
1of x: BEST analysis of what really is happening on the #Boeing737Max issue from my brother in law @davekammeyer, who’s a pilot, software engineer & deep thinker. Bottom line don’t blame software that’s the band aid for many other engineering and economic forces in effect.
— Trevor Sumner (@trevorsumner) March 16, 2019
Recall, after all, that the whole point of the 737 Max project was to be able to say that the new plane was the same as the old plane. From an engineering perspective, the preferred solution was to actually build a new plane. But for business reasons, Boeing didn’t want a “new plane” that would require a lengthy certification process and extensive (and expensive) new pilot training for its customers. The demand was for a plane that was simultaneously new and not new.
But because the new engines wouldn’t fit under the old wings, the new plane wound up having different aerodynamic properties than the old plane. And because the aerodynamics were different, the flight control systems were also different. But treating the whole thing as a fundamentally different plane would have undermined the whole point. So the FAA and Boeing agreed to sort of fudge it.
The new planes are pretty different
As far as we can tell, the 737 Max is a perfectly airworthy plane in the sense that error-free piloting allows it to be operated safely.
But pilots of planes that didn’t crash kept noticing the same basic pattern of behavior that is suspected to have been behind the two crashes, according to a Dallas Morning News review of voluntary aircraft incident reports to a NASA database:
The disclosures found by the News reference problems with an autopilot system, and they all occurred during the ascent after takeoff. Many mentioned the plane suddenly nosing down. While records show these flights occurred in October and November, the airlines the pilots were flying for is redacted from the database.
These pilots all safely disabled the MCAS and kept their planes in the air. But one of the pilots reported to the database that it was “unconscionable that a manufacturer, the FAA, and the airlines would have pilots flying an airplane without adequately training, or even providing available resources and sufficient documentation to understand the highly complex systems that differentiate this aircraft from prior models.”
The training piece is important because a key selling feature of the 737 Max was the idea that since it wasn’t really a new plane, pilots didn’t really need to be retrained for the new equipment. As the New York Times reported, “For many new airplane models, pilots train for hours on giant, multimillion-dollar machines, on-the-ground versions of cockpits that mimic the flying experience and teach them new features” while the experienced 737 Max pilots were allowed light refresher courses that you could do on an iPad.
That let Boeing get the planes into customers’ hands quickly and cheaply, but evidently at the cost of increasing the possibility of pilots not really knowing how to handle the planes, with dire consequences for everyone involved.
The FAA put a lot of faith in Boeing
In a blockbuster March 17 report for the Seattle Times, the newspaper’s aerospace reporter Dominic Gates details the extent to which the FAA delegated crucial evaluations of the 737’s safety to Boeing itself. The delegation, Gates explains, is in part a story of a years-long process during which the FAA, “citing lack of funding and resources, has over the years delegated increasing authority to Boeing to take on more of the work of certifying the safety of its own airplanes.”
But there are indications of failures that were specific to the 737 Max timeline. In particular, Gates reports that “as certification proceeded, managers prodded them to speed the process” and that “when time was too short for FAA technical staff to complete a review, sometimes managers either signed off on the documents themselves or delegated their review back to Boeing.”
Most of all, decisions about what could and could not be delegated were being made by managers concerned about the timeline, rather than by the agency’s technical experts.
It’s not entirely clear at this point why the FAA was so determined to get the 737 cleared quickly (there will be more investigations), but if you recall the political circumstances of this period in Barack Obama’s presidency, you can quickly get a general sense of the issue.
Boeing is not just a big company with a significant lobbying presence in Washington; it’s a major manufacturing company with a strong global export presence and a source of many good-paying union jobs. In short, it was exactly the kind of company the powers that be were eager to promote — with the Obama White House, for example, proudly going to bat for the Export-Import Bank as a key way to sustain America’s aerospace industry.
A story about overweening regulators delaying an iconic American company’s product launch and costing good jobs compared to the European competition would have looked very bad. And the fact that the whole purpose of the plane was to be more fuel-efficient only made getting it off the ground a bigger priority. But the incentives really were reasonably aligned, and Boeing has only caused problems for itself by cutting corners.
Boeing is now in a bad situation
One emblem of the whole situation is that as the 737 Max engineering team piled kludge on top of kludge, they came up with a cockpit warning light that would alert the pilots if the plane’s two angle-of-attack sensors disagreed.
But then, as Jon Ostrower reported for the Air Current, Boeing’s team decided to make the warning light an optional add-on, like how car companies will upcharge you for a moon roof.
The light cost $80,000 extra per plane and neither Lion Air nor Ethiopian chose to buy it, perhaps figuring that Boeing would not sell a plane (nor would the FAA allow it to) that was not basically safe to fly. In the wake of the crashes, Boeing has decided to revisit this decision and make the light standard on all aircraft.
Now, to be clear, Boeing has lost about $40 billion in stock market valuation since the crash, so it’s not like cheating out on the warning light turned out to have been a brilliant business decision or anything.
This, fundamentally, is one reason the FAA has become comfortable working so closely with Boeing on safety regulations: The nature of the airline industry is such that there’s no real money to be made selling airplanes that have a poor safety track record. One could even imagine sketching out a utopian libertarian argument to the effect that there’s no real need for a government role in certifying new airplanes at all, precisely because there’s no reason to think it’s profitable to make unsafe ones.
The real world, of course, is quite a bit different from that, and different individuals and institutions face particular pressures that can lead them to take actions that don’t collectively make sense. Looking back, Boeing probably wishes it had just stuck with the “build a new plane” plan and toughed out a few years of rough sales, rather than ending up in the current situation. Right now the company is, in effect, trying to patch things up piecemeal — a software update here, a new warning light there, etc. — in hopes of persuading global regulatory agencies to let its planes fly again.
But even once that’s done, Boeing faces the task of convincing airlines to actually buy its planes. An informative David Ljunggren article for Reuters reminds us that a somewhat comparable situation arose in 1965 when three then-new Boeing 727 jetliners crashed.
There wasn’t really anything unsound about the 727 planes, but many pilots didn’t fully understand how to operate the new flaps — arguably a parallel to the MCAS situation with the 737 Max — which spurred some additional training and changes to the operation manual. Passengers avoided the planes for months, but eventually came back as there were no more crashes, and the 727 went on to fly safely for decades. Boeing hopes to have a similar happy ending to this saga, but so far it seems to be a long way from that point. And the immediate future likely involves more tough questions.
A political scandal on slow burn
The 737 Max was briefly a topic of political controversy in the United States as foreign regulators grounded the planes, but President Donald Trump — after speaking personally to Boeing’s CEO — declined to follow. Many members of Congress (from both parties) called on him to reconsider, which he rather quickly did, pushing the whole topic off Washington’s front burner.
But Trump is generally friendly to Boeing (he even has a former Boeing executive, Patrick Shanahan, serving as acting defense secretary, despite an ongoing ethics inquiry into charges that Shanahan unfairly favors his former employer), and Republicans are generally averse to harsh regulatory crackdowns. The most important decisions in the mix appear to have been made back during the Obama administration, so it’s also difficult for Democrats to go after this issue. Meanwhile, Washington has been embroiled in wrangling over special counsel Robert Mueller’s investigation, and a new health care battlefield opened up as well.
That said, on March 27, FAA officials faced the Senate Commerce Committee’s Subcommittee on Aviation and Space at a hearing called by subcommittee Chair Ted Cruz (R-TX). Regulators committed at the hearing to revamp the way they certify new planes, in light of the flaws that were revealed in the previous certification process.
The questions at stake, however, are now much bigger than one subcommittee. Billions of dollars are on the line for Boeing, the airlines that fly 737s, and the workers who build the planes. And since a central element of this story is the credibility of the FAA’s process — in the eyes of the American people and of foreign regulatory agencies — it almost certainly won’t get sorted out without more involvement from the actual decision-makers in the US government.