Boeing Crapification: 737 MAX Play-by-Play, Regulatory Capture, and When Will CEO Muilenburg Become the Sacrificial Victim?

By Lambert Strether of Corrente.

At some point in the future, I’d like to do failure matrix for the pathways to misfortune (example of such a matrix here) that precipitated two deadly Boeing 737 MAX crashes on take-off in five months, but I don’t feel that I have enough information yet. (I’m not unsympathathetic to the view that the wholesale 737 MAX grounding was premature on technical grounds, but then trade and even geopolitical factors enter in, given that Boeing is a “national champion.”) We do not yet have results from the cockpit voice and flight data recorders of either aircraft, for example. But what we do know is sufficiently disturbing — a criminal investigation into Boeing had already been initiated after the Lion Air crash, but before the Ethiopian Airlines crash — that I think it’s worthwhile doing a play-by-play on the causes of the crashes, so far as we can know them. About that criminal investigation:

According to the Wall Street Journal, a Washington D.C. grand jury issued a March 11 subpoena requesting emails, correspondence, and other messages from at least one person involved in the development of the aircraft.

“It’s a very, very serious investigation into basically, was there fraud by Boeing in the certification of the 737 MAX 8?” Arthur Rosenberg, an aviation attorney who is representing six families whose relatives died in the Ethiopian Airlines and Lion Air crashes, explained.

“Nobody knows the answer to that yet,” Rosenberg cautioned, adding that he had not yet seen the Justice Department’s subpoena and therefore could not know its full scope.

Rosenberg expects the criminal probe to question whether Boeing fully disclosed to the FAA the engineering of the 737 Max 8’s MCAS flight control system, called MCAS (Maneuvering Characteristics Augmentation System), during the plane’s certification process. The flight control system was designed to prevent the plane from stalling.

Bloomberg comments:

A possible criminal investigation during an aircraft accident investigation is highly unusual. While airline accidents have at times raised criminal issues, such as after the 1996 crash of a ValuJet plane in the Florida Everglades, such cases are the exception.

Before we get to the play-by-play, one more piece of background: CEO Dennis Muilenburg’s latest PR debacle, entitled “Letter from Boeing CEO Dennis Muilenburg to Airlines, Passengers and the Aviation Community.” The most salient material:

Safety is at the core of who we are at Boeing, and ensuring safe and reliable travel on our airplanes is an enduring value and our absolute commitment to everyone. This overarching focus on safety spans and binds together our entire global aerospace industry and communities. We’re united with our airline customers, international regulators and government authorities in our efforts to support the most recent investigation, understand the facts of what happened and help prevent future tragedies. Based on facts from the Lion Air Flight 610 accident and emerging data as it becomes available from the Ethiopian Airlines Flight 302 accident, we’re taking actions to fully ensure the safety of the 737 MAX. We also understand and regret the challenges for our customers and the flying public caused by the fleet’s grounding.

Boeing has been in the business of aviation safety for more than 100 years, and we’ll continue providing the best products, training and support to our global airline customers and pilots. This is an ongoing and relentless commitment to make safe airplanes even safer. Soon we’ll release a software update and related pilot training for the 737 MAX that will address concerns discovered in the aftermath of the Lion Air Flight 610 accident.

Fine words. Are they true? Can Boeing’s “commitment to everyone to ensure ” safe and reliable travel” really be said to be “absolute”? That’s a high bar. Let’s see!

I’ve taken the structure that follows from a tweetstorm by Trevor Sumner (apparently derived from a Facebook post by his brother-law, Dave Kammeyer). However, I’ve added topic headings, changed others, and helpfully numbered them all, so you can correct, enhance, or rearrange topics easily in comments (or even suggest new topics). Let me also caveat that this is an enormous amount of material, and time presses, so this will not be as rich in links as I would normally like it to be. Also note that the level of abstraction for each topic varies significantly: From “The Biosphere” all the way to “Pilot Training.” A proper failure matrix would sort that out.

* * *

(1) The Biosphere: The 737 MAX story begins with a customer requirement for increased fuel efficiency. This is, at bottom, a carbon issue (and hence a greenhouse gas issue, especially as the demand for air travel increases, especially in Asia). New biosphere-driven customer demands will continue to emerge as climate change increases and intensifies, and hence the continued 737 MAX-like debacles should be expected, all else being equal. From CAPA – Centre for Aviation:

The main expected impacts of climate change on aviation result from changes in temperature, precipitation (rain and snow), storm patterns, sea level and wind patterns. In addition, climate change is expected to lead to increased drought, impacts on the supply of water and energy, and changes in wildlife patterns and biodiversity. Consequences for aviation include reduced aircraft performance, changing demand patterns, potential damage to infrastructure, loss of capacity and schedule disruption.

All of these factors will affect aircraft design, manufacturing, maintenance, and use, stressing the system.

(2) Choice of Airframe: The Air Current describes the competitive environment that led Boeing to upgrade the 737 to the 737 MAX, instead of building a new plane:

Boeing wanted to replace the 737. The plan had even earned the endorsement of its now-retired chief executive. “We’re gonna do a new airplane,” Jim McNerney said in February of that same year. “We’re not done evaluating this whole situation yet, but our current bias is to not re-engine, is to move to an all-new airplane at the end of the decade.” History went in a different direction. Airbus, riding its same decades-long incremental strategy and chipping away at Boeing’s market supremacy, had made no secret of its plans to put new engines on the A320. But its own re-engined jet somehow managed to take Boeing by surprise. Airbus and American forced Boeing’s hand. It had to put new engines on the 737 to stay even with its rival.

Why? The earlier butchered launch of the 787:

Boeing justified the decision thusly: There were huge and excruciatingly painful near-term obstacles on its way to a new single-aisle airplane. In the summer of 2011, the 787 Dreamliner wasn’t yet done after billions invested and years of delays. More than 800 airplanes later here in 2019, each 787 costs less to build than sell, but it’s still running a $23 billion production cost deficit.…. The 737 Max was Boeing’s ticket to holding the line on its position — both market and financial — in the near term. Abandoning the 737 would’ve meant walking away from its golden goose that helped finance the astronomical costs of the 787 and the development of the 777X.

So, we might think of Boeing as a runner who’s tripped and fallen: The initial stumble, followed by loss of balance, was the 787; with the 737 MAX, Boeing hit the surface of the track.

(3) Aerodynamic Issues: The Air Current also describes the aerodynamic issues created by the decision to re-engine the 737:

Every airplane development is a series of compromises, but to deliver the 737 Max with its promised fuel efficiency, Boeing had to fit 12 gallons into a 10 gallon jug. Its bigger engines made for creative solutions as it found a way to mount the larger CFM International turbines under the notoriously low-slung jetliner. It lengthened the nose landing gear by eight inches, cleaned up the aerodynamics of the tail cone, added new winglets, fly-by-wire spoilers and big displays for the next generation of pilots. It pushed technology, as it had done time and time again with ever-increasing costs, to deliver a product that made its jets more-efficient and less-costly to fly.

In the case of the 737 Max, with its nose pointed high in the air, the larger engines — generating their own lift — nudged it even higher. The risk Boeing found through analysis and later flight testing was that under certain high-speed conditions both in wind-up turns and wings-level flight, that upward nudge created a greater risk of stalling. Its solution was MCAS, the Maneuvering Characteristics Augmentation System control law that would allow for both generations of 737 to behave the same way. MCAS would automatically trim the horizontal stabilizer to bring the nose down, activated with Angle of Attack data. It’s now at the center of the Lion Air investigation and stalking the periphery of the Ethiopian crash.

(4) Systems Engineering: Amazingly, there is what in a less buttoned-down world that commercial aviation would be called a Boeing 737 fan site, which describes the MCAS system in more technical terms:

MCAS was introduced to counteract the pitch up effect of the LEAP-1B engines at high AoA [Angle of Attack]. The engines were both larger and relocated slightly up and forward from the previous NG CFM56-7 engines to accomodate their larger diameter. This new location and size of the nacelle causes it to produce lift at high AoA; as the nacelle is ahead of the CofG [Center of Gravity] this causes a pitch-up effect which could in turn further increase the AoA and send the aircraft closer towards the stall. MCAS was therefore introduced to give an automatic nose down stabilizer input during steep turns with elevated load factors (high AoA) and during flaps up flight at airspeeds approaching stall.

Unfortunately for Boeing and the passengers its crashed aircraft were carrying, the MCAS system was very poorly implemented. Reading between the lines (I’ve helpfully labeled the pain points):

Boeing have been working on a software modification to MCAS since the Lion Air accident. Unfortunately although originally due for release in January it has still not been released due to both engineering challenges and differences of opinion among some federal and company safety experts over how extensive the changes should be. Apparently there have been discussions about potentially adding [A] enhanced pilot training and possibly mandatory [B] cockpit alerts to the package. There also has been consideration of more-sweeping design changes that would prevent [C] faulty signals from a single sensor from touching off the automated stall-prevention system.

[A] Pilot training was originally not considered necessary, because MCAS was supposed to give 737 MAX the same flight characteristics as earlier 737s; that’s why pilots weren’t told about it. (This also kept the price low.) [B] Such alerts exist now, as part of an optional package, which Lion did not buy. [C] The single sensor was the result of regulatory capture, not to say gaming; see below.

(The MCAS system is currently the system fingered as the cause of both the Lion Air and Ethiopian crashes; we won’t know for sure until the forensics are complete. Here, however, is the scenario for an MCAS-induced crash:

Black box data retrieved after the Lion Air crash indicates that a single faulty sensor — a vane on the outside of the fuselage that measures the plane’s “angle of attack,” the angle between the airflow and the wing — triggered MCAS multiple times during the deadly flight, initiating a tug of war as the system repeatedly pushed the nose of the plane down and the pilots wrestled with the controls to pull it back up, before the final crash.

(5) Regulatory Capture: Commercial aircraft need to be certified by the FAA before launch. The Washington Post labels today’s process “self-certification”:

The FAA’s publication of pilot training requirements for the Max 8 in the fall of 2017 was among the final steps in a multiyear approval process carried out under the agency’s now 10-year-old policy of entrusting Boeing and other aviation manufacturers to certify that their own systems comply with U.S. air safety regulations.

In practice, one Boeing engineer would conduct a test of a particular system on the Max 8, while another Boeing engineer would act as the FAA’s representative, signing on behalf of the U.S. government that the technology complied with federal safety regulations, people familiar with the process said.

(Note that a 10-year-old process would have begun in the Obama administration, so the regulatory process is bipartisan.) I understand that “safety culture” is real and strong, but imagine the same role-playing concept applied to finance: One bankers plays the banker, and the other banker plays Bill Black, and after a time they switch roles…. Clearly a system that will work until it doesn’t. More:

The process was occurring during a period when the Transportation Department’s Office of Inspector General was warning the FAA that its oversight of manufacturers’ work was insufficient.

Four years after self-certification began, fires aboard Boeing’s 787 Dreamliner jets led to the grounding of the fleet and a wave of questions about whether self-certification had affected the FAA’s oversight.

Why “self-certification”? Investigative reporting from the Seattle Times — the article is worth reading in full — explains:

The FAA, citing lack of funding and resources, has over the years delegated increasing authority to Boeing to take on more of the work of certifying the safety of its own airplanes.

Alert readers will note the similarity to the Neoliberal Playbook, where government systems are sabotaged in order to privatize them, but in this case regulatory capture seems to have happened “by littles,” rather than out of open, ideological conviction (as with the UKs’s NHS, or our Post Office, our Veteran’s Administration, etc.).

(6) Transfer of Authority to Boeing: In the case of the 737 Max, regulatory capture was so great that certification authority was transferred to Boeing. In order to be certified, a “System Safety Analysis” for MCAS had to be performed. The Seattle Times:

The safety analysis:

  • Understated the power of the new flight control system, which was designed to swivel the horizontal tail to push the nose of the plane down to avert a stall. When the planes later entered service, MCAS was capable of moving the tail more than four times farther than was stated in the initial safety analysis document.
  • Failed to account for how the system could reset itself each time a pilot responded, thereby missing the potential impact of the system repeatedly pushing the airplane’s nose downward.
  • Assessed a failure of the system as one level below “catastrophic.” But even that “hazardous” danger level should have precluded activation of the system based on input from a single sensor — and yet that’s how it was designed.

So who certified MCAS? Boeing self-certified it. Once again The Seattle Times:

Several FAA technical experts said in interviews that as certification proceeded, managers prodded them to speed the process. Development of the MAX was lagging nine months behind the rival Airbus A320neo. Time was of the essence for Boeing….

“There wasn’t a complete and proper review of the documents,” the former engineer added. “Review was rushed to reach certain certification dates.”

In this atmosphere, the System Safety Analysis on MCAS, just one piece of the mountain of documents needed for certification, was delegated to Boeing.

(I’m skipping a lengthy discussion of even more technical detail for MCAS, which includes discrepancies between what Boeing self-certified, and what the FAA thought that it had certified, along with the MCAS system acting like a ratchet, so it didn’t reset itself, meaning that each time it kicked in, the nose was pitched down even lower. Yikes. Again, the article is worth reading in full; if you’ve ever done tech doc, you’ll want to scream and run.)

(7) Political Economy: This tweet is especially interesting, because even I know that Muddy Waters Research is a famous short seller:

And here we are! There are a myriad of other details, but many of them will only prove out once the black boxes are examined and the forensics are complete.

* * *

It should be clear at this point that the central claims of Muilenburg’s letter are false. I understand that commercial aviation is a business, but if that is so, then Muilenburg’s claim that Boeing’s commitment to safety is “absolute” cannot possibly be true; indeed, the choice to re-engine the 737 had nothing to do with safety. Self-certification makes Boeing “a judge in its own cause,” and that clearly contradicts Muilenburg’s absurd claim that “safety” — as opposed to profit — “is at the core of who we are.” The self-certification debacle that allowed MCAS to be released happened on Muilenburg’s watch and is already causing Boeing immense reputational damage, and a criminal case, not to mention the civil cases that are surely coming, will only increase that damage. Mr. Market, the Beltway, and even Trump, if his trade deals are affected, will all soon be bellowing for a sacrificial victim. Muilenburg should recognize the inevitable and gracefully resign. Given his letter, it looks unlikely that he will do the right thing.

Print Friendly, PDF & Email
This entry was posted in Banana republic, Guest Post, Technology and innovation on by .

About Lambert Strether

Readers, I have had a correspondent characterize my views as realistic cynical. Let me briefly explain them. I believe in universal programs that provide concrete material benefits, especially to the working class. Medicare for All is the prime example, but tuition-free college and a Post Office Bank also fall under this heading. So do a Jobs Guarantee and a Debt Jubilee. Clearly, neither liberal Democrats nor conservative Republicans can deliver on such programs, because the two are different flavors of neoliberalism (“Because markets”). I don’t much care about the “ism” that delivers the benefits, although whichever one does have to put common humanity first, as opposed to markets. Could be a second FDR saving capitalism, democratic socialism leashing and collaring it, or communism razing it. I don’t much care, as long as the benefits are delivered. To me, the key issue — and this is why Medicare for All is always first with me — is the tens of thousands of excess “deaths from despair,” as described by the Case-Deaton study, and other recent studies. That enormous body count makes Medicare for All, at the very least, a moral and strategic imperative. And that level of suffering and organic damage makes the concerns of identity politics — even the worthy fight to help the refugees Bush, Obama, and Clinton’s wars created — bright shiny objects by comparison. Hence my frustration with the news flow — currently in my view the swirling intersection of two, separate Shock Doctrine campaigns, one by the Administration, and the other by out-of-power liberals and their allies in the State and in the press — a news flow that constantly forces me to focus on matters that I regard as of secondary importance to the excess deaths. What kind of political economy is it that halts or even reverses the increases in life expectancy that civilized societies have achieved? I am also very hopeful that the continuing destruction of both party establishments will open the space for voices supporting programs similar to those I have listed; let’s call such voices “the left.” Volatility creates opportunity, especially if the Democrat establishment, which puts markets first and opposes all such programs, isn’t allowed to get back into the saddle. Eyes on the prize! I love the tactical level, and secretly love even the horse race, since I’ve been blogging about it daily for fourteen years, but everything I write has this perspective at the back of it.

107 comments

    1. Big Tap

      Haley is more like a bloviation expert. She does know much about producing hot air so there’s that.

  1. Yikes

    Sacrificial Victims were spread over land and sea in Kenya and Indonesia. Muilenburg and Obbie The Wan both are the criminals who profit.

    1. SoldierSvejk

      Yes, that is why this early sentence does not seem to make sense: ” (I’m not unsympathathetic to the view that the wholesale 737 MAX grounding was premature on technical grounds…”). Not sure how many more fallen aircraft it would take so make someone ‘sympathetic’ to grounding the planes.

  2. dcrane

    That should be “five months” not “five weeks” in the first sentence. Lion Air crashed on 29 October 2018.

    1. dcrane

      (I’m skipping a lengthy discussion of even more technical detail for MCAS, which includes discrepancies between what Boeing self-certified, and what the FAA thought that it had certified, along with the MCAS system acting like a ratchet, so it didn’t reset itself, meaning that each time it kicked in, the nose was pitched up even higher. Yikes. Again, the article is worth reading in full; if you’ve ever done tech doc, you’ll want to scream and run.)

      and that should be “down even lower”, I think. (Talking about the nose here, not the stabilizer.)

  3. Howard Beale IV

    IIRC, one of the big constraints that was leveled was the need to keep the 737, regardless of version, into the same height relative to all other generations of the 737, whereas Airbus kept their height a lot higher than the 737.

    If you look at many 737’s over the years, some of the engine’s nacelles were flat at the bottom to accommodate larger engine. Why? Boeing kept the height the same in order to maintain built-in stairs that, with virtually all airports having adjustable jetways, was basically redundant.

    When you compare an A320xeo against a B737, you’ll find that the Airbus rides higher when it comes to the jetways.

  4. Michael Hudson

    It seems to me that the Boeing 737-Max with the heavier, larger fuel-saving engines is so unbalanced (tilting over and then crashing if not “overridden” by a computer compensation) that it never should have been authorized in the first place.
    When Boeing decided to add a much larger engine, it should have kept the airplane in balance by (1) shifting it forward or backward so that the weight did not tip the plane, and (2) created a larger landing-gear base so that the large engines wouldn’t scrape the ground.
    The problem was that Boeing tried to keep using the old chassis with the larger engines under the wings – rather than changing the wings, moving them forward or aft, and expanding the plane to permit a more appropriate landing gear.
    The computer system has been blamed for not being a “smart enough” workaround to tell the plane not to plunge down when it already is quite close to the ground – with no perception of altitude, not to mention double-checking on the wind speed from both sensors.
    Beyond that ultimate problem is the ultimate regulatory problem: regulatory capture of the FAA by the airline companies. As a result, the FAA represents “its customers” the airplane makers, not the public users and customers. This is like the banks capturing the Fed, the Justice Dept. and Treasury to promote their own interests by claiming that “self-regulation” works. Self-regulation is the polite word for fraudulent self-indulgence.
    I would be surprised if the European Airbus competitors do not mount a campaign to block the 737-Max’s from landing, and insisting that Boeing buy them back. This gives Airbus a few years to grab the market for these planes.
    This probably will throw Trump’s China trade fight into turmoil, as China was the first country to ground the 737-Max’s and is unlikely to permit their recovery without a “real” federal safety oversight program. Maybe Europe, China and other countries henceforth will each demand that their own public agencies certify the plane, so as to represent users and stakeholders, not only stockholders.
    The moral: Neoliberalism Kills.

    1. Lambert Strether Post author

      Rule #2 of Neoliberalism: Go die.

      > “Maybe Europe, China and other countries henceforth will each demand that their own public agencies certify the plane.”

      As if the 737 MAX were the chlorinated chicken of aircraft.

      * * *

      I’m not sure about redesigning the wing and the landing gear. That might be tantamount to designing a new plane. (I do know that the landing gear is so low because the first 737s needed to accommodate airports without jetways, and so there may be other facets of the design that also depend on those original requirements that might have to be changed.)

      1. Cal2

        Rule #3 of Neoliberalism:

        Their profits = Your cancer,

        which presents even more profit taking.

        i.e. Bayer makes the carcinogenic pesticides AND the chemotherapy drugs.

      2. John Beech

        This plane is a product of the 60s. You’re darn tootin’ things have changed in nearly 60 years! Anyway, yes, redesigning the landing gear involves the wing and basically involves a new plane.

    2. John Zelnicker

      @Michael Hudson
      March 19, 2019 at 4:42 pm
      ——-

      “This gives Airbus a few years to grab the market for these planes.”

      That would be great for Mobile as the Airbus A320neo is assembled here.

      1. Octopii

        And provides time for the A220 to ramp up in Mobile as well. Not a direct competitor for the 737 but a very good airplane developed by Bombardier.

      2. Carey

        Also, the MC-21 is in final testing now; still using Western engines, for the moment. One to watch, maybe.

    3. Which is worse - bankers or terrorists

      Engineering logs seem to indicate that larger landing gear cannot be added without re-engineering the plane.

    4. 115 kV

      Regulatory capture is rampant throughout the economy. Boeing self-certification being delegated by the FAA is not unlike the situation with electric transmission utilities.

      After the 2003 northeast & Canada blackout, Congress passed the Energy Policy Act of 2005. It directed FERC to create an “electric reliability organization”. Previously there were voluntary organizations set up after the 1966 blackout to establish operating standards in the industry. One of them was the North American Electric Reliability Council which morphed into the North American Electric Reliability Corporation (NERC) in 2006.

      NERC is headquartered in Atlanta and employs hundreds of people. The standards setting generally takes place in NERC Committees and Subcommittees and sometimes from FERC itself. These are typically packed with industry people, with a patina of diversity that includes some governmental types and large industrial consumers. Let it suffice to say the electric transmission industry itself largely sets the rules how it operates.

      Now consider the article in yesterday’s NYT “How PG&E Ignored California Fire Risks in Favor of Profits“. The transmission circuit featured in the article (the Caribou-Palermo line) that caused the destruction of Paradise is a transmission line that is subject to both FERC and NERC regulation. As described in the article the circuit had many previous failures and was well beyond its design life.

      However, both FERC and NERC have a laser focus on “market players” (think Enron or JP Morgan) and system operations (e.g., prevent collapses like the blackout of 2003). AFIK, neither FERC or NERC have prescriptive standards for routine maintenance or inspection and replacement (i.e., very expensive capital replacement that was not done on the Caribou-Palermo line), these are left to the discretion of the transmission owner. While substantive information about electric reliability is maintained by industry trade groups and submitted to FERC, what is available to the public is generally useless and subjected to scrubbing and polishing (often under the guise of Critical Energy Infrastructure Information).

      We can see how self-policing work, can’t we??? Rent-seeking market players can arbitrage markets, inflating prices consumers pay and make billions in profits, while California burns.

      The neglectful rot in California is endemic in the industry as a whole.

    5. clarky90

      “The Mercedes S Class has over 100 microprocessors attached to sensors…”

      Why New Cars are a Rip-off to Repair

      https://www.youtube.com/watch?v=3zL7P-etsDI

      Here is Scotty Kilmer (a car mechanic for 51 years!) talking about modern cars. I imagine that the Boeing 737 MAX is 100 times worse. Anyway, it is looking at the problem from a more relatable POV.

    1. Robert Hahl

      If an airplane crashes in the forest, and no American were killed, did it make a sound?

  5. Carolinian

    That Seattle Times investigative story is indeed very good and a rare instance of newspaper writers troubling to carefully and cogently explain a technical issue. In hindsight Boeing would have perhaps been better off to leave off the MCAS altogether and depend on pilot retraining to cover the altered handling. One reason they may not have was that crash several years ago of a commuter plane in upstate NY where the plane started to stall and the confused pilot pulled up on the controls rather than making the airplane dive to regain speed. Still one has to believe that no automation is better than badly designed or malfunctioning automation.

    1. allan

      “depend on pilot retraining to cover the altered handling”

      IANAP, but maybe the problem is that “nose up” situations can go south very quickly.
      For those with the stomach for it, there are videos on youtube of the 747 freighter
      that went nose up at Bagram a few years ago (perhaps due to loose cargo shifting backwards on takeoff).
      It was over very quickly.

      1. J7915

        The 737 MAX seems like a fighter, unstable, but lacking ejection seats.
        F100 loose hydraulics, F16 electrical failure, f116? Stealth loose computers eletrical, all have in common the pilots leaves.
        737Max gets into extreme condition, SENSORS or MCAS fails then what??

    2. ChrisPacific

      Yes, I was impressed with it. Unfortunately the investigation precludes Boeing from responding as they did indicate they would have had something to say about it otherwise. But the analysis looks pretty cut and dried:

      1. Boeing underestimated the risk rating for the sensor, excluding the possibility of a catastrophic failure as occurred in the two incidents to date;
      2. Boeing also failed to implement the redundancy that would have been required even for their lower risk rating;
      3. Manual correction by the pilot as a possible risk mitigation was constrained by the fact that pilots weren’t trained on the new system due to commercial factors.

      Fixing any one of those three issues would have averted the disasters, although #3 is pretty precarious as you’re relying on manual pilot actions to correct what is a clear systems defect at that point.

      It sounds like #1 was partly because they failed to account for all the scenarios, like repeat activation raising the risk profile in certain circumstances. This is very easy to do and a robust review process is your best defense. So we could add the tight timelines and rushed process as a contributing factor for #1, and probably the others as well.

      1. XXYY

        People who work on accident investigation would probably agree on 2 things:

        (o) Accidents are invariably a confluence of a myriad of factors that all happened to line up on one day. There is never a single cause of an accident.

        (o) A minor change to some part of the system would have prevented the accident.

        So while there is much to be profitably learned by investigating everything here, an effective “fix” may be surprisingly (or suspiciously) small in scope.

        There will be much clamoring for the whole plane to be resigned or scrapped, for better or worse.

    3. anon in so cal

      The Colgan crash, whose pilot, Renfrew, was chatting with the co-pilot below the allowed altitude? And who had apparently lied about his background, and had a pay-to-play pilot’s license?

      I think the Air France Airbus 447 also had a high-altitude stall (due to a faulty air speed sensor) and needed its nose pushed down, not up (which the copilots didn’t realize).

      Also, very informative article / OP, thanks for posting.

    4. Synoia

      MCAS was added to change the behavior of the plane from to tend to stall as speed increases. That is stall and crash, because such a high speed stall makes polit recovery very, very difficult.

      In addition the MCAS driven amount of elevator change was initially 0.6 to 2.5, which indicates the 0.6 increment was found to be too low.

      1. Carolinian

        Well they are planning to keep it but

        According to a detailed FAA briefing to legislators, Boeing will change the MCAS software to give the system input from both angle-of-attack sensors.

        It will also limit how much MCAS can move the horizontal tail in response to an erroneous signal. And when activated, the system will kick in only for one cycle, rather than multiple times.

        Boeing also plans to update pilot training requirements and flight crew manuals to include MCAS.

        –Seattle Times

        So apparently the greater elevator setting is not so necessary that they are not willing to reduce it. Also the max power setting would normally be on take off when the pilots are required to manually fly the plane.

        1. Synoia

          It is sbout speed, not power.

          I presume that MCAS was developed to solve a problem, nose up behaviour.

    5. Carey

      Yes, that was an excellent Seattle times piece. Surprising to see that kind of truth-telling
      and, especially, *clarity* in an MSM piece these days.

      So what’s the angle?

  6. voislav

    Reports I’ve read indicates that Boeing ignored even the clearly inadequate certification. “Documentation provided to the FAA claims that the MCAS system can only adjust the horizontal tail on the plane by 0.6 degrees out of a maximum of five-degrees of nose-down movement. But that limit was later increased to 2.5-degrees of nose-down movement. Boeing didn’t communicate the change from 0.6-degrees to 2.5-degrees until after Lion Air.”

    Apparently this was done after simulations showed that 0.6 degrees was inadequate and the new 2.5 degree setting was not extensively tested before the planes were rolled out. IANAL, but this may be a serious problem for Boeing. Boeing could also be liable for damages due to 737 groundings and due to delays in delivery of contracted planes.

    Big question is how 737 issues will affect 777X rollout, due at the end of the year. If 777X certification is called into question, this may cause further delays and put it at a further disadvantage against A350.

    1. Lambert Strether Post author

      The 777 has been a great plane. Let’s all pray the MBAs didn’t fuck it up, too.

      If I were Boeing, I’d have a team looking into the 777 certification process right now. And I’d set up a whistleblower line (so the Seattle Times doesn’t get to the story first).

      The analogy has been made between this the 737 MAX story and the Tylenol story. J&J got out in front of the problem and saved the product (and their company). Boeing’s problem is of that order, and Muilenberg — that letter! — seems incapable of understanding that; insular, arrogant. One more reason to fire the dude toot sweet. If he comes out of his next review with a raise — Everything Is Like CalPERS™ — consider shorting Boeing…

      1. Chris

        Thanks, Lambert, for post and comments. I don’t know if this angle has been covered or explored: the relatively new way that Boeing now “manufactures” “tests” and “assembles” parts of its planes. I had dinner with new acquaintance, Boeing engineer for decades (I live near a plant in WA state). For the last few years, this engineer is stationed half year in Russia annually to oversee assembly there. In this newish, more profitable manufacturing system for Boeing, the parts come in from around the world with sketchy quality control, are then assembled by Russian workers this engineer (and other Boeing employees sent from States) supposedly oversees. But the engineer doesn’t speak Russian and has too little access to translators….Needless to say, this engineer is planning an exit as soon as possible. Having grown up in WA state for 60 years with neighbors/friends who were Boeing engineers, assemblers, line workers, etc it makes me heart sick to see the current decimation of talent, rigor and wages with additional far-flung assembly factories (Russia with few translators?! who knew?). Might these manufacturing/assemblying “improvements” also be a contributing factor in these terrifying woes for Boeing?

      2. Tim

        No worries about the 777. I worked at Boeing in the New Airplane Development group in 2000. Those were the good old days when Boeing Engineering requirements routinely ignored FAA requirements that were too lax. It made it hard for our aircraft to compete on cost and performance with Airbus, who always only held to the JAA requirements which came from the FAA.

        As an example, remember the Airbus A300 that went down in Jamaica bay when the pilot snapped the vertical tail off? FAA investigators took a look at the Boeing vertical tail on the 777 and said wow this is way more robust than the A300, why? The engineers in the room said “that would be Tom” (Tom Jones the Analyst for the design who has likely since passed away).

        Tom stated “Well because the pilot can do it (rapid full rudder reversal) means we should design for the aircraft to handle the loads, even though the FAA did not require that, because the pilot instruction manuals tell the pilot not to do it, so AIrbus never design for that and their vertical tail joint was weaker and look what happened.

        So 777 is pre-crapification, even though it was intended to be a 0 excess strength margin design.

  7. PlutoniumKun

    Thanks for this Lambert, fantastically informative and interesting post.

    Self regulation only works when liability is transferred with it – over example, in construction whereby certification by the engineers or architects designing the building are also taking on liability in the event something goes wrong. It seems unlikely that this is the situation with Boeing.

    Allowing this to happen seems the ultimate in short term thinking by Boeing. US manufacturers have always had an advantage over competitors because the FAA was held in such high regard worldwide that it was the de facto world safety regulatory body – every country followed its lead. But this chipping away of its authority has led to a near fatal loss of faith, and will no doubt lead to European and Asian regulatory authorities being strengthened. And no doubt commercial realities will mean they will look much more closely at US manufactured aircraft if there is some benefit to their own manufacturers.

    Airbus will no doubt try to take advantage – just as Boeing (with some justification) tried to focus attention on the Air France Airbus loss which was attributed at least in part to excessive automation. China is pushing hard with its new Comac aircraft, but they seem to be poorly regarded worldwide (only Chinese airlines are buying). The Canadians have missed their chance with the Bombadier C-series.

    1. Arakawa

      Bombardier’s planning seems to rely on forcing their product onto Canadian customers early on (read e.g. about the long and dreary debacle that is ICTS/ART/acronym-du-jour, now being sold to Montréal, and the Toronto streetcar replacement saga). In the case of the C-series, the plan to sell them to Porter Airlines fell through because they would be landing the things right into Downtown Toronto. The airport there currently accepts smaller Dash-8 planes but the prospect of runway expansion to handle C Series jets is the kind of stuff that can make the locals see red and swing elections.

      Then, because the Canadians can be made to put up with lousy performance that would get Bombardier laughed out of Europe or Asia, I’m sure that affects Bombardier’s ability to get its act together for the global market.

      1. eg

        Bombardier does itself no favors on the reliability front in Toronto with its delayed/non-delivery of streetcars …

  8. Darius

    My employer archives a lot of information. Up until now, people have been doing it. Going forward it’s automated. I’m told it now has a 20 percent error rate. Eh. Good enough. That’s crapification in action.

  9. JBird4049

    The more I read of this the more baffling it is. What was there stopping Boeing from just highlighting the changes and installing an easy manual override instead of this hidden change with effectively no way to permanently do so? Especially when in crisis mode? One could make a case of no extra training needed so long as the pilot knows about it and can easily turn it off.

    1. Darius

      I didn’t see this before I posted my response. A more concise statement of my thoughts. This plus more robust redundant sensors. Penny wise and pound foolish.

      The Times thinks Boeing is too big to fail. Without a blockbuster Max, I don’t see how Boeing maintains its current status in the industry.

    2. Synoia

      One could make a case of no extra training needed so long as the pilot knows about it and can easily turn it off.

      That’s the expensive re-certification Boeing wanted to avoid.

    3. Robert Hahl

      That would entail simulator training, that would entail modifying the simulators and the curriculum.

  10. Darius

    I am leaning towards thinking the kludgy design of the 727 Max could have been rolled out with no major problems if Boeing had been up front about design changes, made a robust and conservative MCAS, fully at the command of the pilot, and provided ample training for the new aircraft.

    They still could have saved billions on the airframe. They would have had to acknowledge the significant modifications to the airlines with the attendant training and other costs and delays. They would have lost some sales. They still would have been far ahead of Airbus and light years ahead of where they are now.

    I also think they have been completely afflicted by the defense contractor mentality.

    1. Lambert Strether Post author

      > I also think they have been completely afflicted by the defense contractor mentality.

      Yes, the famous McDonnell-Douglas reverse takeover, where financial engineers inserted their sucking mandibles into an actual engineering culture. The merger took place in 1997, 22 years ago, which is not so long, really. Note also that the finance guys drove the decision to outsource as much 787 manufacturing as possible, which creates headaches for real engineering, so the initial stumble with the 787 that led to the 737 fall is down to them, too.

      Note that Muilenberg came up through the defense side of the company not the commercial aircraft side. He may simply not have been equipped to understand FAA regulation at any deep level, hence the rot that finally surfaced.

  11. VietnamVet

    The 737 Max crashes and Brexit are the chickens coming home to roost. NC is a treasure for your coverage of both.

    Clearly upper management in Chicago only knows short term finance. Boeing stuck with old fashion hydraulic controls in the 737 but faced with an unacceptable flight characteristics of the larger more efficient engines added a fly-by-wire system to compensate for it. The criminal charges are that besides being a faulty design (it relies on one fragile exposed sensor that if out of position keeps triggering dives until switched off) but Boeing hid it and self-certified that it was safe. Adding a discrepancy warning and position indicator for the two independent flight sensors to the cockpit video display is an extra cost feature. Neither of the planes that crashed had the added safety display. All are cost saving measures. Finally, if a faulty sensor triggers dives, the pilot at the controls is busy with both hands on the yoke forcing the airplane to stay in the air with stall and proximity warnings are sounding. The second pilot also must realize what’s going on, immediately turn off the electricity to the screw jack motor and manually turn the stabilizer trim wheel to neutral. You can’t learn this on an iPad. Both pilots should practice it together in a Flight Simulator. If the co-pilot was experienced, unlike the one in the Ethiopian crash; just maybe, they could have survived the repeated attempts by the airplane to dive into the ground on takeoff.

    The tragedy is that corporate media in pursuit of profits will keep us up to date but will never mention the 6 or 8 minutes of terror for the 346 souls aboard the two flights. They will cover the criminal negligence trial if there are ever indictments. But, the news reports never will say that neoliberalism, deregulation, and privatization are the root causes of the deaths.

    1. Lambert Strether Post author

      > if a faulty sensor triggers dives, the pilot at the controls is busy with both hands on the yoke forcing the airplane to stay in the air with stall and proximity warnings are sounding. The second pilot also must realize what’s going on, immediately turn off the electricity to the screw jack motor and manually turn the stabilizer trim wheel to neutral. You can’t learn this on an iPad. Both pilots should practice it together in a Flight Simulator. If the co-pilot was experienced, unlike the one in the Ethiopian crash; just maybe, they could have survived the repeated attempts by the airplane to dive into the ground on takeoff.

      That’s what I mean by horrid UI/UX. Might as well as both pilots to pat their heads and rub their tummies in synch. And since the two pilots have to both understand what’s going on, we’ve multiplied the chances for failure.

      Boeing also clearly did not know its customers. It should be engineering for the sort of pilots who are going to be hired by Lion Air, or any rapidly expanding airline in what we used to all the Third World. Hegemony, it seems, makes you insular and provincial.

    2. EoH

      Added cost, “mandatory” safety feature. Does not seem to square with the [soon to be former?] CEO’s apology-industry written claim to be committed to absolute safety.

      You can’t make this stuff up.

  12. dearieme

    “The FAA, citing lack of funding and resource”: I don’t suppose I’ll survive to see any arm of government not blame lack of funds for its boneheaded or corrupt incompetence.

    But the bigger picture: suppose the FAA is to do its job properly. From where is it going to recruit its staff?

    Smaller picture: it doesn’t really matter whether the cocked-up MCAS killed all those people or not. Even if it’s innocent of the charge, the account of its development and application is a horror story.

    Bigger picture: what other horrors have been hidden by Boeing?

    1. Lambert Strether Post author

      > the account of its development and application is a horror story.

      That’s how I feel. The tech doc department at Boeing sounds like a horrible place to work; MBAs or their goons telling you all the time to do stuff you know is wrong. It’s not surprising people were willing to talk to the Seattle Times; I bet there are more people. (Hey, Seattle Times! How about people testing the 737 MAX in simulators (assuming this is done)).

      Sounds like the MBAs in Chicago have been busy planting land mines everywhere. Somebody stepped on this one; there are others.

  13. oaf

    The unfortunate pilots were made test pilots; the unsuspecting passengers: Guinea pigs. Lab rats. And paid for the privilege. Some others may share this opinion. Change one little thing?…Chaos Theory Rules. Same with weather/climate; folks. That rant is for later.

    oafstradamus

  14. dcrane

    Boeing stuck with old fashion hydraulic controls in the 737 but faced with an unacceptable flight characteristics of the larger more efficient engines added a fly-by-wire system to compensate for it.

    Interestingly, and maybe relevant to the problem of confusion for the pilots, is that Boeing has had another automatic trim-modifier operating on its 737s for some time, the speed-trim system (STS):

    https://leehamnews.com/2019/02/01/bjorns-corner-pitch-stability-part-7/

    This system also modifies the stabilizer position during manual flight. Like MCAS, it was brought in to improve stability under certain flight conditions (the reasons for which are far beyond my knowledge). There is an indication that the pilots on the flight before the Lion Air crash misinterpreted MCAS actions for STS behavior.

      1. Carey

        Someone needs to do a high level but probably non-academic study of corporateSpeak like this tripe: “at the core of who we are..” .

        Maybe there is or will be an American Houellebecq, even as the ship goes down?

  15. drumlin woodchuckles

    At what point does “crapification” become insufficient to describe Boeing’s product and process here? At what point do we have to speak of ” ford-pintofication”?

  16. barrisj

    OK, I’m told to resubmit my crib re: “Boeing options” from the ZeroHedge “tweetstorm” by Trevot Sumner, and include a link…got it:

    Economic problem. Boeing sells an option package that includes an extra AoA vane, and an AoA disagree light, which lets pilots know that this problem was happening. Both 737MAXes that crashed were delivered without this option. No 737MAX with this option has ever crashed

    https://mobile.twitter.com/trevorsumner/status/1106934369158078470

    Ooops! “Options package”? Wait, a “package” that in the interim corrects a potentially catastrophic mfg. defect…and airlines have to pay for it? Whoa, here’s your late capitalism in play.

    1. Lambert Strether Post author

      > Boeing sells an option package that includes an extra AoA vane, and an AoA disagree light

      This is one of the details I could not get to (and we don’t 100% know this is an issue until the forensics are done. Right now, we have narrative. Truly excellent narrative to be sure — if only we thought of government the same way as pilots think of their aircraft! — but narrative nonetheless).

      Let me see if I have this right. Pilots, chime in!

      “Authority” is one of the big words in this discussion; MCAS takes authority away from the pilot (and can do in such a drastic fashion as to crash the plane). Worse, the default case is that it can do so on the basis of a single sensor reading. In a design appropriate to the consequences for failure (i.e., a different design from that described in the “System Safety Analysis” that Boeing self-certified) MCAS would take readings from two sensors, and if they disagreed, authority would revert to the pilot. That’s a general principle at Boeing, and so it’s reasonable for pilots to assume that they retain authority of MCAS has not told them they don’t have it any more.

      Hence, the disagree light, which tells the pilots to take back authority because the sensors are confused. However, I think there are UI/UX issues with that, given that the 737 cockpit is extremely noisy and pilots have a lot to do on take-off. So a light might not be the answer. (The light also strikes me as a kludge; first, MCAS feels to me like a kludge, in that we’re making the aircraft flyable only through software.* Fine for fighter jets, which can be inherently unstable, but perhaps not so fine for commercial aircraft? Then we have a second kludge, a light to tell us that the first kludge has kicked in. I dunno.)

      NOTE * However, it’s also true that automation affects flight characteristics all the time. So I’m not sure how savage to make this indictment.

      1. rowlf

        The AOA indication is Service Bulletin 737-31-1650 (there may be others) and is on the both Pilot Flight Displays (PFDs). Pilots would likely abort a takeoff if they saw the indication come on before getting airborne.

  17. California Bob

    In hindsight, it appears Boeing should have made Mulally CEO. He appears to be competent.

  18. Cal2

    “Boeing has been in the business of aviation safety for more than 100 years,…”

    How many years ago did Wall Street take over the fortunes of the company?

    Why did they move their headquarters from their birthplace of Seattle to Chicago?

    Why did they start assembling planes in South Carolina and China?

    Was it to improve aviation safety?

    Or, to allow the profiteering parasites to feed off the carcass of the company?

    I want to fly on Boeing planes put together by well paid members of the Seattle Machinists Union,
    not low wage peons.

    Let’s not even mention the maintenance of American aircraft in China and El Salvador.

    https://www.bizjournals.com/chicago/news/2018/04/20/southwest-airlines-should-have-inspected-engines.html

    President Trump, here’s a reelection tip:

    “Today I am declaring that all American registered aircraft flying in American airspace must be maintained in the U.S.”

    1. Lambert Strether Post author

      > President Trump, here’s a reelection tip:

      > “Today I am declaring that all American registered aircraft flying in American airspace must be maintained in the U.S.”

      Amazingly, Trump seems to have done OK on this.* First, he didn’t cave to Muilenberg’s (insane, goofy, tone-deaf) request to keep the 737 flying; then he frames the issue as complexity (correct, IMNSHO), and then he manages to nominate a Delta CEO as head of the FAA.

      And your suggestion is very good one. I wonder if he could do that by executive order? And I wonder how many grey-beards would come off the golf courses to help out? I bet a lot.

  19. oaf

    The aircraft is NOT CRAP!!! However. It should have been flown A WHOLE LOT MORE before receiving certification. *Real* test pilots should have their a–es on the line ; operating for a lot more hours at *the edge of the envelope*, as it is known. Stability should be by design; not software*patch*. Patch this!
    What portion of its’ MCAS system flight testing was in computer simulation? Like the so-called Doppler Radar; which *magically* predicts what the future will bring; while the experts pitch it as fact? And make life-or-death decisions on the theoretical data???
    Rush to market; markets rule. We can die.

    1. dcrane

      The aircraft is NOT CRAP!!!

      Agreed, but I think we’re seeing signs that a crapification process has begun on the safety side in this industry. (It has been proceeding for years on the service/amenities side.)

    2. Lambert Strether Post author

      > The aircraft is NOT CRAP!!!

      Didn’t say it was. The headline reads “Boeing Crapification,” not “737 Crapification.”

      That said, the 737 clearly has issues, as Boeing itself knew, since if they’d had their druthers, they would have launched a new plane to replace it. See point #2.

      > What portion of its’ MCAS system flight testing was in computer simulation?

      That is a very good question. If I understand the aerodynamics issues aright, MCAS would be most likely to kick in at takeoff, which raises a host of UI/UX issues because the pilots are very busy at that time. So was MCAS not tested in the simulators? If so, how on earth was a scenario that included sensor failure not included? It may be that there are more issues with Boeing’s engineering process than the documentation issues raised by the Seattle Times, though those are bad enough.

      1. Ron D

        I say the 737-whatever is a flying Turd, and always has been. It has a bad wing design which means it has to fly nose up compared to other models( I always remember that when going to the restroom while going somewhere on one). And because of its poor design it has to takeoff and land at higher speeds. So when flying into someplace like Mexico City it can be quite a harrowing experience, and the smell of cooking brakes is relatively normal.
        Boeing never should have let go of the 757. Now that was a good plane that was simply ahead of its time.

        1. JerryDenim

          “Boeing never should have let go of the 757. Now that was a good plane that was simply ahead of its time.”

          Great comment. For all intensive purposes the 757 airframe was a modern A321, but with more powerful engines. The A320 and A321 Neo was the airplane Boeing was chasing with their “Max” redesign, so it seems like they should have stopped stretching (elongating) and torturing the poor old 737 and rebooted the 757 with a 757 “Max” redesign instead. The 757 airframe offers a bigger fuselage for passenger convenience/comfort and plenty of ground clearance for engines, wing tips and tail. All the things the 737 is missing when compared to the A320/21 Neo. You can slap modern high efficiency turbofans on anything. It’s the other stuff that’s hard.

          1. JerryDenim

            Besides Southwest who operates a one type fleet (all 737) I can’t imagine any of the US carriers really clamoring for a type-compatible (no additional ratings or training required) replacement for the 737 instead of a new and improved 757 meant to outclass the A320 and A321 Neo’s. Seems like a no-brainer, but they probably thought they could hit two slightly different market segments at once with the 737 Max, Southwest and then everybody else that probably would have preferred a A320 Neo. Even Boeing’s hometown airline, Alaska, which used to be an all Boeing 737 Airline operates some number of A320’s now. Every other legacy airline has them in abundance. JetBlue, Frontier, and Spirit are exclusively A320/21 operators.

            Bottom line; Boeing really screwed up pushing the aerodynamically illogical 737 Max instead of a clean sheet design or relaunching a new and improved 757 Max instead.

  20. The Rev Kev

    Considering the fact that all these 737s are grounded as no airline trust them to not kill a plane load of passengers and crew, this is a really big deal. Putting aside the technical and regulatory issues, the fact is that the rest of the world no longer trusts the US in modern aviation so what we have here is a trust issue which is an even bigger deal. We now know that the FAA does not audit the work done for these aircraft but the airlines themselves do it. It cannot be just Boeing but the other aircraft manufacturers as well. Other countries are going to be asking some very hard questions before forking over their billions to a US aircraft manufacturer in future. Worse is when Ethiopia refused to hand over the black boxes to the US but gave them instead to a third party. That was saying that based on how you treated the whole crash, we do not trust you to do the job right and not to change some of the results. It has been done before, ironically enough by France who the Ethiopians gave the black boxes to. And when you lose trust, it takes a very long time to gain it back again – if ever. But will the changes be made to do so? I would guess no.

    1. notabanker

      But if the discount foreign airlines had just trained their pilots and paid for the non-crashintothegroundat500mph upgrade, all of this could have been avoided.

    2. Lambert Strether Post author

      > we have here is a trust issue which is an even bigger deal

      Loss or at least wobbliness of imperial hegemony, like. It’s not just the aircraft, it’s US standards-setting bodies, methods, “safety culture,” even — dare we say it — English as the language of aviation. French is no longer the language of diplomacy, after all, though it had a good run.

      Because markets. Neoliberalism puts everything up for sale. Including regulation. Oversimplifying absurdly: And so you end up with the profit-driven manufacturer buying the regulator, its produce killing people, and the manufacturer canceling its future profits. That’s what the Bearded One would call a contradiction.*

      NOTE * There ought to be a way to reframe contradiction in terms of Net Present Value… which would not be what we think it is, under that model.

  21. Synoia

    Thank you Lambert, this is very complete.

    Can Boeing survive?
    Yes, as a much smaller company.

    What is upsetting to me, is that the Boeing management has sacrificed thousands of Jobs.

    1. Lambert Strether Post author

      > Thank you Lambert, this is very complete.

      I wish it were as complete as it should be! There are a ton of horrid details about sensors, the UI/UX for the MCAS system, 737 cockpit design, decisions by the marketing department, and training and maintenance for Asian airlines that I just couldn’t get to. (Although most of those presume that the forensics have already been done.) But I felt that dollying back for the big picture was important to. Point #1 is important, in that all the factors that drove the 737 decision making are not only still in place, they’re intensifying, so we had better adjust our systems (assuming Boeing remains a going concern — defenestrating Muilenberg would be an excellent way to show we accept the seriousness of customer and international concern).

  22. Bill Smith

    Bloomberg is reporting that :

    “The Indonesia safety committee report said the plane had had multiple failures on previous flights and hadn’t been properly repaired.”

    And the day before when the same plane had the problem that killed everyone the next day:

    “The so-called dead-head pilot on the earlier flight from Bali to Jakarta told the crew to cut power to the motor driving the nose down, according to the people familiar, part of a checklist that all pilots are required to memorize.”

    1. Lambert Strether Post author

      There’s an enormous expansion of air travel in Asia. The lower end — not flag — carriers like Lion Air and also Air Asia are in that business to be cheap; they’re driven by expansion and known to be run by cowboys.

      That said, know your customer. I would translate this into an opportunity for Boeing to sell these airlines a service package for training their ground operations. But it seems that cutting costs is the only thing the MBAs in Chicago understand. Pilots, pipe up!

      1. Bill Smith

        Pilot training and requirements are in the hands of the country, not Boeing.

        If the story that the copilot of the Ethiopian Airlines plane had only 200 hours of experience that is astounding.

        In the US that requirement is 1500 hours. In addition most US airlines would require more than that. And then they slot ‘beginning’ pilots for flights in good (better) weather as high minimums pilot.

        1. Carey

          “them dum furriners shoulda knowed what to do when their new-to-them plane pointed its nose at the ground all by itself, *just after takeoff*, over and over and over…”

          “check the manual, Kenneth!”

          yeesh

      2. Bill Smith

        “sell these airlines a service package”

        That won’t help an airline that is in the business to be cheap.

        The Indonesia airplane was repeatedly reported for problems in prior days/flights that was never fixed.

    2. Basil Pesto

      indeed I was just about to mention this same story. The link is here: https://www.bloomberg.com/news/articles/2019-03-19/how-an-extra-man-in-cockpit-saved-a-737-max-that-later-crashed?utm_campaign=news&utm_medium=bd&utm_source=applenews

      and this quote makes an interesting follow-on to the thread yesterday with 737 Pilot (which Lambert linked to in the first paragraph here):

      “The combination of factors required to bring down a plane in these circumstances suggests other issues may also have occurred in the Ethiopia crash, said Jeffrey Guzzetti, who also directed accident investigations at FAA and is now a consultant.

      “It’s simply implausible that this MCAS deficiency by itself can down a modern jetliner with a trained crew,” Guzzetti said.“

      Setting aside Mr Guzzetti’s background (dismissing his claim here as tendentious right off the bat would strike me as uncharitable), and without wishing to exculpate anyone, it does lend some credence to the idea that Ethiopia Airlines may have some contributory negligence here, staffing the flight with such an inexperienced first officer.

      1. JBird4049

        Setting aside Mr Guzzetti’s background (dismissing his claim here as tendentious right off the bat would strike me as uncharitable), and without wishing to exculpate anyone, it does lend some credence to the idea that Ethiopia Airlines may have some contributory negligence here, staffing the flight with such an inexperienced first officer.

        One can often point to inexperience, incompetence, stupidity, incompetence or just bad luck when some disaster happens, but Boeing counted on perfect performance from flight crews to successfully work with a workaround needed for other workarounds that needed perfect performance to not catastrophically fail. I know enough about complexity that you cannot depend on perfection because something will always fail.

  23. BillC

    Your excellent summary lacks some MCAS details that are not widely reported by the general-audience press.

    Like you, I am a retired software engineer, so I have followed an aviation blog discussion of this issue quite closely since it emerged as a probable software and system design failure. As the blog is open to all, its signal-to-noise ratio is pretty low, but it seems not too difficult for any technically-minded person to separate the wheat from the chaff. My current understanding, which I believe others here are in a position to correct, if necessary:

    A. The requirement for MCAS apparently emerged very late in the MAX’s development, when it became clear that the upper cowling around the larger engines, being moved up and forward with respect to earlier 737 versions, adds nose-up force as the angle of attack (AoA) approaches the upper limits of the MAX’s operating envelope because at such angles, the cowling itself generates lift beyond that of the wing.

    B. As perceived by a pilot flying manually (not on autopilot), this added nose-up force makes it easier to pull back on the control column (“stick”), increasing the AoA further. This is like a car running off the asphalt onto a muddy shoulder: the steering wheel wants to turn the wrong way (toward the ditch) rather than the right way (back on the road).

    C. An FAA regulation prohibits certification of an aircraft that presents the pilot with changing stick forces near stall that nudge the pilot toward the wrong reaction, 14 CFR 25.203(a), IIRC (unfortunately, I can’t find the original blog citation).

    D. MCAS was put in place to satisfy this certification requirement — not to automagically correct stalls without pilot action.

    E. Other means of meeting this requirement exist, ranging from an airframe redesign that avoids the extra nose-up effect of the larger repositioned engines down to a “stick pusher” that increases the force a pilot would need to pull the stick back further in this situation.

    F. Any of the other options would negate one or both of the MAX’s chief selling points: little cost or schedule impact to Boeing (in a rush to meet the Airbus 320 NEO challenge) and to its customers (“No new flight crew training necessary, because to the pilot, the MAX feels just like its 737 predecessors.”) That is, all the other options introduce new hardware to a completed design and the more fundamental changes could require new type certification.

    G. The easiest fix was pure software: at high indicated AoA, under manual control, and with flaps up, automatically rotate the horizontal stabilizer a little bit nose-down, which increases the pressure needed to pull the stick back (nose-up). No need to tell the pilot about this in training or real time, since it’s just to make MAX feel like any other 737.

    H. The design presented for certification described a single small rotation. Testing showed this was insufficient to provide the tactile feedback necessary for certification in all cases, so the software fix was obvious: if the trigger conditions still hold after a 5 sec. pause, do it again.

    I. Apparently nobody asked at that point, “What if the AoA indication is stuck high?” We’re under schedule and cost pressure, so who wants to complexify things by (1) adding additional sanity-checking to the aircraft’s AoA computations or (2) limiting how many times we add a little bit of nose-down.

    J. When these details combine with a consistently erroneous AoA reading, MCAS can — if not repeatedly countermanded or disabled and manually reversed — eventually rotate the horizontal stabilizer to its maximum nose-down position, where it was found in both recent incidents, IIRC.

    Even if the pilots figure out that’s what’s happening amid a cacophony of seemingly contradictory instrument readings and warnings (stick-shaker, trim wheel clacking, alarm chimes, and synthesized voices), the pilots still have to (1) cut power to the electrical trim systems and (2) restore the required trim, which may then require as many as 50 manual turns of a trim wheel. If you’re near the ground, time is short …

    A minority of commenting pilots assert that any competently trained cockpit crew should be able to identify MCAS misbehavior quickly and power off automatic trim per the same checklist that was prescribed for “runaway automatic trim” on every 737 variant, MAX included. Most seem to agree that with aircraft control difficulties, multiple alarms, and disagreement among the pilot’s and first officer’s airspeed and AoA readings almost from the moment of takeoff (not yet officially confirmed), an MCAS-commanded runaway trim event may feel very different from the runaway trim flavors for which pilots have had simulator training, making problem identification difficult even given knowledge of the earlier Lion Air incident.

    I imagine most software developers and engineers have seen cost/schedule pressures lead to short cuts. If their life was at stake, I doubt that many would think self-certification that such a project complies with all relevant safety requirements is a good idea.

    1. ShamanicFallout

      Thank you for that. And just ‘wow’. I don’t really know anything about aircraft/flying but this story is really fascinating and seems to be true a sign of the times. I guess we’ll know what the current ‘temperature’ is out there when the fallout (civil liability, criminal liability, plane orders cancelled/ returned, etc) manifests. If Boeing skates, we’ll know we’ve got a long way to go.

  24. Cheryl from Maryland

    The Post’s article on the FAA and Regulatory Capture is incomplete. The process for the FAA (and probably MANY government agencies) started under Reagan, did not revert to safety under Clinton (make government smaller and all that), and then accelerated under Bush II in 2005 (not a bi-partisan time). In particular, big changes to the FAA were made in 2005 that were executive in nature and did not require Congressional approval. CF:https://www.seattletimes.com/business/delegating-aircraft-safety-assessments-to-boeing-is-nothing-new-for-the-faa/

  25. drfrank

    Yes, but. Part of what we are seeing in this case is a rush to judgement based on less than full evidence and analysis, and so prejudices and ideological positions (which I share actually) are plainly to be seen (and perhaps worth analyzing). “Crapification,” says the headline.

    Yet, I cannot say that I disagree with BA’s business decisions as such in a highly competitive environment as regards the tradeoffs in the development of the MAX and there is a certain absurdity in the idea that Boeing would knowingly take a high reputational risk, in an industry where failure is front page news (contrast banking or pharma failures).

    I have no reason to believe that an FAA fully in charge of all aspects of certification would have prevented these crashes, as banking and drug regulators have not kept us safe either. What seems worthy of note is that neither the airlines that buy the product nor the foreign aviation regulators nor pilots’ associations do their own testing and certification, in an area where more redundancy would be good. Nor is there any kind of private third party watchdog testing, like a Moody’s or S&P, evaluating potentially toxic products and services for a price.

    Finally, I suppose we have to ask ourselves why the price of the stock is holding up fairly well even as the news flow on these tragedies is helping the short sellers. Lest we forget that Boeing is the 5th largest defense contractor in the US.

  26. oaf

    Is engine throttle automated in the flight regime where these accidents occurred? Or are the pilots controlling power? Is the lag in thrust response interacting with the MCAS in an unanticipated way? Aerodynamic lift of nacelles is mentioned several times; there is another lift factor relating to the thrust angle; which is not necessarily aligned with the fuselage axis in flight. Departure procedures often require speed limits and altitude changes; so it is likely multiple power demand levels get set through takeoff and climb until cruise altitude is reached. Does Autopilot/Flight Director integrate with MCAS; or are they independent systems? Even without touching flight controls; power changes affect pitch forces. I am wondering if consequences of manual power changes on an otherwise automated departure were adequately investigated in the certification of the MCAS. Please excuse my ignorance of these details.

  27. oaf

    Regulatory elements that have been getting attention include the use of *standard* weights for passengers; IIRC, 170 lbs for US (and possibly ICAO) passengers comes to mind…. Many aircraft accidents have an element of disregard for proper weight distribution, either accidental, or negligent. For instance: Tail-heavy bad! Intentional loading outside of subsequently approved C.G. and/or max weight limits is a common, if not ubiquitous part of determining certification limits.There is a safety factor in the certificated limits; but banking on this; using estimates; is proven risky or disastrous when actual weights, and distribution thereof, is uncertain. Cargo with false weight values could also occur. One might find incentive to claim lower weights than actual to save on freight charges. How many 170 lb passengers do you know? I am not familiar with scales being used to check aircraft weight and balance before takeoff; only calculations; based on…formulas and charts.
    Scales ARE USED during certain maintenance procedures; for airworthiness certificates; and following certain modifications.

  28. Jack

    Here is an interesting article by a professional pilot blogger Patrick Smith. He calls the 737, “the Frankenplane”, and traces its history all the way back to the 707 in 1959. According to Smith, “We wonder if the 737 MAX even needed to exist in the first place. Somewhere deep down, maybe the heart of this whole fiasco is Boeing’s determination to keep the 737 line going, variant after variant, seemingly forever. I’m not saying this is the reason for what happened in Indonesia or Ethiopia, but the whole 737 program just seems… misguided and unnecessary. Instead of starting from scratch with a new airframe, they took what was essentially conceived as a regional jet in the mid-1960s, and have pushed and pushed and pushed the thing — bigger and bigger engines, fancier avionics and more seats — into roles it was never intended for. The “Frankenplane,” I call it.
    See the article here.
    As a pilot myslef, I feel the airlines have a lot to answer for as well. Their constant “dumbing down” of pilots, which comes from making pilots work long hours for low pay, results in pilots not being the best of the best. And training is a cost to airlines. Training doesn’t result in revenue. Better to have the pilots actually flying, hence Boeing selling this new version of the 737 as not requiring further training. But, training and practice is everything in flying. Flying a plane is actually a relatively easy skill to acquire. Most people can learn to fly a trainer in 5 hours or so. Most people solo (fly the plane without an instructor) with only 10-20 hours of instruction. It takes a lot longer to learn how to drive a car for most people (45 hours is the average). So it really isn’t that difficult….until something goes WRONG. That is when the training kicks in. An often quoted flying truism, is that flying is “99% boredom and 1% stark terror”. What happened with these two crashes is that you had some inexperienced pilots who were not fully trained on the systems (a lot of that blame goes to Boeing). When things start going wrong, information overload can easily occur if you have not been properly trained, even with two pilots.

  29. allan

    “you had some inexperienced pilots”

    The captain, Yared Getachew, had more than 8,000 hours of flying under his belt.
    (It is true that the first officer only had 200.)

    You have to wonder how the average US commercial pilot would have done under the circumstances.

    (Reply to Jack at 11:50 am)

    1. EoH

      Thanks for that correction. We can expect a deluge of blame-the-other-guy PR from the aircraft manufacturer and certification agencies. Billions are on the line for Boeing if a cascade of judgments it made materially contributed to these crashes. The usual strategic corporate bankruptcy might follow. I presume Boeing is considered much TBTF by the USG.

      1. Carey

        Too bad our deeply indebted students and children, too young and naive to know what they’re getting into, and *what monsters they’re dealing with*, do not have that option.
        Thanks, Uncle Joe™!!!!!

  30. JerryDenim

    Great job summarizing and connecting dots Lambert. I might add one more bullet point though. Items #5 and #6 were aided, abetted and perhaps somewhat necessitated by ‘ye ole NeoLiberal playbook’ you spoke of, but more specifically, the current regulatory FAA/Boeing milieu is attributable to years of budget cuts and strategically applied austerity. The old Grover Norquist, ‘… not destroyed, but small and weak enough to be drowned in a shallow bath’ saw. Exact same thing we’ve witnessed with other formally effective regulators like the EPA, the SEC or the IRS.

    I remember having a conversation with an FAA maintenance inspector, an old timer, about ten years ago. He looked to be upwards of seventy, and he told me he was eight years beyond eligibility for a full retirement. He informed me that a few years back he was supervising a team of ten people that was now down to two. Their positions had been cut outright or eliminated after they resigned or transferred when the remaining positions were made miserable by the increased workload and bureaucratic headaches. The inspector said he had not retired yet because he knew he would not be replaced and he felt the work was important. I asked him if his department was atypical and he said it was not. Same thing, across the board, with the exception of the executive level desk jobs in DC and Oklahoma City. Readers can draw their own conclusions but when it comes to funding Federal regulators, I believe you should never attribute anything to incompetence that you could attribute to malice.

    No doubt Neo-Liberal ideologues in high places pushing the corrosive “customer/client” model of regulating along with the requisite deference and obsequious to industry played a large role as well.

    “Chickens coming home to roost” Indeed.

  31. EoH

    I understand the published materials to boil down to this possible scenario:

    To remain competitive and profitable, Boeing needed to improve the fuel efficiency and flight characteristics of a mainstay medium-haul aircraft. Instead of designing a new aircraft, it modified an existing airframe. Among other changes, it added more powerful engines, new lift and control surfaces, and enhanced computerized controls.

    The modified Max aircraft **did not** fly like the earlier version. That meant Boeing would have to disclose information about those changes. It would need to train pilots in them, in how to integrate new protocols into existing ones, and in what to do if the enhanced computer controls malfunctioned, requiring the pilot to regain manual control.

    These steps could have increased cost and time to market, might have involved new certifications, and might have reduced sales. Boeing appears to have relied on enhanced computer flight controls to avoid them.

    The newly enhanced computerized controls meant that the computer would do more of the actual flying – the part that was different from the pre-Max version – and the pilot less. It gave the pilot the virtual – but not real – experience of flying the older aircraft, obviating the need, in Boeing’s judgment, for additional disclosures and training. That worked except when it didn’t. (See, driverless car development.)

    One possible failure mode derives from the Max’s reliance on a single sensor to detect its angle of attack, the aircraft’s nose-up or nose-down deviation from level flight. Reliance on a single sensor would make it harder to detect and correct a fault. (Boeing’s version of commitment to “absolute” safety.)

    In these two crashes, the sensor may have given a faulty reading, indicating that the aircraft’s nose was higher than it should have been for that stage of flight, an attitude that risked a stall. The programmed response was to drop the nose and increase power. A normal reaction to a real stall, this response can become catastrophic when unexpected or when the pilot cannot correct it.

    In both crashes, it appears that the pilot did attempt to correct the computer’s error. Doing so, however, reset the automated control, leading the computer to reread the faulty sensor to mean “stall.” It again dropped the nose and increased speed. The pilot recorrected the error in what would become a deadly loop, a tug of war that ended in a powered dive into the ground.

    1. EoH

      So much for that “absolute safety” choice not to inform or train pilots about what its new automated flight control system would do, apparently regardless of pilot input. From the Guardian today:

      The pilots of the Lion Air Boeing 737 Max that crashed in Indonesia were searching a flight manual to try to find why the plane kept lurching downwards against their commands, according to reports of the cockpit voice recording….

      The final report into the Lion Air crash could be released by July, Indonesian investigators said. The cause has not been determined, but the preliminary report highlighted Boeing’s MCAS system, faulty sensors, and the airline’s maintenance and training.

  32. VietnamVet

    What is interesting is what comes next. The FAA was drowned in the bath tub along with the EPA, FDA, SEC, etc. It doesn’t have the money or staff to recertify the 737 Max. An incompetent Administration that is interested only in extracting resources is in charge. It is clear that Boeing hid the changes to save money and time. Adding a warning indicator that the flight sensors are not in the correct position to the pilot’s display, including it in the preflight checklist, plus flight training would have prevented the Indonesian crash. But these changes would have raised questions on the adequacy of the new flight critical system and may have delayed certification overseas. It is easy to overlook problems if your paycheck is at risk. The Boeing managers who pushed this through deserve jail time for manslaughter.

    Canada said it will recertify the 737 Max before it flies in their airspace. China won’t recertify the Max until the Trump Trade War is over. Also, a delay boosts their replacement airliner. If Chicago and DC paper this over like the 2008 Great Recession; the final nails will have been hammered into the coffin of the hegemon. Trust is gone.

  33. John Beech

    Much has been made of self-certifying. Frankly, for company like Boeing, hanging redesigned engines on the 737 is duck soup, easy as pie, pick your idiom.

    Regarding the improved engine being moved forward and slightly upward, according to MAX pilots that’s not that big a deal. The extra 10k lbs of thrust a side is nice and when the bird when light it’s a rocketship and especially fun to fly.

    What has been mentioned by several is the possibility of the pilot (or co-pilot inadvertently hitting the TOGA paddles (take off/ go around) and getting a massive surge of thrust, which brought MCAS into play – doing exactly what it’s supposed to do, which is ease the nose down automatically because the thrust vector is raising it with max power.

    The analysis from the black box with be illuminating. Meanwhile, have you seen the Bloomberg report the flight before had a jumpseat pilot who had to get involved when the exact same thing happened? https://www.bloomberg.com/news/articles/2019-03-19/how-an-extra-man-in-cockpit-saved-a-737-max-that-later-crashed . . . yikes!

    Anyway, nice article overall, well done Lambert.

  34. EoH

    Regarding the crashed Lion Air flight, the command pilot was struggling to regain control of his aircraft, fighting the computer controls that were overriding his attempts. While doing that, he asked his co-pilot to search the Quick Reference version of the flight manual, looking for reasons why he could not regain control.

    Boeing subsequently admitted that the manual **did not include any reference** to the existence of the malfunctioning system, Maneuvering Characteristics Augmentation System, MCAS, **nor had the pilots been made aware of it** during training to prepare them for switching to the new model 737 from the previous model.

    [Emphasis added.] Was the flight manual with that missing reference restored an extra-cost option, did Boeing assume that its technology could never fail or that the pilots were unnecessary cargo?

    It is difficult to see how any explanation for that and for the extra-cost only option of additional angle-of-attack sensors suggest a policy of “safety,” let alone an “absolute commitment” to it. They suggest a lack of routine foresight and that business concerns were dramatically higher priorities.

    [https://www.thedailybeast.com/boeing-737-max-8-scandal-grows-doomed-lion-air-flight-should-never-have-flown]

Comments are closed.