Software errors could have destroyed Boeing Starliner, NASA says
Two software errors detected after the launch of a Boeing Starliner crew ship during an unpiloted test flight last December, one of which prevented a planned docking with the International Space Station, could have led to catastrophic failures had they not been caught and corrected in time, NASA said Friday.
An independent review board "found the two critical software defects were not detected ahead of flight despite multiple safeguards," according to an agency statement. "Ground intervention prevented loss of vehicle in both cases."
In a teleconference with reporters, Douglas Loverro, director of spaceflight operations at NASA Headquarters, said the issues uncovered by the investigators go beyond the specifics of the software errors and an unexpected communications glitch that initially prevented flight controllers from commanding the spacecraft.
"Too put it bluntly, the issue that we're dealing with is that we have numerous process escapes in the software design, development and test cycle for Starliner," he said. The errors themselves "are likely only symptoms, they are not the real problem. The real problem is that we had numerous process escapes" that allowed the errors to slip through.
The Starliner software is made up of a million lines of code and "as we go forward, that is what we're going to be concentrating on, how do we assure ourselves that all of the software that we've delivered, not just the two routines that were affected by these issues, are fixed."
"Our NASA oversight was insufficient," Loverro concluded. "That's obvious, and we we recognize that. And I think that's good learning for us. The independent review team didn't just have recommendations for Boeing, it's got recommendations for us as well, and we're going to take all those to heart."
Neither Loverro, NASA Administrator Jim Bridenstine nor Boeing Starliner project manager John Mulholland would speculate on whether a second unpiloted test flight might be ordered or even whether a Starliner, piloted or unpiloted, would fly this year. No such decisions will be made until after the safety review concludes at the end of the month.
"We will not speculate right now on a specific launch date," Mulholland said. "What we have to do is fully understand the scope of the corrective actions, implement that into a work plan. Once we get that scope defined ... we'll be able to evaluate a specific launch target."
The Boeing CST-100 Starliner was launched from Cape Canaveral atop a United Launch Alliance Atlas 5 rocket on Dec. 20. The goal of the flight was to put the commercial crew ship through its paces, from launch through rendezvous and docking with the space station to re-entry and splashdown, to clear the capsule for a piloted test flight.
The Atlas 5 put the Starliner onto a sub-orbital trajectory as planned. After release from the rocket's Centaur second stage, the spacecraft was expected to fire its own thrusters to put the craft into a safe orbit. But the critical orbit insertion rocket firing never happened, and the Starliner continued along a trajectory that, without quick corrective action, would have resulted in a catastrophic unplanned re-entry.
After struggling with communications glitches, engineers finally managed to regain control and put the spacecraft in a safe orbit. But by then, too much propellant had been wasted to press ahead with a planned rendezvous with the International Space Station. Instead, flight controllers focused on carrying out as many other tests as possible before bringing the ship down for landing in New Mexico two days later.
The Starliner's failure to execute the orbit insertion burn was blamed on software that incorrectly set the spacecraft's internal clocks based on data retrieved from the Atlas 5's flight control system. The Starliner code should have retrieved the time during the terminal countdown, after the Atlas 5's clocks were precisely set for launch.
Instead, the Starliner computer retrieved the time used during an earlier countdown sequence and as a result, its timer was 11 hours off from the actual time. That, in turn, threw off the timing of downstream post-launch events, including the orbit insertion burn.
With that problem finally corrected, engineers began reviewing other critical software sequences as a precaution and discovered yet another problem. Software used to control thruster firings needed to safely jettison the Starliner's service module just before re-entry was mis-configured, set for the wrong phase of flight.
Had the problem not been found and corrected, the cylindrical service module's thrusters could have fired in the wrong sequence, driving it back into the crew module and possibly triggering a tumble or even damaging the ship's protective heat shield.
While a detailed analysis was not carried out at the time, "nothing good can come from those two spacecraft bumping back into one another," said Jim Chilton, a senior vice president for Boeing Space and Launch.
The timing problem was widely known during the Starliner test flight, but the service module issue was not revealed in any detail until a meeting of the NASA Aerospace Safety and Advisory Panel Thursday, setting off widespread social media calls for more information and "transparency" from NASA's Commercial Crew Program.
NASA responded with the on-line statement and media teleconference Friday.
"It is very unusual for NASA to do a press conference about what the investigation results are as the investigation is underway," Bridenstine said. "But in the interest of transparency and, you know, some of the things that I saw online yesterday, I wanted to make sure that everybody knew kind of where we were in the investigation."
Engineers are still investigating what caused the communications glitches that initially prevented flight controllers from quickly correcting the timing issue. As it turns out, Mulholland said high background radio noise, possibly from cell phone towers, may have played a role.
In any case, "software defects, particularly in complex spacecraft code, are not unexpected," NASA said in its statement. "However, there were numerous instances where the Boeing software quality processes either should have or could have uncovered the defects.
"Due to these breakdowns found in design, code and test of the software, they will require systemic corrective actions. The team has already identified a robust set of 11 top-priority corrective actions. More will be identified after the team completes its additional work."
Said Mulholland: "Nobody is more disappointed in the issues that we uncovered ... than the Starliner team. But to a person, they're committed to resolving these issues in partnership with NASA and the IRT and safely returning to flight."
Since the space shuttle's retirement in 2011, NASA has been forced to buy seats aboard Russian Soyuz spacecraft to ferry U.S. and partner astronauts to and from the International Space Station.
To end sole reliance on Russia for transportation to and from the space station, NASA announced in 2014 that Boeing and SpaceX would share $6.8 billion to develop independent space taxis, the first new U.S. crewed spacecraft since the 1970s.
Under a $2.6 billion contract, SpaceX is building a crewed version of its Dragon cargo ship that will ride into orbit atop the company's Falcon 9 rocket. Boeing's Starliner is being developed under a $4.2 billion contract.
SpaceX carried out a successful unpiloted flight to the space station in March 2019, but suffered a major setback the following April when that same Crew Dragon capsule was destroyed during a ground test. The California rocket builder has recovered from that incident and carried out a successful in-flight abort test in January.
It is widely expected that SpaceX will be ready to launch a Crew Dragon carrying two NASA astronauts — Douglas Hurley and Robert Behnken — sometime this spring.
Boeing's unpiloted test flight in December was only partially successful because of the two software errors and the communications glitch. It's not yet known whether NASA will order a second unpiloted test flight or whether Boeing will be cleared to press ahead for a piloted mission after corrective actions are implemented.
"It's still too early for us to definitively share the root causes and full set of corrective actions needed for the Starliner system," NASA said. "We do expect to have those results at the end of February, as was our initial plan.
"Most critically, we want to assure that these necessary steps are completely understood prior to determining the plan for future flights. Separate from the anomaly investigation, NASA also is still reviewing the data collected during the flight test to help determine that future plan. NASA expects a decision on this review to be complete in the next several weeks."