ACS Side-1 Anomaly Review Board Conclusions STScI TIPS Meeting

21-September-2006 Ken Sembach

Subset of conclusions as summarized in the final ACS Side-1 ARB Report (some items condensed / omitted for this presentation) 1

Anomaly Review Board Members

Roger Chiei, chairperson Randy Stevens, secretary Steve Arslanian Wilber (Dale) Brigham Ken Carpenter Ed Cheung Randy Kimble Olivia Lupie Mal Niedner Mike Prior Art Rankin Tim Schoeneweis Ken Sembach Beverly Serrano Noman Siddiqi Hsiao Smith Renee Taylor Tom Wheeler LMTO/Code 442 LM IS&S/Code 441 HTSI/Code 441 Code 563 Code 667/441 J&T/Code 442 Code 667/442 Code 581/441 Code 667/440 Code 441 J&T/Code 441 BATC STScI RSTX/Code 441 QSS/Code 562 Code 442 Code 303/442 STScI Consultants Chuck Harguth BATC Ken Albin James Simons Andy Hunt Lynette Marbley Henning Leidecker BATC BATC BATC Code 303/442 Code 562 2

•

Conclusions

• • • • • The following facts are based on on-orbit data at the time of the anomaly: • All CEB telemetry collected by the Normal Engineering Data (NED) task went from valid data to all ones within one sample period.

No indication of anomalous secondary voltages prior to ACS Suspend No indication of thermal rise prior to and following the anomaly Anomaly nearly instantaneous; occurred within 100 ms window Anomaly caused the +28V Bus current to decrease ~ 0.8 to 1.3 Amps No out-of-family behavior was observed in the science data prior to the Suspend event • The most likely cause of the ACS Side 1 on-orbit anomaly was a loss of the ACS Side 1 CEB +15V power rail in either an open or shorted condition. This conclusion is substantiated via • • • • engineering evaluations focused ground tests historical performance on-orbit telemetry collected at the time of the anomaly 3

•

Fault Tree Investigation: LVPS Board #3

Loss of CEB Power Rail:

• • • Both WFC and HRC CEBs supplied with power from common LVPS +5V, +15V and -15V source.

Loss of the +15V power rail, with or without degradation of the -15V power rail, results in uncharacteristic performance of CEB A/D conversion and associated serial data stream.

Ground testing of Engineering CEB unit resulted in reproduction of on orbit anomaly 4

• •

Most Likely Location of Fault

The ARB identified LVPS Board #3 as most likely location of the cause of the ACS on-orbit anomaly

• LVPS #3 contains all of the active components that produce and/or interface with the suspected +15V CEB power rail

However, other possible (less likely) locations exist:

• • • MEB Backplane: +15V trace to ground fault MEB to CEB harnessing: +15V wire short to ground fault Short on +15V Input to a CEB prior to power switching relay 5

Most likely location of root cause of anomaly

Possible Fault Locations

Cannot be rule out the possibility that the source of the anomaly may be located in this area LVPS#3 S/N 001 CEB PWR +15V Rail Short or Open -15V Rail +35V Rail MEB2 Backplane Short or Open 2 pins Short Cable Harness 538548 HRC CEB Side Select Relays 2 pins Same as +15V Same as +15V Short WFC CEB Side Select Relays 6 connector pins per power rail 6

Anomaly Review Board Investigated Possible Failures Case # Failure Mode Anomaly match based on ETU testing Implicates +5V Expected Telemetry +15V -15V +35V 1 All Nominal No Problem intermittent or +15V open beyond TLM sample point +5V +15V -15V +35V 2 +15V open (external)

YES

LVPS #3 Board +5V 0V -15V 0V 3 +15V open (internal)

YES MFL2815D

+5V 0V -18V to -20V 0V 4 or 5 6 7 8 or 9 -15V open (external or internal) +15V shorted (internal or external) -15V shorted (internal or external) MFL2815D inhibited or internally destroyed Fault within MFL2815D No MFL2815D or LVPS #3 +5V

YES

No MFL2815D or LVPS #3 or MEB Backplane or Harness to CEBs MFL2815D or LVPS #3 or MEB Backplane or Harness to CEBs +5V +5V No (additional testing in progress)

MFL2815D

Fault on LVPS #3 Board, not in MFL2815D +5V +15V 0V +8V to +11V 0V 0V -6V to -8V 0V 0V +35V 0V +30V to +35V 0V Fault mode matches on orbit anomaly signature 7

• • •

Conclusions

A thorough component-level Failure Mode Evaluation and Analysis (FMEA) was conducted by the ARB on the ACS LVPS3 Board circuitry related to the +/-15V CEB power rails. The FMEA results were compared to a set of voltage rail responses obtained via CEB Engineering Unit lab tests to determine if the postulated failure mode could have resulted in the ACS on-orbit anomaly. Based on the FMEA results, fourteen ACS LVPS3 board parts (nine distinct part types) and associated failure modes have been identified as possibly resulting in the ACS on-orbit anomaly.

• The most likely cause of the ACS failure resides within LVPS Board #3, component U2-7, the MFL2815D Interpoint DC-DC converter. The next most likely cause is a short to ground in the transformer T6 on the same LVPS board.

8

•

Conclusions

ARB quality record review • Identified an MEB level thermal cycle test failure that matched the on-orbit failure scenario exactly. A short occurred on the +15V rail at the +35V T6 step-up transformer.

• The ARB was concerned to find that after the initial replacement of transformer T6, the anomaly again repeated with the new transformer during re-testing with no further part removal and replacement. The ARB has no way to determine quantitatively what potential stress and possible latent damage may have occurred within the T6 transformer over the 3 minutes of heating that it was subjected to during the re test.

• The ARB was concerned to find that the flight LVPS3 board was subjected to uncontrolled heating effects by a heat gun as part of the anomaly investigation. It is conceivable that this anomaly investigation technique produced thermal stresses that resulted in a latent failure on the flight LVPS3 board in question.

9

•

Conclusions

ARB quality record review (continued)

• During ground test inspections, an investigation of the LVPS3 board showed that the board had separated between the Ground/Thermal Plane and the fiberglass board. The ARB cannot help but wonder if such a separation could have occurred that ultimately led to a shorting condition or a stressed component solder joint that resulted in the ACS on-orbit anomaly. Such separation could have been aggravated by thermal transitions. For example, ACS is powered down approximately once every month to perform anneal operations, at which time the LVPS board temperatures transition approximately 30 degrees with the current 6-hour anneals.

10

ARB Recommendations

• • • Modify ACS flight software so the NED history buffer contains 16-bit values exactly as read from the A/D FIFOs. Specifically, do not apply a bit-mask to the data prior to storage in the buffer. Storing the data in an unmodified form would allow better insight into the MEB/CEB interface. Make similar flight software changes to COS and WFC3. (NICMOS and STIS do not have NED history buffers.) New CARD items should be written to prevent invalid “hybrid” power configurations. These include: • • ACS only: prevent commanding of the “off” CEB ACS and STIS: prevent hot switching the CEB internal relays.

Investigate reducing thermal and power cycling by: • • • • Leaving the HRC CEB powered on while doing SBC operations.

Reducing the number of anneals Performing anneals in a “hybrid” mode that would leave Side-2 electronics powered while annealing from Side-1 Investigations have been referred to the Hybrid LEI Task and the STScI ACS Team.

11