Test Reliability Targets and Expected Field Reliability

by | Jul 22, 2022 | Blog

An example of an automotive validation test reliability target is R97C50. In this case, the objective of the validation effort would be to demonstrate at one life on test, with 50% confidence, that the product has a reliability of 97% or better.

Does this mean that a field failure rate of 3% can be expected?

To answer this, we should consider each of the following:

  1. What is the test severity relative to the distribution of stress in the field? If you could know the true field distribution of each important environmental stress factor or usage condition, this would be extremely valuable information. You would use these data to inform the parameters of your validation tests.

    However, this is much easier said than done. Some would argue that this is the most difficult challenge facing a validation engineer. It requires that field measurements be collected on representative parts installed on a representative application running the full range (i.e., representative) of customer usage conditions.

    An internal study conducted by a client revealed that this distribution was unknown in 60% of their product reliability warranty claims. Doesn’t it make sense that their validation test protocols were inadequate for revealing all field-relevant failure modes?

    Now let’s say you have some understanding of the field stress distribution. Would you apply this stress distribution to your test parts? Except for those unique industries whose end-user stress distributions are fairly narrow, the answer is no. To do so would result in a large amount of test time run under non-damaging test conditions. Instead, the conservative (and typical) approach is to test at high severity stress levels representing (e.g.) the 95th %-tile user. In many cases, in order to accelerate a test even further, the test is run at even higher stress levels with reliability estimations projected back to (e.g.) the 95th %-tile stress level using ALT or other quantitative methods.

    For some failure modes and some industries where safety is a consideration, a higher value of test stress severity (e.g., 98th %-tile or more) might be appropriate.

    I’ve found, however, that the choice of test severity is of little consequence if the field stress distribution is not well understood. Collecting representative field data requires a good deal of engineering rigor and resource, but for most products and applications this activity is vital.

  2. Does the reliability target apply to the overall test plan or a few critical tests? Validation test protocols often consist of several accelerated tests each designed to exercise a particular failure mechanism or mode*. Test reliability targets are sometimes intended for only these individual, reliability-critical test exposures. For these, a statistical demonstration of reliability and confidence is required. The remaining tests in the validation protocol can be considered capability tests. These are often short duration tests or tests of low risk or low occurrence environments, and often for mature products. Examples are dust testing, drop tests, shipping damage, connector strength, etc. These often have very small sample sizes, for example 3 to 6 parts.
  3. How was reliability demonstrated on test? Reliability demonstration by testing-to-failure has a number of advantages. It confirms that the observed failure mode(s) are as expected and align with the physics of failure model used to correlate/accelerate the test, it reveals the margin to one life (i.e., do parts fail just after one life or much later), and for the purposes of this discussion, it allows for an accurate estimate of reliability and confidence.

    Reliability demonstration using a success testing strategy however has as its premise that no failures are observed on test. As such, the reliability estimate is conservative irrespective of the stated confidence level. We can only say statistically that the combination of reliability and confidence are at least as high as stated.

Now let’s reconsider the original question. Does a demonstration of only R97C50 on test mean that we can expect 3% field failures? It does not. First, as described above, the spec may be directed at specific tests and thus only those failure modes. Second, though we may have only demonstrated 97% reliability on test, this was accomplished on a very high severity test. Finally, if a success test was used to demonstrate test reliability, this provides even more conservatism to what we can expect in the field.

Regarding test to field reliability conversion. There exist quantitative methods for translating test reliability to field reliability (and vice versa). These methods have limitations: They require that you know or assume a field stress distribution; and they apply only to that stressor and failure mechanism. See Kleyner1 for a good summary and example of one such approach.

  1. Kleyner, A. (2015) Effect of Field Stress Variance on Test to Field Correlation in Accelerated Reliability Demonstration Testing. Quality and Reliability Engineering International (QREI) 2015, 31:783–788 DOI: 10.1002/qre.1635