DIAGNOSTICS VALIDATION & VERIFICATION

Introduction


This section describes the ability and process to exercise a run-time diagnostics knowledge base (DKB) in a simulated run-time environment. The Diagnostics V&V tab provides tools which assist you in simulating fault events and determining diagnostic capability provided by the run-time Diagnostician. These tools help to verify and validate the diagnostic capability provided by the combination of the Diagnostic Knowledge Base and the Diagnostician. The Maintenance Exerciser lets you define single or multiple fault events, simulate the events, and view the runtime Diagnostician in action as it reads test results, selects the optimum next test to be executed, reads those results, until it derives the best fault call-out possible. You can see what test result data was used by the Diagnostician, and view either the Repair action, the Repairable Item call-out, or the pin-level fault call-out. Another way to use this tool is to allow it to run in batch mode. In batch mode, the tool will run all fault events and accumulate fault isolation resolution statistics, and will also generate a log file which profiles the diagnostic results of each simulated fault event.

What is the Diagnostician?


The Diagnostician is a run-time software system that provides a fault isolation call-out based upon reading the results of tests. This Maintenance Exerciser allows you to exercise the run-time behavior in simulation mode by allowing you to simulate fault scenarios and test scenarios. The simulation demonstrates how the Diagnostician interacts with test results to isolate faults.

What is the Software Structure of the Diagnostician?


The Diagnostician is structured as a library of functions. This library of functions is compiled to different operating systems. For example, for Windows 3.x, Windows 95 and Windows NT, the Diagnostician is compiled as a Dynamic Link Library. This allows for tremendous flexibility for integration onto actual run-time platforms and software systems. Key run-time features of the Diagnostician include:

1. Test results can be entered into the Diagnostician from any source, in any order, and as many or few at a time as the application warrants. This tremendous flexibility is provided by the internal structure of the model that is used for diagnostic reasoning - the model is in a matrix format, as opposed to a "fault tree" structure.
 
2. The Diagnostician uses both pass and fail cones of evidence to obtain fault isolation call-out. It also includes a minimum set covering algorithm to speed processing and improve diagnostic resolution.
 
3. The flexibility of the internal structure enables operating modes that fit into a broader set of operating environments than typical expert diagnostic systems, especially in automatic test equipment and embedded applications, which typically involve reading of large sets of test result data provided in an instance, as opposed to one test result at a time which is used to flow through a static test tree or fault flow diagram.
 
4. The model structure and the algorithms allow for multiple independent and multiple dependent, or common mode failures to be isolated.
 
Use of the Maintenance Exerciser for Validation and Verification of Test Programs


One of the most time and resource consuming aspects of test program set development is the validation and verification of test programs. TPS V&V is normally accomplished at the end of TPS development through a process of physical insertion of faults on a UUT and running of the TPS on the test system to verify that the fault is detected and isolated by the test program to the specified ambiguity group size. Thus TPS V&V requires 1) the completed test program, 2) the unit under test, 3) test equipment assets.

The Maintenance Exerciser can be used to validate and verify test program diagnostics: This approach:
 
- allows for test program V&V early in the TPS development process
- decreases or eliminates the need for test equipment assets
- decreases or eliminates the need for the unit under test
- eliminates the need for physical insertion of faults into the UUT

This approach is based upon uncoupling the diagnostic logic associated with a test program from the actual running of the tests. By separating the diagnostic logic from the test program, the diagnostic logic can be verified independent from the test program, therefore not requiring either physical test assets or the UUT.
 
In this approach, the diagnostic logic and the ability of specified tests to isolate to specific ambiguity groups is verified using a model-based reasoning capability. A structural model of the UUT is transformed into a model which maps the propagation of all faults to circuit locations where they are observable, and 2) maps the structural coverage of test locations, or specified tests. In using the Diagnostic Profiler, you define tests to be performed and test locations associated with test outputs. As you performs this process, you have on-line access to statistics on fault isolation and ambiguity group size and composition. You can also print out test specification reports showing the structural and functional coverage requirements of each specified test.
 
After defining and mapping the functionality of the tests onto into the model base, the model base is converted to a run-time diagnostic knowledge base (DKB). At this point, the DKB is a representation of: 1) the UUT's inherent fault propagation characteristics, and 2) the fault coverage of defined tests.
 
The Maintenance Exerciser is used to verify the run-time diagnostic capability of the test program. The Maintenance Exerciser consists of two components: the Fault Propagation Simulator and the DKB Exerciser. The Fault Propagation Simulator (FPSIM) allows the user to simulate the insertion of a fault at any physical point, or any failure modeled failure mode in the circuit. FPSIM generates the test results, for the specified tests, that correspond to the existence of a fault at the selected location. These are output to ASCII test results files which are interpretable by the second component, the V&V Exerciser. The V&V Exerciser reads the Diagnostic Knowledge Base, and the test results, and based upon these determines the current ambiguity group. The Exerciser uses the actual Diagnostician run-time algorithms to perform fault isolation, therefore providing a very accurate representation of run-time. If desired, you can then simulate repair of the item in that if the test program includes branches to external programs or modules which display repair procedures, the specific repair procedure can be displayed, further verifying the integrity of the test program.
 
This process of simulated fault insertion and resultant fault isolation can be run for random fault selections and the percentage of fault isolation to the required ambiguity group size calculated and compared to specifications.
 
Thus, a total diagnostic simulation/validation process is effected: (1) insertion of a fault and the generation of simulated pass/fail test results, (2) simulation of the diagnostic reasoning process (diagnostic logic) in response to the pass/fail data, and (3) declaration of a fault call-out (i.e. which component or components caused the simulated failure).
 
Using the Maintenance Exerciser


To initiate the Maintenance Exerciser:

1. Double-click on the Maintenance Exerciser tool in the tool list on the Diagnostics V&V tab. The Maintenance Exerciser tool screen will appear, with the run-time DKB of the current project and candidate loaded. This screen is shown below.
 

You must define fault events to be simulated and test scenarios before running any simulations.

 
Define Failure Events


You must define failure events before using the Maintenance Simulator. To define failure events:

1. Press the Failure Events pushbutton at the upper left side of the Maintenance Simulator screen. The Failure Events definition screen will appear, as shown below.
 
Failure events are defined at the fault location or failure mode level. The list of failure events that have been already defined for the loaded model are listed in the leftmost window. The upper-middle box contains a list of all of the parts in the currently loaded diagnostic knowledge base.

2. Highlight a part (component) from the upper-middle list, the Part List, and press Select Part. This will cause the fault list associated with the selected part to be listed in the middle lower list box.
 
3. Highlight one or more pin-level faults and press the Add Selected Fault button. The selected fault(s) will appear in the rightmost list box.
 
Note: a fault event can include multiple independent faults. Simple add additional faults to the Selected Fault Locations List.
 
4. Save the fault event. Point and click on the box at the top of the Failure Events list. Type in a name describing the failure event that you have just defined, and press the Save an Event button. The event will now be saved and will appear in the Failure Events List.
 
5. You can delete failure events and retrieve failure events using the so-named buttons.
 
6. To return to the Maintenance Simulator screen, press the Maintenance Simulator button.

 
Define Test Scenarios


During Diagnostic Profiling, you defined a number of tests that may be applied to the design in run-time. The definition of these tests become an intrinsic part of the Diagnostic Knowledge Base. Several different test scenarios may be applicable to run-time environments. One of the functions of the Maintenance Simulator is to evaluate test effectiveness of various test scenarios.

To define a Test Scenario:

1. Press the Test Scenario button from the main screen. A new screen will appear, as shown below. At the leftmost window is the list of Test Scenarios that have been already defined for the current DKB. The middle box contains a list of all of the tests that have been defined in the current DKB.
 

2. To define a Test Scenario, highlight those tests which you wish to include in the scenario from the middle box, and press Add Selected Tests. The selected tests will appear in the rightmost list box.
 
3. Save the test scenario. Point and click on the box at the top of the Test Scenarios list. Type in a name describing the test scenario that you have just defined, and press the Save Scenario button. The event will now be saved and will appear in the Test Scenarios list.
 
4. You can delete test scenarios and retrieve test scenarios using the so-named buttons.

5. To return to the Maintenance Simulator screen, press the Maintenance Simulator button.

The purpose of having different test scenario's is to evaluate, on a fault-by-fault basis, the diagnostic capability that results from having different test resources available. Trade-offs on use of various levels of test capability can be clearly evaluated.
 
Simulate a Diagnostic Scenario


To simulate a diagnostic session based upon a specific fault event and test scenario:

1. Select a Fault Event from the list by clicking on that event
 
2. Select a Test Scenario from the list by clicking on that scenario.
 
3. Press Simulate.

The status line will indicate the processing that is being performed. First the fault scenario is simulated, next the simulated test results are fed into the Diagnostician.

Note that at the right side of the screen there are several pieces of information which indicate the current and resulting diagnostic resolution of the scenario.
 
4. The results of the simulation are shown at the right side of the screen. The following information is available:
 
"Suspect Count" Reflects the number of replaceable parts in the current ambiguity group. You may note that this number is zero (0) until a fault is detected, at which point it goes to a relatively high number and decreases as test results are interpreted by the Diagnostician.
 
"Test Count" Reflects the number of test procedures that have been executed during the diagnostic session to arrive at the fault call-out. Note that the Diagnostician reads only those tests results associated with tests that have diagnostic significance.
 
In the rightmost window, you can display four different pieces of information related to the details of the diagnostic session:
 
"Suspects" Displays a list of the current (or final) suspect replaceable parts. Next to each item in the list is a percentage in parenthesis. This percentage represents the probability that that item is the faulty item in the ambiguity group. The probability is based on both part failure rate and the amount of evidence (supporting test results) indicting the part.

"Faults" Displays a list of the current (or final) suspect faults (pin level).
 
"Tests Used" Displays the names of the test procedures which have been run (simulated). Note: the Diagnostician optimizes the fault isolation process by requiring only those tests that have diagnostic significance given the current fault.
 
"Measurements" Displays the specific Pass/Fail test results interpreted by the Diagnostician for each measurement location of each test procedure that has been used by the Diagnostician.
 
You can toggle among these choices either before the diagnostic simulation starts (before clicking on Simulate) or after the diagnostic simulation has been completed.
 
Four of the Diagnostician's operating modes can be modified for the simulation. These include:
 
"Diagnose to Fault" Continues diagnostic session seeking to isolate to the lowest pin-level fault, as opposed to seeking to isolate to a replaceable repair item, normally a part. Normally, the Diagnostician ceases the diagnostic session when a fault has been isolated to a single repair item.
 
"Traverse Hierarchy" Causes Diagnostician to treat the current model as a hierarchical model, in other words, to traverse to lower level child models. If lower level models are not available, this mode should not be used, as inconsistent/incorrect results may occur.
 
"Use Go-Sequence" Causes Diagnostician to Use the Go-Sequence as a group of initial tests to be performed. The Go-Sequence is defined by the user using the Go-Sequence button at the top of the main screen. This is a way to define a set of tests as a functional test.
 
"Stop Diag on Setup" After the Go-Sequence has been performed, if the Diagnostician sees a change in the Set-Up field (a run-time linkage), the Diagnostician will discontinue the diagnostic session. (This allows the you to restrict available diagnostic routing associated with test procedures.)
 
"Go Sequence" This button causes the display of another screen, shown below, which allows you to define a Go/No-Go test sequence. It presents you with a list of all tests that have been defined for the loaded knowledge base, and lets you define which tests to be included in the Go-Sequence as well as the specific order of the tests.

 

Log Diagnostic Session


The Diagnostic Session can be recorded in a log file. This enables you to keep a record of diagnostic sessions across various fault scenarios. The Log File produced is in standard ASCII format, and can be imported to most word processors and reformatted as desired. The Log File will be generated in the subdirectory location where the project's candidate data is stored. This is normally in \Profiler\Projects\Project Name\Candidate Name. The log file is named "VV.LOG."

To Log a diagnostic session, press the LOG pushbutton at the lower right corner of the Maintenance Exerciser screen. You must press the LOG pushbutton each time you want the current session to be logged. The data to be logged for each session is appended to the log file.
 
The Log file identifies for each logged session, the following information
 
<SESSION> The name of the run-time DKB loaded during the diagnostic session
<FAULTS> Identification of the fault(s) simulated in the current diagnostic session.
<TESTS> The tests for which test results were used, and the order in which the tests were requested by the Diagnostician.
<SYMPTOMS> Those test's measurements which resulted in failed test result data.
<SUSPECTS> The fault call-out or ambiguity group which the Diagnostic Session resulted in.
<END> End of logged data for current simulated fault.
 
A sample of this data for several logged sessions is provided below.

<SESSION>
DKB=JAST.dkb
<FAULTS>
U1.2
<TESTS>
Functional Test 1
Test Point 3
Test Point 2
Test Point 5
<SYMPTOMS>
[Functional Test 1]T01
[Functional Test 1]T02
[Functional Test 1]T03
[Test Point 2]TP2
[Test Point 3]TP3
[Test Point 3]TP4
[Test Point 5]TP5
<SUSPECTS>
U1 ( 100% )
<END>

<SESSION>
DKB=JAST.dkb
<FAULTS>
U14.4
<TESTS>
Functional Test 1
Test Point 2
Test Point 1
<SYMPTOMS>
[Functional Test 1]T01
[Functional Test 1]T03
[Test Point 2]TP2
<SUSPECTS>
U16 ( 66.7% )
U14 ( 33.3% )
<END>

<SESSION>
DKB=JAST.dkb
<FAULTS>
U19.3
<TESTS>
Functional Test 1
Test Point 1
<SYMPTOMS>
[Functional Test 1]T03
<SUSPECTS>
U19 ( 75% )
U16 ( 25% )
<END>

<SESSION>
DKB=JAST.dkb
<FAULTS>
U3.2(Fail High)
<TESTS>
Functional Test 1
Test Point 2
Test Point 1
<SYMPTOMS>
[Functional Test 1]T01
[Test Point 1]TP1
[Test Point 2]TP2
<SUSPECTS>
U3 ( 100% )
<END>

<SESSION>
DKB=JAST.dkb
<FAULTS>
U16.3
<TESTS>
Functional Test 1
Test Point 2
Test Point 1
<SYMPTOMS>
[Functional Test 1]T01
[Test Point 2]TP2
<SUSPECTS>
U16 ( 100% )
<END>

<SESSION>
DKB=JAST.dkb
<FAULTS>
U12.3
<TESTS>
Functional Test 1
Test Point 2
<SYMPTOMS>
[Functional Test 1]T01
[Functional Test 1]T03
<SUSPECTS>
U12 ( 33.3% )
U15 ( 25% )
U7 ( 16.7% )
U9 ( 16.7% )
U11 ( 8.3% )
<END>