7-Step Method for Electronic Troubleshooting
Ken Reed System Improvements, Inc. 238 South Peters Road, Suite 301 Knoxville, Tennessee 37923 USA Phone: 865-539-2139 Fax: 865-539-4335 e-mail:
[email protected] web: www.taproot.com Copyright © 2007 by System Improvements, Inc. All Rights Reserved. Duplication Prohibited.
1
Scenario #1 • You are enjoying a leisurely day by the pool. Man, it’s hot out here! • Your wife calls from the kitchen door. “The thermostat for the air conditioner conditioner is broke. broke. It’s getting hot in here!” • Consequences: Stress and discomfort
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
1
Scenario #2 • You are the maintenance supervisor at a coal mine. You must move move 600 tons tons / hr out out of the mine. • You get a call from from the control room. room. “The conveyor belt belt is down down again. It just shut off.” • You have about 1 hour before the hopper is empty and the rail cars are no longer being filled. • Consequences: $400,000 / hr. ©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
Scenario #3 • You are the Reactor Control Chief on a nuclear submarine. submarine. You awaken awaken when when you hear all the ventilation fans turn off, and a loudspeaker announcement, “Reactor Scram.” • The ship is currently operating under the north polar ice, with no place to surface within 6 hours. • You have 3 ! hours left on battery power. • Consequences: Real ugly ©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
2
Which is worse?? Fix it NOW!!!
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
Ken Reed
Senior Associate System Improvements ©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
3
Who am I? • 21 years Nuclear Submarine Experience – Retired Master Chief Electronics Technician – Served on: • Ballistic Missile Boat • Fast Attacks • Nuclear Research sub
– Last boat – USS SEAWOLF ©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
Electronic Troubleshooting • Topics: – Electronic vs Other Equipment – Before you start… – Troubleshooting Methods – Organizing the Team – Seven Step Process
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
4
Note About This Presentation • I am going by my experience, which often dictates a worst-case “must fix” scenario • If your needs are not this critical, take what you need from what you see today
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
Troubleshooting Differences Mechanical • Easier to see system • Can put your hands on it • Not much calculation • Often, small number of alternatives
Electronic • Hard to visualize what a part does • Black boxes, miniaturized • May require extensive calculation • Many directions to go
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
5
Troubleshooting Similarities • • • •
Requires knowledgeable troubleshooters Systems becoming more complex Lines blur between mechanical and electronic Must always look at entire system interaction
• Operators are ALWAYS breaking the gear!
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
Why Does Troubleshooting Fail?? • • • • • •
No formalized process in place No one in charge and looking at the Big Picture No one knows initial conditions No one is sure what has already been done Everything performed in series Pre-conceived idea of what caused problem
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
6
Before you start… • Specify the goals of the troubleshooting • Have plan in place to: – – – –
Gather initial data Notify team Preserve the evidence / indications Record troubleshooting progress / results
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
Organizing Your Team • Man in charge must be: – Designated and agreed upon – Equipment expert, but not necessarily the “guru” – Able to interface with team and management – Able to see “big picture. Do NOT get pulled into “weeds” of troubleshooting effort – Knowledgeable of entire system interactions – Organized and firm – Able to multi-task ©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
7
Organizing Your Team • Team may have all or some of the following: – Expert on the equipment that has failed – Equipment operator • May be the one operating the gear when it failed
– Parts procurement – Data entry / extraction / computer guy – LO/TO guy – Runner (“gopher”)
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
7-Step Process General Comments • Write down everything. Do not depend on your memory.
• Get initial data immediately.
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
8
In Parallel: •
While waiting for initial steps to be completed, what can be done? • • • •
Gather maintenance history Get technical manual Obtain test equipment Immediately get observed symptoms from the operators
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
7-Step Process 1. 2. 3. 4. 5. 6. 7.
Symptom Recognition Symptom Elaboration List of Possible Faulty Functions Localize the Faulty Function Localize the Faulty Circuit / Component Failure Analysis Retest Requirements
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
9
1. Symptom Recognition
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
Immediate Action • Immediate discovery of the problem by operator and supervisor – Who knows? – Who should know? – Who must be informed? – Is equipment in a safe condition?
• Procedure to record problem at operator level – Is there a policy in place? ©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
10
Troubleshooting Team Arrives • This is much easier than you think • Only list what actually caused the operator to recognize that there was a fault
• Do not include: – Operating conditions – Other indications
• What are goals? – Repair? – Restore operation? ©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
2. Symptom Elaboration
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
11
Symptom Elaboration • Operating conditions at the time – Start-up, shut-down, etc – Power level, fluid levels – Special operations in progress – Recent maintenance – Anything different than normal – Exact sequence of switch operation
• Other symptoms noted:
Include anything you think might be pertinent.
– Alarms – Warnings – Blown fuses – Smells – Sounds ©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
Symptom Elaboration • Man in charge – Be ready to give options. – Can we operate in reduced status? • Lower throughput, less reliable, more likely to break something else, etc.
– Can we by-pass the failed module (permission?) – How long to implement each option? – Risks for each option? – How long can we operate with each option? – What additional safeguards should be put in place? ©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
12
Symptom Elaboration • Don’t operate anything yet • What permissions do you need to continue? • This step in the process is where many troubleshooting plans break down. – Not all indications are listed – Operator impressions / observations are not immediately obtained – Nothing is written down
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
3. Possible Faulty Functions
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
13
Possible Faulty Functions • List all major functional blocks that could cause the problem: Don’t – Power supply – Sensor – Amplifier – Etc
eliminate anything unless you can prove it!
• We are not yet looking for the one specific failed component ©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
Possible Faulty Functions • This is where Equifactor® can be used – Quickly look through “Symptoms” for problems available in the tables – Gives you ideas what possible causes you should look for – This is quick method of verifying what you think could be the cause – Makes you think of problems outside your experience ©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
14
Possible Faulty Functions • You may want to have a Troubleshooting Guide assembled for most frequent or critical failures – Allows you to quickly get to this point – Could be pulled directly from Equifactor®, or even maintained in an Equifactor® Custom Table – Often in quick, flowchart format
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
4. Localize Faulty Function
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
15
Localize Faulty Function • Actual troubleshooting • Permissions obtained • Write down expected before you take a reading – Prevents “seeing what you expect” – You’ll talk yourself into whatever you observe
• Write down EVERYTHING you do – May need to turn over – Will want to look at it tomorrow ©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
Localize Faulty Function • Man in charge – Be ready to give further options. – More options may be available, now that you know what major component has failed. • Replace major function without knowing what exact component failed? • What is risk of damaging new part without full failure mode known?
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
16
Localize Faulty Function
Equifactor® Tables may have ideas on how this type of fault was troubleshot last time
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
Functional Troubleshooting Methodologies • Linear INPUT Signal
Output Signal
– Test at each element – Left to right – Check input, output, power applied for each element
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
17
Functional Troubleshooting Methodologies • Half-split INPUT Signal
Output Signal Check Good
Check Bad
– Test each element – Start in the middle – Check input, output, power applied for each element ©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
Functional Troubleshooting Methodologies • Easter Egg INPUT Signal
Most – Used because: Popular?
Output Signal
• Not trained in any particular method • No plan in place, therefore, just get going • Not sure what has already been done
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
18
5. Localize Faulty Circuit / Component
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
Localize Faulty Circuit / Component • This is troubleshooting to component level • Troubleshoot down to lowest possible (expected) level • May require specialized test gear • Again, record everything you do
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
19
6. Failure Analysis
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
Failure Analysis • Prove to the team that the failure you found would cause ALL the symptoms you found. This prevents: – Replacing wrong component – Having multiple undiscovered faults – Talking yourself into your favorite failure
This is rarely done well
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
20
7. Retest Requirements
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
Retest Requirements • What retests will be required to prove the gear works? – Prove the new component is good – Prove there are no other associated faults – Prove you did not cause other damage during the troubleshooting or repair
• May be required to officially recertify the system
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
21
Retest Requirements • Correct retests are required to ensure reactor safety. – Contained in a variety of manuals. • Maintenance & Replacement Instructions. • Equipment Technical Manuals. • Equifactor® may provide guidance when a specific retest was used in the past • Can be researched and pre-approved. • Agreement on the correct retest requirements is a combined effort. ©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
Plant Relationships CAUTION: Actions taken during troubleshooting efforts can affect various plant capabilities.
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
22
Date:_________
TROUBLESHOOTING RECORD Trouble shoote rs -
1. Symptom Recognitio n _______________________________________________ _______________________________________________ _______________________________________________ ________________________________________ _______ _______________________________________________ 2. Symptom Elaboration : _______________________________________________ _______________________________________________ _______________________________________________ _______________________________ ________________ ___________________________________________________________ ___________________________________________________________ ___________________________________________________________
3. Possible Faulty Functions : _________________ _________________ _________________ __________________ _________________
! ! ! ! ! ! !
Maint. History PlantOperatingManual TechnicalManual RepairParts Retests Area se t-up Test Equipment
_ ___________________ _ ___________________ _ ___________________ ___________________ _ ___________________
List operational options: Options Lower reli ability? By-passinterlock? How long to implement? Permission required? List risks AdditionalSafeguards
4. Localize Faulty Function: Test Point
Expected
Actual
Sat/Unsat
______________________________________ __________________________ _________ ____________ ___________________________________________ ©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
5. Localize Faulty Circuit
:
Test Point
Expected
Actual
Sat/Unsat
___________________________ _____________________________________ ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ ___________________
6. Failu re Analysis
________ _________________
____________________
:
________________________________________________________________ ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ _________________ _________________________ __________________________________________
______________________ _____ _________________
___________________________________________________________ _____ _______________________________________________ __________ _______
7. Retest Requirements
:
________________________________________________________________ ________________________________________________________________ ______________________________________ __________________________ ________________________________________________________________ _______________________________________ _________________________ ________________________________ ______________________________ __ ________________ _____ ________________ ___________________________ _____________________ _____ ______________________________________ ___________________________________________________________ _____ __________________________
__________ ____________________________
©Copyright 2007 by System Improvements, Inc., Knoxville, TN. All Rights Reserved Worldwide. Duplication Prohibited.
23
