Measurement Systems Analysis - Appendices

 

Measurement System Analysis How-to Guide - Appendices Version 6.1  August 2013

 

 

MSA How-to Guide

 © 2013 Rolls-Royce Rolls-Royce plc  The information in this document is the property of Rolls-Royce plc and may not be copied or communicated to a third party, or used for any purpose other than that for which it is supplied without the express written consent of Rolls-Royce plc.   This information is given in good faith based upon the latest information available to Rolls-Royce plc, no warranty or representation is given concerning such information, which must not be taken as establishing any contractual or other commitment binding upon Rolls-Royce plc or any of its subsidiary or associated companies.  

2

| © 2013 Rolls-Royce plc

 

 

This appendices provides supplementary information on how to carry out analysis using Minitab statistical software – software  – together  together with some of the more detailed analysis of the statistical output. 

Step 11  Be Prepared 

CONTINUOU CONTINUOUS S DATA :    A ppend ppendii x 1: Setting up and randomising the spreadsheet in Minitab 

CONTINU CONTINUOU OUS S DATA :    A ppend ppendii x 2: Entering the data in Minitab 

ppendii x 3:  A ppend Carrying out Gauge R&R in Minitab 

Step 22  Plan the Study  Step 33  Conduct the Study 

Step 44    Type of Study 

Continuous Data: Gauge R&R for continuous data 

 A TT TTR R I B UT UTE E D A TA :    A ppend ppendii x 6: Setting up and randomising the spreadsheet in Minitab 

 A TT TTR R IB UT UTE E D A TA    A ppend ppendii x 7: Entering the data in Minitab 

Attribute Data:   Attribute agreement analysis for attribute data 

ppendii x 8:  A ppend Carrying out  Attribute  Agreement  Analysis Minitab  in

 A ppend ppendii x 4: Supplementary Information on Interpreting the Graphical Output from Gauge R&R in Minitab 

Step 55  Taking action if the results are unacceptable 

Step 66    Maintaining the improvement 

ppendii x 9:  A ppend Supplementary Information on Interpreting the Output from Attribute Agreement  Analysis 

© 2013 Rolls-Royce plc | 3 

 

 

MSA How-to Guide In This Section: 

Continuous Data 

4

1.

Setting-up and Randomising the Spreadsheet in Minitab 

2.

Entering the Data in Minitab 

3.

Carrying out Gauge R&R in Minitab

4.

Supplementary Information on interpreting the output 

5.

FAQ FA Q for Gauge RR 

| © 2013 Rolls-Royce plc

 

  Continuous Data  Appendix 1: Setting-up and Randomising the Spreadsheet in Minitab   Setting up and randomising the worksheet for a Gauge R&R Study for Variable Data  1. The starting point for setting up the worksheet for a variable data Gauge R&R study is the same regardless of which randomisation method for the worksheet is required. To begin go to:  Stat > Quality Tools > Gage Study > Create Gage R&R Study Worksheet  

2.

Complete the dialogue box for the required detail:  1) Enter the quantity of parts to be studied 

2) Enter the identity of the parts to be used 

3) Enter the number of people in the study 

4) Enter the identities of the people in the study 

5) Enter the number of times that each person will check each part 

© 2013 Rolls-Royce plc | 5 

 

 

MSA How-to Guide Continuous Data  Appendix 1: Setting-up and Randomising the Spreadsheet in Minitab  3.

Click on the options box 

Options 

This gives you 3 options to randomise the worksheet:  

 At this stage you must decide which randomisation method to use taking into  most ost economical use consideration the practicalities of running the experiment and the m of people’s time.  described ribed below.  The different options are each desc

6

| © 2013 Rolls-Royce plc

 

  Continuous Data  Appendix 1: Setting-up and Randomising the Spreadsheet in Minitab  a.

Do not randomise: As it states, this option does not randomise the data. This option will sequence the parts then the people for each part and provide a run order column as shown sh own below.

1) Use the ‘Options’ to confirm selection  3) Note the sequence of parts and people 

2) Make the selection and click OK will then generate a worksheet for the study 

© 2013 Rolls-Royce plc | 7 

 

 

MSA How-to Guide Continuous Data  Appendix 1: Setting-up and Randomising the Spreadsheet in Minitab  b.

Randomise all runs: runs: This will completely randomise the the order that the measurements are taken in as shown below. This is useful to prevent the appraisers from memorising their previous measurements and also to reduce the impact of time related factors. It does however require all of the appraisers to be present at once which can be impractical in many situations such as where different shifts are worked.  As a facilitator, it can also be useful to preserve the ‘standard’ (un (un-randomised) -randomised) order by selecting the option ‘Store standard run order in worksheet’.   2) Check the box to include standard order column 

1) Use the ‘Options’ to confirm selection 

4) Note the sequence of parts and people 

3) Make the selection and click OK will then generate a worksheet for the study 

Following data collection and analysis the standard order can be used to re-sort the recorded data so that the pattern of collection may give an insight into what happened. This should only be done if the measurement system analysis study is not clearly acceptable and in this case can be useful in identifying combinations which were awkward for the appraisers.  8

| © 2013 Rolls-Royce plc

 

  Continuous Data  Appendix 1: Setting-up and Randomising the Spreadsheet in Minitab   c.

Randomise runs within operator: This will prevent memory of measurements by the people undertaking the study but preserves the appraiser sequence. This enables a study appraiser’s time to be managed as managed  as only one appraiser needs to be present at specified times. As a facilitator, itit can also be useful to preserve the ‘standard’ (non-randomised) (non-randomised) order by selecting the option.  

2) Check the box to include standard order column 

1) Use the ‘Options’ to confirm selection 

4) Note the sequence of parts and people 

3) Make the selection and click OK will then generate a worksheet for the study 

This is the most commonly used option; On completion of Note 4, Minitab returns you to previous dialogue box (Create Gage R&R Worksheet) then press OK on this screen. Minitab will now generate the worksheet, you will need to add in your data column and collect the data before running the study.  © 2013 Rolls-Royce plc | 9 

 

 

MSA How-to Guide Continuous Data  Appendix 2: Entering the Data in Minitab  Maintaining Data Integrity  It is often overlooked that data integrity starts when the data is entered. In statistical software such as Minitab it is common to see data formatting and entry errors causing issues. The two most common issues to be aware of are as follows: 1) Areas of the the worksheet have have been previously used OR the wrong sort of data has been entered resulting in the column being in the wrong data format  turns ns  A ‘space’ was typed in and tur the column type to text 

The wrong type of data format for columns then has the effect of hiding columns that are expected to be numeric (or vice versa) when conducting an MSA study.   The most common occurrences of this is when a space is added somewhere in the column or when the letter O is used instead of 0 (zero). In both cases, even if the typing error is rectified this will change the format of the column from numeric to text format.  Text format columns can be identified by the addition of a ‘T’ to the column number as in the example above.  10

| © 2013 Rolls-Royce plc

 

  Continuous Data  Appendix 2: Entering the Data in Minitab  2)

The second issue is that of human mistakes when entering the data.  To guard against this, the facilitator of the measurement study must control the study to maintain the concentration, time, speed and discipline required to type/record each data point. In addition to this, it is possible to assist the person entering the data to select the correct cell by highlighting the line (descriptions and details) of the active entry.  

Left click on the row number will highlight the complete row 

The example shown has used the option ‘randomise runs within operators’ with the next entry being from appraiser 2 for part identity 9.   It can also be very beneficial to record comments when entries are typed. This additional information can be useful for analysis where the Measurement System is not acceptable and further investigation is needed. 

Data Type Considerations  Types of data are very specific to each MSA study. For Gauge R&R  studies   studies the data is variable and has to be the same units of measure as the operating process.   For Attribute Agreement Analysis For Attribute Analysis this  this is attribute data BUT this can be in the format of whole (count or scale) numbers or text values.   © 2013 Rolls-Royce plc |

11 

 

 

MSA How-to Guide Continuous Data  Appendix 3: Running the Analysis  To run the analysis then use the menu commands: Stat> Quality Tools>

Gage Gage Study> R&R Study (crossed) 

12

| © 2013 Rolls-Royce plc

 

  Continuous Data  Appendix 3: Running the Analysis    A dialogue box will appear. Enter the data into the fields as shown shown below: 

1. Minitab will display in this window, appropriate elements of the worksheet for selection, transfer the columns for parts appraisers and measurement data as shown. 

2. Click against ANOVA in “Method of  Analysis” This should be the default as Minitab Minitab opens the dialogue box, however should always be checked 3. Then click OK 

© 2013 Rolls-Royce plc |

13 

 

 

MSA How-to Guide Continuous Data  Appendix 3: Running the Analysis   Click on “Options” to enter the tolerance of the characteristic being measured. For our example this is 0.5mm.  –  Lower spec” Enter 0.5 into the “Upper spec – 

Click on “Do not display percentage contribution” &  &   “Do not display percentage study variation”. 

 

only once. Click [This “OK” simplifies the graphical output to remove graphs not necessary for our analysis]  14

| © 2013 Rolls-Royce plc

 

  Continuous Data  Appendix 3: Running the Analysis  Click on “Gage Info” to enter the relevant equipment, team and information for the study. It is also good practice to record the date date of the study for future reference  

Fill out the details requested   Click “OK” then “OK” again. 

© 2013 Rolls-Royce plc |

15 

 

 

MSA How-to Guide Continuous Data  Appendix 3: The Numerical Output   The Graphical output will appear as below. Click on Show sessions folder icon output”

to review the numerical

 

Gage R&R (ANOVA) for Measurement G ag age nam e: e: Date of study study :

Re ep ported by : T olerance: M is isc:

v e rrn nie r caliper 16th A ug 2006

HA SF A D M ea easurement S y st stem A n na aly si sis

Components of Variation

Measurement by Parts Parts % Tolerance

200

   t   n   e   c   r 100   e    P

2.00 1.75 1.50

0 Gage R&R 

Repeat

Reprod

1

Part-to-Part

2

3

R Chart by Appraiser   e   g   n   a    R 0.10   e    l   p 0.05   m   a    S 0.00

1

2

4

7

8

9

10

2.00 UCL=0.0789  _  R=0.0307 LCL=0

1.75 1.50

Parts

1

2  Appraiser  Appraiser

 Xbar Char Chartt by Appraiser Appraiser 1

2

3

Parts

| © 2013 Rolls-Royce plc

3

 Parts * Appraiser Interaction  _   _  UCL=1.8504 X=1.819 LCL=1.7876

  1   0   1   0   1   0      2      3     4      5     6      7     8     9     1      2      3     4      5     6      7     8     9     1      2      3     4      5     6      7     8     9     1

16

6 Parts

Measurement by Appraiser Appraiser

3

  1   0   1   0   1   0      2      3     4      5     6      7     8     9     1      2      3     4      5     6      7     8     9     1      2      3     4      5     6      7     8     9     1

  n 2.00   a   e    M   e 1.75    l   p   m   a    S 1.50

5

 Appra iser

2.00

1

  e   g   a   r   e 1.75   v    A

2 3

1.50 1

2

3

4

5 6 Parts

7

8

9

10

 

  Continuous Data  Appendix 4: Supplementary Information on interpreting the Graphical Analysis Output from Gauge R&R Studies  In the following appendix the results and interpretation is explained for each of the six graphs in the Minitab Graphical Output. Please note that all of the statistical analysis and graphs should be considered before making conclusions for the study and for the potential actions required. Note also that ‘no action required’ is a possible and valid conclusion.   The first graph to look at is the “Components of Variation” on the top right of the graphical output. 

This graph shows where most of the variation in the study came from.   The Gauge R&R column shows the % Tolerance taken up by the measurement system variation. Remember this was 76.99%  C

If the Part-to-Part columns are high (or very high compared to the others) this tells us that most of the variation in the study was due to the fact that the parts being measured were not n ot identical (which we would expect).   o m p o n e n ts

If the Repeat columns are high compared to the others, this indicates that there is a problem with Repeatability (i.e. one or more of the appraisers is inconsistent with themselves). The remaining graphs will help us investigate this further.  If the Reprod columns c olumns are high compared to the others, this indicates that there is a problem with Reproducibility (i.e. some of the appraisers are inconsistent with each other). The remaining graphs will help us investigate this further. 

o f V a ri ati o n

The Gauge R&R column is the variation component total for Repeat and Reprod. In cases such as this example where a problem is identified with the repeatability and/or the reproducibility of the measurement system then the remaining graphs should be examined to investigate further.   Where no problem is identified from the analysis of the components of variation then there is no need to examine the remaining graphs   © 2013 Rolls-Royce plc |

17 

 

 

MSA How-to Guide Continuous Data  Appendix 4: Supplementary Information on interpreting the Graphical Analysis Output from Gauge R&R Studies  Summary – Components of Variation Graph 

* 30% is a generally used acceptance criteria, however manufacturing standards may have tighter requirements. Be sure to consult the relevant measurement standards for your area. Details are contained within the SABRe Supplier Management System Requirements document.

18

| © 2013 Rolls-Royce plc

 

 

  Continuous Data  Appendix 4: Supplementary Information on interpreting the Graphical Analysis Output from Gauge R&R Studies  Next we will look at the R Chart by Appraiser graph. This chart shows the Range of the results for each appraiser for each of the 10 parts.  

Interpreting the Graphical Data  For a perfectly consistent measurement system, all of the ranges on the graph would be zero i.e. each part would be measured the same giving no (zero) range. G ra

However, it is unlikely that they will all be zero, therefore we use this chart to help us identify any measurements of concern.   p h ic a l

We interpret this graph by saying that any point which is above the upper red line is worth investigating, as this indicates that the range of results for that appraiser and part was higher than expected.  

D at a

So in the case study example we can see that appraiser 2 has a bigger range than the other appraisers. Julie and the team make note to ask appraiser 2 if they did anything different from the instructions given and move on the next graph  

 

Refer to SPC “How to” for more information on R charts.

© 2013 Rolls-Royce plc |

19 

 

 

MSA How-to Guide Continuous Data  Appendix 4: Supplementary Information on interpreting the Graphical Analysis Output from Gauge R&R Studies  Summary – R Chart by Appraiser  

20

| © 2013 Rolls-Royce plc

 

  Continuous Data  Appendix 4: Supplementary Information on interpreting the Graphical Analysis Output from Gauge R&R Studies  Next Julie and the team look at the Xbar Chart by Appraiser graph  –  –   this is the graph from the case study.

 

 

In te rp re

This graph shows the average measurement for each part and appraiser. 

ta tio n o f g ra

Ideally we want the patterns of the data to be identical for all 3 appraisers. If they are not, we should investigate. The 2 red lines on the chart are control limits. We would expect all at least 50% of the points to lie outside the control limits (red lines) on this chart. This is different and the opposite to the conventional use of SPC charts.

p h ic a l o u tp u t

 As we can, even though we do have the majority of parts par ts outside the control limits we can also see the patterns for each appraiser look different. This is again an indication of poor Reproducibility.  Julie and the team note the results of this graph and ask each appraiser what they did to identify differences then move to the next one. 

If you are not sure on interpreting control limits on an Xbar chart then ask a local Black Belt to help you choose the most appropriate type of MSA. 

© 2013 Rolls-Royce plc |

21 

 

 

MSA How-to Guide Continuous Data  Appendix 4: Supplementary Information on interpreting the Graphical Analysis Output from Gauge R&R Studies  Summary – Xbar Chart by Appraiser  

22

| © 2013 Rolls-Royce plc

 

  Continuous Data  Appendix 4: Supplementary Information on interpreting the Graphical Analysis Output from Gauge R&R Studies  Next Julie and the team look at the Measure by Part graph  –  – this  this is the graph from the case study. 

Interpreting the Graphical Data  So in the case study example we can see that:   This graph shows circles for all measured values of each part, together with the average values for each part (shown by ‘crossed circles’.) 

In te rp re ta ti o n o f g ra p

The average values (crossed circles) are connected by the straight lines  h ic a l o

The graph allows us to compare how consistent the measurements for each of the parts were in the study.  

ut p u t

If the measurement system is consistent, there should be very little scatter between the measurements for each individual part (in other words, the circles for each part should almost be on top of each other or overlapping). We interpret this graph by saying that any part for which there is a noticeably larger spread in the results, re sults, might be worth investigating.   In this case parts 10, 8, 4 & 5 appear to have greater variation than the rest of the to parts. The team need consider parts were more difficult measure? Julie andtothe team why note these this finding and move on to the next graph.  © 2013 Rolls-Royce plc |

23 

 

 

MSA How-to Guide Continuous Data  Appendix 4: Supplementary Information on interpreting the Graphical Analysis Output from Gauge R&R Studies  Summary – Measurement by Parts 

24

| © 2013 Rolls-Royce plc

 

  Continuous Data  Appendix 4: Supplementary Information on interpreting the Graphical Analysis Output from Gauge R&R Studies  Interpreting the Graphical Data  Next we will look at the Measurement by Appraiser graph:  

In te

This graph shows a box plot for all measured values of each item, together with the average measurement for each appraiser (shown by ‘crossed circles’). rp re ta ti o n

The average values (crossed circles) are connected by the straight lines. If the measurement system is perfectly consistent, we would expect the average values for the 3 appraisers to be the same  –  –   in which case the connecting con necting lines would be horizontal. o f g ra p h ic a l o

We would also expect the spread of results (boxes and whiskers) for all 3 appraisers to be the same (however, unlike the previous graph, we wouldn’t necessarily expect the spread to be small, as the results for all of the parts are shown against each appraiser).  

u tp ut

We interpret this graph by saying that if, for any appraiser, there is a noticeably larger spread in results, or the average value is noticeably different from the others, this might be worth investigating. So in the case study example we can see that appraiser 2 has a larger spread of result than the other appraisers. We can also see the average values line is not straight indicating a Reproducibility problem.   Again Julie and the team note the findings and move onto the final graph.  © 2013 Rolls-Royce plc |

25 

 

 

MSA How-to Guide Continuous Data  Appendix 4: Supplementary Information on interpreting the Graphical Analysis Output from Gauge R&R Studies  Summary – Measure by Appraiser Graph 

26

| © 2013 Rolls-Royce plc

 

  Continuous Data  Appendix 4: Supplementary Information on interpreting the Graphical Analysis Output from Gauge R&R Studies  Interpreting the Graphical Data  Finally, we will look at the Part*Appraiser Interaction graph:  

In te rp ret a ti o n

This graph overlays the average measurements for each item as measured by each person.  o f g ra p

If the measurement system is perfectly consistent, we would expect all of the lines to be on top of each other so only one line is seen. Overlaying lines is the ideal situation here, however there can be occasions when parallel lines occur. This would indicate that the parts and appraiser interaction is consistent BUT bias between appraisers exists. 

h ic a l o u tp u t

We interpret this graph by saying that if any of the lines is noticeably separate from the other 2 lines (for one or more of the parts), this is worth investigating.  Julie and the team agree that this graph confirms some of their thoughts from the previous graphs. They are now ready to summarise their findings. 

© 2013 Rolls-Royce plc |

27 

 

 

MSA How-to Guide Continuous Data  Appendix 4: Supplementary Information on interpreting the Graphical Analysis Output from Gauge R&R Studies  Summary – Part*Appraiser Interaction 

28

| © 2013 Rolls-Royce plc

 

  Continuous Data  Appendix 4: Supplementary Information on interpreting the Graphical Analysis Output from Gauge R&R Studies  Use of Gauge Run Charts  One additional graphical tool which can be used to assess differences in measurements between different operators and different parts is a Gauge Run Chart.   You can use a gauge run chart in combination with a Gage R&R Study to help determine what is causing the variability in the measuring system.  Create a gauge run chart as follows: 1.

To begin go to Stat > Quality Tools > Gage Study > Gage Run Chart  

2.

Complete the dialogue box for the required details   Click on ‘Gage Info’ to enter the relevant equipment and study references  Enter the columns for parts appraisers and measurement data as shown. 

Then click ‘OK’

If known the historical process mean can be entered and will be plotted as a reference line on the graph 

© 2013 Rolls-Royce plc |

29 

 

 

MSA How-to Guide Continuous Data  Appendix 4: Supplementary Information on interpreting the Graphical Analysis Output from Gauge R&R Studies  Use of Gauge Run Charts  The graphical output will appear as shown: 

This is a plot of all of the observations by operator (denoted by different colours) and by part number (each box numbered 1 – 1  – 10  10 represents one of the 10 parts).   The horizontal reference line is the t he overall mean of the measurements.   The plot allows you to see if any patterns are evident in the data. For instance, you might see that one operator consistently measures higher than the others or that the measurements on certain parts vary more when compared to other parts. Here for example you can see that for part 4, appraiser 1 (in black) has measured higher than the other two appraisers. You can also see for part 10 noticeable difference between the measurements of the three appraisers.   Looking at the repeatability within appraisers, for parts 2 and 8 it can be seen that appraiser 2 (in red) noticeably differs in their three measurements indicating a repeatability problem.  30

| © 2013 Rolls-Royce plc

 

  Continuous Data  Appendix 5: FAQ for Gauge R&R (Typical Manufacturing Questions)   Q: “Where can I find the acceptance criteria for MSA in manufacturing?”    A: “The criteria and other information are contained within SABRe Supplier Management System Requirements document. Further guidance information on measurement and inspection is available on the supplier portal, specifically in the Guide to Dimensional Dimensional Measurement Equipment document.” Q: “We only have 2 operators using this gauge, shall I take part in the study to make the numbers up?”   A: “Not unless the operator is trained to use the gauge and familiar with the component being measured. Having an untrained operator may result in the failure of the study due to the stability and/or reproducibility of the measurements due to the untrained operator. It would be far better to compromise on the amount of measurement readings than to perform a study study that is not representative of the way the process works”   ?”  Q: “We cannot ge t access to will 10 parts, 5 do  A: “Lowering the sample size affect will the uncertainty of the test. However there are times when this may be required. In difficult situations a compromise may be required but the analyst should be mindful of this when interpreting the data. For example if a gauge R&R returns 19% of tolerance with only 5 parts there is a reasonable case for either acquiring further parts for study or asking the operators to repeat the measurements 4 or 5 times rather than the usual 3”   Q: “We can’t get parts that represent the full process variation as the only ones we have are from the same batch. What should we do?”    A: “If the sample does not represent the true process variation then this will affect the results of gauge R&R against study variance, the number of distinct categories and the limits on the X bar chart on the R&R output. If this is the case 2 options exist. Either proceed and study the % Gauge R&R against tolerance.””  tolerance. Q: “Our gauge measures hundreds of features, do I have to run gauge R&R study on each of them or can I use read- across methodology?”   A: “The Quality Management System requires that all product features/characteristics are measured using capable measurement systems. That said there are situations where similar features measured with the same gauge presents an opportunity to demonstrate capability without direct study of every single feature as long as this is done robustly. This must however be done in a robust and traceable way. Read across is not permitted on CMM equipment, not because the CMM’s tend to be incapable but because of the risk of program errors due to the manual nature of the program creation.””  creation. © 2013 Rolls-Royce plc |

31 

 

 

MSA How-to Guide Continuous Data  Appendix 5: FAQ for Gauge R&R  Q: “The gauge is automated (there is no ope rator influence)! What do I do ?”   A: ““First First be sure that there really is not any operator influence  –  – for  for instance if there is a setup process which is manual then this may lead to reproducibility problems. If the gauge is completely automated then a study can be performed with only 1 operator. The gauge R&R study will report only on the repeatability element of the gauge R&R. If there is the opportunity to study two gauges simultaneously (e.g. 2 CMM’s) these can be identified within the R&R study to allow the reproducibility due to different equipment rather than operator.” operator. ”  Q: “I have a surface finish gauge, and it keeps failing R&R. What do I do ?”   A:”Some gauges are notoriously difficult to perform gauge R&R on. Some gauges are  A:”Some the best available for a given measurement. In this situation contact a measurement practitioner or Metrologist.” Metrologist. ” 

 

Q: “Under what circumstances should whenever I repeat the gauge R&R?”  A: “ A  A gauge R&R should be repeated the process (either measurement process or manufacturing process) changes significantly or the part tolerance is changed. Also when turnover of labour is high”   Q: “My gauge R&R against tolerance is very good but I   only get 1 distinct category. Is my measurement process good or not ?”   A: “It is likely that the parts selected for the study are not representative of the total process variation. This will affect the %R&R against study variance, the number of distinct categories and the X bar chart. If the parts are representative then the measurement process is not adequate for the application of SPC analysis as the majority of the variation seen will be from the measurement system and not the underlying process.” process.”

 

Q: “How many decimal places on my gauge shall I use when conducting the gauge study?”   A: “As a minimum you should ensure that the study represents the requirements on the the part drawing but the more the better. For instance if the drawing requires measurement to 2 decimal places, run the gauge R&R to 3.”  Q: “My gauge passes the gauge R&R study. Is this all I need to consider ?” ?”   A: “No –  gauge R&R will not highlight gauge bias. For instance it is possible to be repeatibly wrong. Comparison with a known standard will enable you to study the amount of gauge bias. Calibration is not done on production parts, real parts can introduce large differences.”  32

| © 2013 Rolls-Royce plc

 

 

In This Section: 

Attribute Data  6. Setting-up and Randomising the Spreadsheet in Minitab  7. Entering the Data in Minitab  8. Carrying Attribute Agreement Analysis out in Minitab 9. Supplementary Information on how to interpret the output 

© 2013 Rolls-Royce plc |

33 

 

 

MSA How-to Guide Attribute Data  Appendix 6: Setting-up and Randomising the Spreadsheet in Minitab  Attribute Agreement Analysis Worksheet Configurations

1.

34

Setting up the worksheet for an attribute data, Attribute Agreement Analysis is very similar to a Gauge R&R study where, the worksheet construction all start from an identical point independent of which randomisation for the worksheet is used. This is found at: Stat > Quality Tools > Create Attribute Agreement Analysis Worksheet 

| © 2013 Rolls-Royce plc

 

  Attribute Data  Appendix 6: Setting-up and Randomising the Spreadsheet in Minitab   Complete the dialogue box for the required detail: 

2.

1) There are 4 drop down selections here (see below)

 

3) Enter the identity of the parts to be used and the text standard 

2) Enter the quantity of parts to be studied

 

4) Enter the number of people in the study  5) Enter the identities of the people in the study 

6) Enter the number of times each person is to inspect each part 

Dropdown Selection   unknown: Sample Standard/attribute unknown: This provides a worksheet very similar to that of a Gauge R&R study but does need to have an additional column manually  added so that a ‘standard’ agreement can be compared c ompared to it. This is the most flexible option.   Sample Standard/attribute in text : This constructs the worksheet with the additional column of text ‘standards’ for comparison, com parison, i.e. When comparing judgements such as good or bad against the standard which is also stated as good or bad. This is the option used in the following pages.   Sample Standard/attribute in numbers: numbers : This constructs the worksheet with the additional column column of numerical ‘standards’ for comparison, i.e. When judgements are made using a scale (often 1 to 5 or 1 to 10), and the standard should be an exact

 

match. Sample Standard/attribute in worksheet : This favours manual lists already in the worksheet and provides selection of those column references for the study.   © 2013 Rolls-Royce plc |

35 

 

 

MSA How-to Guide Attribute Data  Appendix 6: Setting-up and Randomising the Spreadsheet in Minitab 

3. Click on the options box 

Options 

This gives you 3 options to randomise the worksheet:  

36

| © 2013 Rolls-Royce plc

 

 

Attribute Data  Appendix 6: Setting-up and Randomising the Spreadsheet in Minitab  a.

Do not randomise: As it states, this option does not randomise the data. This option will sequence the parts then the people for each part and provide a run order column as shown below. 1) Use the ‘Options’ to confirm selection 

3) Note the sequence of parts, people and the ‘standard’ 

2) Make the selection and click OK to generate a worksheet for the study 

© 2013 Rolls-Royce plc |

37 

 

 

MSA How-to Guide Attribute Data  Appendix 6: Setting-up and Randomising the Spreadsheet in Minitab   b.

Randomise all runs: runs: This will completely randomise the order that the measurements are taken in as shown below. This is useful to prevent the appraisers from memorising their previous measurements and also to reduce the impact of time related factors. It does however require all of the appraisers to be present at once which can be impractical in many situations such as where different shifts are worked.  As a facilitator, it can also be useful to preserve the ‘standard’ (un (un-randomised) -randomised) order by selecting the option ‘Store standard run order in worksheet’.  

2) Check the box to include standard order column 

4) Note the sequence of parts, people and the ‘standard’ 

1) Use the ‘Options’ to confirm selection 

3) Make the selection and click OK to generate a worksheet for the study 

Following data collection and analysis the standard order can be used to re-sort the recorded data so that the pattern of collection may give an insight into what happened. This should only be done if the measurement system analysis study is not clearly acceptable and in this case can be useful in identifying combinations which were awkward for the appraisers.  38

| © 2013 Rolls-Royce plc

 

  Attribute Data  Appendix 6: Setting-up and Randomising the Spreadsheet in Minitab  c.

Randomise runs within operator: This will prevent memory of measurements by the people undertaking the study but preserves the appraiser sequence. This enables a study appraiser’s time to be managed as managed  as only one appraiser needs to be present at specified times. As a facilitator, it can also be useful to preserve the ‘standard’ (non-randomised) (non-randomised) order by selecting the option.  

2) Check the box to include standard order column 

4) Note the sequence of parts, people and the ‘standard’ 

1) Use the ‘Options’ to confirm selection 

3) Make the selection and click OK to generate a worksheet for the study 

This is the most commonly used option. option . 

© 2013 Rolls-Royce plc |

39 

 

 

MSA How-to Guide Attribute Data  Appendix 7: Entering the Data in Minitab  Maintaining Data Integrity  It is often overlooked that data integrity starts when the data is entered. In statistical software such as Minitab it is common to see data formatting and entry errors causing issues. The two most common issues to be aware of are as follows: f ollows: 1) Areas of the the worksheet have have been previously used OR the wrong sort of data has been entered resulting in the column being in the wrong data format   A ‘space’ was typed in and turns the column type to text 

The wrong type of data format for columns then has the effect of hiding columns that are expected to be numeric (or vice versa) when conducting an MSA study.   The most common occurrences of this is when a space is added somewhere in the column or when the letter O is used instead of 0 (zero). In both cases, even if the typing error is rectified this will change the format of the column from numeric to text format.  Text format columns can be identified by the addition of a ‘T’ to the column number as in the example above.

40

| © 2013 Rolls-Royce plc

 

 

  Attribute Data  Appendix 7: Entering the Data in Minitab  2) The second issue is that of human mistakes mistakes when entering entering the data.  To guard against this, the facilitator of the measurement study must control the study to maintain the concentration, time, speed and discipline required to type/record each data point. In addition to this, it is possible to assist the person entering the data to select the correct cell by highlighting the line (descriptions and details) of the active entry.  

Left click on the row number will highlight the complete row 

The example shown has used the option ‘randomise runs within operators’ with the next entry being from appraiser 2 for part identity 9.   It can also be very beneficial to record comments when entries are typed. This additional information can be useful for analysis where the Measurement System is not acceptable and further investigation is needed.  

Data Type Considerations  specific to each MSA MSA study. For Gauge R&R  studies   studies the data Types of data are very specific is variable and has to be the same units of measure as the operating process.   For Attribute Agreement Analysis For Attribute Analysis this  this is attribute data BUT this can be in the format of whole (count or scale) numbers or text values.   © 2013 Rolls-Royce plc |

41 

 

 

MSA How-to Guide Attribute Data  Appendix 8: Running the Analysis  To run an Attribute Measurement System Analysis (MSA) then use the menu commands: Stat> Quality Tools> 

Attribute Agreement Analysis

42

| © 2013 Rolls-Royce plc

 

 

  Attribute Data  Appendix 8: Running the Analysis    A dialogue box will below::  will appear. Enter the data into the fields as shown below Click to confirm data is listed downwards (stacked)   This is the format Minitab generates for the worksheet

 

Enter the column containing the ‘standard’ to be compared to  Only check this box when a scale or multiple class of judgement is used 

© 2013 Rolls-Royce plc |

43 

 

 

MSA How-to Guide Attribute Data  Appendix 8: Running the Analysis  It is good practice to use the ‘Information’ button to record details about the MSA study directly into the displayed results – results – click  click on Information button  

The information dialogue will appear, so that you can complete the details as appropriate. Once completed, click OK button to close the information screen, then click OK again to close on main window.  44

| © 2013 Rolls-Royce plc

 

  Attribute Data  Appendix 8: The Graphical Output   The Graphical output will appear as below. Click on Show sessions folder icon numerical output” 

to review the

© 2013 Rolls-Royce plc |

45 

 

 

MSA How-to Guide Attribute Data  Appendix 8: The Numerical Output   Click on Show graphs folder icon graphical output. 

46

| © 2013 Rolls-Royce plc

to return to the

 

  Attribute Data  Appendix 8: Running the Worksheet   Click on Show worksheet folder icon

to return to the worksheet 

© 2013 Rolls-Royce plc |

47 

 

 

MSA How-to Guide Attribute Data  Appendix 9: Supplementary Information on interpreting the Output of Attribute Agreement Analysis  Attribute Agreement Analysis Worksheet Configurations 1.

Setting up the worksheet for an attribute data, Attribute Agreement Analysis is very similar to a Gauge R&R study where, the worksheet construction all start from an identical point independent of which randomisation for the worksheet is used. This is found at: 

 S ta tatt > Qu Qual alii ty Tools > C r eat eate e A ttri ttribute bute Ag A g reemen reementt A nalys i s W or orkk s heet  

48

| © 2013 Rolls-Royce plc

 

  Attribute Data  Appendix   Appendix 6 9: Supplementary Information on interpreting the Output of Attribute Agreement Analysis 

In addition to the ‘How to guide: Measurement System Analysis’ the following pages give supplementary information on some of the statistical concepts to deepen your understanding of how to fully interpret the output

 

Interpretation of graphical output 

The ‘blue’ dots have been explained as the actual proportion of agreement within the sample of parts appraised appraised in the study. Also shown are the ‘red’ lines which end in a cross. These indicate the confidence intervals for each Appraiser   and for each Appraiser vs. Standard.  The actual numbers for each confidence interval are recorded in the session window and used in plotting this graph.  These confidence intervals take into account the fact that the actual agreement % calculated (as represented by the blue dot) is based only on a relatively small sample of data. If it was possible to know the ‘true’ agreement %  of the appraiser (based on every part they ever inspected) then this % would be likely to be different from the % seen in the study sample. The confidence interval indicates the possible range of values that the ‘true’ % agreement could be. Minitab defaults defaul ts to a confidence level of 95%. This means that we can interpret the confidence interval for Appraiser 1 for example as saying “I am 95% confident that the true % of within appraiser agreement for appraiser 1 is between 62.1% and 96.8% (which is the range indicated by the blue crosses and red line).  © 2013 Rolls-Royce plc |

49 

 

 

MSA How-to Guide Attribute Data  Appendix 9: Supplementary Information on interpreting the Output of Attribute Agreement Analysis  Each ‘red’ line and two crosses should be as short as possible indicating less poten potential tial error in the ‘blue’ dot point indication, when using 95% confidence (the default level for most statistical analysis).

 

These confidence intervals are calculated using the F distribution which also requires a thing called degrees of freedom. The larger the degrees of freedom, which can be influenced by using a larger selection of items for inspection (sample size) and the quantity of matched agreements (such as Pass / Pass and the standard also as Pass) also contribute to reducing the confidence intervals.  The F distribution is not symmetrical and therefore the confidence intervals can look a little odd, one (usually the lower) will be longer than the other.  

Interpreting the Lower Confidence Interval   quality of the measurement system is It may be desirable in some instances where the highly critical to use the lower confidence interval (lower red cross) rather than the observed agreement (blue dot) when assessing the measurement system against the rules of thumb described on page 46 . This assesses the measurement system based on the worst case scenario. You must however take into consideration that the sample size will significantly influence this. Where the sample size is small (less than 20 parts for example) the lower confidence intervals will nearly always fall below 70% even when the average agreement within the sample is fairly good). In these circumstances you should discuss with a Black Belt an appropriate sample size to use for the assessment of critical measurement systems. If the measurement needed is highly critical then consideration should also be given to whether it is possible to redesign the measurement system to use variable data rather than attribute and assess the system using Gauge R&R instead. 

50

| © 2013 Rolls-Royce plc

 

  Attribute Data  Appendix 9: Supplementary Information on interpreting the Output of Attribute Agreement Analysis  Each of the percentage results shown in the session window tables have been explained in the ‘How to guide: Measurement System Analysis’. This section will explain the other statistics such as the ‘Kappa’ listed and the P -value. Each of the tables is constructed with the same format and therefore the ‘Within Appraisers’ table will be used for these explanations. 

Interpretation of the numerical output 

The Kappa (correctly quoted as Fleiss Kappa) works on a scale of -1 to 1, where -1 indicates total disagreement between each measurement run and the standard. A value of 0 (zero) indicates a 50:50 chance of correctly assessing the part which implies appraisers are ‘guessing’ whether to pass or fail the part. The best possible outcome is a Kappa value or 1 which indicates total agreement to each round of measures and to the standard.

Within Appraisers table is shown for a simple Fail or Pass judgement which gives a Kappa value similar BUT not the same as the percent listing.  Fleiss’ Kappa becomes very useful when a scale or multiple class of judgement is used which then has a Kappa value indicating agreement for each scale value.   the e scale, shades A  An example of this could be shade judgements, where the ends of th or B (light), I or J (dark) have very high Kappa values near to 1 and middle shades such as E or F have lower (approximately 0.778291 say) showing for that person these were harder to judge as correct.   © 2013 Rolls-Royce plc | 51 

 

 

MSA How-to Guide Attribute Data  Appendix 9: Supplementary Information on interpreting the Output of Attribute Agreement Analysis  P-value Judgements 

Interpretation of the numerical output 

The P-value (which is short for the probability value) is a back-up statistic describing the chance of having a Fleiss Kappa near 0 (zero).    As the tabular notation shows, the judgement is made about the ‘chance’ (probability) where the lower the P (vs. > 0) column p-value shows a low number, the less statistical chance the Kappa values are 0 (zero) or less.   In other words, the p-values of 0.0000 are very good and what should be seen.  

52

| © 2013 Rolls-Royce plc

 

 

Measurement System Analysis How-to Guide - Appendices Change History

Revision

Date

Description of Change

Author

Owner

Approval

V6.1

20/08/2013

Guide reformatted for SABRe

D Prodger

D Prodger

D Prodger

Document update policy This document may be updated periodically. Major amendments will be shown as an update from one revision number to a higher revision number (e.g. revision 1 to revision 2) and therefore the content of the higher revision will be regarded as the latest requirements. A minor amendment will be shown as a number change after a decimal point (e.g. revision 1.1 to revision 1.2) and therefore any of these revisions may be regarded as the latest requirements until a major amendment is introduced  

© Rolls-Royce plc 2013 The information in this document is the property of Rolls-Royce plc and may not be copied, communicated to a third party or used for any purpose other than that for which it is supplied, without the express written consent of Rolls-Royce plc. While the information is given in good faith based upon the latest information available to Rolls-Royce plc, no warranty or representation is given concerning such information, which must not be taken as establishing any contractual or other commitment binding upon Rolls-Royce plc or any of its subsidiary or associated companies. 

© 2013 Rolls-Royce plc |

53