DO Qualification Kit ®
Simulink Verification and Validation™ Tool Qualification Plan
R2012a
How to Contact MathWorks
Web Newsgroup www.mathworks.com/contact_TS.html Technical Support www.mathworks.com
comp.soft-sys.matlab
[email protected] [email protected] [email protected] [email protected] [email protected]
Product enhancement suggestions Bug reports Documentation error reports Order status, license renewals, passcodes Sales, pricing, and general information
508-647-7000 (Phone) 508-647-7001 (Fax) The MathWorks, Inc. 3 Apple Hill Drive Natick, MA 01760-2098 For contact information about worldwide offices, see the MathWorks Web site. DO Qualification Kit: Simulink® Verification and Validation™ Tool Qualification Plan © COPYRIGHT 2009–2012 by The MathWorks, Inc. The software described in this document is furnished under a license agreement. The software may be used or copied only under the terms of the license agreement. No part of this manual may be photocopied or reproduced in any form without prior written consent from The MathWorks, Inc. FEDERAL ACQUISITION: This provision applies to all acquisitions of the Program and Documentation by, for, or through the federal government of the United States. By accepting delivery of the Program or Documentation, the government hereby agrees that this software or documentation qualifies as commercial computer software or commercial computer software documentation as such terms are used or defined in FAR 12.212, DFARS Part 227.72, and DFARS 252.227-7014. Accordingly, the terms and conditions of this Agreement and only those rights specified in this Agreement, shall pertain to and govern the use, modification, reproduction, release, performance, display, and disclosure of the Program and Documentation by the federal government (or other entity acquiring for or through the federal government) and shall supersede any conflicting contractual terms or conditions. If this License fails to meet the government’s needs or is inconsistent in any respect with federal procurement law, the government agrees to return the Program and Documentation, unused, to The MathWorks, Inc.
Trademarks
MATLAB and Simulink are registered trademarks of The MathWorks, Inc. See www.mathworks.com/trademarks for a list of additional trademarks. Other product or brand names may be trademarks or registered trademarks of their respective holders. Patents
MathWorks products are protected by one or more U.S. patents. Please see www.mathworks.com/patents for more information.
Revision History
March 2009 September 2009 April 2010 March 2010 September 2010 April 2011 September 2011 March 2012
Online Online Online Online Online Online Online Online
only only only only only only only only
New for Version 1.0 (Applies to Release 2009a) Revised for Version 1.1 (Applies to Release 2009b) Rereleased for Version 1.1.1 (Applies to Release 2009bSP1) Revised for Version 1.2 (Applies to Release 2010a) Revised for Version 1.3 (Applies to Release 2010b) Revised for Version 1.4 (Applies to Release 2011a) Revised for Version 1.5 (Applies to Release 2011b) Revised for Version 1.6 (Applies to Release 2012a)
Contents Introduction
1 Tool Operational Requirements
2 Certification Considerations
3 Requirement for Qualification . . . . . . . . . . . . . . . . . . . . . . DO-178B Checks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Model Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3-2 3-2 3-3
Certification Credit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . DO-178B Checks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Model Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3-5 3-5 3-9
Tool Development Lifecycle
4 Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4-2
Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . DO-178B Checks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Model Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4-3 4-3 4-3
Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . DO-178B Checks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4-5 4-5
v
Model Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4-6
Tool Lifecycle Data
5 DO-178B Checks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5-2
Model Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5-4
Schedule
6
vi
Contents
1 Introduction This document comprises the Tool Qualification Plan (reference DO-178B Section 12.2.3.1) for the following capabilities of the Simulink® Verification and Validation™ verification tool: • DO-178B checks • Model coverage This document is intended for use in the DO-178B tool qualification process for verification tools. See also the DO Qualification Kit User’s Guide.
1
1-2
Introduction
2 Tool Operational Requirements The Tool Operational Requirements for the following capabilities in the Simulink Verification and Validation product are documented in DO Qualification Kit: Simulink Verification and Validation Tool Operational Requirements: • DO-178B checks • Model coverage
2
2-2
Tool Operational Requirements
3 Certification Considerations • “Requirement for Qualification” on page 3-2 • “Certification Credit” on page 3-5 This section provides certification considerations for the following capabilities of the Simulink Verification and Validation verification tool: • DO-178B checks • Model coverage
3
Certification Considerations
Requirement for Qualification In this section... “DO-178B Checks” on page 3-2 “Model Coverage” on page 3-3
DO-178B Checks To determine whether a tool must be qualified, you must answer the following questions. If you answer yes to all three questions, you must qualify the tool. Question
DO-178B Checks
Can the tool insert an error into the airborne software or fail to detect an existing error in the software within the scope of its intended usage?
Yes1
Will the output of the tool not be verified as specified in Section 6 of DO-178B?
Yes
Are processes of DO-178B eliminated, reduced, or automated by the use of the tool? Will you use output from the tool to meet an objective or replace an objective of DO-178B, Annex A?
Yes
Given that the answer to all the preceding questions is yes, the DO-178B checks in the Simulink Verification and Validation product must be qualified. To determine the type of qualification (development tool or verification tool qualification) needed, you must answer the following question about the tool. Question
DO-178B Checks
Is the tool output part of the airborne software, such that the output can insert an error into the software?
No
1. The DO-178B checks might fail to detect an error.
3-2
Requirement for Qualification
Because the answer to the preceding question is no, the DO-178B checks in the Simulink Verification and Validation product must be qualified as a verification tool.
Model Coverage To determine whether a tool must be qualified, you must answer the following questions. If you answer yes to all three questions, you must qualify the tool. Question
Model Coverage
Can the tool insert an error into the airborne software or fail to detect an existing error in the software within the scope of its intended usage?
Yes2
Will the output of the tool not be verified as specified in Section 6 of DO-178B?
Yes
Are processes of DO-178B eliminated, reduced, or automated by the use of the tool? Will you use output from the tool to meet an objective or replace an objective of DO-178B, Annex A?
Yes
Given that the answer to all the preceding questions is yes, the model coverage capability in the Simulink Verification and Validation product must be qualified. To determine the type of qualification (development tool or verification tool qualification) needed, you must answer the following question about the tool. Question
Model Coverage
Is the tool output part of the airborne software, such that the output can insert an error into the software?
No
2. Model coverage might fail to detect an error.
3-3
3
Certification Considerations
Because the answer to the preceding question is no, the model coverage capability in the Simulink Verification and Validation product must be qualified as a verification tool.
3-4
Certification Credit
Certification Credit In this section... “DO-178B Checks” on page 3-5 “Model Coverage” on page 3-9
DO-178B Checks The following table shows the certification credit (see DO-178B Annex A Objectives), being taken for the DO-178B checks in the Simulink Verification and Validation product. Note The DO-178B checks can contain two sections: an analysis section for reviewing the model and an action section for automatically fixing warnings and failures. The DO Qualification Kit covers the DO-178B check analysis, not the check actions. The DO Qualification Kit does not cover Model Advisor check exclusions.
Certification Credit for DO-178B Checks Annex A Table
Objective
DO-178B Software Reference Levels
Credit Taken
Table A-3
High-level requirements are accurate and consistent
Section 6.3.1b
A, B, C, D
Full or Partial1 – The DO–178B checks verify the accuracy and consistency of the model statically. A combination of Model Advisor checks, simulation against the higher-level requirements, and review of the System Design Description can be used to take full credit for this objective.
Table A-3
High-level requirements are
Section 6.3.1c
A, B
Full or Partial1, 3 – The DO–178B checks verify that the code generator
3-5
3
Certification Considerations
Certification Credit for DO-178B Checks (Continued) compatible with target computer
3-6
settings related to the CPU are correct. A combination of Model Advisor checks and review of the System Design Description can be used to take full credit for this objective.
Table A-3
High-level requirements are verifiable
Section 6.3.1d
A, B, C
Full or Partial1 – The DO–178B checks verify parameter tunability, test point visibility, and in some cases can find unreachable decisions. A combination of Model Advisor checks and model coverage during simulation can be used to take full credit for this objective.
Table A-3
High-level requirements conform to standards
Section 6.3.1e
A, B, C
Full or Partial1 – The DO–178B checks verify conformance to standards that have dedicated checks. For any modeling standards that do not have Model Advisor checks, this verification may be completed via manual reviews of the System Design Description.
Table A-3
High-level requirements are traceable to system requirements
Section 6.3.1f
A, B, C, D
Partial1 — The DO–178B checks verify that the requirements links are consistent; the actual traceability must be verified independently by reviewing the “Requirements Traceability” section of the System Design Description.
Certification Credit
Certification Credit for DO-178B Checks (Continued) Table A-3
Algorithms are accurate
Section 6.3.1g
A, B, C
Full or Partial1 — The DO–178B checks verify the accuracy of data types used within the model statically. A combination of Model Advisor checks, simulation against the higher-level requirements, and review of the System Design Description can be used to take full credit for this objective.
Table A-4
Low-level requirements are accurate and consistent
Section 6.3.2b
A, B, C
Full or Partial2— The DO–178B checks verify the accuracy and consistency of the model statically. A combination of Model Advisor checks, simulation against the higher-level requirements, and review of the System Design Description can be used to take full credit for this objective.
Table A-4
Low-level requirements are compatible with target computer
Section 6.3.2c
A, B
Full or Partial2, 3 — The DO–178B checks verify that the code generator settings related to the CPU are correct. A combination of Model Advisor checks and review of the System Design Description can be used to take full credit for this objective.
Table A-4
Low-level requirements are verifiable
Section 6.3.2d
A, B
Full or Partial2 — The DO–178B checks verify parameter tunability, test point visibility, and in some cases can find unreachable decisions. A combination of Model Advisor checks and model coverage during simulation can be used to take full credit for this objective.
3-7
3
Certification Considerations
Certification Credit for DO-178B Checks (Continued)
3-8
Table A-4
Low-level requirements conform to standards
Section 6.3.2e
A, B, C
Full or Partial2 — The DO–178B checks verify conformance to standards that have dedicated checks. For any modeling standards that do not have Model Advisor checks, this verification may be completed via manual reviews of the System Design Description.
Table A-4
Low-level requirements are traceable to high-level requirements
Section 6.3.2f
A, B, C
Partial2 - The DO-178B checks verify that the requirements links are consistent; the actual traceability must be verified independently by reviewing the “Requirements Traceability” section of the System Design Description.
Table A-4
Algorithms are accurate
Section 6.3.2g
A, B, C
Full or Partial2 - The DO-178B checks verify the accuracy of data types used within the model statically. A combination of Model Advisor checks, simulation against the higher-level requirements, and review of the System Design Description can be used to take full credit for this objective.
Table A-4
Software architecture is consistent
Section 6.3.3b
A, B, C
Full or Partial2 – The DO-178B checks verify that the architecture of the model is consistent statically. A combination of Model Advisor checks, simulation against the higher-level requirements, and review of the System Design Description can be used to take full credit for this objective.
Certification Credit
Certification Credit for DO-178B Checks (Continued) Table A-4
Software architecture conforms to standards
Section 6.3.3e
A, B, C
Full or Partial2 - The DO-178B checks verify conformance to standards that have dedicated checks. For any modeling standards that do not have Model Advisor checks, this verification may be completed via manual reviews of the System Design Description.
Table A-5
Source code is traceable to low-level requirements
Section 6.3.4e
A, B, C
Partial2, 3 – The DO-178B checks verify that the code generator settings are appropriate for generating traceable code; the actual traceability must be verified independently.
Notes: 1 This credit is taken only if the Simulink and Stateflow® models are considered high-level software requirements for the project. 2 This credit is taken only if the Simulink and Stateflow models are considered low-level software requirements for the project. 3 This credit is taken only if the Embedded Coder™ product is used to automatically generate code from the models.
Model Coverage The following table shows the certification credit (see DO-178B Annex A Objectives), being taken for the model coverage capability of the Simulink Verification and Validation product.
3-9
3
Certification Considerations
Certification Credit for Model Coverage
3-10
Annex A Table
Objective
DO-178B Software Reference Levels
Credit Taken
Table A-3
High-level requirements are verifiable
Section 6.3.1d
A, B, C
Full or Partial1 – During simulation, model coverage verifies that all conditions and decisions in the model can be exercised. A combination of Model Advisor checks and model coverage during simulation can be used to take full credit for this objective.
Table A-4
Low-level requirements are verifiable
Section 6.3.2d
A, B
Full or Partial2 — During simulation, model coverage verifies that all conditions and decisions in the model can be exercised. A combination of Model Advisor checks and model coverage during simulation can be used to take full credit for this objective.
Table A-4
Software architecture is verifiable
Section 6.3.3d
A, B
Full or Partial2 — During simulation, model coverage verifies that all conditions and decisions in the model can be exercised. A combination of Model Advisor checks and model coverage during simulation can be used to take full credit for this objective.
Certification Credit
Certification Credit for Model Coverage (Continued) Annex A Table
Objective
DO-178B Software Reference Levels
Credit Taken
Table A-7
Test cases and procedures are correct
Sections 6.3.6a and 6.3.6b
A, B, C
Partial – During simulation, model coverage verifies that all conditions and decisions in the model have been exercised and provides the data ranges achieved. The adequacy of the data ranges and the expected results are not verified by model coverage. The model coverage report may be used to verify the correctness and completeness of test cases generated by the Simulink Design Verifier™ product3.
Table A-7
Test coverage of high-level requirements is achieved
Section 6.4.4.1
A, B, C, D
Partial1 – During simulation, model coverage verifies that all conditions and decisions in the model have been exercised and provides the data ranges achieved. The test cases executed on the model must be repeated on the object code to complete this objective.
Table A-7
Test coverage of low-level requirements is achieved
Section 6.4.4.1
A, B, C
Partial2 – During simulation, model coverage verifies that all conditions and decisions in the model have been exercised and provides the data ranges achieved. The test cases executed on the model must be repeated on the object code to complete this objective.
Notes: 1 This credit is taken only if the Simulink and Stateflow models are considered high-level software requirements for the project. 2 This credit is taken only if the Simulink and Stateflow models are considered low-level software requirements for the project. 3 The Simulink Design Verifier product is not a qualified tool. However, executing the Simulink
3-11
3
Certification Considerations
Certification Credit for Model Coverage (Continued) Annex A Table
Objective
DO-178B Software Reference Levels
Credit Taken
Design Verifier automatically generated tests on the model and assessing the results, while using the qualified model coverage tool, provides credit for demonstrating completeness and correctness of those test cases.
3-12
4 Tool Development Lifecycle • “Planning” on page 4-2 • “Requirements” on page 4-3 • “Verification” on page 4-5
4
Tool Development Lifecycle
Planning The Plan for Software Aspects of Certification (PSAC) designates that the following capabilities of the Simulink Verification and Validation product will be qualified as verification tools: • DO-178B checks • Model coverage This document provides the Tool Qualification Plan for these capabilities of the Simulink Verification and Validation product.
4-2
Requirements
Requirements In this section... “DO-178B Checks” on page 4-3 “Model Coverage” on page 4-3
DO-178B Checks • Tool Operational Requirements for the DO-178B checks in the Simulink Verification and Validation product are in: DO Qualification Kit: Simulink Verification and Validation Tool Operational Requirements The applicant will:
-
Review the Tool Operational Requirements for applicability to the project under consideration.
-
Configure the Tool Operational Requirements in a configuration management system.
• User information for the DO-178B checks in the Simulink Verification and Validation product is in: Simulink Verification and Validation User’s Guide • Instructions for installing the Simulink Verification and Validation product are in the following MathWorks® document: Installation Guide
Model Coverage • Tool Operational Requirements for the model coverage capability of the Simulink Verification and Validation product are in: DO Qualification Kit: Simulink Verification and Validation Tool Operational Requirements The applicant will:
4-3
4
Tool Development Lifecycle
-
Review the Tool Operational Requirements for applicability to the project under consideration.
-
Configure the Tool Operational Requirements in a configuration management system.
• User information for the model coverage capability of the Simulink Verification and Validation product is in: Simulink Verification and Validation User’s Guide • Instructions for installing the Simulink Verification and Validation product are in the following MathWorks document: Installation Guide
4-4
Verification
Verification In this section... “DO-178B Checks” on page 4-5 “Model Coverage” on page 4-6
DO-178B Checks Requirements-based test cases and procedures will be developed from the: DO Qualification Kit: Simulink Verification and Validation Tool Operational Requirements The test cases and procedures will be developed in the form of the Simulink models that exercise the DO-178B checks under consideration in the Model Advisor. The test cases and procedures are documented in: DO Qualification Kit: Simulink Verification and Validation Test Cases, Procedures, and Results The applicant will: • Review the test cases and procedures for applicability to the project under consideration. • Configure the test cases and procedures in a configuration management system. • Execute the test cases and procedures in the installed environment. Executing the Simulink Report Generator™ report listed in the following table generates tool verification results in the specified test report. Test File
Test Report
qualkitdo_slvnv_tcpr1.rpt1
qualkitdo_slvnv_qualificationreport1.html
The applicant will:
4-5
4
Tool Development Lifecycle
• Review the test results for correctness. • Configure the test results in a configuration management system.
Model Coverage Requirements-based test cases and procedures will be developed from the: DO Qualification Kit: Simulink Verification and Validation Tool Operational Requirements The test cases and procedures will be developed in the form of the Simulink models that exercise the model coverage capability. The test cases and procedures are documented in: DO Qualification Kit: Simulink Verification and Validation Test Cases, Procedures, and Results The applicant will: • Review the test cases and procedures for applicability to the project under consideration. • Configure the test cases and procedures in a configuration management system. • Execute the test cases and procedures in the installed environment. Executing the Simulink Report Generator reports listed in the following table generates tool verification results in the specified test reports. Test File
4-6
Test Report
qualkitdo_slvnv_tcpr2.rpt1
qualkitdo_slvnv_qualificationreport2.html
qualkitdo_slvnv_tcpr3.rpt
2
qualkitdo_slvnv_qualificationreport3.html
qualkitdo_slvnv_tcpr4.rpt
1
qualkitdo_slvnv_qualificationreport4.html
qualkitdo_slvnv_tcpr5.rpt
2
qualkitdo_slvnv_qualificationreport5.html
Verification
Test File
Test Report
qualkitdo_slvnv_tcpr6.rpt
3
qualkitdo_slvnv_qualificationreport6.html
Notes: 1 Requires a Simulink Fixed Point™ license. 2 Requires a Stateflow license. 3 Requires a Simulink Design Verifier license. The applicant will: • Review the test results for correctness. • Configure the test results in a configuration management system.
4-7
4
4-8
Tool Development Lifecycle
5 Tool Lifecycle Data • “DO-178B Checks” on page 5-2 • “Model Coverage” on page 5-4
5
Tool Lifecycle Data
DO-178B Checks The following table shows the lifecycle data for the DO-178B checks in the Simulink Verification and Validation product. The table maps the documents and artifacts to DO-178B lifecycle data items. Simulink Verification and Validation — DO-178B Checks Lifecycle Data Data
Available/ Submit
DO-178B Documents/ Reference Artifacts
Plan for Software Aspects of Certification (PSAC)
Submit
Sections 12.2, 12.2.3a, & 12.2.4
Tool Qualification Plan
Submit*
Sections 12.2.3a(1), 12.2.3.1, & 12.2.4
DO Qualification Kit: Simulink Verification and Validation Tool Qualification Plan (this document)
Tool Operational Requirements
Available
Sections 12.2.3c(2) & 12.2.3.2
DO Qualification Kit: Simulink Verification and Validation Tool Operational Requirements
Test Cases and Procedures
Available*
Section 12.2.3c
DO Qualification Kit: Simulink Verification and Validation Test Cases, Procedures, and Results qualkitdo_slvnv_tcpr1.rpt
Test Results
Available*
Section 12.2.3c
DO Qualification Kit: Simulink Verification and Validation Test Cases, Procedures, and Results qualkitdo_slvnv_ qualificationreport1.html
5-2
DO-178B Checks
Simulink Verification and Validation — DO-178B Checks Lifecycle Data (Continued) Data
Available/ Submit
DO-178B Documents/ Reference Artifacts
Software Accomplishment Summary (SAS)
Submit
Section 12.2.4
Tool Qualification Accomplishment Summary
Submit
Sections 12.2.3c(3) & 12.2.4
Notes: * Optional for verification tool qualification ** To be created by applicant The applicant must deliver data marked “Submit” to the certification authorities. Data marked “Available” must be available at the applicant’s or tool vendor’s site for inspection by the certification authorities.
5-3
5
Tool Lifecycle Data
Model Coverage The following table shows the lifecycle data for the model coverage capability of the Simulink Verification and Validation product. The table maps the documents and artifacts to DO-178B lifecycle data items. Simulink Verification and Validation — Model Coverage Lifecycle Data Data
Available/ Submit
DO-178B Documents/ Reference Artifacts
Plan for Software Aspects of Certification (PSAC)
Submit
Sections 12.2, 12.2.3a, & 12.2.4
Tool Qualification Plan
Submit*
Sections 12.2.3a(1), 12.2.3.1, & 12.2.4
DO Qualification Kit: Simulink Verification and Validation Tool Qualification Plan (this document)
Tool Operational Requirements
Available
Sections 12.2.3c(2) & 12.2.3.2
DO Qualification Kit: Simulink Verification and Validation Tool Operational Requirements
Test Cases and Procedures
Available*
Section 12.2.3c
DO Qualification Kit: Simulink Verification and Validation Test Cases, Procedures, and Results qualkitdo_slvnv_tcpr2.rpt, qualkitdo_slvnv_tcpr3.rpt, qualkitdo_slvnv_tcpr4.rpt, qualkitdo_slvnv_tcpr5.rpt, qualkitdo_slvnv_tcpr6.rpt
5-4
Model Coverage
Simulink Verification and Validation — Model Coverage Lifecycle Data (Continued) Data
Available/ Submit
DO-178B Documents/ Reference Artifacts
Test Results
Available*
Section 12.2.3c
DO Qualification Kit: Simulink Verification and Validation Test Cases, Procedures, and Results qualkitdo_slvnv_ qualificationreport2.html, qualkitdo_slvnv_ qualificationreport3.html, qualkitdo_slvnv_ qualificationreport4.html, qualkitdo_slvnv_ qualificationreport5.html, qualkitdo_slvnv_ qualificationreport6.html
Software Accomplishment Summary (SAS)
Submit
Section 12.2.4
Tool Qualification Accomplishment Summary
Submit
Sections 12.2.3c(3) & 12.2.4
Notes: * Optional for verification tool qualification ** To be created by applicant The applicant must deliver data marked “Submit” to the certification authorities. Data marked “Available” must be available at the applicant’s or tool vendor’s site for inspection by the certification authorities.
5-5
5
5-6
Tool Lifecycle Data
6 Schedule
