Introduction to Metrics that Matter™ for Microsoft Certified Trainers
Metrics that Matter for MCTs
Contents Essential MTM Links MCT Requirements - Overview Accessing MTM as an MCT First Steps: Registering for MTM Managing your MCT Profile How to Schedule Classes in MTM When Scheduling Classes at a CPLS Acknowledging a Certified Class If you are scheduled for a class that you will not be teaching How to Schedule Classes at Non-CPLS Location How to Review Your Class Criterion How to Add Client/Customer Information How to Use MTM in the Classroom How to Use Metrics that Matter in the Classroom How to Introduce Metrics that Matter to Students Using MTM Reports How to Accessing Reports in MTM Important Reports in MTM Scheduling Reports for Auto-Delivery by E-mail Using the Filtering Tool in MTM Additional Information and FAQ Quick Reference Card FAQ Additional Services Available Appendix
© 2008. KnowledgeAdvisors. All Rights Reserved.
www.KnowledgeAdvisors.com
3 4 5 5 6 7 7 7 7 8 8 9 10 10 11 12 12 13 16 17 19 19 19 21 22
Page 2
Metrics that Matter for MCTs
Metrics that Matter (MTM) Quick Reference Card
Instructor EssentialChecklist: MTM Links Essential MTM Links
MCT Registration Page http://www.mtmlearning.com/mtmenrollment/mctenrollment.htm Resetting your MTM Password https://www.metricsthatmatter.com/mtm/Popup/ForgotPassword.aspx MCT Instructor Survey Portal www.metricsthatmatter.com/mctsurvey/coursesurvey MCT Reporting Log In https://www.metricsthatmatter.com/mtm/default.aspx MTM Help Guides http://www.mtmlearning.com/lghelp/html/introduction.htm KnowledgeAdvisors User Contacts o User Support (8:30 AM – 5:00 PM CST)
[email protected] +1 (800) 561 3341 (within the US) +1 (312) 676 4400 (worldwide) o General Information
[email protected]
© 2008. KnowledgeAdvisors. All Rights Reserved.
www.KnowledgeAdvisors.com
Page 3
Metrics that Matter for MCTs
MCT Requirements - Overview TM
What is required of each MCT in using Metrics that Matter ? The following is an overview of the MCT requirements regarding MTM. Each of these steps is explained in detail in the following pages of this guide. 1. Registration: You must receive a login ID and password from KnowledgeAdvisors to access the MTM system. a. New Instructors: If you do not have login information to access MTM then you must register to use the tool and receive login information. To register, access your Microsoft portal page. b. Existing Instructors: Using the Profile Management tool, add your MSP # to your existing MTM account 2. Manage MCT Profile: (2 minutes) Log in and modify your profile. This must be done for your information to be included in MCT reporting, and also so that clients can schedule you for training. This includes updating your MCP ID, and your home region. 3. Manage Classes: (1 minute per class) Each class will need to be scheduled in the MTM system prior to collecting evaluations. Classes will be scheduled by either by the CPLS, or by you, the Instructor. 4. Provide students with the URL to submit evaluations: (2 minutes per class) For each class the instructor will be responsible for communicating the appropriate URL to the students to collect their feedback. The CPLS scheduling the class might communicate the URL to you, or you can find the URL by logging in to your own account. 5. Monitor results: (Ongoing) As an MCT you will also have the ability to monitor all of the feedback entered by students in any class that you teach.
© 2008. KnowledgeAdvisors. All Rights Reserved.
www.KnowledgeAdvisors.com
Page 4
Metrics that Matter for MCTs
Metrics that Matter (MTM) Quick Reference Card S
Instructor Checklist: Accessing MTM as an MCT Accessing MTM MCT Accessing MTM asas anan MCT
Registering for MTM You must have an MTM login ID and password to access the MTM system. Step 1: Go to the MCT Registration Page: http://www.mtmlearning.com/mtmenrollment/mctenrollment.htm Step 2: Fill in all requested fields. The following fields are required: Full name Email* MCP ID The following optional fields are also very useful, in applicable Your current MTM username** Date of first class *** Region Date of first class * Email address: the login information to access the system will be sent via email to the address provided here. **You may have already been provided an MTM user account through a CPLS if you have already used the system to collect feedback in the past. Please provide that login information here, if applicable. *** Date of first class taught in MTM: this information will only be used to help prioritize requests. Step 3: Receive login information. Approximately one week from date of registration. Step 4: Log in to Metrics that Matter and update your profile. See instructions under ―How to Manage Your Profile‖ on page 6 of this document. Regarding Lost or Forgotten Logins If you have lost or forgotten your MTM login, please contact
[email protected] to research this inquiry.
© 2008. KnowledgeAdvisors. All Rights Reserved.
www.KnowledgeAdvisors.com
Page 5
Metrics that Matter for MCTs Managing yourChecklist: MCT Profile Instructor
Metrics that Matter (MTM) Quick Reference Card
Managing your MCT Profile
An MCT’s profile in MTM includes all of the information that will be used to identify the instructor in the system, not only in online reporting features, but also as part of the MCT program and for scheduling purposes at a CPLS. It is critical that this information be updated as soon as login information is received. When a CPLS schedules a class, they will search for an instructor by Last Name and MCP ID – this information is not in MTM until updated by the MCT. By providing your name and MCP ID to a CPLS you are giving them permission to find your name and schedule you for training. How to Start Managing your Profile in MTM: 1. Go to the MTM Login Page at: https://www.metricsthatmatter.com/mtm/default.aspx 2. Type in your user name and password 3. Arrive at the Dashboard tab in MTM. Click on the My Settings tab
4. You will arrive at the My Settings Menu 5. Insure that your name is accurate, and reflects the name you will plan on having the CPLS do a search on. 6. Also take this opportunity to update Email & Password information 7. Click Update Settings to commit any changes 8. Navigate to the My Setting menu and click on My Instructor Certifications 9. Input your MCP ID and MCT Region to reflect those assigned by Microsoft. Click Update Settings to commit any changes
© 2008. KnowledgeAdvisors. All Rights Reserved.
www.KnowledgeAdvisors.com
Page 6
Metrics that Matter for MCTs
Metrics that Matter (MTM)How to Schedule Instructor Checklist: Classes in MTM
How to Schedule Classes in MTM
Quick Reference Card S
When Scheduling Classes at a CPLS In order for a CPLS to receive credit from Microsoft for offering the course you will be teaching, the courses should be scheduled under the CPLS’s account. 1. Provide the Last Name, as it appears in MTM, and your MCP ID to the CPLS MTM Administrator. They will use this to identify your user account in MTM a. Note: The CPLS will not be able to schedule you for a class until you have entered your MCP ID into your MTM account (see How to Manage Your Profile above). 2. Confirm with the CPLS that the correct official courseware has been scheduled for this class 3. An email will notify you that you have been scheduled for the class. Note: Classes may be scheduled at separate Learning Provider
Acknowledging a Certified Class Once the class has been scheduled, you must acknowledge that it is scheduled correctly and linked to your class. 1. Click the Administration tab in MTM
2. From the Schedule Training Menu, select Acknowledge Class from the Schedule Classes box
3. 4. 5. 6.
Edit the class Start/End date to include the date of the class Optionally, input specific information like the Course Name in the search Press Search From the list of classes, select the specific one in question and click Acknowledge Selected
If you are scheduled for a class that you will not be teaching 1. Click the Administration tab in MTM 2. From the Schedule Training Menu, select Acknowledge Class from the Schedule Classes box 3. Edit the class Start/End date to include the date of the class 4. Optionally, input specific information like the Course Name in the search 5. Press Search 6. Click the Remove button to reject the class
© 2008. KnowledgeAdvisors. All Rights Reserved.
www.KnowledgeAdvisors.com
Page 7
Metrics that Matter for MCTs How to Schedule Classes at Non-CPLS Location If an instructor is delivering training for a customer that does not utilize the Metrics that Matter system, then you - the instructor - will be responsible for scheduling each delivery in MTM. This describes the process that an independent instructor will follow to schedule classes when their customer/learning provider is not using MTM. Note: that by following this method only the instructor will be able to collect data for this class – not the learning provider. 1. Click the Administration tab in MTM 2. From the Schedule Training Menu, select Add from the Schedule Classes box
3. Chose a Client/Customer from the Drop Down Box* a. If there are no Client/Customers listed in the drop down box, you must Create a Client/Customer. 4. Chose your correct Course from the Drop Down Box* a. Note that only Microsoft Official Courseware is available for scheduling 5. Chose a Start Date for the class. This is the day the class begins.* 6. Chose an End Date for the class. If the class finished the same day, this can be the same as the Start Date* 7. Choose the appropriate Learning Methodology* 8. Select whether you would like to limit the number of Evaluations Submitted to the # of Students Enrolled 9. Include the # of Students Enrolled in the course. 10. Click the Add Button. * Required Information
How to Review Your Class Criterion 1. Click the Administration tab in MTM 2. From the Schedule Training Menu, select Search from the Schedule Classes box 3. Chose the Start Date for the class
4. Chose the End Date for the class 5. Click Search
© 2008. KnowledgeAdvisors. All Rights Reserved.
www.KnowledgeAdvisors.com
Page 8
Metrics that Matter for MCTs 6. The Edit action mirrors the Class Add menu 7. The Summary action will provide you with an overview of the class information. It will also provide you the information regarding the surveys attached to your class. Of note is the generic, direct link to your class survey 8. The Audit page identifies actions taken by users to the class details such as ―add‖ or ―modify‖ changes
How to Add Client/Customer Information Your classes must be tied to a Client/Customer before you can successfully schedule a class. 1. Click the Administration tab in MTM 2. From the Schedule Training Menu, select Add from the Schedule Classes box 3. From the Class Add screen, click the link in the upper right hand corner labeled ―Create a Client/Customer‖
4. The Add Client / Customer screen consists of the following information: 5. Client/Customer Name (Example: Learning Provider Organization or Company for whom the training is being provided. This must be a unique field)* 6. Site Name* (Example: A specific ―location‖ at which the training takes place) 7. City* 8. Time Zone 9. Language * Required Information
© 2008. KnowledgeAdvisors. All Rights Reserved.
www.KnowledgeAdvisors.com
Page 9
Metrics that Matter for MCTs
Metrics that Matter (MTM) How to Use MTM Instructor Checklist: in the Classroom
How to Use MTM in the Classroom
Quick Reference Card S
How to Use Metrics that Matter in the Classroom To collect feedback from the students you will need to provide them with a URL to access the evaluation form online. Step 1: Generating the URL. The URL that should be given to the students can be found in one of two ways: 1. If at a CPLS, the center will provide each instructor with the appropriate URL Or – 2. The instructor can click Manage Classes. Search for the class, then select the Summary action and click Go. See page 9 of this guide which displays the exact location of the URL. Step 2: Providing the URL to the students. This can be provided to the students a few ways: Simply share this information verbally in the classroom Write the information on the whiteboard or overhead, so they can type the page into their browser Distribute via email to each student so they can click on the link and access the webpage Save the URL as a favorite or the homepage on the machines in the classroom. NOTE: This is the most preferable option as it eliminates the possibility of errant typos by the student. Step 3: Accessing the Student URL online. When the students first access the site, they will see a screen that asks for their email address. Please explain that this is only for the purpose of sending a follow up evaluation that will be sent 60 days after training has completed.
© 2008. KnowledgeAdvisors. All Rights Reserved.
www.KnowledgeAdvisors.com
Page 10
Metrics that Matter for MCTs
Step 4: Introducing the Evaluation to the students in the classroom. The next page shows an Instructor Checklist that can be used to make sure that the main talking points are covered with the students.
How to Introduce Metrics that Matter (MTM) to Students
Metrics that Matter (MTM) is the online course evaluation tool we are using to collect all evaluations. It is: Faster than completing a paper evaluation More confidential A more accurate way to collect data
Instructions to Students Let’s review the steps to complete your evaluation. 1. Go to URL address __________________________. Reminder - ALL data is confidential. 2. Input your (student) Email address - used only for a follow-up evaluation: Instructor Note - Sixty days following a learning event, students will be sent a brief, follow-up evaluation and again, responses will be kept confidential. Emails will not be used for marketing purposes. (If student does not have email address, user may type in dummy address). It is very important to Microsoft that data be collected both on the Post event and Follow Up evaluations therefore please encourage students to enter their email address. 3. Select the appropriate class from the drop-down list. 4. Please complete your evaluation. 5. Click Submit upon completion. Do not click Submit multiple times, it may take a few seconds for the next screen to appear. 6. Wait for Thank You to appear on the screen notify the instructor to receive your certificate. Thank You for attending the Class!
© 2008. KnowledgeAdvisors. All Rights Reserved.
www.KnowledgeAdvisors.com
Page 11
Metrics that Matter for MCTs
Instructor Checklist: Using MTM Reports Using MTM Reports
Metrics that Matter (MTM) Quick Reference Card S
Metrics that Matter has made available many online reporting features that you can utilize to track your performance, and use this information to improve future training programs and prove the value that you provide as an instructor. The reporting suite looks at survey responses from students and provides insight into the results of the class. Microsoft uses particular measurements in order to evaluate different elements of a class, including Instructor performance. Following are the steps an MCT can follow in order to view the Instructor Performance score most valuable to Microsoft.
How to Accessing Reports in MTM 1. From the Dashboard tab, click the Reporting tab
2. Under the My Recent Reports tab will display any recently ran reports. If you haven’t recently ran any reports, it will provide a list of Featured Reports, which will be most useful to you as an MCT
3. Clicking on any of the report names will enable you to open the report and begin inputting your criterion to retrieve your class data.
© 2008. KnowledgeAdvisors. All Rights Reserved.
www.KnowledgeAdvisors.com
Page 12
Metrics that Matter for MCTs Important Reports in MTM Access all of these reports from the My Recent Reports Menu under the Reporting tab
Evaluation Retriever View by class and by student the actual evaluations submitted by the students. Useful for the most granular reporting needs. a. Chose a Start and End date in which your class was taught b. Click Search to produce a list of all the classes you taught during that timeframe
c.
Click on the class that is populated in the list below the Search button
d. If you would prefer to have this e-mailed to you instead of shown in MTM, select Auto Email in the Report Output box e. The name of the report is customizable, so identifying information can be included to help make the reports unique
Class Summary Report Summarize class data by Question Category and by Specific Question a. Chose a Start and End date in which your class was taught b. Click Search to produce a list of all the classes you taught during that timeframe c.
Click on the class that is populated in the list below the Search button
© 2008. KnowledgeAdvisors. All Rights Reserved.
www.KnowledgeAdvisors.com
Page 13
Metrics that Matter for MCTs d. Chose which of the Question Categories you would like to include in the query. This can be as few as one question category, and as many as all. e. Click Generate Report
Learner Comments View the comments students have included in open ended questions on the class survey. Comments are grouped by question at the course or overall level. a. Used to view the comments students have included in open ended questions on the class survey b. Chose a Start and End date in which your class was taught c.
Click Search to produce a list of all the classes you taught during that timeframe
d. Click on the class that is populated in the list below the Search button
Quick Question Report Summarize all responses to questions for a single class or custom query of classes a. Used to summarize all question responses onto one report b. Used to view the comments students have included in open ended questions on the class survey c.
Chose a Start and End date in which your class was taught
d. Click Search to produce a list of all the classes you taught during that timeframe e. Chose which of the Question Categories you would like to include in the query. This can be as few as one question category, and as many as all. f.
You may also chose to Aggregate (sum) the instructor data in this report, and optionally view All Comments.
g. Click Generate Report h. Will display the number of responses to each answer selection, as well as the % of respondents and the Average student score. i.
Available at the end of the report is a link to View Comments as well.
Variance Report Measures the percentage of learners whose responses fall within the bounds set in the report. Excellent resource to measure response data associated with your Overall NSAT measurement. a. Excellent resource to measure response data associated with your Overall NSAT measurement. b. In Report Type, select Instructors for an overall view of performance, and Course or Class for more granular levels c.
In the Filter section,
© 2008. KnowledgeAdvisors. All Rights Reserved.
www.KnowledgeAdvisors.com
Page 14
Metrics that Matter for MCTs i.
Select Microsoft and Courseware Library – MS as the Content Provider
ii.
Select ―Overall, how satisfied are you with this course? [Instructor Data]‖ as the Question
iii.
In the Pop-up Box, click the radio button for ―My report should display all data from students that selected these answers for this question only‖
iv.
Ensure all check-boxes are marked
v.
Click Close
vi.
The variance report will display your question scores so that
1. VSAT Scores (8 and above) = Green 2. SAT Scores (Between 4 and 8) = Yellow 3. NSAT Scores (Lower than 4) = Red 4. Note that this is only representative of the part of Overall NSAT calculation associated with this question in the evaluation
© 2008. KnowledgeAdvisors. All Rights Reserved.
www.KnowledgeAdvisors.com
Page 15
Metrics that Matter for MCTs Scheduling Reports for Auto-Delivery by E-mail A powerful tool to employ to greatly reduce the amount of time spent administering reports can be the utilization of Automatic Report Deliver function. 1. From the Reporting Tab, under the Manage Reports menu, select Schedule Class Reports
2. From this tool, any of these reports can be scheduled to be automatically delivered to the e-mail of your choice 3. To enable the reports to be scheduled automatically, check the Schedule in MTM box 4. Select the # of Days after Class Ends to determine when you’ll receive the report. Most students will respond to surveys within 7 days. Note that more data may be collected after the report is sent. 5. Also select what Time of Day the report is delivered to your inbox and whether the report is attached to the e-mail or embedded
© 2008. KnowledgeAdvisors. All Rights Reserved.
www.KnowledgeAdvisors.com
Page 16
Metrics that Matter for MCTs 6. In the Filter / Email Settings, chose the same Query criterion as you would in the report query. Including Filtering Preferences
Using the Filtering Tool in MTM The filtering tool allows the user to select specific criterion to include into their reports, and is used in any of our manually ran Aggregate Reports, such as the Variance Report, or on the Scheduled Class Based Report. 1. From the Reporting Menu, chose an Aggregate Report, such as the Comparative Performance. This report will show you the average score of different criterion in a graphical way. 2. At the bottom of this Report Query is the My Filters box. This will include any filters that the user has created, and any filters that have been created for the user by a system administrator (i.e. at a CPLS)
3. Clicking the Create New Filter button will open the Filter Dialog.
© 2008. KnowledgeAdvisors. All Rights Reserved.
www.KnowledgeAdvisors.com
Page 17
Metrics that Matter for MCTs
4. Each attribute opens a separate dialog box, populated with all the values of that attribute associated with your user. 5.
Find the criterion you would like to filter to from the Available Filter Items and check the box to include it. Click Add to Filter.
6. Once finished click Save. 7. Once all the criterion for filtering have been selected, chose a Filter Name, and Save the filter. 8. The filter will now be available in the My Filter box. To include it in the Report Query, select the Checkbox next to the Filter. 9. Click Generate Report.
© 2008. KnowledgeAdvisors. All Rights Reserved.
www.KnowledgeAdvisors.com
Page 18
Metrics that Matter for MCTs
Metrics that Matter (MTM) Quick Reference Card
Instructor Checklist: Additional Information and FAQ Additional Information and FAQ
How to Get Help on Metrics that Matter The following support tools are provided and available to you: Welcome Kit: Introduction to Metrics that Matter for Microsoft Certified Trainers User Guides: Available online using your login ID and Password by clicking on: MTM Help from the top blue selection bar The Help tab available from within every tool, for help and information on that specific tool. Using the Search feature available from the top blue selection bar on each page of the system. KnowledgeAdvisors User Contacts o User Support (8:30 AM – 5:00 PM CST)
[email protected] +1 (800) 561 3341 (within the US) +1 (312) 676 4400 (worldwide) o General Information
[email protected] You will receive a response within 24 hours of your query.
FAQ I have Multiple MTM Accounts or was given a new account by a CPLS. How do I manage my information? In cases of multiple accounts that have collected data about your classes, KnowledgeAdvisors can merge the multiple accounts into one. Please contact
[email protected] and provide information regarding any accounts you have and which account you would like these to be merged into. While this data cleanup is taking place, collect data under your preferred account and provide that account data for any new classes for which you are scheduled. How can I review students’ feedback immediately after class and evals were entered? The last 5 classes that you’ve collected data for will be displayed on your home page under the heading My Classes. Select any of the following reports to the right of a class: Evaluation Retriever – Displays individual evaluations for each student Class Summary – Combines all students’ responses for the class into one report Learner Comments – Displays all comments from students regarding the class Quick Question – Shows distribution on for the scale-based questions and class averages. Performance to Goals – Compares class scores to user preset goals on the My MTM > Set Goals tab.
© 2008. KnowledgeAdvisors. All Rights Reserved.
www.KnowledgeAdvisors.com
Page 19
Metrics that Matter for MCTs To view reports for other classes and generate reports from the Dashboard 1. Select a Report from the My Favorites widget
2. Enter a date range a. The date range that you enter will search for all classes ending between those two dates 3. Generate the report a. For the Evaluation Retrieval report, click the View All link to view all students’ evaluations. How do you administer classes? Please see our MTM Quick Reference Guide on Classes How do you retrieve reports about your classes? Please see our MTM Quick Reference Guide on Reporting
© 2008. KnowledgeAdvisors. All Rights Reserved.
www.KnowledgeAdvisors.com
Page 20
Metrics that Matter for MCTs Additional Services Available Through Metrics that Matter you have online access to a variety of reports to help you measure and manage your performance. KnowledgeAdvisors provides many additional services that you may find useful:
Marketing information: How to use the information available in Metrics that Matter to generate additional business and prove the value in your training;
Professional Development: Attend training on analyzing return on investment;
Enhanced Client Reporting: Generating custom reports for clients, possibly to be used as an additional fee-based service that you provide;
Analyzing Metrics: Gain a better understanding of learning analytics, or utilize KnowledgeAdvisors’ team of experts to provide deeper analysis on your results to help improve your performance;
Referrals: If you have a client that would also be interested in using Metrics that Matter.
Please contact KnowledgeAdvisors at
[email protected] or access our website at www.knowledgeadvisors.com if you are interested in learning more about any of the services listed above.
© 2008. KnowledgeAdvisors. All Rights Reserved.
www.KnowledgeAdvisors.com
Page 21
Metrics that Matter for MCTs Appendix What is Metrics that Matter? KnowledgeAdvisors is a practical, scalable and repeatable way to quantify learning investments. ® Our proprietary Metrics that Matter technology helps organizations continuously measure the satisfaction, effectiveness, impact, business results and ROI on learning investments. Through customizable templates and creative technology, Metrics that Matter™ automates the administrative aspects of measurement by collecting, storing, processing and reporting results— leaving time for analysis, training improvements and showcasing value to stakeholders. Metrics that Matter™ (MTM) is KnowledgeAdvisors’ market leading learning analytics and measurement tool – an online system that automates the process of collecting and analyzing feedback from learners. MTM wraps technology around timeless and industry proven methodologies by Dr. Donald Kirkpatrick and Dr. Jack Phillips. The result is a powerful way of measuring the impact of training at the point of delivery and back on the job. Leveraging standard data collection instruments that can be customized to gather data from various respondents (learners, managers) at various points in time (at time of training, back on the job), users of MTM can easily view over 100 web-based reports. Data from LMS, ERP, CRM, MRP, and other systems can be integrated into MTM for more comprehensive analysis. Linking measurements back to the Learning Levels gives the manager of a learning organization the power to continuously improve training and proactively showcase value to the stakeholders outside the learning organization. MTM utilizes standard measurement tools to collect feedback from learners in general at two points in time: Post Event Survey: collect feedback from learners immediately after class to understand if the elements of the training experience were conducive to learning, did they actually learn anything, and do they plan to use that information. Follow Up Survey: 60 days after training has completed, we then follow up to ask additional questions to understand if the learners were actually able to use what they learned, and how it impacted their job performance. Working with premier thought leaders like Kirkpatrick and Phillips, these measurement tools combined provide insight into each of the levels of learning, as illustrated below.
Benefits of Using Metrics that Matter Streamlined evaluation process, quick and easy to implement. Drive superior business results by making decisions based on quantifiable metrics. Improve client relationships by demonstrating value and sharing key feedback. Engage clients more proactively based on access to real-time feedback. Access critical marketing information to drive additional business. Access to industry benchmarks that enable you to compare performance to the rest of the industry.
© 2008. KnowledgeAdvisors. All Rights Reserved.
www.KnowledgeAdvisors.com
Page 22
