ThinkCodex

Tag: theory

  • 5 Methodology Models to Measure the Effectiveness of Trainings: A Comprehensive Guide

    5 Methodology Models to Measure the Effectiveness of Trainings: A Comprehensive Guide

    Written by: Andrew Lau (CEO of Think Codex)

    As companies invest more in Learning and Development (L&D) programs to help employees acquire new skills and stay competitive, measuring the success of these programs becomes increasingly important. In this article, we’ll explore different methodology models that L&D professionals can implement and the critical metrics they need to measure to demonstrate the value of their programs. We start off by examining the most widely used learning effectiveness model – Kirkpatrick Model and then comparing other models against it. 

    Kirkpatrick Model by Donald Kirkpatrick

    The Kirkpatrick Model is one of the most commonly used methodology models in L&D. Developed by Donald Kirkpatrick in the 1950s, this model has four levels of evaluation:

    The Kirkpatrick Model is a valuable framework for evaluating the effectiveness of L&D programs. However, it’s important to note that the levels are hierarchical, and each level builds on the previous one. For example, learners are less likely to learn from the program if they don’t react positively to the program. They are less likely to change their behavior if they don’t learn from it. And if they don’t change their behavior, the program is less likely to positively impact the organization’s results.

    Comparative Methodology Models

    In addition to the Kirkpatrick Model, there are other methodology models that L&D professionals can implement to evaluate their programs:

    Phillips Model by Jack Phillips

    This model takes a step further by not just measuring Levels 1 to 4 which is similar to the Kirkpatrick Model but has a Level 5 measuring the return on investment (ROI) of the training program. The Level 5 ROI measurement uses data from L2 Learning, L3 Behaviour & L4 Results to create a model for finding out what monetary returns the organization is actually getting back from the training dollars spent.

    L1 – L4: Following Kirkpatrick Model
    L5: Return on Investment

    In the form of calculation,

    ROI (%) = (Net benefits of the training program / Total program cost)  x 100

    In the end, the Level 5 calculation comes up with a percentage (%) of ROI. While this is something that is indeed good to measure, there remain a number of challenges in implementing Level 5. Among them:

    1. High cost of effort to measure L5 – it takes extensive data collection and working with a number of stakeholders to get accurate data.

    2. Delayed Results – as a lot of effort is required, the results of L5 can often be delayed beyond the expected timelines of evaluation by stakeholders.

    3. Accuracy of measurement – while calculating, there are a number of assumptions made to put both tangible and intangible factors in focus. Some of these are subjective and up to interpretations depending on stakeholders. It’s important to note that this challenge appears in L4 of the Kirkpatrick & Phillips model to a lesser extent.

    4. Ideal but not pragmatic – while it’s great to have these measurements, very few training programs warrant L5 measurements. Jack Phillips himself noted that only 5-10% of training programs require measurements at L5. 

    5. Lack of data or cooperation from the customer – sometimes a customer just does not have the data required to make a L5 measurement as not all companies track the data of their operations. In the absence of such data, it is often a painstaking process of creating new data sets which can require additional work for stakeholders – which leads to resistance from them.

    Brinkerhoff Model by Robert Brinkerhoff

    This model is also known as Brinkerhoff’s Success Case Model (SCM) which emphasizes the importance of post-training support and reinforcement to ensure that learners apply what they have learned. Brinkerhoff states that traditional evaluation models such as Kirkpatrick & Phillips models were wrong to just focus on assessing the scope of a training’s effect. He says that the true measure of business results is from performance. SCM is not concerned about finding the average performance of training participants (Kirkpatrick & Phillips Models), but focuses on looking for extreme examples on both ends of measurements.

    As such this model measures the performance of the most successful participants and the least successful participants as use cases to understand how well did a learning program perform. The model looks at contrasts or extremes from both ends by seeking to understand 2 important areas:

    1. In a best-case scenario, how well did a program perform? This is derived from analyzing participants who show the most increase in performance after a training program.

    2. In a worst-case scenario, how badly did the program perform? This is derived from analyzing participants who may show the least or no increase in performance after a training program.

    Brikenhorff Model
    Brikenhorff Model

    How this model differs from Kirkpatrick & Phillips models is that it requires a few things:

    • The development of an Impact Model – the definition of what success looks like.

    • Creating a Success Case Study.

    • Creating & implementing surveys that can identify best-case and worst-case scenarios.

    • Conducting interviews.

    • Documenting successful cases.

    • Drawing insights, summarizing conclusions, and providing recommendations to stakeholders. 

    Kaufman Model by Roger Kaufman

    This model is based on Kirkpatrick’s model and seeks to improve on it by looking at measurements at a detailed level (L1a and L1b) and big picture level (L5). It’s broken into 5 levels – though if counted, there are actually 6 levels.

    • L1a Input: This level covers training materials.

    • L1b Process: This level covers the delivery of the learning process.

    • L2 Acquisition (Micro): Individual & small group payoffs. Did the learner “acquire” the learning?

    • L3 Application (Micro): How well the participants utilize what they have learned on the job.

    • L4 Organizational Results (Macro): Organizational level payoffs – business results.

    • L5 Customer / Societal Results (Mega): Customers & Society payoffs – the 2 major stakeholders that an organization serves.

    Here are some key differences in the Kaufman Model:

    1. Group ‘clients’ into Micro (Individual & Small Group), Macro (Organization), and Mega (Customers & Society). This is found in L2, L3, L4 & L5.

    2. Divides Kirkpatrick’s Level 1 (Reaction) into Kaufman L1a (Input) & L1b (Process).

    3. Introduces a new Level 5 (Customer & Societal Results) which looks at a much broader picture.

    So what are the pros of the Kaufman model? Firstly the split in Kirkpatrick’s L1 (Reaction) breaks down the experience into 2 Kaufman factors – supporting materials (L1a Input) and delivery (L1b Process). This helps to isolate the issues of experience by looking at the quality of materials or the quality of delivery.

    What are the cons of the Kaufman model? While aspirational and good from a high-level view, trying to measure customer & societal payoffs (L5) require extensive effort, time, and budget. There’s also a high level of ambiguity as there are too many factors at play when we talk especially about customer & societal impacts. In fact, by measuring at L5, it’s hard to isolate these impact factors. This goes against Kaufman’s original intent in where else the goal in L1a & L1b was to isolate factors.

    So on one hand, the goal was to improve on Kirkpatrick’s model by isolation (L1a & L1b), L5 instead went against the same goal of isolation, thus giving conflicting views on the Kaufman Model.

    Where does Gamification Fit Into all these Models?

    Firstly, what is gamification? Gamification is the use of game phycology & game mechanics in non-game environments while achieving non-game outcomes. Such a use case is Learning & Development.

    Coming back to learning evaluation models, at the core – most models measure 4 things:

    • Experience – What was the experience like?

    • Learning – What did the person learn?

    • Behaviour – Is there a change in behavior at work?

    • Results – What individual, team, or business outcomes can be seen?

    Gamification’s methodology itself is not an evaluation model but instead is considered one of the most effective ways to drive impact at the levels of Experience, Learning & Behaviour which ultimately leads to Outcome. Let’s break it down to how gamification does that.

    • Experience – a great experience comes from the right type of engagement. Gamification and game-based learning use the methodology Creation (Ideation & Experimentation) as opposed to just Consumption (Information Download) found in most learning methodologies.

    • Learning – the most important learning comes from not just the acquisition of the knowledge but rather the application of the knowledge. The application of knowledge is typically where challenges and problems will appear. As such, it’s critical that application can happen within the learning session itself. Game-based learning methodologies such as Business Simulations (emphasis on Business) allow participants immediate application & feedback. Coupled with debriefs that tie it back to organizational context, learning becomes highly effective.

    • Behaviour – one of the challenges of getting a learner to change their current behaviors is their current habits. Often what is learned is forgotten or replaced with resistance to old habits. Gamification’s iterative and habit-looping techniques get the learner to learn new & better habits which are then overlaid on top of old habits. When learners are back at work, they then begin to practice what they learn from the new habits.

    Results – a changed behavior comes down to 3 factors:

    1. The ability to retain the knowledge learned

    2. The retrieval of new habits

    3. The confidence to try out what was learned.

    The confidence factor comes from the retention of knowledge and retrieval of new habits. If a learner can remember what they learned and also act out what they learned, then there is confidence to behave in a new way – which then leads to business results.

    How to Choose the Right Model

    1. Do you have data in place for measurements? If you know where the data is, then a number of models might work.

    2. Do you want to create new data sets for measurements? This can often be a complex and long process.

    3. Do you have the cooperation of the different stakeholders? Measurements not only require input from the HR professional but stakeholders from the business. Many often have no

    4. Do you understand the different models? Do you know what is measurement required and how success is defined in each of the models and how it will fit into your organization?

    5. Does the effort required makes it worth getting the results? In essence, this is a cost-benefit analysis. Is the results worth the trouble of going through the measurements – some of which require extensive effort.

    Using Different Models For Different Learning Programmes

    Should we use different models to measure different training programs? From an outcome perspective, this is ideal. From a practical perspective, the costs may far outweigh the benefit. Our recommendation is using different models should be determined by:

    1. What type of results do you really need?

    2. What type of effort are you willing to put in?

    3. What type of monetary cost are you willing to invest?

    4. What type of support do you have in the organization?

    5. What level of expertise do you or your vendor have on the different models?

    Why Measure L&D Metrics?

    Before delving into the different methods for measuring L&D metrics, it’s important to understand why it’s necessary. Here are some key reasons:

    • Measuring L&D metrics helps organizations understand the effectiveness of their training programs and identify areas where improvement is needed.

    • By measuring the ROI of L&D initiatives, organizations can justify the investment in these programs and secure funding for future initiatives.

    • Improve employee engagement which can lead to increased productivity and reduced turnover.

    • Increase employee performance because when they have the skills and knowledge they need to do their jobs well, they are more likely to perform better.

    Measuring the effectiveness of L&D programs is crucial to ensure that they are achieving their intended goals. By using key metrics and frameworks such as the Kirkpatrick Model, organizations can get a better understanding of the impact of their L&D programs. Subscribe to Think Codex’s content and explore our other resources to learn how to measure their effectiveness and achieve your goals using frameworks like the Kirkpatrick Model.

  • Strategic Thinking: When Theory Meets Workplace

    Strategic Thinking: When Theory Meets Workplace

    By Andrew Lau

    Strategic thinking has been an important and famous topic with regards to organisations, so much so that it has been mentioned countless times in management, innovation and corporate articles and blogs.

    There are numerous workshops and trainings with regards to this area and we would like to shed some light on this topic as well and how simulations actually play a part in honing this skill.

    We would start by introducing what is strategic thinking, its crucial components, an example of how it applies to our daily life and how simulations are able to effectively address this skill. We would like you to take note that there are countless amount of research with regards to strategic thinking, with numerous amount of segregation, components and definition.

    We take on the theories of strategic thinking from Liedtka, as we believe that it is the most wholesome and well defined one yet.

    What is Strategic Thinking?

    The best way of learning what is strategic thinking is by knowing what strategic thinking is not. On the surface level, when strategic thinking is mentioned, we have in our minds that it is thinking about the strategy itself. It may appear to us that when we approach a project, planning up a strategy with regards to it denotes strategic thinking, but that is not the case.

    Strategic thinking at an actual fact is a way of thinking, more specifically a mindset with regards to approaching a certain task or project, or in more layman terms, a more organised way of thinking. 

    You may ask, “what is the difference between both of the definition that you had just spouted?”

    Well, the difference between making a strategy and approaching a project with a strategic mindset is the flexibility and adaptability that you possess when critical moments appear (i.e. when something does not go as planned).

    A strategic mindset makes you think from your core of what is important to you, what are your strengths, what are your tolerance level for risk, who are your stakeholders, how are you approaching your future, and how you would turn your challenges into opportunities; which a mere strategy (with multiple backup plans) could not have addressed. It gives you a wholesome picture of what is happening within and around you for you to make the best possible decision at that moment in time.

    And as Liedtka personally has said:

    Strategic thinking, on the other hand, is a synthesizing process, utlizing intuition, and creativity, whose outcome is “an integrated perspective of the enterprise”

    We hoped that we have managed to clear some of the misconceptions with regards to strategic thinking. Now let us move on to the elements of it to fully understand it better.

    Components of Strategic Thinking

    In order for strategic thinking to happen, all the following components have to be realised. Some could be more salient at different points of time depending on the circumstances. They are:

    1. Systems Perspective: Having a mental model of the complete end-to-end systems of value creation, and understands the interdependencies within it.

    2. Intent-Focused: A strategic intent is differentiated and implies a unique point of view about the future, which communicates a sense of direction, discovery and destiny to the team or individual.

    3. Intelligent Opportunism: The ability to adapt without having to depend on top management strategies, which leave open to the possibility of new strategies emerging.

    4. Thinking In Time: After having seen the future that we want, what must we keep from our past, lose from that past, and create in our present to get there?

    5. Hypothesis Driven: Mirroring the “scientific method” by generating hypothesis and engaging in testing as central activities. Hypothesis generation asks the questions “What if…?” while hypothesis testing follows the critical question “If… then…?”

    Understanding and practicing these five components will give you a powerful source of competitive advantage by (as quoted by Liedtka)”

    Their whole system perspective should allow them to redesign the processes for greater efficiency and effectiveness. Their intent-focus will make them more determined and less distracted by their rivals. Their ability to think in time will improve the quality of their decision-making and the speed of implementation. A capacity for hypothesis generation and testing will incorporate both creative and critical thinking into their processes. Intelligent opportunism will make them more responsive to local opportunities… This meets the three fundamental strategically valuable capability: (1) they create superior values for customers, (2) they are hard for competitors to imitate, (3) they make the organisation more adaptable to change.

    How can strategic thinking be applied in our daily lives?

    Let us take a scenario here. You are currently in the marketing department of a FMCG industry and you are considering a career change because you have seen that your peers are switching companies and you feel that you have done enough time within this company. So most likely you would think of, “should I take a leap?” or “where should I go to next?” Ideally, it makes sense to venture into the same industry at the next company but you would like to try something new as well hence, you are in a dilemma. So let us apply the components of strategic thinking into this situation. It is important to take note that there are no right or wrong to this and would depend on individual differences.

    The conventional definition of strategic thinking would get you to immediately come out with a long term plan, and mini goals to get there. What are your backup options if one the other does not work as planned. That would be your strategy. As you go down the line, the same cycle dissatisfaction happens again and you revamp or come out with another strategy. There are some things to be questioned here.

    • Have you looked at the big picture? Such as what are some of your strengths and your important networks that you could cascade into opportunities? What you think you are good at versus what people value about you?
    • Have you looked into your past and find out what you are good and poor at and how have you changed since then?
    • Have you looked into the future of job needs? And what kind of skills do you need to equip yourself with in order to meet the future job needs?
    • Do you have a specific/few direction(s) which you would you want to head towards?
    • Have you gathered enough information to garner the resources needed to revise your strategy when your needs changes?
    • Have you evaluate and manage to risk involved, for each of the path you take?

    It may sound overwhelming at first, but as more practice is involved with regards to the strategic way of thinking, the easier it would be for you to utilise them. They should come naturally as it is a mindset by itself.

    After answering all the questions, you should have a vivid picture /clear sound of what you should do next. This helps you to stay true to yourself and not ending up making a career change based on what you have seen on social media or small talks with your peers. In addition, it gives you a clear picture of the skills/knowledge/stakeholders that are at your arsenal and how they can be transferable to the next workplace that may or may not be relevant to your current industry.

    Here is an example of how strategic thinking is applied in the scenario above:

    First you look at the big picture and draft out the important stakeholders that have brought you to this day. That would be your uncle who provides you sound advice, your previous boss who introduced you this job, the sales manager who introduced the ropes of the company to you and your trusted team partner who backs you up in time of need. Interview them on why did they chose you? What do they see in you? And how have you improved over time?. Then draft out your strengths, skills and interests.

    Take a look on how your skills match other forms of interests that you have. Know what is the best environment for you to work with and research on the transformation of job needs in the near future. Then take a look on where you need to improve or further polish to meet those needs. For example marketing has been going digital and customers’ demands are getting more specific, thus you feel that you need to hone your data analysis skill in order to segment customers’ needs more effectively. 

    Do not be afraid to look back at your past and check on what do you dislike and where your achievements were. Be sure to put that in mind when you are drafting your plan. For example, you are comfortable with technology and is always updated with the current technologies, at the same time hated being too comfortable in one place. 

    By now, you would have an intent on where you want to go. If your traits includes risk-taking and future-minded, then you would naturally be more comfortable in going to an industry that is more futuristic and different from your current industry (e.g. tech). If your trait consists of stability then consider taking another role within the same industry or vice versa. Be sure to check the risks involved in each decision as well. 

    With that, you are all geared up to face the situation when there are changes in job needs within the economy. For example, you would have an adequate amount of knowledge to face the digital marketing world as well as the competencies to carry out corporate marketing plans. The skills that you have obtained by then could further propel you to help bring existing companies into digitalisation, and many more opportunities to come. 

    In conclusion, we can see in the example that there is no one right way towards strategic thinking and each components of the theory could emerge at different points in time. Two persons with similar traits / strengths may have totally different career pathways due to differences in their pasts and traits. However, having an organised way of thinking helps you to get a firm grasp of the situation and how you could respond to them in an effective way.

    Promotional Content Beyond This Point

    How does simulation effectively addresses strategic thinking?

    Strategic thinking does possess its complexity and therefore, requires a medium complex yet simple enough to transfer this skill towards the learner. We do not think traditional classroom teachings are adequate in addressing the full scope of strategic thinking because the fluency of this mindset has to be practiced in the real world setting to successfully grasp its mechanics. At the same time, the the workplace may be too hectic or comfortable that the learner falls back to his or her comfort mindset. Therefore, we believe that simulations possess the complexity, yet simple enough in a controlled environment to allow learners to practice this skill first, so they would have the confidence to apply them within their workplace.

    With the alignment of Liedtka’s theory to our Strategic Thinking Simulation, Allocate, we are able to transfer the theoretical components in stages towards the learners and at the same time, allows them to test out the theories within a safe and realistic environment.

    In addition, we have also aligned specific tools within the programme to help the learners translate the theories (in the head) into paper (visualising) so that they could act on them (execute). With this, we are sure that the strategic thinking mindset can be brought back into the workplace. And the benefit continues when they are shared within the working space.