[ad_1]
By Bogdan Yamkovenko, efficacy research manager of Khan Academy.
Mastery learning
The success of educational technology must be measured not only by time spent on a platform but also by learning. But learning is not always straightforward to measure. Khan Academy is designed based on principles of mastery learning, which encourages learners to engage in instruction and practice until they reach high levels of success in practice and assessment activities before moving on to the next skill. Our previous studies suggest that spending 30+ minutes per week on Khan is associated with better learning outcomes.1 Our most recent research indicates that beyond time spent, focusing on getting skills to proficient or higher is associated with success on an external learning assessment.
Focusing on proficiency: a better way to measure learning outcomes on Khan Academy
A trustworthy measure of math learning on Khan Academy is one that is correlated with other known, valid assessments of math achievement. One such valid assessment is the nationally normed MAP Growth mathematics standardized assessment. We analyzed the relationship between MAP Growth mathematics scores and various indicators of mastery progress on Khan Academy.
Khan Academy’s mastery learning system focuses on skills that increase in mastery levels, from “attempted” to “familiar” to “proficient” to “mastered.” Learners advance through these levels as they get more questions correct on exercises, quizzes, unit tests, and course challenges.2 Educational research on mastery learning3 tells us that learners who work on a skill until a high assessment score is achieved will have better long-term retention of the material.
In our research, we found that regardless of the number of skills learners worked on, the proportion of math skills completed at a proficient/mastered level correlated with MAP Growth mathematics scores. This pattern held regardless of the overall number of skills a learner worked on or the amount of time spent on Khan Academy (see figure below). We also saw this pattern hold across grades. This is consistent with mastery learning research that suggests getting to higher levels of achievement on a skill before moving on is related to better long-term retention of information.3
We now measure the impact of Khan Academy based on the amount of time spent on our platform and the proportion of skills completed at a proficient/mastered level on Khan Academy. As shown in the figure above, skills to proficient/mastered measure is notable because of its correlation with an external valid measure of math achievement, which is evidence of what researcher’s call “convergent validity.” This metric also aligns with mastery learning research and is consistent with the learning design principles we use on our platform.
Breadth versus depth: How should learners approach practice on Khan Academy
A typical Khan Academy exercise has several items that are focused on the same skill. Students advance to a certain level (familiar, proficient) on that skill depending on how many of these items they answer correctly. If a student gets to familiar in our mastery system once they complete an exercise, they can try the exercise again to reach proficient on the skill, or they can choose to move on to the next skill.
Stopping at a familiar level versus getting to a proficient or higher level may capture the difference between shallow learning and deeper learning since the latter requires more practice. Our analyses indicate that it is better to focus on fewer skills but reach at least proficient levels on those skills than to try and work on multiple skills but only reach a familiar level of understanding. Our analysis shows that in a typical 30-minute session, a learner can work on two exercises and level up to proficient on two skills. If the first attempt at completing an exercise is unsuccessful in getting to proficient, we recommend that a learner try again rather than move on to the next skill.
Khan Academy’s five-year goal
Our five-year goal is to accelerate the learning of 10 percent of U.S. students by 50 percent in math and/or science. We use “skills to proficient+”, the number of skills a learner leveled up to proficient or mastered, as our internal learning metric because it is both easy to understand and associated with greater gains on an external mathematics assessment. We hold ourselves accountable for moving the needle on student learning outcomes at scale as we make progress towards achieving our five-year goal.
Sources
1. Read more about MAP Accelerator study fundings here.
2. Learn more about Khan Academy’s mastery mechanics here.
3. Block, J. H., & Burns, R. B. (1976). Mastery learning. Review of research in education, 4, 3-49.
[ad_2]