Explore the effectiveness of a multi-level assessment protocol for online learning

Ya Mo; Eulho Jung; Meehyun Yoon

doi:10.30935/cedtech/17547

Ya Mo ¹ ^* , Eulho Jung ², Meehyun Yoon ³

More Detail

CONT ED TECHNOLOGY, Volume 17, Issue 4, Article No: ep615

https://doi.org/10.30935/cedtech/17547

Submitted: 01 October 2024, Published Online: 15 December 2025

OPEN ACCESS 48 Views 22 Downloads

Download Full Text (PDF)

Abstract

This study examines the effectiveness of a multi-level assessment protocol for online learning, particularly the use of reflection and self-assessment as learning and assessment tools, by capturing and interpreting five graduate students’ experiences in an online assessment course. Data were collected through surveys, reflection questions, self-assessments, pre-assessments, and module assessments. Likert-scale survey questions and selected-response reflection questions or assessment items were analyzed by counting responses or tallying correct answers. Open-ended survey questions, reflection questions, and assessment items were analyzed qualitatively using thematic analysis. Results reveal that combining traditional tests with reflection enhances assessments as learning tools, emphasizing the necessity of carefully formulated reflection questions. For self-assessment with constructed-response tests, providing students with illustrative responses and detailed rubrics that clearly state the number and scope of key points expected in responses can help standardize students’ subjective judgments. Furthermore, the unanimous preference for hands-on projects among students and the frustration expressed by some regarding asynchronous discussions underscore the importance of offering diverse, flexible, and engaging activities across various modalities to support learning outcomes. Overall, the findings affirm the enhancement of assessments as learning tools through the amalgamation of traditional tests with reflection and self-assessment, while also pinpointing the need for thoughtful implementation.

Keywords: online learning, assessment, reflection, self-assessment

References

Akhtar, H., & Kovacs, K. (2024). Measurement precision and user experience with adaptive versus non-adaptive psychometric tests. Personality and Individual Differences, 225, Article 112675. https://doi.org/10.1016/j.paid.2024.112675
Alleyne Bayne, G., & Inan, F. A. (2022). Development of the online course overload indicator and the student mental fatigue survey. The International Review of Research in Open and Distributed Learning, 23(4), 75-92. https://doi.org/10.19173/irrodl.v23i4.6223
Arabyat, R. M., Qawasmeh, B. R., Al-Azzam, S. I., Nusair, M. B., & Alzoubi, K. H. (2022). Faculty members’ perceptions and attitudes towards anti-plagiarism detection tools: Applying the theory of planned behavior. Journal of Empirical Research on Human Research Ethics, 17(3), 275-283. https://doi.org/10.1177/15562646221078655
Arend, B. (2007). Course assessment practices and student learning strategies in online courses. Journal of Asynchronous Learning Networks, 11(4), 3-13.
Ash, S. L., & Clayton, P. H. (2009). Generating, deepening, and documenting learning: The power of critical reflection for applied learning. Journal of Applied Learning in Higher Education, 1(1), 25-48. https://doi.org/10.57186/jalhe_2009_v1a2p25-48
Beebe, R., Vonderwell, S., & Boboc, M. (2010). Emerging patterns in transferring assessment practices from F2F to online environments. Electronic Journal of e-Learning, 8(1), 1-12.
Benabbes, S., Algazo, M. A., & Alghazo, S. M. (2025). Exploring the impact of digital scaffolding on collaborative writing practices. Humanities and Social Sciences Communications, 12, Article 1606. https://doi.org/10.1057/s41599-025-05606-0
Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education: Principles, Policies, and Practice, 5(1), 7-74. https://doi.org/10.1080/0969595980050102
Bloom, B. S. (1956). Taxonomy of educational objectives. Vol. 1: Cognitive domain. McKay.
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77-101. https://doi.org/10.1191/1478088706qp063oa
Chen, N., Wei, C., Wu, K., & Uden, L. (2009). Effects of high-level prompts and peer assessment on online learners’ reflection levels. Computers & Education, 52, 283-291. https://doi.org/10.1016/j.compedu.2008.08.007
Creswell, J. (1998). Qualitative inquiry and research design: Choosing among five traditions. SAGE.
Darling-Hammond, L., Flook, L., Cook-Harvey, C., Barron, B., & Osher, D. (2020). Implications for educational practice of the science of learning and development. Applied Developmental Science, 24(2), 97-140. https://doi.org/10.1080/10888691.2018.1537791
Davies, J., & Graff, M. (2005). Performance in e-learning: Online participation and student grades. British Journal of Educational Technology, 36, 657-663. https://doi.org/10.1111/j.1467-8535.2005.00542.x
Diarsini, M. S., Artini, L. P., Padmadewi, N. N., Ratminingsih, N. M., Utami, I. G. A. L. P., & Marsakawati, N. P. E. (2022). Challenges and opportunities of online assessment implementation during COVID-19 pandemic in Indonesia based on recent studies. European Journal of Education and Pedagogy, 3(6), 82-88. https://doi.org/10.24018/ejedu.2022.3.6.421
Gomez, M. J., Ruipérez-Valiente, J. A., & Clemente, F. J. G. (2022). A systematic literature review of game-based assessment studies: Trends and challenges. IEEE Transactions on Learning Technologies, 16(4), 500-515. https://doi.org/10.1109/TLT.2022.3226661
Guerrero-Roldán, A. E., & Noguera, I. (2018). A model for aligning assessment with competences and learning activities in online courses. The Internet and Higher Education, 38, 36-46. https://doi.org/10.1016/j.iheduc.2018.04.005
Hafeez, M., Tahira, F., & Leghari, M. F. A. (2021). Challenges faced by the teachers and students in online learning during COVID-19. Pakistan Journal of Educational Research and Evaluation, 9(1), 83-102. https://doi.org/10.21831/cp.v41i1.35411
Harsy, A., & Hoofnagle, A. (2020). Comparing mastery-based testing with traditional testing in Calculus II. International Journal for the Scholarship of Teaching and Learning, 14(2), Article 10. https://doi.org/10.20429/ijsotl.2020.140210
Heil, J., & Ifenthaler, D. (2023). Online assessment in higher education: A systematic review. Online Learning, 27(1), 187-218. https://doi.org/10.24059/olj.v27i1.3398
Hew, K. F., & Cheung, W. S. (2014). Students’ and instructors’ use of massive open online courses (MOOCs): Motivations and challenges. Educational Research Review, 12, 45-58. https://doi.org/10.1016/j.edurev.2014.05.001
Holden, O. L., Norris, M. E., & Kuhlmeier, V. A. (2021). Academic integrity in online assessment: A research review. Frontiers in Education, 6. https://doi.org/10.3389/feduc.2021.639814
Kayler, M., & Weller, K. (2007). Pedagogy, self-assessment, and online discussion groups. Journal of Educational Technology & Society, 10(1), 136-147.
Kearns, L. (2012). Student assessment in online learning: Challenges and effective practices. Journal of Online Learning and Teaching, 8(3), 198-208.
Keppell,M., Au, E., Ma, A., & Chan, C. (2006). Peer learning and learning-oriented assessment in technology-enhanced environments. Assessment & Evaluation in Higher Education, 31(4), 453-464. https://doi.org/10.1080/02602930600679159
Kim, K. J., & Frick, T. W. (2011). Changes in student motivation during online learning. Journal of Educational Computing Research, 44(1), 1-23. https://doi.org/10.2190/EC.44.1.a
Kirkwood, A., & Price, L. (2016). Technology-enabled learning implementation handbook (version 1). Commonwealth of Learning.
Langenfeld, T. (2020). Internet-based proctored assessment: Security and fairness issues. Educational Measurement: Issues and Practice, 39(3), 24-27. https://doi.org/10.1111/emip.12359
Li, C., Guo, J., Zhang, G., Wang, Y., Sun, Y., & Bie, R. (2019). A blockchain system for e-learning assessment and certification. In Proceedings of 2019 IEEE International Conference on Smart Internet of Things (pp. 212-219). IEEE. https://doi.org/10.1109/SmartIoT.2019.00040
Miles, M. B., Huberman, A. M., & Saldana, J. (2014). Qualitative Data Analysis: A Methods Sourcebook (3rd ed.). Sage.
Nicol, D. J., Thomson, A., & Breslin, C. (2014). Rethinking feedback practices in higher education: A peer review perspective. Assessment & Evaluation in Higher Education, 39(1), 102-122. https://doi.org/10.1080/02602938.2013.795518
Noorbehbahani, F., Mohammadi, A., & Aminazadeh, M. (2022). A systematic review of research on cheating in online exams from 2010 to 2021. Education and Information Technologies, 27(6), 8413-8460. https://doi.org/10.1007/s10639-022-10927-7
Oncu, S., & Cakir, H. (2011). Research in online learning environments: Priorities and methodologies. Computers & Education, 57(1), 1098-1108. https://doi.org/10.1016/j.compedu.2010.12.009
Ortega-Ruipérez, B., & Correa-Gorospe, J. M. (2024). Peer assessment to promote self-regulated learning with technology in higher education: Systematic review for improving course design. Frontiers in Education, 9. https://doi.org/10.3389/feduc.2024.1376505
Öztürk, M. (2024). Rethinking online assessment quality from pre-service teachers perspectives. Open Praxis, 16(4), 696-711. https://doi.org/10.55982/openpraxis.16.4.689
Ramaswami, G., Susnjak, T., & Mathrani, A. (2023). Effectiveness of a learning analytics dashboard for increasing student engagement levels. Journal of Learning Analytics, 10(3), 115-134. https://doi.org/10.18608/jla.2023.7935
Reeves, T. C. (2000). Alternative assessment approaches for online learning environments in higher education. Journal of Educational Computing Research, 23(1), 101-111. https://doi.org/10.2190/GYMQ-78FA-WMTX-J06C
Rhode, J., Richter, S., & Miller, T. (2017). Designing personalized online teaching professional development through self-assessment. TechTrends, 61(5), 444-451. https://doi.org/10.1007/s11528-017-0211-3
Roberts, T. S. (Ed.). (2006). Self, peer and group assessment in e-learning. IGI Global. https://doi.org/10.4018/978-1-59140-965-6
Salas-Bustos, D. A., Coral-Padilla, S. J., Bustos-Lozano, H. L., & Belén, M. (2025). The role of formative assessment in higher education: Strategies to improve learning and knowledge retention. Pakistan Journal of Life and Social Sciences, 23, 5441-5455. https://doi.org/10.57239/PJLSS-2025-23.1.00425
Swan, K. (2001). Virtual interaction: Design factors affecting student satisfaction and perceived learning in asynchronous online courses. Distance Education, 22(2), 306-331. https://doi.org/10.1080/0158791010220208
The University of Wisconsin-Madison. (n.d). Best practices and sample questions for course evaluation surveys. The University of Wisconsin-Madison. https://assessment.wisc.edu/best-practices-and-sample-questions-for-course-evaluation-surveys/
Tsai, F.-H., Tsai, C.-C., & Lin, K.-Y. (2015). The evaluation of different gaming modes and feedback types on game-based formative assessment in an online learning environment. Computers & Education, 81, 259-269. https://doi.org/10.1016/j.compedu.2014.10.013
Tseng, S.-C., & Tsai, C.-C. (2010). Taiwan college students’ self-efficacy and motivation of learning in online peer assessment environments. The Internet and Higher Education, 13(3), 164-169. https://doi.org/10.1016/j.iheduc.2010.01.001
VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist, 46(4), 197-221. https://doi.org/10.1080/00461520.2011.611369
Vlachopoulos, D., & Makri, A. (2024). A systematic literature review on authentic assessment in higher education: Best practices for the development of 21^st century skills, and policy considerations. Studies in Educational Evaluation, 83, Article 101425. https://doi.org/10.1016/j.stueduc.2024.101425
Vonderwell, S. Liang, X., & Alderman, K. (2007) Asynchronous discussions and assessment in online learning, Journal of Research on Technology in Education, 39(3), 309-328. https://doi.org/10.1080/15391523.2007.10782485
Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Harvard University Press.
Wang, S., Wang, F., Zhu, Z., Wang, J., Tran, T., & Du, Z. (2024). Artificial intelligence in education: A systematic literature review. Expert Systems with Applications, 252, Article 124167. https://doi.org/10.1016/j.eswa.2024.124167
Webb, N. L. (2002). Alignment study in language arts, mathematics, science, and social studies of state standards and assessment for four states. Council of Chief State School Officers.
Yang, H., & Wong, R. (2024). An in-depth literature review of e-portfolio implementation in higher education: Processes, barriers, and strategies. Issues and Trends in Learning Technologies, 12(1). https://doi.org/10.2458/itlt.5809
Yin, R. K. (1989). Case study research: Design and methods. SAGE.
Zhang, J., Desrochers, M. N., & Fensken, M. (2025). Evaluation of teacher, self-assessment, versus combined feedback to increase students’ behavioral observation skills. International Journal of Teaching and Learning in Higher Education, 35(1), Article 4.
Zhang, P., & Tur, G. (2024). A systematic review of e-portfolio use during the pandemic: Inspiration for post-COVID-19 practices. Open Praxis, 16(3), 429-444. https://doi.org/10.55982/openpraxis.16.3.656
Zimmerman, B. J. (2000). Attaining self-regulation: A social cognitive perspective. In M. Boekaerts, P. R. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 13-39). Academic Press. https://doi.org/10.1016/B978-012109890-2/50031-7

Citation

Mo, Y., Jung, E., & Yoon, M. (2025). Explore the effectiveness of a multi-level assessment protocol for online learning. Contemporary Educational Technology, 17(4), ep615. https://doi.org/10.30935/cedtech/17547

Mo, Y., Jung, E., and Yoon, M. (2025). Explore the effectiveness of a multi-level assessment protocol for online learning. Contemporary Educational Technology, 17(4), ep615. https://doi.org/10.30935/cedtech/17547

Mo Y, Jung E, Yoon M. Explore the effectiveness of a multi-level assessment protocol for online learning. CONT ED TECHNOLOGY. 2025;17(4), ep615. https://doi.org/10.30935/cedtech/17547

Mo, Ya, Eulho Jung, and Meehyun Yoon. "Explore the effectiveness of a multi-level assessment protocol for online learning". Contemporary Educational Technology 2025 17 no. 4 (2025): ep615. https://doi.org/10.30935/cedtech/17547

Mo, Ya et al. "Explore the effectiveness of a multi-level assessment protocol for online learning". Contemporary Educational Technology, vol. 17, no. 4, 2025, ep615. https://doi.org/10.30935/cedtech/17547

Mo Y, Jung E, Yoon M. Explore the effectiveness of a multi-level assessment protocol for online learning. CONT ED TECHNOLOGY. 2025;17(4):ep615. https://doi.org/10.30935/cedtech/17547