A Comprehensive Framework for Evaluating VR-based Automotive Training: Insights from Vocational Education

Abstract

The integration of Virtual Reality (VR) in vocational education has introduced new possibilities for immersive skill development, yet there remains a lack of standardized evaluation frameworks to assess its effectiveness. This study addresses this gap by comparing existing evaluation methods for VR-based training, analyzing student performance data, and proposing a refined assessment model for vocational education. The research involved 83 students (ages 13–17) from three Malaysian institutions: E-Access International College (n=42), Langkawi Vocational College (n=25), and Faculty of Engineering and Built Environment (n=16). Participants underwent VR-based training covering three essential engine maintenance procedures: replacing a valve cover gasket, removing a gasket from the exhaust manifold, and replacing a gasket on the intake manifold. A mixed-methods evaluation approach was employed, incorporating quantitative performance analysis and qualitative learner feedback. The findings revealed a consistent negative correlation between task completion time and overall scores across three examinations, with the strongest correlation observed in Exam 1 (r = -0.411), followed by Exam 2 (r = -0.297) and Exam 3 (r = -0.235). This indicates that faster task completion generally led to higher proficiency. The highest score recorded was 100 (93.26%), while the lowest was 69 (64.49%), with an overall mean score of 84.35 (Exam 1), 86.10 (Exam 2), and 90.36 (Exam 3) out of 107. The internal consistency measured by Cronbach’s alpha was 0.543, highlighting the need for a more structured assessment approach. A comparative review of existing evaluation models including Kirkpatrick’s model, Bloom’s Taxonomy, and the Technology Acceptance Model (TAM) that revealed limitations in their applicability to immersive VR training. To address this, the study introduces a new VR-based training evaluation framework that integrates task standardization, adaptive difficulty levels, and real-time performance feedback. This framework provides a more comprehensive approach to measuring skill acquisition and training effectiveness in vocational settings. The findings advocate for the broader adoption of VR in vocational education and emphasize the need for a structured evaluation model to ensure its practical relevance and scalability.