Journal of Contemporary Issues in Business and Government

Abstract

How has academic research in rating quality evolved over the last decade? Time has
witnessed that previous researchers actively contributed to the development of
knowledge particularly in ascertaining educational assessment to be updated with latest
research-based practices. Thus, this review seeks to provide a bird’s-eye view of
research development on rating quality over the last ten years focusing on factors
influencing raters’ rating quality within the context of rater-mediated language
assessment. This systematic literature review was conducted with the aid of Preferred
Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines
through three stages, namely identification, screening and eligibility. Accordingly, the
searching process has resulted in 43 articles to be thoroughly reviewed retrieved from
two powerful database, Scopus and World of Science (WoS). Five major factors have
emerged in response to the objective and they include rating experience, first language,
rater training, familiarity and teaching experience. Analysis indicated that these factors
lead to contradicting findings in terms of raters’ rating quality except for rater training
factor. Only rater training was proven to be successful in mitigating rater effect and
enhancing raters’ variability, severity and reliability. However, other factors were
discovered to be inconclusive depending on whether they leave any impact on raters’
rating quality. The direction for future studies is also discussed suggesting the inclusion
of more qualitative or mixed-method studies conducted to be reviewed using other
possible techniques.