The study aimed to validate the factors that affect the inter-rater reliability of secondary school certificate (SSC) papers in high-stake testing. For this purpose, papers of Urdu and English of Board of Intermediate and Secondary Education (BISE) were selected. A survey method was used to collect marking on the same set of papers from each rater and their response to the questionnaire. The population of the study has comprised the raters for English and Urdu subjects. A sample of 98 raters was selected randomly from the list. Instruments of the study were solved papers of annual examination for both subjects and a questionnaire related to the factor that affects inter-rater reliability. According to consistency estimates of the inter-rater reliability approach, the Spearman correlation coefficient was applied to examine consistency in the scoring of raters. It was found that raters' training effect the inter-rater reliability in their scoring. However, other factors did not affect inter-rater reliability in scoring. Based on the findings of the study, it is recommended to ensure the participation of all raters in the training session.