Self-assessment has been used to assess second language proficiency; however, as sources of measurement errors vary, they may threaten the validity and reliability of the tools. The present paper investigated the role of experiences in using Japanese as a second language in the naturalistic acquisition context on the accuracy of the self-assessment. Results revealed that experiential factors played a significant role in the measurement errors introduced by the self-assessment. The asymmetry pattern emerged, whereby less experienced second language speakers appeared to overestimate their ability, whereas those with more experience underestimated their language skills. The Rasch analysis identified poor-fit items in the self-assessment survey, and the subsequent qualitative analysis of the items indicated that the greater misalignment was related to items including more difficult tasks, with which participants had relatively little experience. Implications for development and use of self-assessment are thus discussed in relation to experiential factors in self-assessment.