This dissertation reports development and validation studies of a Web-based standardized test of Japanese as a foreign language (JFL), designed to measure learners' off-line grammatical and pragmatic knowledge in multiple-choice format. Targeting Japanese majors in the U.S. universities and colleges, the test is designed to explore possible relationships between JFL learners' grammatical knowledge and domains of pragmatic knowledge. Pragmatic knowledge is defined as knowledge of "[linguistic] forms necessary for productively conveying speech intentions, and assigning pragmatic meaning to interlocutor utterance" (Rover 2005), and is operationalized as the ability to associate linguistic features in specific contexts with appropriate speech acts, routines, and speech styles. Grammatical knowledge includes well-formedness of isolated sentences at the morpho-syntactic level, independent of contextual information. Research questions address reliability of the test, the relationship between JFL learners' grammatical and pragmatic knowledge (Bardovi-Harlig and Dornyei 1998), as well as the relationship between the overall scores of this multiple-choice test and the ACTFL Proficiency Guidelines Levels for speaking. The data from 110 participants of JFL learners in four different universities revealed that the test was overall reliable. The total scores of the test positively correlated with length of study/exposure, levels of instruction, and proficiency ratings based on the ACTFL Guidelines. As for the relationship among grammatical and domains of pragmatic knowledge, the study found complex interrelationships: correlation analyses showed that the four subsections (Grammar, Routines, Speech Acts, and Speech Styles) are significantly correlated overall, but the strengths of correlation varied depending on the levels of instruction and proficiency levels. The Routines section was somewhat separate and was not highly correlated with other sections at upper levels. The data also suggest that, as far as JFL is concerned, the knowledge of speech styles is in closer relationship with the knowledge of grammar and speech acts, compared with studies on learners of English as a Second/Foreign Language. Exploratory factor analysis, however, found unexpected patterns of factor loadings, not matching the internal structure of the test, suggesting that the hypothesized components of language ability may not be empirically distinguishable. Construct validity of the test is defensible only for the overall score being a valid indication of communicative language ability "in general." The extent to which each section scores are valid indications of corresponding hypothesized separate components of the language ability needs to be further investigated in future studies. Pedagogical implications for testing knowledge of natives' prototypical communication patterns, limitations of the study, and future development of the test are discussed. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]