This paper discusses the development of an assessment to satisfy the International Civil Aviation Organization (ICAO) Language Proficiency Requirements. The Versant Aviation English Test utilizes speech recognition technology and a computerized testing platform, such that test administration and scoring are fully automated. Developed in collaboration with the U.S. Federal Aviation Administration, this 25-minute test is delivered via a telephone or computer. Two issues of interest are discussed. The first concerns the practicalities of assessing candidates in each of six separate dimensions of spoken proficiency: Pronunciation, Structure, Vocabulary, Fluency, Comprehension, and Interactions. Although an automated scoring system can objectively segregate these skills, we question whether human raters have the capacity to do this in oral interviews. The second issue discussed is how an automated test can provide a valid assessment of spoken interactions. Tasks were designed to simulate the information exchange between pilots and controllers on which candidates' proficiency in "Interactions" could be measured, for example, by eliciting functions such as correcting miscommunications and providing clarification. It is argued that candidate ability can be probed and estimated in a fair and standardized way by presenting a series of independent items which are targeted in difficulty at the various ICAO levels. (Contains 5 tables and 1 figure.)