Classroom practices, materials and teaching methods in language classes have changed a lot in the last decades and continue to evolve; however, the commonly used techniques to test students' foreign language skills have not changed much regardless of the recent awareness in Bloom's taxonomy. Testing units at schools rely mostly on multiple choice questions (MCQ) because these types of questions are reliable, cost effective and time savers; however, these measure only surface information in a particular skill while other skills such as critical thinking and synthesis cannot be evaluated using MCQs. On the other hand, open-ended question (OEQ) tests include analysis and synthesis and a higher level of cognitive processing, with some washback effects such as less reliable results and more time and effort in scoring. This study aims to compare language learners' performances on MCQ and OEQ tests administered to 116 students studying at a language preparatory school in Eskisehir. Four separate tests including grammar and reading questions in both forms were prepared and administered to the same students respectively. Research results showed that there was a significant difference between OEQ and MCQ tests in terms of item difficulty and item discrimination levels. In both grammar and reading assessment, MCQ tests were found to be easier than OEQ tests.