This paper examines a procedure measuring student competencies in numeracy using school-based assessments, and demonstrates how the procedure informs the school system on quality improvement. The sample consisted of 7648 students, attending three different types of urban schools including government, formal private and low cost private in poor informal settlements in Kenya. The numeracy tool measures six curriculum outcome areas. Each outcome area is defined by 2-11 measured items based on the emphasis given to an area in the curriculum. Within each of the outcome areas, items are also hypothesised to measure categories of the cognitive domain of learning. Using the Rasch, we examine the item difficulties, person location and differential item functioning. The paper provides important implications to provision of evidence to inform teaching in poor urban informal settlements in Kenya and this can also be replicated in other sub-Saharan African countries.