Students' graph comprehension may be affected by the background of the students who are the readers or interpreters of the graph, their knowledge of the context in which the graph is set, and the inferential processes required by the graph operation. This research study investigated these aspects of graph comprehension for 152 first year undergraduate South African physics students by comparing their questionnaire responses to tasks requiring corresponding mathematics and kinematics graph operations. Interviews with 14 participants aided in interpreting the quantitative data. Participants' gender, year of school completion and study course served as reader characteristics. Their responses were interpreted in terms of their contextual knowledge (understanding kinematics and mathematics concepts, especially graphs, as well as aspects regarding the nature of these subjects) and the inferential processes (visual decoding and judgement) required when performing graph operations. Four graph operations were investigated, namely reading coordinates, determining the gradient of and the area under a line graph and connecting representations. The results of the empirical study indicated that reader characteristics had small to medium effects on the students' responses while their contextual knowledge and the inferential processes largely determined their performance of graph operations. The participants generally transferred their mathematics knowledge on coordinate reading and representations of straight line functions to the kinematics contexts, but not in the cases of parabolic and hyperbolic functions or area under graphs. Insufficient understanding of the gradient concept contributed to weak performances on this graph operation in both the mathematics and kinematics contexts. From this comprehensive study it is deduced that participants' problems with kinematics graphs are mainly due to insufficient contextual knowledge that is foundational to the physical-mathematical model of linear motion represented by the graphs. These deficiencies hamper students in knowing what inferential processes to perform and how to do them. The dependence of the results on the graph contexts and operations reveals the complexity of students' graph comprehension in kinematics.