Concept inventories (CI) are validated, research-based, multiple-choice tests, which are widely used to assess the effectiveness of pedagogical practices in bringing about conceptual change. In order to be a useful diagnostic tool, a CI must reflect only the student understanding of the conceptual material. The Force Concept Inventory (FCI) is arguably the standard for testing conceptual understanding of Newtonian mechanics. Studies in the United States and United Kingdom have shown the existence of a gender gap in FCI scores and gains between male and female students. This study aimed to examine whether such a gap exists for Canadian students at a mid-sized university. Four-hundred and thirty-four men and 379 women taking first-term introductory physics courses from the past nine years were assessed with the FCI prior to and after receiving instruction. A gender gap in the pre-instruction and post instruction scores was revealed in favour of male students (p < 0.01). There also existed a gender disparity in the learning gains between the two tests, where males had significantly higher gains (p < 0.01), although the effect size was small. Further analysis found that both male and female students who studied in classes that included interactive engagement methods had somewhat higher gains than students in traditional lecture courses, but that the interactive engagement methods did not eliminate the gender gap between male and female students (p < 0.01). Our results sound a cross-disciplinary note of caution for anyone using concept inventories as research or self-assessment tools.