Connecting neuroscience and education is a desire in contemporary society, related to the recurring calls for education to become more evidence-based. Research in educational neuroscience strives towards such interdisciplinary knowledge production and to an enhanced interaction between neuroscience research and educational practice. However, various problems and difficulties in achieving these collaborations are often reported. Discrepancies, hierarchies, misconceptions and communication problems can be described as creating a 'discourse of difficulty'. The aim of this paper is to trace the specific difficulties that have created this discourse, and to problematize these difficulties in ways that enable new conceptions of what might be entailed by interaction and mutual knowledge development between the fields of neuroscience and education, and between academic theory and educational practice. The most significant difficulty is caused by a binary understanding of the concept of difference in relation to understanding the fields. Instead of understanding the fields in opposition to each other, I will suggest an understanding that implies difference emerging in each of the collaborating fields as the self-differing effects of the encounter. In the concluding discussion, I will argue that an understanding of the concept of difference as a process of mutual transformation can be essential for reciprocity and bi-directionality in collaborations. Instead of producing contradictions and hierarchies between scientific fields and between theory and practice, such an understanding of difference might facilitate an investigation of the polarizations that always position something as of lesser value, and ultimately, creates the gaps that collaborations want to bridge.