Randomised control trials (RCTs) are an evidence-based research approach which has not yet been adopted and widely used in open and distance education to inform educational policy and practice. Despite the challenges entailed in their application, RCTs hold the power to robustly evaluate the effects of educational interventions in distance learning and conclude on whether (or not) these interventions should be adopted and used extensively. The aim of this paper is to spark discussions around the use of RCTs in distance learning by illustrating their benefits and drawbacks including challenges in adopting RCTs in education. To achieve this aim, a RCT was implemented to examine whether a small-scale intervention in four language modules could improve attendance at an end-of-module speaking assessment, and in consequence, performance, completion and pass rates. Results raise the need for further research in order to identify what type of interventions should be designed and put into practice to elicit a positive impact on learners. The paper concludes with a discussion on why RCTs should be brought to the forefront as a viable method for the effective evaluation of the impact of open learning analytic interventions.