Teachers usually expect that any form of a continuous assessment (CA) should contribute significantly to the student learning process in introductory programming courses. To foster teachers to go beyond the current practices of a CA applied when to programming, from 2011 to 2014, we investigated the use of the Boca Online Contest Administrator (BOCA) system, an online judge used in programming marathons, and the CA of program correctness in the cohorts of an introductory programming course. Empirical results show that there is no significant difference between student's performance when comparing the cohorts that used a CA and did not use a CA, and when comparing the cohorts that used and did not used the BOCA system but used a CA. An in-depth analysis revealed the potential and limitations of the use of a CA and the BOCA system, unveiling the need for the adoption of assessment practices and environments that build cumulative knowledge through multiple means of assessment, allowing profitable interactions among students, and between students and their teacher regarding the students' solutions.