HealthNovember 20, 2015

Firecracker Step 1 and COMLEX performance analysis

In August 2014 and again in October 2015 we surveyed medical students—Firecrackers and non-Firecrackers—about their test performance and scores. Through this initiative, we gathered USMLE Step 1 scores (actual score reports) from 1,115 students and COMLEX scores from 180 students. We’ve used this data to update our 2014 analysis on the efficacy of Firecracker for boosting exam scores. These are the results—including (for the first time!) results on Firecracker usage and COMLEX outcomes.

In this analysis “Firecrackers” were users who answered at least 1000 questions in the Firecracker program, which correlates to about 1 month of use. “Non-Firecrackers” were students who answered fewer than 1000 questions in Firecracker or did not have Firecracker accounts. We compared the distributions of non-Firecracker Step 1 scores with the distribution of Firecracker Step 1 scores.

Firecracker and Step 1

Graph of USMLE Step 1 scores and percent respondents

In this set we have data for 473 Firecracker users, and 627 non-Firecracker students. The mean non-Firecracker Step 1 score was a 228.7 with a standard deviation of 23.7. The mean Firecracker score was a 240.3 with a standard deviation of 16.8. On average, Firecracker users out-scored non-Firecrackers by almost 12 points.

We also examined how, for Firecrackers, usage correlated with Step 1 score.

Number of recalls:

Graph of USMLE Step 1 scores and number of recalls

r=0.357,n=627,p<0.001

Number of unique questions answered on Firecracker:

Graph of USMLE Step 1 scores and number of questions

r=0/340,n=627,p<0.0001

Final average question score (for all questions answered by that Firecracker):

Graph of USMLE Step 1 scores and final average question scores

r=0.363,n=627,p<0.001

There is a clear correlation, in each case, with increased usage and increased Step 1 score.

Firecracker and COMLEX 1

In this set we have data for 65 Firecracker users and 81 non-Firecracker students.

The mean non-Firecracker COMLEX Level 1 score was a 527.4 with a standard deviation of 81.2. The mean Firecracker COMLEX Level 1 score was a 586.8 with a standard deviation of 79.6. On average. Firecracker users out-scored non-Firecrackers by almost 60 points on the COMLEX 1 exam.

Graph of COMLEX Level 1 scores

This is how Firecracker usage correlated with COMLEX scores for Firecrackers.

Number of recalls:

Graph of COMLEX Level 1 scores and number of recalls

r=0.467,n=65,p<0.0001

Number of unique questions answered on Firecracker:

Graph of COMLEX Level 1 scores and number of questions

r=0.38,n=65,p=0.0016

Final average question score (for all questions answered by that Firecracker):

Graph of COMLEX Level 1 scores and final average question score

r=0.384,n=65,p<0.001

Total number of weeks student was active on Firecracker:

Graph of COMLEX Level 1 scores and number of active weeks

r=0.312, n=65, p=0.011

Again, we see a clear correlation between Firecracker usage and improved COMLEX score.

The white papers on these results, with appendices and details like overall average scores on these exams, will be launching in the next couple of weeks.

Solutions
Firecracker

A personalized, digital tutoring experience brought to you by Lippincott.

We put together content and resources written by experts—specifically, medical students who scored high on their own exams and have experience-based insights—and provide it to students in MD, PA, and DO programs to unclog exam prep and curb Stage 4 panic.