HealthSeptember 16, 2016

2016 Firecracker Step 1 performance analysis

Since August 2014, we’ve surveyed medical students annually about their Step 1 scores. We’ve gathered score reports from both Firecracker users and non-Firecracker users and used these reports and our own data to determine if Firecracker usage creates better Step 1 outcomes.

Today we’re releasing the updated performance analysis based on this student data. This most recent data set includes 228 Firecracker users and 499 non-Firecracker users. We wanted to focus our analysis on users who have been active on Firecracker in past year because of the significant changes we have made to the platform since summer 2015—the launch of Firecracker MD and Firecracker DO.

We continue to see significant correlations between the extent to which a student learns, retains, and applies Firecracker’s material and their performance on the Step 1 exam. The reported national average for 2015 Step 1 scores was 229. Our data shows that Firecracker MD users scored 15 points higher on their exam than non-Firecrackers, with an average score of 245. Further, Firecracker MD users scored 3 points higher than Firecracker Legacy users.

Graph of USMLE Step 1 scores and percent respondents

And still, Firecracker users don’t report studying more than non-Firecrackers, and these correlations persist even after adjusting for MCAT scores as a baseline measurement.

Graph of MCAT and Step 1 scores

This spring we launched Dedicated Test Prep mode and initiated an overhaul and expansion of our Q-bank. We looked at Firecracker MD and Firecracker DO students’ use of Dedicated Test Prep mode, and their performance on CVs (USMLE style questions, clinical cases). We found a strong correlation between use of Dedicated Test Prep mode and better Step 1 outcomes. We also found that the Step 1 practice exam blocks were highly predictive of Step 1 scores:

Graph of Step 1 scores and score on first diagnostic block

We did not find a correlation between the flat number of CVs answered and Step 1 outcomes. But we did find that performance on flashcards on medical concepts was a strong indicator of performance on related clinical case questions.

Table of logit regression results

Overall we found that consistent Firecracker usage correlated with better performance on clinical vignettes, and ultimately better Step 1 outcomes. You can read the full paper here. It includes details on the above findings as well as an appendix that covers a product glossary and additional data.

Solutions
Firecracker

A personalized, digital tutoring experience brought to you by Lippincott.

We put together content and resources written by experts—specifically, medical students who scored high on their own exams and have experience-based insights—and provide it to students in MD, PA, and DO programs to unclog exam prep and curb Stage 4 panic.