Why formative assessment is the key to math growth all year long – Watch On-Demand Video Now

On Research and Results in Education

At DreamBox Learning, we partner with teachers and schools across the US and throughout Canada to improve student thinking, learning, and achievement in mathematics.  Our innovative math program is used in virtually every educational setting you can imagine: public, private, charter, community programs, and home-schools. The types of schools using DreamBox very closely mirrors the national composition of charters, public, and private schools. In short, a very small percentage of the schools in all 50 states using DreamBox are charter schools. All of these schools have different structures, systems, and designs because historically states have given local control to schools and districts to act in the best interests of their students. Learning guardians in these schools have confidence in DreamBox because they see how our lessons engage students as unique individuals and empower them as thinkers.  Educators trust our unique adaptive platform because they know teachers and education researchers built it as a promising example of how technology can enhance learning. Our mission is to partner with schools to cause great learning outcomes for students, regardless of their zip code. And our CEO, Jessie Woolley-Wilson, does an incredible job sharing our vision as she did when interviewed by Bill Gates at SXSWedu in 2013.

One of the earliest users of DreamBox Learning Math is Rocketship, a non-profit charter school network whose co-Founder, John Danner, is also a board member for DreamBox. I spent over ten years in public education prior to coming to DreamBox, but I didn’t know about Rocketship until I arrived here.  Since joining DreamBox, I’ve seen the national attention and scrutiny the Rocketship school network has attracted, and I’ve worked with their team as they continue to refine their model and structures. Early on, one important thing Rocketship provided to DreamBox was a third-party efficacy study they commissioned on their own.  This research study took the form of a randomized controlled trial conducted by SRI involving over 500 first graders and spanning the course of a couple of months.  The results were impressive: student gains were equivalent to progressing 5.5 points in percentile ranking in just 16 weeks (the equivalent of 4 weeks of a typical math class). Naturally, we have shared this research as one piece of evidence of DreamBox’s success.

But as we’ve expanded from being a PreK-2 program to releasing content this fall that extends through eighth grade math and beyond, other schools have not initiated third-party efficacy studies like Rocketship did. As a former classroom teacher and district administrator, I know it’s difficult to find a powerful resource like DreamBox that can help all students and support teachers, but then make the tough decision to conduct research that necessarily prevents every child from accessing the resource. Because other schools haven’t independently contracted with research firms such as SRI, we have begun working with the American Institutes for Research (AIR) to conduct efficacy studies. We do know that schools continually collect their own evidence of the impact of DreamBox to decide whether to continue using our program, and we’ve published case studies for many of these schools. While these case studies are not controlled quantitative analyses, they do provide educators with examples of how technology is being used to impact achievement and innovate. Beyond these case studies, we are very interested in seeing more reports like the SRI study with Rocketship.

In December 2013, the US Department of Education’s Institute of Education Sciences (IES) reviewed the SRI study with Rocketship and concluded it was flawed. Based on this single study, the IES published a report on the What Works Clearinghouse (WWC) with the conclusion: “DreamBox Learning was found to have no discernible effects on mathematics achievement.” After SRI provided additional information to the IES, it was determined that SRI’s original conclusions and report were actually valid, and the WWC report was corrected in March of 2014.  The conclusion in the corrected report reads: “DreamBox Learning was found to have potentially positive effects on mathematics achievement.” As an educator who spent several years consulting the WWC for information during textbook adoptions and curriculum revisions, I was disappointed that the WWC’s initial conclusion wasn’t as measured – with words such as “potentially” – as their corrected conclusion. But what’s far more disappointing is how the initial incorrect report on the WWC has contributed to a continued misunderstanding and misrepresentation of DreamBox Learning’s efficacy.

At DreamBox, we spent little time addressing the WWC report directly both in December and again when the corrected report was released in March. Because the conclusions were drawn from only one study, we chose to focus instead on our continued work with teachers and students to improve math achievement. But we needed to speak up when a new EPI report was issued last week that inaccurately reflects our efficacy, vision, and objectives. We believe the author, Dr. Gordon Lafer, is coming from a positive place of advocating change and improvements in how we look at school systems. We value educational improvement, and the innovation we support across the country ties back to the deep passion we all have for helping make the learning experience better for every student.

Beyond the main points of Dr. Lafer’s report, he discussed the DreamBox WWC report. Unfortunately, he primarily discussed information contained in the incorrect report from December rather than the corrected report published in March. Regarding the correction he noted, “…the Rocketship-commissioned authors produced additional data that convinced federal researchers to upgrade their assessment of DreamBox’s impacts…” At DreamBox, we must point out that the second WWC report was not an “upgrade” or revision to the December report. It was indeed a correction that completely replaces the December report rather than adds to it. The original report no longer exists on the WWC because it was incorrect.  We were pleased that SRI and IES worked to correct the report, and we’re proud to be one the few elementary math software programs validated on the WWC with an efficacy study that meets the rigorous IES criteria without reservations. If you use the WWC “Find What Works” search database and filter for the math topics, you’ll see that DreamBox is one of the few elementary math software programs to have been validated even though we are a relatively young company.

As a leader in education technology, it is our responsibility to be open and transparent in our work to all our partners and stakeholders – district and school administrators, teachers, students, parents, associations, technology providers, policy makers, investors and many more. While we hope more schools using DreamBox will conduct third-party research like Rocketship did, we know that thoughtful teacher and administrators consider much more than simply one research study on our website or the WWC when making decisions that profoundly impact students. Educators dig into our lessons, watch students learning with DreamBox, consider the industry awards we’ve received, and consult what other teachers and parents say in reviews on places such as Common Sense Media’s Graphite. Educators see the DreamBox difference because they see their students learning.  We anticipate having additional efficacy research to share with our trusted school partners. It’s these trusted partnerships that allow us to do what we love to do: focus on students and equitable learning.