Key Implementation Factors for Edtech Success - Project Red

Project RED III results are in!  Last month at ISTE, we released initial findings.  The complete narrative and analysis will be published and in 2017.  Project RED is unique in that it is the only research that studied:

• In multiple states
• Three years of longitudinal data
• Builds on a 1000 school study
• Reports financial ROI
• Correlates education practice to state high stakes test results
• Has findings with measurable statistical significance.

In 2010 our initial Project RED study demonstrated the non-negotiable imperative that proper implementation of education technologies was essential in raising student achievement and maximizing the return on investment. The results provided 9 Key Implementation Factors (KIFs) that most strongly influenced the results.  We further uncovered that through a full digital conversion, a large number of typical school district expenditures can be redeployed to offset the cost of implementing ubiquitous technology. Chapter 9 of the original research goes deeply into those discoveries.  Also significant was the finding that the lower the student to computer ratio, the better the return on investment.

In Project RED II we realized the need to examining the KIFs in action. Through a nationally competitive application process, we selected 20 Signature Districts based on their readiness, and their commitment to implementing the KIFs. From that group 17 districts remained committed to the 3 year program.

The goal of Project RED III was to independently analyze relationships between the KIFs, student achievement, and the overall return on investment. Project RED enlisted the support of the University of Memphis, Center for Research in Education Policy to analyze the year-over-year data provided by the Signature Districts, as well as publicly verifiable data such as state test score data, dropout rates, etc. Following is a 30,000 foot view of what they found, noting the disclaimer that, as is true with most research, some data was incomplete or not reported, and that questions emerge that will require further study.

ROI Analysis-Academics

Generally, the Signature Districts demonstrated gains academically and financially.  Five showed both positive financial and academic results.  The other twelve showed either some financial or academic improvements. 

Fourteen of the participating districts reported verifiable academic district data for Year 1 and Year 3, allowing for comparisons to be made in five ROI categories.  Four of the five areas showed promising academic results between years 1 and 3.

1.  Disciplinary Actions
     Eleven of the 14 districts, or 79% saw a decrease in the number of disciplinary action.

2.  Drop-out Rates
     Nine of the 14 districts, or 64% of districts saw a decrease in dropout rates.

3.  AP Classes Participation
     Eight of the 14 districts, 57% of districts saw an increase in the number of students taking AP classes.

4.  Graduation Rates
     Eleven of the 14 districts, or 79% saw a decrease in the number of disciplinary action.

Dual Enrollment is the one success measure that showed a negative trend, with more districts reporting a decrease rather than an increase in participation. Although there were some reporting issues with the data in this category, we question if an increase in enrollment in AP courses may be affecting the number of participants in Dual Enrollment Courses.

ROI Analysis-Financial

Nine annual expenditure categories were examined. A third of the Signature Districts saw their overall expenditures decline. Of the nine categories studied, four showed the most promising results regarding decrease in expenditures between years 1 and 3.

1.  Printing Supplies
When transforming learning through a 1:1 implementation, and operations through a district digital conversion, the need for printing should be reduced. This should reduce the expenditures for printers, printing supplies, and copy machines, as well as copy machine and printer maintenance as noted below.

2.  Copy Machine and Printer Maintenance
See the response to number 1.

3.  Tech Expenditures
We anticipated that technology expenditures would increase in all of the Signature District as they expanded their 1:1 programs. Surprisingly, these expenditures decreased in more than half of the districts.

4.  Textbooks
Sixteen districts reported verifiable data. While nine of the 16, or 56% of districts reported decreases in textbook expenditures, 44% actually indicated an increase. The result seems to indicate that some of the Signature Districts are still struggling to make the shift away from textbooks.

When looking at the nine categories individually, five showed at least as many increases as decreases in expenditures between years 1 and 3. For example, an equal number of districts reported increases and decreases in expenditures for printers, and seven districts reported increases in copy machine expenditures. More investigation will be necessary to determine the specific factors that led to these results, but it is clear there is still room for growth in cost avoidance and reallocation of assets in all of the Signature Districts.

Academic Achievement Analysis

Generalized Findings

While the impact of the Signature District program was almost universally positive in increasing proficiency levels in mathematics and science, the proficiency levels in high school reading did not improve, and in some cases decreased. Basic reading skills, such as decoding, can be effective developed through online skill development software, but increased use of such technology carries the risk of replacing the conceptual thinking work needed for deeper understanding of text. What type of applications are used, and how they are applied may affect high school reading proficiency outcomes. As University of Memphis quoted in their report, “Reading comprehension is much harder to learn when someone (or something) else is comprehending the material for you” (Diamond & Mandel, 1996).  Therefore, to increase reading proficiency levels through the use of technology in the upper grades, technology must be used to encourage students to interact more deeply with the texts presented to them.

The University of Memphis also found that “mandating” the use of technology, with regular checks for compliance was not effective. There is, however, a connection between technology and increased academic achievement when the teacher uses technology more organically, and in innovative ways.

The increased use of Communication Tools, such as email or other digital platforms that students use to talk with the teacher and work with other students, showed a positive relationship with proficiency levels in elementary school science, middle school science, and high school mathematics.

High School Academic Findings

• The use of technology and the increased use of change management strategies by the principal in high school mathematics and science showed a positive relationship with proficiency levels.
• A written plan that includes systematic collection and usage of data had a statistically significant and positive effect on reading and mathematics scores.

Middle School Academic Findings

• The use of technology, together with the increased use of change management strategies by the principal, showed a positive relationship with proficiency levels in mathematics.
• The increased use of online formative assessments showed a positive and educationally meaningful relationship with proficiency levels of students in reading.
• One consistent statistically significant finding for reading, mathematics, and science on the survey was that involving teachers in the planning of 1:1 professional development opportunities is not leading to improved academic performance in comparison to the state.

It is important to note that all of the statistically significant findings showed an inverse relationship between improvements the central office identified in the 1:1 program, and the actual academic improvements the middle schools showed on their state standardized tests. This may be an indication of a delay between improvements made on the district level, and the impact they have on the students in the classroom. It could also be that there is a disconnect between the perceptions of central office administrators, and the actual proficiency students demonstrated on their state standardized test.  

Elementary Findings

• The increased use of online formative assessments showed a positive and statistically significant relationship with proficiency levels of elementary students in reading, mathematics, and science relative to the state. 
• Increased use of search engines had a positive relationship with increased proficiency levels in reading, mathematics, and science relative to the state for elementary students. 
• For the elementary schools, a statistically significant connection was found between the use of technology as a component of the teacher evaluation system, and increased proficiency levels in reading, mathematics, and science in comparison to the state. 

The reported use of technology in elementary school intervention classes did not show any measurable impact on academic performance.  This is in contrast to the findings of our original research. The type of technology intervention used, and the frequency of use can play a major role in its effectiveness. Therefore, it will be necessary look at what was implemented and how it was used in order to better understand the results. 

Leslie Wilson, founder and CEO of One-to-One Institute, has served education for 38+ years in top level, key decision-making roles at state and local levels. Recognized as an international expert in education technology, Wilson is a frequent writer, presenter and interviewee. Among her many publications, she co-authored, “Project RED-The Technology Factor, Nine Keys to Student Achievement and Cost Effectiveness” which is the most broadly used research around successful implementation of 1:1 technologies in schools.