A STATEWIDE INDICATOR SYSTEM

FOR POSTSECONDARY SUCCESS

Over the past eight years, the Massachusetts Department of Elementary and Secondary Education (DESE) has developed a remarkable statewide system to help educators support students for postsecondary success.

Beginning in 2015, the Early Warning Indicator System (EWIS) team expanded its initial focus on K12 milestones and high school graduation to address students’ likelihood of enrolling in college, taking credit-bearing courses without remediation, and persisting into the second year of college. In addition to the ABCs of attendance, behavior, and course performance, other indicators were added that allow educators to dig deeper into students’ academic experiences:

~ Did they earn at least a B in key courses?
~ Which math courses did they take?
~ Did they take any AP courses?

State funding and two consecutive federal State Longitudinal Data System (SDLS) grants over the past eight years have allowed a three-person EWIS team and a network of colleagues and partners to construct a robust, well-connected system that helps educators support students better. Strong support from DESE policy leaders and IT staff, teamwork across the Department, and partnership with practitioners have been crucial in the development and adoption of the statewide EWIS. This effort illustrates the state’s longstanding priority of using data to improve schools.

Helping educators use the data to support students has been just as important as including valid predictive indicators. 

The EWIS team spearheads much of the day-to-day EWIS work, offering a variety of user-focused training and support resources:

  • A statewide workshop in September helps school staff focus on students most at risk of missing key academic milestones in the coming year, if interventions are not provided. In 2019, this workshop will expand into several regional events promoting integration of EWIS and school climate surveys.
  • Year-round support through specialized trainings, technical assistance, webinars, and an implementation binder are linked to interactive web-based tutorials. A newsletter and the Department’s website highlight new training offerings.
  • Input from and collaboration with local educators—who hold the key to identifying and addressing root causes of student risk factors—have been essential in the system’s dissemination. Local practitioners have shared strategies through trainings and tutorials.
  • In 2018, ten districts won 16-month competitive stipends to enhance data practices using the Early Warning Implementation Cycle. Working with these districts has helped the EWIS team deepen its understanding of local needs, resources, and priorities.

The Massachusetts EWIS uses data available in the state’s secure data warehouse; this avoids placing an unnecessary burden on schools and districts. The data warehouse provides a “one stop shop” where educators can learn about student and subgroup outcomes on everything from the ABCs, to which colleges their former students attend. Access to the secure data warehouse is managed locally at school districts.

Organizational Focus

The goal of the Massachusetts public K-12 and adult education system is to prepare all students for success in life. The statewide agency works with approximately 1,850 schools, 400 LEAs, 80,000 educators, nearly one million students in K-12, and tens of thousands of others in adult education programs to improve educational outcomes.

Personnel Required

Primarily three individuals are involved in the execution of the Early Warning Indicator System (EWIS) and in the development of its analytics, policy, and implementation across the Commonwealth.

Population Served

EWIS serves educators – specifically administrators, counselors, and teachers – across the Commonwealth of Massachusetts.

Organization's Accomplishment

The Massachusetts EWIS has two components:

  1. A comprehensive array of predictive analytics covering every student enrolled in grades 1-12, with multiple academic outcomes and a suite of reports tailored for school and district use.
  2. The Early Warning Implementation Cycle, providing implementation resources that guide practitioners as they incorporate and strengthen cycle-of-inquiry practices.
Important Factors for Success

Critical to success of this work has been the interweaving of technical components and training. Technical components include collection of dependable student-level data across many different systems; analysis to form student-level predictions; and delivery of user-friendly reports to school-level and district-level practitioners in a timely manner. It has also been essential to offer a wide range of training and assistance opportunities as school- and district-level users learn EWIS and practice the cycle of inquiry.

Requirements for Success

The work has required three staff members, tremendous support from data collection and analysis staff at the Department, an IT staff capable of delivering the reports through a data warehouse system designed for tens of thousands of potential users, and a statewide commitment to prioritize the use of data for school improvement. Staffing has been state-supported (for two staff members partially dedicated to this project) and funded by an expiring SLDS grant (for one staff member dedicated to this project). There is also financial support for statewide, regional and local events to train users in the usage of the system.

Timeline

The DESE collaborated with the American Institutes for Research (AIR) to develop EWIS risk models using funds from a 2011 federal State Longitudinal Data Systems (SLDS) grant. A DESE team worked closely with AIR to determine the appropriate models for each grade level using a rigorous statistical method. An EWIS Advisory Group with members from DESE and other state agencies reviewed the research findings and discussed key decisions. DESE updates and validates the model annually.

Starting in 2015, DESE expanded the EWIS risk models to include risk information for postsecondary outcomes using funding from a second SLDS grant. Postsecondary risk information was first available to districts in 2016. Since 2016, DESE has continued to improve delivery and training on the system, developed several new resources, and increased technical assistance offerings to strengthen practitioners’ implementation practices.

Beginning in May 2018, DESE offered a 16-month competitive grant to ten districts to enhance their data practices using the Early Warning Implementation Cycle. Highlights and examples from their work have been incorporated into resources.

Overcoming Challenges

The team overcomes challenges through persistence, flexibility, and a user-centered focus. Resources and professional development experiences are designed with the users in mind. Developing strong relationships with internal IT and data collection partners has allowed the Department to provide cohesive and user-friendly reports to the field.

EWIS risk levels are available to educators in late August, coinciding with the start of the school year. At an annual fall convening, workshops are offered for users at all levels, from an introductory workshop to specialized sessions for seasoned users. Technical assistance offered throughout the year centers on resources and strategies for monitoring ongoing risk, monitoring interventions, and adapting supports based on emerging information.

A challenge for users has been incorporating EWIS into existing routines. The EWIS team mitigates this by positioning EWIS to be as user-friendly as possible, with a variety of approaches and resources; providing a range of engaging experiences and technical assistance; and partnering with other teams to incorporate components of EWIS into their supports. Examples include the EWIS Monitoring Tool and other monitoring guidance; EWIS e-newsletter; Enhancing Data Use grant; videos; and other resources on the Massachusetts EWIS website.

Lessons Learned

~ Create a small internal team with a range of skills, a strong sense of mission, and a focus on delivery.

~ Listen to and partner closely with users.

~ Promote the use of data as a process that can lead to positive outcomes for young people. Rather than “another thing to do,” it is a resource.

~ Build, refine and adapt. The team has built and released reports and resources over time in response to users’ unmet needs. Resources are refined and adapted based on user feedback.

~ Use clear and succinct messages, such as “Focus on the ABCs: attendance, behavior and course performance.” These factors will improve students’ lives in the long run.

~ Cultivate relationships with organizations across the state for outreach and training opportunities (e.g. state associations of school counselors or secondary school administrators).

Future Goals

To continue the important work of data use on the individual and aggregate level, using early warning indicators in a cycle of inquiry to improve student performance, particularly in the most challenging schools and districts.

Major Takeaways

~ Use measurable outcomes for students; this can test the integrity of the work.

~ Cultivate partnerships with educators in the field so that they share feedback, which helps improve delivery systems.

~ Focus on understanding underlying causes of risk, in particular all of those within the scope of educators, school leaders, and district leaders.