McREL Blog

Data Snapshot: National Differences in ELL NAEP Mathematics Scores

By Jane Barker

Most of the commentary following the release of national NAEP scores last week focused on flat performance among the nation’s 4th grade students. Of course, reading past the byline yields more interesting fodder for discussion.

In the case of English Language Learners, NAEP captures national variation in achievement that's difficult to ignore - and explain. Using NAEP data and published ETS ELL statistics, I pulled together a couple of graphs comparing 4th grade ELL performance among those states with the nation’s largest ELL populations (Arizona, California, Florida, Illinois, New York, and Texas) and those states with the nation’s fastest growing ELL populations (Alabama, Indiana, Kentucky, Nebraska, North and South Carolina, and Tennessee). Among the latter states, rates of ELL population growth between 1994 and 2005 ranged from just over 300 percent (Nebraska) to just shy of 715 percent (South Carolina).

While you might expect both groups to demonstrate relatively similar patterns of achievement over time, NAEP outcomes suggest otherwise. Clearly, this is more than a demographic story: several states with fast-growing ELL populations seem to be doing fairly well (North and South Carolina, Kentucky) in comparison to more heavily populated states that are lagging behind the national leaders (Arizona, California, Illinois, New York). And internal variation within each group? All over the board. Critics have pointed out that NAEP is sensitive to differences in curricula and standards in addition to more controversial policies (namely, distribution of instructional resources and social promotion).  Obviously, graphs are just a starting point, but it’s worth pointing out that an honest NAEP discussion should serve as a starting point for uncomfortable questions about educational equity.

Jane is a Research Associate within McREL's Research & Evaluation department.

NCLB and Science/Social Studies Instruction in High-Risk Schools

Unsurprisingly, NCLB reauthorization hasn’t garnered much press recently. As this recent Education Week column points out, the stimulus package took the heat off of Congress, leading to speculation that any proposed reauthorization bill won’t gain much traction this year. Legislation aside, the NCLB debate is ongoing. Case in point: the National Center for Analysis of Longitudinal Data in Education Research, or CALDER, recently organized a conference inviting researchers to consider differences in educational outcomes and delivery since the passage of NCLB. Presentations from the conference, co-hosted by the National Center on Performance Incentives, were recently posted here.

 

Findings presented at the conference touch upon a range of subjects, worthy of at least a few minutes’ perusal (if not a few hours’). Dale Ballou and Jeffrey Springer, both of Vanderbilt, contributed a representative presentation on science and social studies instruction in high-risk schools. Conventional wisdom suggests that schools struggling to meet AYP goals would be tempted to neglect instruction in subjects which do not impact accountability outcomes – social studies, and until recently, science. Using data from South Carolina, which adopted a no-stakes science and social studies test in 2003, and Virginia, which incorporated science and social studies test results in accreditation requirements beginning in 1998, the authors present conclusive evidence to the contrary.

 

Essentially, post-NCLB science and social studies scores improved at pace with math and reading scores in each state. In Virginia, the scores for these subjects improved at a faster rate in high-risk than in low-risk schools, except among scores designated as ‘advanced’, where the gap widened slightly. While scores improved in South Carolina, the gap remained relatively constant between low- and high-risk schools. The only evidence of trade-off: high-risk South Carolina elementary schools, where the authors found that reading instruction displaced non-core subjects. On average, South Carolina schools studied managed to close this gap by high school, which may suggest that the extra reading instruction paid off. While the authors caution that their results can’t be generalized to all states, their study lends weight to the argument that accountability requirements which reward improvement in math and reading don’t always come at the expense of science and social studies.

Charter school market at a crossroads?

by Bryan Goodwin

The Wall Street Journal reports that the number of charter schools in the U.S. is likely to mushroom in the next few years as a result of U.S. Secretary of Education Arne Duncan warning states that if they’re unfriendly to charter schools, they shouldn’t expect to see much of the $5 billion in federal stimulus for schools. Not surprising, many states are now scrambling to create charter-friendly environments.

Last month, the Center for Research on Education Outcomes (CREDO) at Stanford University, released a nationwide study analyzing charter school performance. The report was notable for a number of reasons—starting with its methodology. To control for any possible “cherry picking” (i.e., enrolling easier-to-teach students), the study compared the mathematics performance of students in charter schools with their “virtual twins”—students with similar demographic, socioeconomic, and special needs status—in traditional public schools.

Using this analysis, the study painted a mixed picture.

It found that only about one in six (17%) of the 2,400 charter schools studied were actually successful in helping their students perform better than their “virtual twins” in traditional public schools. About half (46%) offered little or no bump for their students compared with their “twins.” And nearly two-fifths (37%) appeared to have a negative impact on achievement; their students learned at lower rates than their comparable peers in traditional schools.

Whither market forces?

So what happened to the market forces of choice and competition that were supposed to make charter schools better than public schools? It appears that these market mechanisms have blunted in at least two significant ways.

First, according to the Stanford researchers, too few charter school authorizers are shutting down low-performing charter schools. Consider, for example, that in states where multiple agencies are licensed to grant charters, charters turned in their lowest performance—presumably because weaker schools have been able to shop around for more permissive entities under which to operate.

Second, it seems that parents, who were supposed to pull their kids from ineffective schools and create market-based incentives to provide better outcomes, have yet to become informed consumers of schools. Conversely, parents choose charters schools on the basis of more than just the academic performance of their students. Indeed, the Stanford study notes that it is often parents and communities who most strongly resist closing low-performing schools, arguing that shutting down their school “does not serve the best interests of currently enrolled students.”

A fragmented market

At the moment, the charter school market resembles what Harvard professor Michael Porter describes in Competitive Strategy (a common business school primer) as a “fragmented market.” No single provider—or even handful of providers—has achieved significant market share. According to the National Alliance for Public Charter Schools Web site, 77.5 percent of charter schools are “free-standing,” not associated with an education or charter management organization, such as Edison or KIPP.

My own quick scan of charter school Web sites (see below) suggests that combined, the top four largest charter and education management organizations operate 320 schools—or just 6.9 percent of the 4,618 charter schools nationwide. Throw in the next six on the list and you find that the top 10 companies still only control 11.4 percent of the market.

Top 10 charter or education management companies*

1. Edison Schools (97 schools)
2. KIPP (82 schools)
3. Imagine Schools (73 schools)
4. Big Picture Learning (68 schools)
5. National Heritage Academies (57 schools)
6. White Hat Management (51 schools)
7. EdVisions (40 schools)
8. Aspire (21 schools)
9. (tie) Green Dot (19 schools), Charter Schools USA (19 schools)

Shakeout coming?

Fragmented markets, like this one, can be ripe for “shakeouts”—with increased competition forcing smaller, less effective companies out of business. After the dust settles, usually just a handful of big players are left (consider, for example, video rental stores, roadside motels, and airlines).

So can federal stimulus dollars infused into the charter school market create more competition and ultimately a “survival-of-the-fittest” shakeout?

Probably not.

Simply incentivizing states to allow more charters is not likely to change the underlying conditions that Porter says creates fragmented markets:

• Low barriers to entry—The abundance of charter school authorizers, especially those with lax oversight, makes it easy for a variety of providers, regardless of their demonstrated competence, to enter the market.

• Few economies of scale—For the moment, given that most costs of running a school are tied to salaries and personnel, operating in multiple locations doesn’t offer much advantage in terms of marketing, curriculum development, or teacher training; this could change, however, if parents or authorizers were to demand better demonstrated results (which are typically expensive to document) of charter school operators.

• Diverse market needs—Because parents often enroll their children in charter schools to serve the unique needs of their child, it could be difficult for any single type of charter school to serve a large population of students.

• Local regulation—Charter schools are typically authorized by local districts or state granting agencies, each with their own criteria or rules, which may favor local, “mom and pop” providers.

Killing charters with kindness?

A fragmented market is not always a bad thing—it can provide fertile ground for innovation and experimentation. However, as Duncan told a gathering at the annual conference of the National Alliance of Public Charter Schools that “The charter movement is putting itself at risk by allowing too many second-rate and third-rate schools to exist.”

Ironically, a little more regulation and oversight (anathema in the eyes of some charter proponents) might help to create a more mature market, allowing the effective 17 percent of charter models become the norm, not the exception. By more regulation, I don’t mean red tape restricting hiring policies or the number of hours the schools can operate, but rather, as suggested by the Stanford report, encouraging charter authorizers to raise the bar by more consistently closing ineffective schools (which is, after all, the second half of the charter school “bargain”—less red tape for better results).

States might take that a little further, entering into multi-state compacts that could both raise the barriers to entry by demanding higher results from charter schools, while at the same time, creating a consistent set of criteria for charter applications across multiple that could make it easier for effective charter providers to enter new markets.

To help authorizers make better decisions and parents make more informed choices, better information is needed across the system. The Stanford report recommends, for example, that national metrics be created that would allow for better comparisons of schools and identification of high and low performers. Such metrics could allow parents to more accurately gauge a school’s contribution to their children’s academic success and weigh that value against other less tangible benefits they may perceive their school provides.

While some charter authorizers may have adopted a laissez-faire approach to regulation early on to encourage the growth of the market, the movement may now be at a cross roads. Those who support it may need to decide whether a proliferation of new charter schools is in their best interest, as quantity comes at the expense of quality. More to the point, they should decide whether a continued hands-off approach to regulation is in the best benefit of the charter market or conversely, killing it with kindness.

* I didn’t include Expeditionary Learning or Core Knowledge on this list, both of those organizations offer curriculum to charter schools (35 and 23 schools, respectively by my count), but don’t function as management organizations, actually running the schools. If I’ve missed any CMOs or EMOs, please let me know. I’d be happy to update this chart.

Bryan Goodwin is McREL's Vice President of Communications & Marketing.

Summer Brain Drain Revisited

As part of Atlantic Monthly’s annual ‘ideas’ issue, out this month, Derek Thompson offers up a provocative list of not-so-quick fixes for the nation's educational system ("10 Crazy Ideas for Fixing Our Education System"). While Thompson's list mixes solutions old and new, readers might be surprised by the suggestion topping the list: the elimination of summer vacation.

Perhaps his suggestion is a bit extreme, but Thompson’s reasoning has a basis in sound research. Several high-profile studies from the past few years have noted that achievement gap margins tend to widen over the summer break. For a good summary of the reasons why, see this article in yesterday’s Washington Post. Middle class children are more likely to have books in the home and to attend high-quality summer programs in the summer, offsetting the loss in reading skills that occurs while students are on vacation. The 2007 study cited in the article found that differences in summer experiences explained two-thirds of the achievement gap between advantaged and disadvantaged 9th graders.

Even high-quality programs are more likely to focus on reading than math, which explains why the reading achievement gap is more prevalent at summer’s end. Children (and adults) have more difficulty retaining specific processes than basic concepts over long periods of time – e.g., solving a quadratic equation versus reading a passage for comprehension. As a result, the greatest summer losses across the board are typically in math computation and spelling.

For more information on the effort to promote summer learning, check out the National Center for Summer Learning (www.summerlearning.org) ahead of the July 9th National Summer Learning Day. In addition to research and policy briefs, the site offers suggestions for effective programming, leveraging community partnerships, and professional development options.

What went wrong in Miami?

The Miami Herald reported recently that former superintendent Rudy Crew’s “$100 million investment in Miami-Dade County’s lowest performing public schools failed to boost student achievement, according to the school district’s final report on the program.”

“The School Improvement Zone” focused on eight low-performing high schools and their feeder schools for a total of 39 schools. The effort extended the school day by one hour and the school year by 10 days and required 150 minutes of reading instruction each day. On paper, the Zone effort appears to have focused on the right things, such as improving reading instruction and giving kids and teachers extra time for learning (a la promising schools such as KIPP).

Yet, three years and $100 million later, the district had little to show for its effort. According to a 166-page internal report from the district’s program evaluation office, annual academic growth rates for 8th graders attending Zone schools was actually lower than students in comparison “control” group schools.

So, what exactly happened?

To be fair, some of the Zone schools, especially elementary schools, did improve student achievement, just not markedly over comparison schools. And it's important to note that schools in the comparison or “control” group weren’t sitting idly by; many were focused on improving reading instruction through the federal Reading First program. In effect, the most significant (and costly) difference between the Zone schools and others in the district was the additional time added to the school day and calendar.

Why didn’t the extra time help?

For starters, some students played hooky. The evaluation report notes “excessive absences” (even more than might be expected for the end of school) among students in Zone schools during the added days. It also notes that some students—especially those who were high achieving, felt like their required 10 extra days of school was punishment for a crime they did not commit.

One might also ask whether 60 minutes a day is really enough extra time to offer struggling students the support they need when other schools, such as K.I.P.P., provide as much as 60 percent more instructional time over traditional public schools (staying open from 7:30 a.m. to 5:00 p.m. on weekdays, every other Saturday morning, and for three additional weeks in the summer).

The anemic results of the Miami Zone schools may offer a cautionary tale for those, like President Obama and Arne Duncan, who have advocated for longer school days and calendars. While more instruction may sound like a good idea, U.S. schools already appear to offer students more instructional time than many other higher-performing nations. Moreover, lengthening the school calendar by half measures—tacking on an extra hour a day and some poorly attended days in the summer—is probably insufficient to provide struggling students the extra support they need to succeed.

To top it off, such changes—or any large-scale change, for that matter—are likely doomed to failure if they do not address school culture. Consider these data from Miami:

• 34 percent of teachers in Zone schools said “staff morale is high” versus 55 percent in control schools

• 43 percent of parents of children in Zone schools think their “school maintains high academic standards” versus 62 percent in control schools

• 56 percent of students in Zone schools said “this school is safe” versus 70 percent in control schools

In the conclusion of their report, evaluators noted that low morale among staff in Zone schools likely contributed to the effort’s “lack of efficacy.” Attempting to reform struggling schools without changing their culture is like trying to lose weight simply by buying a new track suit and joining a new gym; it may be an important first step, but only as effective as the changes in values, beliefs, and behaviors that must accompany it.

Motivation, Feedback and Achievement: Exploring Recent Headlines

Practice makes perfect – so why are so many middle school students more intent on honing their video game reflexes this summer than their math skills? Research suggests that academic motivation tends to decline in American schools as students reach middle school – a quandary for educators, given the links between intrinsic motivation, practice, and achievement (for a practical take on this subject, see David Brooks’ recent New York Times op-ed, “Genius: The Modern View”). Teaching students to delay gratification is critical to this effort – making the development of self-regulation a hot topic in early education. While educators have struggled to determine whether or not instructional strategies can successfully teach children to internalize intrinsic motivation prior to middle school, evidence suggests that focusing on effort rather than ability (‘you’re working hard’ vs. ‘you’re good at this’) is an especially beneficial practice. Other voices have called for parents and educators to explicitly teach children about the malleability of intelligence and the relationship between practice and achievement.

These are complicated issues. As we are all aware, motivation varies by context, culture, and personal predilection: a student intrinsically motivated to finish a Harry Potter novel may simultaneously rely on extrinsic motivation (conveniently enhanced by threat of a pop quiz) to finish Wuthering Heights. Unfortunately, extrinsic motivators (grades, social pressure) begin to break down in middle school, making the development of internal drive all the more valuable. Think this might be an interesting topic for light summer reading? Richard Nisbett’s new book, Intelligence and How to Get It, offers a thought-provoking discussion of the sources of individual variation in student motivation.

A Denver school’s “Rap to Roots” program … the rest of the story

Last week, the top, front-page headline of the Denver Post read “Rhythm Scholars: Rap helps kids boost their academics—from multiplication to Shakespeare.” The story described an afterschool “Rap to Roots” program at the Wyatt-Edison Charter School in Denver’s inner-city Five Points neighborhood. The Post reported that the program “debuted this month in Colorado after success in cities such as Cleveland and Chicago."

In a nutshell, the program aims to boost kids’ academic achievement by using rap music to motivate students to stay in school and to teach them about literary conventions, technology, and African-American history. The article reported that the program “did significantly better in standardized testing, attention spans in the classroom, and some [students] improved their writing skills.”

The story got noticed both in and outside Denver. Colleen O’Connell, the Post reporter who wrote the article, said that a national TV news network contacted her, expressing interest in covering the story. The next day, ASCD Smart Brief picked up the story and blasted it out to 175,000 readers. And within hours of the article being posted on the Post Web site, more than 80 comments had been posted, ranging from favorable support to equating the use of rap in schools with the demise of Western civilization.

My first reaction to the story was mixed. On the one hand, I could appreciate what the program was trying to do: offer urban kids an opportunity to express themselves creatively, motivate them, and provide them with a positive way to spend after-school hours (“Rap to Roots” is part of the school’s “Open Door Youth Gang Alternatives” program).

On the other hand, the rap lyrics written by one of the program’s 13-year-old scholars didn’t seem to be on par with the expectations of Colorado’s new language arts standards, which require seventh graders to “write fully-developed literary … text” and to “edit writing for correct grammar, capitalization, punctuation, spelling, [and] sentence structure … .” 

I worried that this nifty idea might be just another example of well-meaning people setting low expectations for urban kids. So I contacted Michael Schenkelberg, the program’s director, to get, as the late Paul Harvey used to say … the rest of the story.

First off, I learned that the program is more of an adaptation than an adoption of what Schenkelberg had been doing in Cleveland under the auspices of The Initiative for Cultural Arts in Education (ICARE), supported by the Young Audiences of Northeast Ohio, a nonprofit arts-in-education organization. ICARE is essentially an artist-in-residence program. It places artists in classrooms for weeks at a time, during which they collaborate with teachers to enhance classroom learning through arts education that can include dance, music, drama, and visual arts.

An evaluation study of the ICARE programfound that the longer students were exposed to the program, the higher they scored on standardized reading, math, and science tests. In addition, 87 percent of teachers said that their students worked more creatively because of the ICARE program. While the study produced correlational, not causal data, the findings suggest the program may have some positive effect on student learning.

However, in contrast to ICARE, the Rap to Roots program is—at least in its initial iteration—strictly an afterschool program that is, for the moment at least, focused on one type of artistic expression. Instead of placing artists in residence in a single classroom, guest artists work with students from multiple grades (fourth through eighth). Those differences, Schenkelberg explained, make it difficult to involve classroom teachers in offering feedback on student work, which might help them take their writing to the next level.

Indeed, at this point, the student work produced through the program is really just a means to a larger end: providing kids with a creative outlet that sparks their interest to learn and stay in school. Someday Schenkelberg hopes to expand the program to look more like the ICARE program—perhaps by bringing in local artists, musicians, and writers to work side-by-side in classrooms with teachers whose students have often seen their arts programs cut or scaled back.

Bottom line: while the Post should be commended for its efforts to provide a “good news” story on education (and place it on the front page), its headline writers probably connected a few too many dots in proclaiming “Rap helps kids boost their academics.”

Nonetheless, “Rap to Roots” shows promise, especially if the program is able to successfully replicate the components that made ICARE successful in Cleveland. Moreover, Schenkelberg should get credit for recognizing that all students deserve an opportunity to explore and develop their creative sides.