Are Massachusetts's public school students all-stars or also-rans in science compared to their fellow students around the country and the world?
The answer depends on who's talking and which standardized test is under the microscope. One measure of how well Bay State kids are performing recently came from the 8th grade science test designed by the National Assessment of Educational Progress (NAEP), which tested students and their peers across the United States. Another benchmark is offered by the Program for International Student Assessment (PISA), which just released its results showing how Massachusetts 15-year-olds stack up in science, math, and reading compared to others across the country and around the world.
Results from the NAEP test, administered in 2015 to more than 100,000 8th graders in 46 states, had Utah leading the nation, with 47% of its 8th graders scoring "Proficient" and 3% scoring "Advanced." Not only the did Beehive State outscore Massachusetts (which had 2,200 students take the test), but it also made considerable gains from the 2011 NAEP test, when 41% of its 8th graders scored "Proficient," and 2% scored "Advanced."
Massachusetts 8th graders placed fourth, behind New Hampshire and Minnesota. The test is given every four years by the U.S. Department of Education (DOE).
"It's an independent thermometer on how well American kids are actually doing [understanding their science curriculum]," said Frederick Hess, a scholar at Washington, D.C.-based American Enterprise Institute, a think tank that researches public policy issues, and an executive editor at Education Next, a journal published at Harvard University's Kennedy School of Government. "It's an apples-to-apples comparison, and it's really useful for getting an honest, straightforward audit on how well kids are doing across the states."
"We have amazing science teachers," said Richard Scott, K-12 Science Education Specialist for Utah's Board of Education. "They're devoted to what they do."
He attributed Utah's success to a new open-source curriculum science textbook used by its public schools, which is aligned to the state's science standards. Scott says many of Utah's middle school science teachers also have an academic background in science.
Bay State 8th graders' performance on the 2015 NAEP science test was stalled compared to their performance in 2011. In 2015, 41% of Massachusetts 8th graders were deemed "Proficient" (up from 40% in 2011) and 3% were ranked "Advanced" (down from 4% in 2011).
In Massachusetts, reaction to the results were mixed.
"What's interesting is that there's been an improvement [in the NAEP science test] nationwide by 4 percentage points since 2009, but Massachusetts is slipping relative to the rest of the country," said Paul Peterson, a Harvard University education policy professor and the senior editor at Education Next. "Massachusetts seems to be stagnating and riding on its laurels. These results are a warning to the policy leaders in Massachusetts: Their schools aren't getting any better."
"The strength our students are showing in science will continue to be a critical part of growing a strong Massachusetts economy and workforce," Gov. Charlie Baker stated in a Massachusetts Department of Elementary and Secondary Education (DESE) press release reporting the results.
DESE Commissioner Mitchell Chester was more tempered: "It's clear that while our NAEP results continue to be strong relative to other states, the latest round of both NAEP and MCAS scores show that we have room to improve in science."
DESE spokesperson Jacqueline Reis noted that the Massachusetts Board of Education voted a year ago to change the state's science standards. She cited a DESE press release that stated: "While not tied to NAEP, the new Massachusetts standards include practices that could help engage students more deeply in the subjects of science and technology/engineering."
Unlike NAEP, which measured students' knowledge of a science curriculum, the PISA test assessed what students "know both in and out of school," says Patrick Gonzales, the test's U.S. administrator. More than 540,000 15-year-olds from 72 countries, provinces, or cities worldwide took the test. Students took the two-hour computer-based test in October and November 2015.
"It's [a measure] of what's been learned both in and out school, maybe even from their parents, and sees if they can apply that knowledge," he said. "It's as real-world assessment as you can get on a test."
U.S. 15-year-olds have taken the test since it was first given. Massachusetts has participated twice, both last year and in 2012, as an independent entity, with its results reported separately from the United States. More than 1,600 Bay State students took the test, and another 5,700 from across the United States also participated.
The 2015 exam focused on science, Gonzales says, but two other subjects were also tested: math and reading literacy and comprehension. Some teenagers took the science and reading tests while others took the science and math tests; an even smaller sample of the students, Gonzales says, were assessed in all three areas.
Bay State 15-year-olds -- all from public schools -- earned a score of 529 on a 1,000-point scale on the science portion, 527 on reading, and 500 on the math test.
Massachusetts's science score ranked it above the United States average at 496, but below Singapore, which took top honors in science with 556. The Commonwealth's reading score placed it well above the United States average of 472, but below Singapore, which scored 535. In math, the Bay State ranked higher than the United States score of 470, but again below Singapore's top score of 564.
Worrisome to some is the Bay State's math score, which was 14 points lower than in 2012.
"The gains Massachusetts was registering for the last 15 to 20 years aren't there anymore," Harvard's Peterson said. "Massachusetts is no longer moving in an upward direction."
While Massachusetts "is still the state with probably the best public schools in the country, there are signs it's beginning to slip," he added.
Tom Loveless, a senior fellow with The Brookings Institution, a Washington, D.C.-based think tank that examines public policy issues, suggests parents look at long-term trends with these tests. He also urges caution when considering their results.
"I'd take PISA with a grain of salt," he said. "It's a very narrow test that measures what kids can do with the knowledge they've gained in reading, math, and science. Two-thirds of the kids who are tested are only sophomores, and it's given in the fall, so they've only had one year of high school." The fall testing time meant students had not been back in school long following their summer vacation. "It's not a good test of what's going on in the [nation's] high schools."
Gonzales disagreed: "What we're asking students to do is what you would want them to be able to do, like solve their everyday problems, whether it's scheduling work or buying Christmas presents." But he also urged caution when parents consider PISA test results: "There are many factors that go into student achievement. [PISA] is another lens into what students know and can do, but it's not the only lens."
In reviewing 2015's PISA results across the three areas tested, Asian countries, such as Singapore, Japan, and Taiwan, and Chinese provinces like Hong Kong and Macau, often find their students among the top five in performance, with a few exceptions. Some of this is due to culture, Loveless suggested.
"The kids [in East Asia] are used to taking high-stakes tests, and there you have a much more academically oriented culture," he said. "The Confucian culture respects intellectualism, and that doesn't exist in the United States."
The United States, he says, "thinks it's more important to be well-rounded than just educated, more socially inclined, and play sports. We convey that to our children and our schools convey that, too."