ACT is expanding its use of experimental sections.  College Board continues to give some students experimental sections, but only if they don’t pay more to do the essay.  It’s time to come clean about experimental sections.

Ever been asked to participate in research? Maybe you answered some questions after a screening of Joe Dirt 3 and they gave you some popcorn? Or perhaps you underwent a little light sleep deprivation in college to get some extra cash for Smashmouth tickets?  Participants in experiments might do it for their love of science, but they usually get paid.

Not if they are SAT or ACT students, however. Both the College Board and ACT have students conduct research for them, by giving them experimental or, to use the technical term, “pretest” questions. According to the testing company Prometric, pretest questions are used to “to ensure that every candidate receives an exam of equitable difficulty” and to test new questions. Pretest questions do not count in students’ scores, nor should they since they are still being vetted to make sure they are good enough for an official exam. Test takers are doing research work for the College Board and ACT, but instead of paying them, the testing companies charge them.

Experimental sections are not new. The very first SAT, given in 1926, had a 30 minute experimental section. Through more than a dozen revisions to the exam over the next 90 years, the section remained on the test. All students, except those with accommodations (e.g., extra time) had an experimental section on their exam. It was indistinguishable and could more or less fall anywhere in the test and be Math or Verbal (or Math, Reading, or Writing in the 2005-15 version of the test). By the end of the test, students would know what type of section did not count, since there would be an extra section of it, but they did not know which one it was. It was annoying, but it was fair.

Historically, ACT has not used an experimental section on most tests. The format of the exam made it trickier to do so, since the ACT follows a set order (English, Math, Reading, Science) and the sections vary in length.  It would be impossible to slip an experimental section in somewhere, unless everyone had the exact same experimental section.  What ACT has done for years, instead, is add a fifth, twenty-minute section to the June exam.  Students are told that the questions do not count, but could they please try their best on the section. Since it comes at the end of the test, it does not tire students our for the multiple-choice sections that do count, as the old SAT experimental section did. It is true that the ACT experimental section could fatigue students who have opted to do the essay, since it precedes the essay, but as we’ve discussed elsewhere, the ACT essay really doesn’t play a role in admissions decisions.

In a recent update, ACT announced that starting in September all students, except those with accommodations, will have an experimental section, whether they opt to do the ACT Writing section or not. It will still be twenty minutes long; it still won’t count; and students will still be told it does not count.

If only the College Board could be this transparent.

When the SAT underwent its redesign, which made it look a lot like an ACT–a fixed order of sections with varying lengths–test prep experts assumed that meant the demise of the experimental section.  The College Board released test specifications for the new exam, but nowhere in the 210-page document was there an indication that there would be experimental questions on the new test.  At the very end of the document, pretesting questions comes up, but the specifications make it sound like questions will be pretested only in unofficial formats, settings that are “like” the national SAT administration.

All questions are . . . pretested on a motivated sample of students
that resembles the SAT population and is sufficient in size to allow the College Board to evaluate the materials statistically in terms of difficulty, to discern whether the questions can differentiate between lower- and higher-achieving students, and to ensure that students from different racial/ethnic groups do not differentially respond to the questions. The questions are administered to students in test administrations like those in which the SAT is given. [emphasis added] The data from 1,000 to 3,000 students responding to each question are used to evaluate question performance.

This paragraph made it sound like there would be no pretest questions on the official exam, which was why it was so surprising when rumors that experimental questions weren’t dead started leaking from College Board employees in 2015.

We confirmed the existence of the experimental section of the redesigned SAT  in 2016, at a College Board session in Boston for SAT test center administrators. The leader of the session explained that the new test would indeed contain an experimental section and that, like the ACT, it would be a 20-minute section added on after the fourth section of the SAT. We publicized this news in The Washington Post, at the time.

It is not hard to understand why the College Board has not advertised the experimental questions. It wants the best data it can get, and if a lot of students opt out of the fifth section by leaving it blank, that will compromise the section, although the psychometricians at College Board have surely built this contingency into their analysis. The College Board is not putting this additional section on the test to torture teenagers but to make their test better, which in the long run and in the aggregate is perhaps to the benefit of students.

There are, however, three aspects of the way the College Board has chosen to handle its experimental section that should upset anyone who thinks that high school students are already too stressed out by testing.

First, the College Board does not tell students about the experimental section before test day. The section comes as a surprise, which can in and of itself cause anxiety. We could find no mention of it on the College Board website, at Khan Academy, or in the Official SAT Study Guide, where each exam has four sections, not five. When we asked why students cannot learn about the experimental section on the College Board website, the testing company retorted that they could . . . and sent a link for an 18-page pdf of their terms and conditions. If you want to see where they talk about “pretest” questions, the paragraph is on page three.

Second, not all students have to do the experimental section and, worse yet, they can pay their way out of the doing it.  Only examinees who do not pay the $14.00 extra to do the essay (next year, the essay will be $17.00 extra) get an experimental section.

Third, and worst of all, the College Board continues to be disingenuous about whether the experimental section counts in students’ scores. Here is what the proctor tells students, word for word:  “Your test will contain some questions that won’t be used to compute your scores.  These questions may appear in any section.  To give you the extra time you need to answer more questions, your test will include a fifth section with regular and pretest questions” [emphasis added]. To translate, “regular” here presumably means questions that count, also called “operational items.” It may be that by calling them “regular” instead of “operational,” College Board is trying to set up some plausible deniability should someone accuse them of being dishonest about this section.

So, does the fifth section contain operational questions?  It does not appear to do so.  We had several people at multiple sites intentionally leave their experimental section blank.* Some of us had a math section; some had a writing and language section.  Not one of us answered a question on the fifth section.  If any of those questions counted, the blanks would have shown up on the score reports.  Not one of us had a blank answer on our score reports.  In other words, none of the questions in the experimental section counted. None. They had no effect on our scores.

We cannot tell students to leave the section blank, however, on the off chance that a section might count, if say the College Board gave an SAT with a typo on it that invalidated a whole section of the test, as it did in June 2015. It is possible that, after the fact, the College Board could decide to count questions on the experimental section.

We would like to see the College Board take ACT’s approach to its experimental section:  warn students ahead of time that it is there and let them know it does not count in their score. If it cannot do that, it should at least be more transparent about how it operates on the test.

* We took the exams in March and May, which, as stated in the terms and conditions**, is when non-students can take the SAT

** We read the terms and conditions, but we doubt teenagers do

 

 

Leave a Reply