Test Prep Professionals

(Washington Post, by Valerie Strauss & James Murphy)  ||

 

The College Board is about to unveil its newly redesigned SAT college admissions test, with the first students set to take it next Saturday. We already know a lot about the new exam, which was designed to align heavily with the Common Core State Standards and looks a lot more like the knowledge-based ACT test than the previous SAT version. Here’s more, from the College Board website:

Total Testing Time	3 hours and 45 minutes	3 hours (plus 50 minutes for the Essay [optional])
Components	Critical Reading Writing Mathematics Essay	Evidence-Based Reading and Writing Reading Test Writing and Language Test Math Essay (optional)
Important Features	Emphasis on general reasoning skills Emphasis on vocabulary, often in limited contexts Complex scoring (a point for a correct answer and a deduction for an incorrect answer; blank responses have no impact on scores)	Focus on the knowledge, skills, and understandings that research has identified as most important for college and career readiness and success Greater emphasis on the meaning of words in extended contexts and on how word choice shapes meaning, tone, and impact Rights-only scoring (no penalty for guessing)
Essay	Required and given at the beginning of the SAT 25 minutes to write the essay Tests writing skill; students take a position on a presented issue	Optional and given at the end of the SAT; postsecondary institutions determine whether they will require the Essay for admission 50 minutes to write the essay Tests reading, analysis, and writing skills; students produce a written analysis of a provided source text
Score Reporting	Scale ranging from 600 to 2400 Scale ranging from 200 to 800 for Critical Reading; 200 to 800 for Mathematics; 200 to 800 for Writing Essay results scaled to multiple-choice Writing	Scale ranging from 400 to 1600 Scale ranging from 200 to 800 for Evidence-Based Reading and Writing; 200 to 800 for Math; 2 to 8 on each of three dimensions for Essay Essay results reported separately
Subscore Reporting	None	Subscores for every test, providing added insight for students, parents, admission officers, educators, and counselors

 

But there is something you probably don’t know about the new SAT, which is explained in this post by  James S. Murphy, director of tutoring for the Princeton Review in New England. He has almost two decades of experience in test prep. I asked the College Board for a response to a key part of the post, and got an email that provided very little information. The email exchange is at the end of this post.

By James S. Murphy

When David Coleman became the president of the College Board in 2012, he brought with him a promise of change, and not just any change—the positive kind! Although I have been critical of him and the College Board in the past, only a curmudgeon would deny that he has delivered on that promise. The new College Board is more transparent, more cooperative, and, most cheeringly, more committed than ever before to making its signature test, the SAT, an instrument for creating rather than curtailing opportunity for students, particularly low-income and first-generation students, who have historically been ill-served by an exam that has been very amenable to coaching by test-prep professionals like me. From the removal of difficult vocabulary words and a winnowing of math topics in order to facilitate deeper focus on essential skills to the release of free practice materials and easier access to test fee waivers, the College Board has made a sequence of moves that are decidedly good for students. It recently announced the creation of an August test date, beginning in 2017, which means that students don’t need to sacrifice time best spent on schoolwork and extracurricular activities to test prep.

And then the College Board quietly let out one more piece of news about the new SAT. On February 4th, at a session in Boston for test center coordinators who administer the SAT on the ground in high schools, the leader of the session explained that some students at each site would be completing a fifth multiple-choice section, even though all the sample exams that have been released have four sections.

The point of the announcement was to ensure the proper administration of the exam on March 5, the day the SAT debuts nationally (in some states, students will take the test during the school day in the preceding week). A subsequent investigation revealed that this information appears in one place on the College Board website, a document for school counselors, which can be downloaded on the site.

I was in the audience and asked if this section would be an experimental section. The answer was quick, quiet, and short: “Yes.”

As Erik Jacobsen notes at his invaluable website dedicated to admissions exams, the SAT has had an experimental section from its start in 1926. All students did an experimental section, except for those who had extended time to complete the exam. They did not know which section was the experimental, and their performance on it had no impact on their score. Within the testing industry, these questions are called “pretest,” and the questions that count toward a student’s score are called “operational.” According to the testing company Prometric, pretesting questions are used to “to ensure that every candidate receives an exam of equitable difficulty” and to test new questions.

Good as this procedure might be for the test, experimental questions have never been great for students, who basically paid to do some research work for the College Board. They paid not just in money but in time and energy. The experimental section, which mimicked either a Math, Reading, or Writing section on the most recent version of the SAT, demanded the same amount of work as the other sections and left students just as tired, but it was meaningless and potentially deleterious for the student. Even Prometric acknowledges, “Increasing the number of items on an examination can increase candidate anxiety and fatigue in answering the questions on the exam.”

The testing company also claims that “most test development professionals would recommend that the pretesting process be disclosed to candidates prior to an exam administration.” In the past, the College Board did just this. The experimental section of the exam was no secret. In all the public relations the company has done for the new SAT, however, no mention has been made of an experimental section. This omission led test-prep professionals to conclude that the experimental section was dead. One more reason to love the new SAT!

Now we know that this is not true, at least not for many of the students who will not be doing the optional essay. The College Board has not been transparent about the inclusion of this section. Even in that one place it mentions experimental questions—the counselors’ guide available for download as a PDF — you need to be familiar with the language of psychometrics to even know that what you’re actually reading is the announcement of experimental questions.

 The SAT will be given in a standard testing room (to students with no testing accommodations) and consist of four components — five if the optional 50-minute Essay is taken — with each component timed separately. The timed portion of the SAT with Essay (excluding breaks) is three hours and 50 minutes. To allow for pretesting, some students taking the SAT with no Essay will take a fifth, 20-minute section. Any section of the SAT may contain both operational and pretest items.

The College Board document defines neither “operational” nor “pretest.” Nor does this paragraph make it clear whether all the experimental questions will appear only on the fifth section, at the start or end of the test, or will be dispersed throughout the exam. During the session, I asked if all the questions on the extra section won’t count and was told they would not. This paragraph is less clear on that issue, since it suggests that experimental (“pretest”) questions can show up on any section.

When The Washington Post asked for clarification on this question, they were sent the counselor’s paragraph, verbatim. Once again, the terminology was not defined and it was not clarified that “pretest” does not mean before the exam, but experimental.

College Board can call them what it wants, but experimental questions are back, and they might be even worse than the old version because this time not everyone will be subject to them. Only examinees who pay more to write the essay can be sure they will not have to do twenty minutes more work on the multiple choice material than others do.

It is understandable why the College Board has not advertised the experimental questions. It wants the best data it can get, and if a lot of students opt out of the fifth section by leaving it blank or using the answer sheet to make a picture, that will compromise the section, although I am quite certain that the clever psychometricians at College Board have built this contingency into their analysis. The College Board is not putting this additional section on the test to torture teenagers, either, but to make their test better, which in the long run and in the aggregate is perhaps to the benefit of students.

Perhaps. It is hard, however, not to think about the experience of those students who, in less than a month, will be taking the SAT with an extra section while some of their peers won’t. As Dan Edmonds, the Senior Director of Research and Development at Noodle, an education company, argued, “Difficult pre-test questions that a student doesn’t realize have no impact on scoring could throw that student off for the entire test.” Panic is often just a question away for some SAT examinees, and this new section creates more opportunity for it to set in. Edmonds also noted, as Prometric does, that “test fatigue on long tests is a very real factor; the students who don’t have to do the extra work . . . could have a real advantage over students who do have to do the extra work.”

College Board’s decision not to inform students about the experimental section creates another source of inequity. There will be those who know about it and those who do not. All students at The Princeton Review, for instance, have been warned about the extra section. Noodle has surely done the same. Edmonds comments, “Students who are aware that the section isn’t scored–in other words, students who’ve received good test advice or preparation (and who will therefore almost certainly be disproportionately from wealthier families)–will have an advantage over students who don’t realize that these sections have no impact on their scores.”

This advice matters only if, as I was told at the session in Boston, all the questions on the additional 20-minute sections do not count. The obfuscation of this issue by the College Board, in both its statement to me and in its counselors’ handbook, makes that sound less certain. So much for transparency, so much for opportunity.

_________________________

I asked the College Board to comment on a key section of Murphy’s original draft:

I am running a post on the experimental section of the new SAT. Can you comment on the following:

Here’s how it will work:  On the new SAT, the essay is optional.  Students who opt to do the essay will be in separate rooms from those who do not. They will not have to do an experimental section.  Many, but not necessarily all, examinees who opt out of the essay will begin the exam with a twenty-minute section that could cover math, reading, or writing. Students will, as in the past, not be told that this section will not count toward their score on the SAT, and they will not receive any information on how they performed on the section after the exam.

Here’s the response:

Thanks for your note, Valerie. James Murphy from Princeton Review has already been in touch with us for this story. This sentence is inaccurate: Many, but not necessarily all, examinees who opt out of the essay will begin the exam with a twenty-minute section that could cover math, reading, or writing.

That’s all we have to offer at this time.

Thanks,

Kate

Kate Levin
Director, Assessment Communications

I asked what was wrong with the sentence and got this response:

Hi, Valerie:

The SAT will be given in a standard testing room (to students with no testing accommodations) and consist of four components — five if the optional 50-minute Essay is taken — with each component timed separately. The timed portion of the SAT with Essay (excluding breaks) is three hours and 50 minutes. To allow for pretesting, some students taking the SAT with no Essay will take a fifth, 20-minute section. Any section of the SAT may contain both operational and pretest items.

Thanks,

Kate

Kate Levin
Director, Assessment Communications