Charades of Evaluation: mis-connecting cause and effect

updated 4/23/19

"…passing a failing student is the #1 worst thing a teacher can do. … Changing grades is the most undermining contribution to a student’s failure, but above all else – it invalidates your data. Putting aside creating and submitting inaccurate school data for the moment, entering a 'false grade' will make it virtually impossible to reliably measure any improvement of your skills as a teacher. Your improvement will now be based on unsound and worthless data." -- M. Cubbin (8/2/15) The Business of School

"… Is the Customer Always Right?" -- Farrington, Frank (1915) in Merck Report, Volume 24 pg 134-135

Pseudo-Evaluation. Data are not fundamental. On their face, data cannot be distinguished from outputs of a random number generator. Data are like shell chips on a beach mixed in with sand; or, like foam on the tide. (For an article relating “data” and “objectivity,” see Can Criminal or Immoral Behavior Be Dealt With Objectively?)

Far more important to know is what the processes are by which the putative data are collected. And even more critical is knowing which theories connect the data and collection process to what supposedly they indicate.

Much “data-collection” is like trying to identify proportions of bird-species during a fall migration. If the process were merely to tally varieties of southward flying objects, we might well end up confounding red-winged blackbirds with jet planes and monarch butterflies. (See Is It Really a Test? Or Just Another Task?)

In the above epigraph, Cubbin presumes a connection, presumably ideally possible, between school grades, student failure, and teacher skill. Using teachers as graders cuts costs, but is begging for inconsistency: not, because teachers may not “know their stuff.” But, because many institutional processes can overrule even the best of teachers’ judgments, e.g. administors’ prerogative, special education policies, or political involvement in the grading process. There is little consensus, when interest-group push comes to shove, on either goals or concepts appropriate to education. (See What Does a Consensus Mean, Anyway?" )

Requiring teachers to grade then grading the teachers is like judging baseball coaches using their players' batting averages. There will likely be only a tenuous, if any, connection between the data and any causal relationship to coaching (teaching) efforts. (See Power Failure: Losing the Series; Blaming the Bat Boys )

The Diploma-Holder Markets: is the customer always right? An important assumption Cubbin seems to make is that markets for test-passers are comprised of persons looking for those who possess certain proven skills. This is only a minor proportion of the markets for certificate- or diploma-holders.

Consider these other markets for whom actual skill levels are a distant, if even, a second consideration to applicant grade-point average:

a. Colleges, public, private or commercial, who have external, e.g. federal, or foundational, funds available for applicants with a certain grade-point average -- especially if the recipient institutions have tight budgets;

b. Institutions legally required to have certified staff but faced with employee scarcities, e.g. hospitals, clinics, civil-service;

c. School districts needing both adults certified as teachers, and students with birth and health certificates, in order to be run at even somewhat remove from peak efficiency;

d. Government administrations pursuing certain public policy initiatives that depend on items a, b and c, preceding; e.g. special education, affirmative action, STEM (Science, Technology & Mathematics); and last but not least,

e. the children applicants, legacies, to colleges which favor (paying) parents who are past graduates.

If skills really counted, there would be something like board standard examinations to be passed; normally, to be retaken at standard intervals. Teacher grades would not be accepted in place of board exam results. (See The Dangers of Diplomas)

But where the sheepskin alone is most important, rarely will the sheep’s diet be.


For examples and to pursue the issues raised in this essay, see

1. “Data-Driven”: a slogan to distract from organizational disagreement?;

2. Classification Error in Evaluation Practice:
the impact of the "false positive" on educational practice and policy

Cordially
--- EGR

The Worm Turns: Testing the Candidates

The current governmental infatuation with one-size-fits-all standardized testing is a concern of many. But there is a remedy for this illicit love affair. Require every candidate for political office to take a battery of standardized tests. Then we will widely and repeatedly publicize the result. And to make sure no deserving individual escapes the net, we will also require aspiring high level government appointees, such as candidates for Secretary of Education, to also take the tests.

We could require every aspiring office holder to take the same tests he/she prescribes for others. Before being allowed to run for President, for example, Dubya would have had to pass the self-same tests he championed for high schoolers. Imagine him sweating and scratching his noggin.

Similarly we could require every aspiring state Secretary of Education to pass the battery of tests they propose requiring of aspiring teachers. In Pennsylvania, for example, he or she would have to pass separate NTE tests in Reading, Writing, and Listening Skills, (The later would be a tough one for any politician.).

An alternate plan is to design office specific skill and knowledge tests. A candidate for President of the United States, for instance, would be tested on their knowledge of world geography test, US and world history, basic economics, the environment and so forth. We could turn to ETS or the Psychological Corporation to craft the items. And they would be finalized only after a painstaking vetting. Blue ribbon boards would appraise and reappraise every question. Then every candidate’s test results would be announced to the world.

We could also test their moral and ethical views. Admittedly, dishonest answers would be a problem and safeguards clearly are required. One possibility is to administer this particular test while test-takers are hooked up to lie detectors. Imagine a candidate sweating and squirming as the polygraph relentlessly tells the tale. “Is that your actual answer? Is that your final honest answer?” (Philadelphia’s infamous late Duce/Mayor Frank Rizzo once failed a lie detector test while trying to prove the device’s reliability. Evidently the polygraph was more discerning than voters.) Alternatively, we could inject test-takers with scopolamine, a truth serum favored by secret policemen the world over. The test would be administered orally as the subjects drift guilelessly on a tripped out cloud.

Regardless of the method, however, we would have to be absolutely certain that our subjects answer truthfully. And we should keep in mind that most of them would be unaccustomed to doing this.

That, in broad outline, is the plan. But it needs filling in. That’s where you can help. Tell us what you think. Should aspirants for public office take the same tests they prescribe for others, or should they be required to take brand new custom designed tests of character and job-related knowledge. And should we test just once, or every year the person is in office? (Longitudinal testing has the obvious advantage of measuring whether or not the subject is improving while “serving.”)

You also might like to suggest specific test items. They need not be multiple choice. All types of questions typically found on standardized tests are welcome. Rush your comments and test item suggestions to The Worm Turns Foundation, c/o Newfoundations, P.O. Box 94, Oreland, PA 19075, or post them here.

To examine these issues further, see Justice Through Testing .


Best Wishes,

--- G K Clabaugh

The Public School’s Real Mission: Inculcating A Common Faith?

Faith is believing what you know ain't so.
 -- Mark Twain, Pudd'nhead Wilson's Calendar

The reason bad public schools are bad is that they badly governed, from top to bottom. Happily, bad public schools are in the minority, despite what some newspapers and other lickspittle TV pundits have to say about them.

“Badly governed” does not mean “governed with bad intention.” What it does mean is that school leaders seldom are willing to face facts which threaten their piece of mind. These school governors, -- state and local administrators, superintendents and principals -- are brim full of ideologies and nostrums they’ve inherited and which they insist on foisting off on the vulnerable.

Part of this is their not unreasonable belief that that much of the general public is in the same delusional state. No parent wants their kids learning things which would disturb the parents’ peace of mind. (This is why so-called “critical thinking” gets many a salute, but has yet to make any headway into the public school curriculum.)

In the olden days of kings and princes, the Established Church of a nation saw to it to inculcate the belief in their congregations that their ruler, as malignant as he might be, sat in power by the Will of God. Subjects were to accept their lot in life and forswear violence as an instrument of change. (See In Days of Yorengore)

In today’s United States of America, where both aristocracy and religious sectarianism in government are purported to be absent, it is the role of the Public Schools (and many a private and parochial school) to teach little more than that our “citizens” -- never “subjects” -- are obliged to accept their governors as the Will of the People.

As good citizens, they are to reject any attempts at governmental change which might involve “violent” activity -- no matter what Thomas Jefferson ever advocated. This "Citizen’s Faith" is what is most important; the recently tediously advocated knowledge-and-skills-focus on the schools has always been given little more than lip-service.

If getting knowledge and skills were really the mission of the schools, then there would be a far more careful preselection of students than our governors and their clients, including parents of schoolchildren are prepared to pay for. Kids are put in grades by age. This is a traditional and cheap sorting method. If some don’t do so well on their report cards, everyone who does not work in a school knows at whom to point the finger of blame.

Here is a method for raising test grades substantially in just two years:

a. for the next school year, retain in grade every public school student who does not have B+ average on standardized tests;

b. any student who is failing this year should be held back for two years;

c. make age 7 the beginning year for first grade; and

d. pull back the curriculum so that any intellectually rigorously formal curriculum is not encountered before 9th grade.

If these steps were to be taken, students would encounter the present high-stakes testing almost two years later in their academic careers and more likely do far better in them, especially in the middle schools.

I don’t expect the above steps to be even considered. Why? Because our school and political leaders, along with many, many parents of school children, just have too many mental impediments -- myths, misinformation, and narrowly focused esteem issues -- to deal with providing their students, their children, with an appropriate, humane, invigorating education.

For references and to examine these issues further, see The Capacity to Benefit from Formal Academic Schooling: 
two ideologies of distribution 




Cordially
--- EGR

What is a Fair Grade? Should Effort Count?

What's sauce for the goose, is sauce for the gander. -- Proverb

Instructors at even “top-rank” universities report that their students fuss and fume if they get anything less than an A on an assignment. They expect that just putting out the effort – or claiming to have done so – merits a B so that anything more should bring it up to an A. A professor’s insisting that there may be standards that should influence judgments of quality is dismissed as some kind of cranky, outmoded elitism.

The retorts are obvious. The easiest one is to ask students if they would want to go to a doctor who had “put in the effort” during his or her studies, but never really learned the material. The students see the point but somehow continue to believe that their work should be exempted from conclusions drawn from such examples.

So far as most students are concerned, grades are competitive prizes, to judge from their comparison and boasting about GPA’s. However, everyone’s entitled to the honor of a diploma. This is a benefit that must be fairly shared quite independently of its use as a marker of accomplishment

To examine these issues further, see Fair Share vs. Fair Play: 
Two Competing Conceptions of Justice

-- GKC (EGR)

Grades: an Illusion of Value?

updated 012621

Princeton University students were upset. Their alma mater had decided to fight back against grade inflation by putting a cap on the number of A’s students could be given. But students felt they were being deprived of what they were entitled to. (New York Times Metropolitan Section, Sunday, January 31, 2010)

In many disciplines academic grades have little significance because professors in those fields cannot themselves agree on what standards should be. How much knowledge a student has acquired may play only a small part in the process of chasing a grade. No small factor is how the student capitulates in the face of faculty demands for deference. In many institutions of “higher learning” it is not reasoned argument and factual knowledge that wins the grade, but rather, -- to forego more obvious, vulgar expressions – obsequiousness.

Students have learned to expect a certain quid pro quo: they will give rambling, incoherent college teachers good reviews on the grounds they are “OK" or "nice people.” In return the students expect to be rewarded with A’s and B’s despite their ignorance or low levels of production, for having been “enthusiastic" or "attentive.”

If they care at all, prospective employers may use an applicant’s academic grades as an inexpensive selection method. Using grades to screen new employees helps those making hiring decisions cover their "assets." No one gets criticized for hiring a bad employee if that employee came in with a sterling transcript! And transcript review is a lot cheaper than an apprenticeship.

To examine these issues further, see The Teacher as Technician: Will Technology Improve Schooling?

Cordially
EG

What? No More Brownie Points! It’s … It’s UnAmerican!

The Sunday November 28th, 2010 New York Times, Week In Review p. 3 reports that because some kids with B+ and A averages cannot pass standardized tests, and because other students with F, E, and D averages do pass them, that awarding grade points for homework done (and other not directly academic activity) will be tightly curtailed in certain school districts.

This is dangerously revolutionary! It threatens to invert the pyramid of scholastic honors by giving top rank to hard-scrabble, lower-class NERDS, (and too many Asian students)! Such a policy fails to appreciate the social graces hard-won by students from Middle Class and Elite families who know how to show appreciation for the (often, very subtle) humor, insights and wisdoms of whoever is in charge. A “Yes, sir!” or an “Of course, ma’am!” is worth a hundred solved quadratic equations.

Mere technical skills are not what move a person up in the corporate world. Technicians, accountants and engineers are just hired help. A head full of facts is not necessary, or even desirable to achieve even the highest political office! Schools that serve the fetish of skills and facts are schools that will graduate workers destined to suffer resizing!

To examine these issues further, see Productivity, Politics and Hypocrisy in American Public Education


Cordially
--- EGR