“Data-Driven”: a slogan to distract from organizational disagreement?

“There is a difference between numbers and numbers that matter. This is what separates data from metrics.”-- Jeff Bladt and Bob Filbin (3/4/1) Know the Difference Between Your Data and Your Metrics

Assumptions in using data. A number of assumptions are necessary to turn “raw data” into something useable. What must be developed, if not already on hand, is a workable consensus on Terminology (concepts), Goals, and Methods of implementation. (For short, let us call this TGM-consensus.)

A workable TGM-consensus is one with sufficient depth of agreement on all the items, TGM, to enable control of production. (See The Indeterminacy of Consensus: masking ambiguity and vagueness in decision)

Whether or not TGM-consensus can be developed is highly sensitive to the kind of organization one is dealing with. In many, not all, businesses there is TGM-consensus. The proof is in the pudding, as the old saying goes: do they consistently -- not necessarily always -- produce a saleable product? If so, there is TGM-consensus.

In politics, religion, the “soft-sciences” and all levels of education[1], different agendas compete: widespread TGM-consensus is often lacking. Again, the proof is in the pudding: is there is persistent debate as to what the pudding should look like, the ingredients needed, the production procedures, and the evaluation methods? Then TGM-consensus is likely rare. And, neither vociferous admonitions, nor seductive pleas to become “data-driven” will make up for these lacks.

Much current promotion emphasizing “data-driven” approaches are little more than ploys to get worried persons to adopt prepackaged TGM-programs without critical pre-evaluation. Such pre-packaged TGM-programs, unless carefully examined for congruence with the TGM-environment to which it will be applied, will likely result -- to judge from past examples -- in a ritual charade of evaluation. (See Causal Charades: organizational rituals of evaluation)

Dealing with “Raw” Data. Here is some “raw” data.

@@&@*$@@&&@@*$$*&@@$$*&*@*@@$&$*@*$@*$*&&*@*&*&$**
What can you do with it (them)? What do you have to know about it (them)? Suppose I let you know that it (they) is (are) “data” gotten trying to measure someone’s physical development? What other information do you need?
Would it matter, if instead of “physical development” I had written “moral development?” Of course it would. The “data” inasmuch as it is relevant data -- some of the symbols might be mere construct effects of the instrumentation -- do not resolve by themselves any questions about development. [2]

A More Everyday Example. A clear example can be given as follows: suppose we have two persons standing together at normal speaking distance, facing each other. Call them Harry and John. Some noise issues from Harry. Consider the following possible descriptions of Harry's behavior:

a. Harry emitted the sound-sequence: /2aym+ gowing+3 hówm1 /.

b. Harry said, "I'm going home."

c. Harry told John he was going home.

d. Harry informed John that he was going home.

e. Harry surprised John with the statement that he was going home.

We can easily imagine a situation where all of these descriptions are true of what Harry is doing. But given a, -- which is the "physical" data of Harry's behavior in b, c, d and e -- neither b nor c nor d nor e need be true. What supporting information do we need to draw any inferences from the data? Here are some possibilities: [3]

1. If Harry is a babbling idiot, a might be true and none of the rest.

2. If Harry is reciting aloud a line from a script, a and b might be true and none of the rest.

3. If John already knew that Harry was going home, a, b, and c might be true but none of the rest.

4. If John is never surprised by anything Harry does, but did not already know he was going home, a, b, c, and d but not e might be true.

Data is the mere tip of an “iceberg.” If you want to trust data, you have to trust a lot more: parts which are usually submerged; and, often hard to fathom.

Why is data often ignored? What may be perplexing is that even in organizations which have long traditions of TGM-consensus, possibly very relevant data is paid little attention to. Why might that be?

Ask , perhaps cautiously, “Qui Bono?” Who stands to benefit, who stands to risk loss, if the data are paid attention to? One reason for the exaggerated drum-beating for “data-driven” undertakings is distraction. The organization’s present TGM-consensus may be either decrepit, or fail to address the burning issues. So, an emphasis on likely irrelevant data draws attention away from deeper disagreement about either terminology, or goals or method. [4] Addressing TGM issues would upset someone’s applecart.

The most vociferous advocates for improvement may be those most wanting to stifle it.

To examine these issues further, see Moral Education: Indoctrination vs. Cognitive Development?)

Cordially
--- EGR

NOTES
[1] See Charades of Evaluation: mis-connecting cause and effect

[2] To see how this code-string can be used as data see the article reference given at the end of the essay, Moral Education: Indoctrination vs. Cognitive Development?

[3] See Measurability and Educational Concerns

[4] Quantification is appealed to especially if it is believed to promote one’s agenda in discussion and investigation. However, Deborah Stone offers some telling criticisms about this practice especially as it is used to squelch open discussion. See for example, "Using Quantitative Procedures Wisely".

Charades of Evaluation: mis-connecting cause and effect

updated 4/23/19

"…passing a failing student is the #1 worst thing a teacher can do. … Changing grades is the most undermining contribution to a student’s failure, but above all else – it invalidates your data. Putting aside creating and submitting inaccurate school data for the moment, entering a 'false grade' will make it virtually impossible to reliably measure any improvement of your skills as a teacher. Your improvement will now be based on unsound and worthless data." -- M. Cubbin (8/2/15) The Business of School

"… Is the Customer Always Right?" -- Farrington, Frank (1915) in Merck Report, Volume 24 pg 134-135

Pseudo-Evaluation. Data are not fundamental. On their face, data cannot be distinguished from outputs of a random number generator. Data are like shell chips on a beach mixed in with sand; or, like foam on the tide. (For an article relating “data” and “objectivity,” see Can Criminal or Immoral Behavior Be Dealt With Objectively?)

Far more important to know is what the processes are by which the putative data are collected. And even more critical is knowing which theories connect the data and collection process to what supposedly they indicate.

Much “data-collection” is like trying to identify proportions of bird-species during a fall migration. If the process were merely to tally varieties of southward flying objects, we might well end up confounding red-winged blackbirds with jet planes and monarch butterflies. (See Is It Really a Test? Or Just Another Task?)

In the above epigraph, Cubbin presumes a connection, presumably ideally possible, between school grades, student failure, and teacher skill. Using teachers as graders cuts costs, but is begging for inconsistency: not, because teachers may not “know their stuff.” But, because many institutional processes can overrule even the best of teachers’ judgments, e.g. administors’ prerogative, special education policies, or political involvement in the grading process. There is little consensus, when interest-group push comes to shove, on either goals or concepts appropriate to education. (See What Does a Consensus Mean, Anyway?" )

Requiring teachers to grade then grading the teachers is like judging baseball coaches using their players' batting averages. There will likely be only a tenuous, if any, connection between the data and any causal relationship to coaching (teaching) efforts. (See Power Failure: Losing the Series; Blaming the Bat Boys )

The Diploma-Holder Markets: is the customer always right? An important assumption Cubbin seems to make is that markets for test-passers are comprised of persons looking for those who possess certain proven skills. This is only a minor proportion of the markets for certificate- or diploma-holders.

Consider these other markets for whom actual skill levels are a distant, if even, a second consideration to applicant grade-point average:

a. Colleges, public, private or commercial, who have external, e.g. federal, or foundational, funds available for applicants with a certain grade-point average -- especially if the recipient institutions have tight budgets;

b. Institutions legally required to have certified staff but faced with employee scarcities, e.g. hospitals, clinics, civil-service;

c. School districts needing both adults certified as teachers, and students with birth and health certificates, in order to be run at even somewhat remove from peak efficiency;

d. Government administrations pursuing certain public policy initiatives that depend on items a, b and c, preceding; e.g. special education, affirmative action, STEM (Science, Technology & Mathematics); and last but not least,

e. the children applicants, legacies, to colleges which favor (paying) parents who are past graduates.

If skills really counted, there would be something like board standard examinations to be passed; normally, to be retaken at standard intervals. Teacher grades would not be accepted in place of board exam results. (See The Dangers of Diplomas)

But where the sheepskin alone is most important, rarely will the sheep’s diet be.


For examples and to pursue the issues raised in this essay, see

1. “Data-Driven”: a slogan to distract from organizational disagreement?;

2. Classification Error in Evaluation Practice:
the impact of the "false positive" on educational practice and policy

Cordially
--- EGR