“Data-Driven”: a slogan to distract from organizational disagreement?

“There is a difference between numbers and numbers that matter. This is what separates data from metrics.”-- Jeff Bladt and Bob Filbin (3/4/1) Know the Difference Between Your Data and Your Metrics

Assumptions in using data. A number of assumptions are necessary to turn “raw data” into something useable. What must be developed, if not already on hand, is a workable consensus on Terminology (concepts), Goals, and Methods of implementation. (For short, let us call this TGM-consensus.)

A workable TGM-consensus is one with sufficient depth of agreement on all the items, TGM, to enable control of production. (See The Indeterminacy of Consensus: masking ambiguity and vagueness in decision)

Whether or not TGM-consensus can be developed is highly sensitive to the kind of organization one is dealing with. In many, not all, businesses there is TGM-consensus. The proof is in the pudding, as the old saying goes: do they consistently -- not necessarily always -- produce a saleable product? If so, there is TGM-consensus.

In politics, religion, the “soft-sciences” and all levels of education[1], different agendas compete: widespread TGM-consensus is often lacking. Again, the proof is in the pudding: is there is persistent debate as to what the pudding should look like, the ingredients needed, the production procedures, and the evaluation methods? Then TGM-consensus is likely rare. And, neither vociferous admonitions, nor seductive pleas to become “data-driven” will make up for these lacks.

Much current promotion emphasizing “data-driven” approaches are little more than ploys to get worried persons to adopt prepackaged TGM-programs without critical pre-evaluation. Such pre-packaged TGM-programs, unless carefully examined for congruence with the TGM-environment to which it will be applied, will likely result -- to judge from past examples -- in a ritual charade of evaluation. (See Causal Charades: organizational rituals of evaluation)

Dealing with “Raw” Data. Here is some “raw” data.

@@&@*$@@&&@@*$$*&@@$$*&*@*@@$&$*@*$@*$*&&*@*&*&$**
What can you do with it (them)? What do you have to know about it (them)? Suppose I let you know that it (they) is (are) “data” gotten trying to measure someone’s physical development? What other information do you need?
Would it matter, if instead of “physical development” I had written “moral development?” Of course it would. The “data” inasmuch as it is relevant data -- some of the symbols might be mere construct effects of the instrumentation -- do not resolve by themselves any questions about development. [2]

A More Everyday Example. A clear example can be given as follows: suppose we have two persons standing together at normal speaking distance, facing each other. Call them Harry and John. Some noise issues from Harry. Consider the following possible descriptions of Harry's behavior:

a. Harry emitted the sound-sequence: /2aym+ gowing+3 hówm1 /.

b. Harry said, "I'm going home."

c. Harry told John he was going home.

d. Harry informed John that he was going home.

e. Harry surprised John with the statement that he was going home.

We can easily imagine a situation where all of these descriptions are true of what Harry is doing. But given a, -- which is the "physical" data of Harry's behavior in b, c, d and e -- neither b nor c nor d nor e need be true. What supporting information do we need to draw any inferences from the data? Here are some possibilities: [3]

1. If Harry is a babbling idiot, a might be true and none of the rest.

2. If Harry is reciting aloud a line from a script, a and b might be true and none of the rest.

3. If John already knew that Harry was going home, a, b, and c might be true but none of the rest.

4. If John is never surprised by anything Harry does, but did not already know he was going home, a, b, c, and d but not e might be true.

Data is the mere tip of an “iceberg.” If you want to trust data, you have to trust a lot more: parts which are usually submerged; and, often hard to fathom.

Why is data often ignored? What may be perplexing is that even in organizations which have long traditions of TGM-consensus, possibly very relevant data is paid little attention to. Why might that be?

Ask , perhaps cautiously, “Qui Bono?” Who stands to benefit, who stands to risk loss, if the data are paid attention to? One reason for the exaggerated drum-beating for “data-driven” undertakings is distraction. The organization’s present TGM-consensus may be either decrepit, or fail to address the burning issues. So, an emphasis on likely irrelevant data draws attention away from deeper disagreement about either terminology, or goals or method. [4] Addressing TGM issues would upset someone’s applecart.

The most vociferous advocates for improvement may be those most wanting to stifle it.

To examine these issues further, see Moral Education: Indoctrination vs. Cognitive Development?)

Cordially
--- EGR

NOTES
[1] See Charades of Evaluation: mis-connecting cause and effect

[2] To see how this code-string can be used as data see the article reference given at the end of the essay, Moral Education: Indoctrination vs. Cognitive Development?

[3] See Measurability and Educational Concerns

[4] Quantification is appealed to especially if it is believed to promote one’s agenda in discussion and investigation. However, Deborah Stone offers some telling criticisms about this practice especially as it is used to squelch open discussion. See for example, "Using Quantitative Procedures Wisely".

An Ancient "Ghost", Yet to be "Exorcised": Precision

Indicate precisely what you mean to say,
Yours sincerely wasting away.
-- Paul McCartney, "When I'm Sixty-Four"

A Comparison? An interesting picture is given by McCartney's lyric: there is

a. something that a person means to say; and, also, a second thing that is
b. an indicator of that something meant.

What is being asked for is precision of the indicator to the meaning. But how would we judge that?

If I ask you to tell me precisely how long your left foot is, you grab a rule and hold it alongside your left foot and make a comparison with your heel at the beginning of the number scale and the measurement mark most in line with the end of your toe. We know how to go about that.

But what kind of comparison can we make between what you mean to say and something offered as an indicator of it?

This is a problem very similar to what we encounter if we believe that when we read aloud a text, we are describing the thoughts of its author. What can we compare? Can we independently get at the author's thoughts to see if the words he chose are "fitting" to them? (See "An Ancient View of Authorship".)

Impossible Correspondences? This is actually an old chestnut in a different guise: the correspondence theory of knowledge. If we imagine that "knowing" indicates a correspondence between your thoughts and "objects in the real world" we face the problem how we can determine that that correspondence exists. And why would we ever want to "correct" ourselves?

But, actually, we have a very similar problem with our foot and the ruler. Some rulers are scored in full inches; others, in fractions of the inch. We are trained to think that using a finer scale gives us more precision. Sometimes it does. But consider this: what happens if the length of something varies depending on the fineness of the scale we use to measure it?

Things like that don't happen, you say? Wrong. Measure any natural coastline. Begin by indicating its length in miles but stepping a mile long ruler (do this on a map) from one end point to the next. If you use a 1/2-mile long scale the coastline measurement will INCREASE! Why? Because you will catch more variations in the coastline at a half-mile than at a mile.

Increase the fineness of the measurement scale and the length of the coast will increase even more as every little brook and stream and inlet gets marked by that scale. Theoretically, it would seem that every coastline expands to infinity as our scales become finer and finer -- except for the fact that as we approach molecular levels there would be variation, if not confusion, as to what would be the length of the coastline.

(You can check this out on Wikipedia at "How Long Is the Coast of Britain? Statistical Self-Similarity and Fractional Dimension")

Is This Proof of No Meaning, No Knowledge, No Length? Not at all. But it does show that the ideas of precision that were drummed into us as part of our basic education deserve a more critical review than we might think.

If you've ever wondered what the criticism meant when you were told that your speech, or your writing or your ideas were "imprecise," you were not exhibiting stupidity. What precision is, exactly, as regards speech, writing or thought, is seldom explained: either from lack of knowledge on the part of the critic; or, from the critic's desire to maintain a position of dominance over you -- no doubt for reasons of "moral discipline."

In much schooling, quantification is appealed to precisely because it seems to enhance "precision" in discussion and investigation. However, Deborah Stone offers some telling criticisms about this practice especially as it is used to support the domination of some people by others. See for example, "Using Quantitative Procedures Wisely".

Cordially,

--- EGR

Measuring Educational Outcomes: truth, tricks and hype

It sometimes seems that the only real result of math education is to convince its worst students – and a substantial number of us mediocre ones – that numbers are magic. “Anything can be measured,” goes the myth; and, once a number is associated with something, it becomes much more real, much more defined. All problems are soluble once they have become mathematized – even TV pushes this philosophy on the dramatic series, Numbers.

This hocus-pocus has some startingly impressive roots. Psychologist Edward L. Thorndike – unskilled in basic algebra -- wrote: "Whatever exists at all exists in some amount. To know it thoroughly involves knowing its quantity as well as its quality." Heavy philosophy!

But what is the quantity of a chess game? Or of the relation "<>"? Do chess games or "<>" not exist? Do chess masters wait with bated breath for someone to mathematize a chess game so that they can “know it thoroughly?”

The harm done with this exaggeration is that it creates the expectation that when and only when a technical, measurable solution to a problem can be devised, can that problem be solved.

So it is that problems must be left to mathematical geniuses, since their solution is beyond the ken of most people who can only stumble through math counting on their fingers.

To examine these issues further, see Measurability and Educational Concerns

Cordially

---EGR

Functional Analysis in Education: Science or Just Plain Old B.S.?

edited 040620

Inch worm, inch worm
Measuring the marigolds ...
--F Loesser (1952)

Instead of saying, “Johnny’s bad behavior comes from his parents’ being too strict,” we can put on the appearance of being highly learned, a scientist, even, by saying, “Johnny’s behavior is a function of the way his parents punish him.”

Rather than teachers’ being asked what might provoke disruptive behavior in their classes, they are instructed to look for “functional relations” in environmental variables, as though formulating it this way would make it any less speculative

Then there is Euler’s argument for the existence of God: "(a+ bⁿ)/n = x, therefore God exists". (Maybe God is supposed to equal x.)

Educational research abounds with formulas and statistics. Are they meant to be informative or merely impressive?

To examine these issues further, see The Functional Analysis of Informal Models

See, also, The Functional Analysis of Behavior

Cordially
EGR