“There is a difference between numbers and numbers that matter. This is what separates data from metrics.”-- Jeff Bladt and Bob Filbin (3/4/1) Know the Difference Between Your Data and Your MetricsAssumptions in using data. A number of assumptions are necessary to turn “raw data” into something useable. What must be developed, if not already on hand, is a workable consensus on Terminology (concepts), Goals, and Methods of implementation. (For short, let us call this TGM-consensus.)
Whether or not TGM-consensus can be developed is highly sensitive to the kind of organization one is dealing with. In many, not all, businesses there is TGM-consensus. The proof is in the pudding, as the old saying goes: do they consistently -- not necessarily always -- produce a saleable product? If so, there is TGM-consensus.
In politics, religion, the “soft-sciences” and all levels of education[1], different agendas compete: widespread TGM-consensus is often lacking. Again, the proof is in the pudding: is there is persistent debate as to what the pudding should look like, the ingredients needed, the production procedures, and the evaluation methods? Then TGM-consensus is likely rare. And, neither vociferous admonitions, nor seductive pleas to become “data-driven” will make up for these lacks.
Much current promotion emphasizing “data-driven” approaches are little more than ploys to get worried persons to adopt prepackaged TGM-programs without critical pre-evaluation. Such pre-packaged TGM-programs, unless carefully examined for congruence with the TGM-environment to which it will be applied, will likely result -- to judge from past examples -- in a ritual charade of evaluation. (See Causal Charades: organizational rituals of evaluation)
Dealing with “Raw” Data. Here is some “raw” data.
What can you do with it (them)? What do you have to know about it (them)? Suppose I let you know that it (they) is (are) “data” gotten trying to measure someone’s physical development? What other information do you need?
Would it matter, if instead of “physical development” I had written “moral development?” Of course it would. The “data” inasmuch as it is relevant data -- some of the symbols might be mere construct effects of the instrumentation -- do not resolve by themselves any questions about development. [2]
A More Everyday Example. A clear example can be given as follows: suppose we have two persons standing together at normal speaking distance, facing each other. Call them Harry and John. Some noise issues from Harry. Consider the following possible descriptions of Harry's behavior:
a. Harry emitted the sound-sequence: /2aym+ gowing+3 hówm1 /.We can easily imagine a situation where all of these descriptions are true of what Harry is doing. But given a, -- which is the "physical" data of Harry's behavior in b, c, d and e -- neither b nor c nor d nor e need be true. What supporting information do we need to draw any inferences from the data? Here are some possibilities: [3]
b. Harry said, "I'm going home."
c. Harry told John he was going home.
d. Harry informed John that he was going home.
e. Harry surprised John with the statement that he was going home.
1. If Harry is a babbling idiot, a might be true and none of the rest.Data is the mere tip of an “iceberg.” If you want to trust data, you have to trust a lot more: parts which are usually submerged; and, often hard to fathom.
2. If Harry is reciting aloud a line from a script, a and b might be true and none of the rest.
3. If John already knew that Harry was going home, a, b, and c might be true but none of the rest.
4. If John is never surprised by anything Harry does, but did not already know he was going home, a, b, c, and d but not e might be true.
Why is data often ignored? What may be perplexing is that even in organizations which have long traditions of TGM-consensus, possibly very relevant data is paid little attention to. Why might that be?
Ask , perhaps cautiously, “Qui Bono?” Who stands to benefit, who stands to risk loss, if the data are paid attention to? One reason for the exaggerated drum-beating for “data-driven” undertakings is distraction. The organization’s present TGM-consensus may be either decrepit, or fail to address the burning issues. So, an emphasis on likely irrelevant data draws attention away from deeper disagreement about either terminology, or goals or method. [4] Addressing TGM issues would upset someone’s applecart.
The most vociferous advocates for improvement may be those most wanting to stifle it.
To examine these issues further, see Moral Education: Indoctrination vs. Cognitive Development?)
Cordially
--- EGR
NOTES
[1] See Charades of Evaluation: mis-connecting cause and effect
[2] To see how this code-string can be used as data see the article reference given at the end of the essay, Moral Education: Indoctrination vs. Cognitive Development?
[3] See Measurability and Educational Concerns
[4] Quantification is appealed to especially if it is believed to promote one’s agenda in discussion and investigation. However, Deborah Stone offers some telling criticisms about this practice especially as it is used to squelch open discussion. See for example, "Using Quantitative Procedures Wisely".
