Data be damned

When will you learn that there isn’t a word for everything?
– Nicole Krauss, The History of Love

Consider the following. In his book, Getting It Wrong From the Beginning, author Kieran Egan tells of a study in education, conducted in the 1990s, that concluded the following: “To develop competence in an area of inquiry, students must (a) have a deep foundation of factual knowledge, (b) understand facts and ideas in the context of a conceptual framework, and (c) organize knowledge in ways that facilitate retrieval and application”. (168-9) Sound familiar? Does anything about those findings bother you?

Egan
For the record, Egan has been extremely valuable to me as I have tried to articulate the many problems I have had with reform in education as I have seen it practiced both throughout my career in the classroom and while I was a member of the provincial government. In both instances, I watched enormous resources brought to bear in ways that have, to my mind, done little more than further complicate a situation that was exceedingly complicated to begin with.

So what about the study Egan notes? If you are lost within the language employed in reporting the results, wondering what the problem is, or simply curious, consider the following: what does it mean to be “competent in an area of inquiry”, maybe botany, let’s say? I admit I don’t know much about the subject but someone judged competent would, I think, need to know many things about plants since botany, by definition, is the science of plant life. A working knowledge of biology would seem essential as well since botany is a more specialized pursuit within the larger field. Finally, if someone were judged competent, I think we would need to see proof of some kind, whether through good advice on how to grow my geranium or an explanation, maybe, of why some plants need more sunlight than others.

Botanist
I’ve tried to abbreviate an example that Egan takes far more time explaining but I’m hoping my version will suffice. The study I’ve noted represents just one brand of educational research with which I have a problem. If you take a look at the “conclusions” reached through the research, the “findings” could quite properly appear in a dictionary defining “competence in an area of inquiry”. In other words, a study was undertaken to prove that competence is properly defined as competence. While I cannot point to a specific “study”, I’m pretty sure there are any number of them out there that have spent enormous sums of money and considerable time proving that young people who are read to frequently and who are raised in a home where a great deal of reading material is available are more likely to be good readers.

The fourth chapter concerns the teacher and the classroom and notes that an indisputable conclusion of research is that the quality of teaching makes a considerable difference in children’s learning.
– from the abstract for “Becoming a Nation of Readers: The Report of the Commission on Reading” (1985)

If any or all of these examples strike you as absurd, welcome to one manifestation of educational research: that which purports to lend the weight of scientific validation to an already established definition or a commonly understood reality. The fact that someone perceives a need to provide such a thing is as frustrating to me as the fact that it can gain acceptance and validation among those commonly tasked with making decisions that have an impact on so many facets of our public schools. The determination to forego any comment on educational practice unless “data” accompanies the comment leads naturally to my next point.

If, like me, you have spent any time trying to offer alternatives to certain practices and/or trends in education these days, you have, no doubt, been questioned regarding data. The thing is, I don’t really mind the question; rather, my issue is how one defines “data”.

Some 7 or 8 blogs ago I started in on this issue and I warned any potential readers that this was going to be long. Now, however, I can circle back to a point I tried to make early in the argument: avenues to understanding and to action should not be restricted to those things bounded by a model of research that is either ineffective or inapplicable. Education has been subjected to a bevy of research that imagines “learning” can be understood in much the way a physicist might determine the conditions under which water can be brought to a boil. Can it?

boiling water
In the latter example, the scientist can do something that, to my mind, is impossible in education: control the variables. Even then, the identification of the variables can be more complicated than we might assume: what vessel is being used to hold the water? At what atmospheric pressure is the experiment being conducted? What is the temperature in the surrounding environment? My guess is, a proper physicist might have a few more questions that I wouldn’t think of.

In the educational example, how can we even begin to identify the variables? The scientific method demands that a hypothesis, followed by observation, lead to some statement regarding the truth or falsity of the hypothesis. In social science research, rather than an absolute proof, trends and/or possibilities might be the result. The problem remains the same, however: supportable results, within the necessarily limiting confines of traditional scientific research methods, demand a control of variables. In something as complex as a classroom, how could that ever be deemed possible?

One technique is to narrow the focus. Consider: a special class is established to help students who did poorly on a standardized exam, specifically to prepare them for the next administration of that exam so that they might do better. Lo and behold, a year later, students rewriting the exam do better. The conclusion? The intervention (creation of this special class) worked.

statistics
I won’t even try to list the many factors which might have had an impact on the improved results. Suffice to say that, to my mind, claiming that the improved results are directly attributable to the class that was established is roughly equivalent to suggesting that a certain team won the Stanley Cup because the guy who scored the crucial goal just got married and that made him a better goal scorer.

In the quotation I provided at the outset, a character in Nicole Krauss’s novel questions whether we can really have a word for everything. Are there elements of human experience that are truly inexpressible? Personally, I think there are but that’s a topic for another time, perhaps. I chose the line because it hints at another problem: what if we give a name to something without being entirely able to say exactly what we intend, even when we have a definition? “Education” is just such a word.

The devil’s in the data

Data is not information, information is not knowledge, knowledge is not understanding, understanding is not wisdom.
– Clifford Stoll

Data 3
I feel the need to do a bit of a recap at this point, just because the argument I’m trying to make demands so much background. First, I contend that society’s embracing of science as the truest measure of the truth or falsity of something has had unintended negative consequences. Overall, I am a great believer in the power of science to illuminate a great many dark corners in human understanding. That being said, I do not believe it is the lone measure or revealer of truth and/or understanding. It may seem old-fashioned to some, but I believe in intuition, personal experience, imagination, insight, judgement et al as alternative – and sometimes superior – avenues to understanding, as well as sufficient foundations upon which to consider certain actions based on any one, or a combination, of the same. In other words, actions can be considered reasonable and necessary sometimes, even if we don’t have “empirical data” (assuming all things can be reduced to the empirical) to support them. Subjective analysis can sometimes trump the objective.

Second, the overarching confidence we have come to have in scientific and technological innovation to promote “improvement” in society and in our individual lives has morphed into an ideology without our noticing. The orthodoxy of this “techno-faith” allows the adherents to dismiss any assertion or comment that is not supported by “research” or “data”. To clarify, I’m not suggesting that we dispense with either; good research is what leads to advance and innovation, and data, when valid, can serve us well. Unfortunately, in our simplistic encounters with these things, objections need be little more than tag lines to shut down the opposition. If you happen to be a teacher these days and you’ve had questions about certain programs or policies you’ve probably had the experience of raising your concerns with one of the theorists or “specialists” only to be greeted with a sentence that begins: “but the research shows . . .”

Which leads nicely to my third point, namely, that those who appeal to the research are asserting, through such an exclamation, their role as “experts’. It wouldn’t matter if the person asking the question had thirty or more years as a classroom teacher upon which to base a quibble, question or challenge; if the research says something else is true, personal experience cannot offer an alternative interpretation, nor can it qualify data-driven claims. The empirical result trumps all and it is to that “empiricism” I now wish to turn.

quanta
Scientific “truth” achieves such status when the outcome of a particular observation proves to be repeatable, regardless of who is doing the observing. The core concept at work here is the scientific method, which says, at its most basic: hypothesize, test, conclude. If we are dealing with the physical world, regardless of size, I have no quibble. If physicists tell me that everything in the universe is built on something called quanta, I’ll take their word for it. When educational theorists, on the other hand, tell me they have discovered the solution to a given educational dilemma and, further, that they have the research to support their claims, it takes significant will power not to immediately snort with derision (a battle I’ve been known to lose).

Key to the validity of any scientific experiment is control of the possible factors that could affect whatever is being tested. If you’ve seen the movie “The Fly”, you have an exaggerated example of what can happen when something unaccounted for ends up in the mix. If you don’t know the film, the scientist is seeking to teleport from location A to B. A fly manages to enter the teleportation chamber and, in the process of being dismantled and then reassembled, fly and human DNA end up mingling. The result, of course, is horrifying, to say the least. An exaggeration, yes, but still a useful illustration of the importance of accounting for all pertinent factors. Such accounting has always been recognized as problematic in the hard sciences but, even more so, in the soft sciences, what we all know as the social sciences.

Data 1
Most people who have attended university probably took some kind of introductory course in psychology or, maybe, sociology. If so, you would have learned about probabilities, trends and the like. History, once the bastion of irrefutable truth in the minds of many, has been recast as “historiography” in many instances, so as to acknowledge the interpretative element at work in any accounting of the past. “Research” in the social science context, does not pretend to provide absolutes; it works to continually refine understanding even as, at some level, it would argue that, when dealing with something as complex as human beings and human society, definitive and unassailable conclusions are pretty hard to come by, impossible to have, in fact.

Still, such subtlety is the preserve, for the most part, of the university and academic journals. In popular culture, to a greater or lesser degree, we have all become experts in psychology of some sort. Who has not spoken of – or been privy to someone else speaking of – what is or isn’t true when it comes to men and women? Each of us, at some time or other, I’m guessing, has made a statement something along the lines of: “oh, you know men/women; they’re just like that (whatever ‘that’ might be at that moment)”. I can’t blame anyone for feeling expert in this regard (we all have a lifetime’s experience of being one or the other, after all) but few would imagine, at the same time, that they are expert psychologists, in a developed sense.

data 2
Education presents a very special problem in the world of research and data. While social scientists may feel it necessary to qualify both expectations and results, they can still be fairly clear on just what it is they are examining: how do people react to certain stimulus alone? While in crowds? With friends? Strangers? But what happens when an entire research industry grows up around something that cannot easily be defined? As Hamlet says: ”Aye, there’s the rub.” (to be continued)