The Importance of Validity

About three weeks ago I got into an argument about the scientific validity and pragmatic usefulness of MBTI. It suffices, for the purposes of this post, to just say that MBTI is a psychological personality test. Even though I have strong opinions about the measure itself, MBTI is not the issue here.

What is the issue is the relationship of scientific validity and practical usefulness of a concept or theory? Is there a correlation? To me it seems obvious there’s at least some correlation, but how much? And how much does it matter? What are valid theories better at – if anything? And can an invalid theory have practical uses? These are the kinds of questions that the debate highlighted for me. And since nowadays science seems to pop up everywhere, the importance of such questions should be as high as ever – or higher.

To start off, the first step should be scientific validity itself. What do we mean by that? Of course, this simple question in itself has been the subject of a raging debate for centuries. The likes of Popper, Kuhn etc. have all offered their take on the issue. Unfortunately space constraints prevent from discussing those points, and all I can do is offer my very humble opinion. As I see it, scientific method is just a method of questioning and testing taken to the extreme. What scientific validity means for me is that in the defined frame of reference, a theory gives more often than not the same results given the same circumstances. Frame of reference is an important limiting factor, as theories usually have a very limited scope. It makes no sense to have a quantum theory of the human mind, since the unit of analysis in the mind is not the quark, nor will it in the foreseeable future to be so (complexity is a limiting factor here). A theory that works in every situation is either wrong or computationally too demanding. For that reason, we must allow for a little laxity in accuracy. It would be foolish to allow no inaccuracies or mistakes, since then we would have to severely constrain the meaning of the term “science”. Very general or vague statements aside, no theories would be left after the Armageddon of mistake-cleansing.

Next step: why is validity important? Simply put, it seems to promise a lot in the way of consistency. A valid theory outputs the same results in the same situation. Therefore, a valid theory outputs either truths or falsities (in the majority of cases). An invalid theory, however, produces whatever results it happens to produce – there are only weak correlations with the parameters of the situation. How can we know, then, whether the output is true or false?

If validity were a binary variable this questioning would hardly be worth a post. Unfortunately, validity is a continuum. There’s no clear line between validity and its counterpart. It’s more a case of two distinct polar opposites – and a huge swathe of grey area in between. What to say of those theories and their validity?

The answer, it seems to me, hinges on what we can gain and lose from accepting a theory. Assuming it is valid – though not yet proven so – what are the gains? And in the opposite situation – if the theory turns out to be false – what are the losses? And here the title comes into play: it’s an issue about the economy of concepts.

If the single grey-area theory is all we have, the situation is the following: we are lost in the wild, and we have but a single map. We have never seen the map before, and have no idea whether it is accurate or not – or if it portrays the area we’re lost in. The question being – is an unfamiliar map better than no map at all? To this question, I believe, the answer is to take the map. After all, going without a map is a random walk, and unlikely to be successful. Even the crudest of maps, however, had something right. Some information trumps lack of it, even though we may follow the wrong path. But eventually, the map will lead us somewhere, and if we’re willing to update on the information – redraw the map – we’ll be better off than starting from nothing.

However, in the modern society there rarely is just one map. There are more often than not several competing theories which to choose from. In this situation we ought to choose the best map. And that is a question of validity (also of accuracy & reliability, but let’s combine all those academic terms into ‘validity’ for now). It is simply inefficient to choose the less valid theory. That will only result in fewer correct predictions – and if unlucky, more of really, really false ones(like phlogiston or elan vital). That is a risk I’m not willing to accept. In the world of theories, a tested case is better than a new bag of tricks.

P.S.
This whole discussion leaves non-theories, like thought experiments aside. This is deliberate. The space just won’t allow for that side. But I’ll be following that lead shortly, I hope.