# When is a test "biased"?

Data Colada recently posted a comment (written by Uri Simonsohn) on a supposed ‘bias’ lurking a common default Bayesian alternative to the t-test.  Point #1 was that the Bayesian T-test is ‘biased’ against low power results. Several smart people have made sophisticated and pretty sensible critiques of Simonsohn’s arguments, which I won’t rehash here. Instead, I want to point out the obvious problem: the claim of ‘bias’ is based on choosing the sample size in a way that is, well, biased.

Here’s the central graph under contention.

Now, there are several things to notice. First, not only does the probability that the results support the null increase as the effect size goes down, the probability of the results supporting the alternative go down as well. This is barely noticeable in this graph, but that’s just a function of the particular power level the author chose.  With a power of 0.85 (which is a better estimate of what a person ought to be doing in their experiment anyway), the size of these changes is reversed; the probability of supporting the alternative decreases 5 points from .73 to .68, while the increase in the probability of supporting the null increases just 4 points, from .005 to .042.  I got these numbers using Simonsohn’s publicly available R code).  So it seems that only really low power tests are ‘biased’ against small effect sizes?

But this brings up the deeper question: why make this comparison at all? In particular, why use the notion of ‘power’ to set your sample size?   ‘Power’ is supposed to control for type II errors in a t test.  Why should it have any particular privileged status in setting the size of an experiment?  The answer is that it shouldn’t.  These two concepts have been juxtaposed here without any clear justification.  One could just  as easily choose sample sizes to keep a constant probability that the default test favors the null.  Then one would find that the t-test is ‘biased’ against large effect sizes, in that the power would be higher for small effects.

I might be missing something, but I don’t see what. You have different ways of using models to estimate the parameters for an experiment to control the probabilities of certain kinds of errors.  Parameters that control for one type of error won’t necessarily control for another, but it’s rather extreme to call this a ‘bias’.

I’ve been reading Matt Crawford’s fascinating op-ed in the New York Times, and thought I’d write a quick response.  I should mention that I know Matt, and think of him as a friend; I’ve collaborated on projects with and am a former colleague of Beth Crawford, who is married to Matt. With that stipulation, here’s my initial response to the article.  What I love here is the economic and adversarial analysis of advertising–The idea that a war is being waged for our attention, and that as individuals we need to fight back. This comes out in language like “a straightforward conflict between me an L’Oréal”.

It’s fascinating to think of attention in terms of combat—a war waged across our public spaces.  However, there is an alternative viewpoint Matt also raises, which I ultimately find more compelling. This is that our attention is used transactionally, as fungible payment for goods and services.  On this view, the war between between “us” and “the companies” is no different than it has been, just fought over different resources. The obvious example  motivating this perspective is the way we pay for phone apps.  Many, many apps now have a ‘paid’ version, and a ‘free’ version. Though the language is misleading, most of us seem to understand that we are paying either way—we just pay with our attention and vigilance, or we pay with cash up front. Depending on our needs, beliefs, and interests, we choose how to pay for our goods. In this, we ware clear winners: we get the choice to pay for a service, but we get choice over what currency we use.  In many cases, I prefer paying in attention. It might seem that there is a difference between this example and the airport, but as Crawford points out, there really isn’t: at the airport as in the app store, we can pay for silence.

The risks of ubiquitous advertising have been actively discussed at least since billboards were a great new idea.  The idea that attention is a resource is already an idea actively in play by parties on both sides of this culture war.  Has anything really changed lately, or has Crawford simply noticed what’s been going on for fifty years?  I’m looking forward to reading the book to see (I hope) a more careful analysis of the relevant components of payment, of attention, and of social structure that underlies the emotions of frustration, stress, and anger which are foremost in the times piece. It seems to me that a great contribution would be the thinking through of this economic character of this transaction, where we sell our attention for goods and services.  Do we get something worth what we pay? Can we trust people to use good judgment in making these sorts of transactions? What kinds of laws or other institutional safeguards are important for understanding the interests of companies and

Instead answering these sorts of questions, a large portion of the op-ed is occupied tallying up supposed losses to society caused by this battle for our attentional space. I found this tallying less persuasive. For instance, among the losses Crawford claims the sealing off of public spaces—the massive reduction in the capacity for spontaneous conversations and interactions in airports, train stations, shopping malls, and other places the public congregates.  For my part, while I belief that such a sealing off has indeed occurred, I am skeptical of the link to advertising or other claims on our attention.  I find that when a person sits next to me in an airport or on a plane, unless I’m in an unusual mood I quickly to the very same thing that they do—plug in earphones. I often don’t even turn anything on—the earphones simply indicate my personal preference for privacy over the intrusion of a ‘spontaneous interaction’.  Sometimes people-watching is fun; sometimes conversations are rewarding (I am currently reading a book that was recommended to me by a stranger on an airplane); but mostly, we as a society seem to prefer to avoid these chance encounters and awkward interactions.

Looking around right now, in the Washington Dulles airport, I am in a nearly silent place—no ads, no televisions, no one talking. I didn’t have to pay for this space–I just found it. I travel about 50 times a year, leading to around 200 flights.  I have found that nearly every airport I travel in has free silent spaces, where the people who prefer quiet tend to congregate, but which are generally uncrowded. One has to seek these space out, but they are always there—unmarked, except by the absence of frantic people and televisions.  Often they move with the traffic, but the same few places regularly become eddies in the flow of people, and so if one knows where to look one can quickly get to privacy. The simple fact that such silence exists raises a small point and a bigger point. The small point is that one can pay for experiences one wants in any of three currencies: money, attention, or time. The bigger point is evident in the activities people engage in in these spaces. At this moment, I see 16 people sharing my pseudo-private space.  14 of them (including one Amish gentleman) are silently consulting their phones or laptops. Reading, texting, playing games, reading, working.  Different people are doing different things. One person is staring at the other people blearily, perhaps wishing she was asleep.  The one thing no one is doing?  Chatting up the people around them, initiating spontaneous and unwanted intrusions into each others private lives.

So: Crawford has one intuition, and I have another. How do we resolve this difference?  Unfortunately, the analytical tools on offer do not allow us to track down mechanisms with any confidence. I have no idea which of us is right about why our culture has shifted away from spontaneous discourse (or vouyerism) in public spaces, just as I have no idea whether this is a good or bad thing from the perspective of the broader cultural goals Crawford implicates in the debate.  What I do think is clear is that these issues are rapidly changing and of central importance. Crawford provides value raising them.  I just wish they were raised with more of a focus on rigorous analysis over anecdote, and on a clear articulation of the public goods rather than emotional responses may not be shared. At his worst moments, Crawford can come off as a cranky curmudgeon, annoyed with ‘kids these days’ and sure that every new direction is a wrong turn.  At his best moments, he sees with clear, fresh eyes solutions to problems you didn’t know you had, and makes clear the dangers of cultural choices that seemed entirely safe, or even obvious. My belief and expectation is that book itself will provide a deeper analysis of the economic tradeoffs involved, and (maybe more so) invoke a larger and more profound conceptual framework for understanding the personal and social implications of an increasingly transactional approach to attention, outside of the ‘corporations vs. the little guy’ framing presented in the article.

# The ignorance of the Ipsos Mori Ignorance Index

A number of news outlets and otherwise sophisticated blogs are reporting on the Ipsos Mori "Index of Ignorance", where the polling group purports to gather data about how people misunderstand and ‘overestimate’ proportions of various values in their societies, such as the percentage of Muslims, or the rate of teen pregnancy.  For instance, British people estimated that 21% of the population of Britain is Muslim, where the actual figure is about 5%.  The poll goes so far as to rank these countries based on their ‘ignorance’.

There is a news story here, but it is the exact opposite of the reported one: people are shockingly good at estimating the proportion of events in their countries. I guess I can’t blame Ipsos for getting it so wrong, because in order to see what’s going on, you need a basic understanding of psychophysics.