Science, culture, complexity

Tag: ignorance

  • A little ignorance can be a good thing

    Picture a city where most drivers use the same navigation app. At 9 am, the app says one side street is the quickest shortcut to get from area A to area B. Thousands of commuters accept this option and drive through that street, and soon it becomes jammed with traffic. As a result, the route that was until a few minutes ago a shortcut on the app becomes undesirable, even as nearby streets remain underused.

    Now imagine a small change. Say some drivers don’t fully trust the label “fast”, or they see slightly different estimates, so they spread out across several streets. As a result no single street bears the entire traffic load, and in turn the average travel time (across all commuters) falls even though no road has been widened and no signal has been reconfigured.

    According to a preprint paper uploaded to arXiv in March, this limited ignorance on the drivers’ part effectively reduced congestion created by completely informed yet selfish choices.

    This view of the traffic should be reminiscent of any public space you’ve navigated in India, where seemingly small selfish choices add up to inconveniences that affect many people. Motorists spill into junctions when the signal is on red, with motorcyclists in particular clogging the sidewalk and even blocking the path of oncoming traffic. Commuters rush at bus and train doors rather than queuing in, thus slowing the boarding process as well as rendering existing queues useless. Families often reserve extra seats or rush baggage carousels and stall others. Each act yields a momentary private gain but together they congest and frustrate.

    Curiously, the new study posits that if people who route themselves through a crowded network know a little less, the whole system might work a little better.

    Prior work has already shown that when every user pursues the quickest route for themselves, the system settles into a state that isn’t globally optimal. The study began by quantifying this shortfall with a “price of anarchy”. The authors — all from Ohio State University — also defined a parallel idea that they called the “price of ignorance”, which tracked how outcomes change when users are uncertain about the network’s links.

    Next, the model mixed two kinds of links. “Slow” links had a fixed travel time that didn’t change with traffic. “Fast” links were faster when they were underused but slowed down in a linear fashion as more users pile on. The study’s model then assumed that users didn’t know for sure which kind of link they were facing. Instead, the model continued, they planned their route using a perceived cost that blended the two possibilities. The authors ‘measured’ the users’ ignorance using a single parameter denoted ɑ. If ɑ = 0, the users had no ignorance and complete knowledge; if ɑ = 1, the users were completely ignorant. The more the users were ignorant, the more they’d think all routes are equally suitable.

    The network took the form of a directed square lattice; ‘directed’ means users could only move through it in a fixed way, from left to right. Each link between two points in the lattice was said to be “fast” with probability p and “slow” with probability 1 – p. Users chose those routes that they believed would minimise travel time. The authors evaluated the true average travel time and compared it to the ɑ = 0, and defined the “price of ignorance” as the ratio of the average travel time with and without ignorance.

    The main finding was stark: the authors found that a small amount of ignorance always helped. For all compositions of the network, increasing ignorance from 0 reduced the average travel time up to a particular threshold. The paper proved analytically that for every probability of a lane being “fast” (i.e. for all values of p), any ɑ ≤ 2/3 guaranteed the price of ignorance was PI ≤ 1. That is to say, a limited ‘amount’ of ignorance softened the traffic by redirecting it into more tempting “fast” links. In this scenario, because some users overestimated the “slow” links and “underestimated” the fast ones, the traffic spread more evenly across the network. This reduced congestion on the fast links — which is good.

    The team also found a special sweet spot, so to speak. Say the network has a tipping point between “not enough fast links” and “plenty of fast links”. Near this tipping point, if users’ ignorance is around ⅔, their self-chosen routes spread out in just the right way. The result matches the best possible routing that a planner might pick. Put another way: imagine some links are marked “fast” and others are marked “slow”. If everyone fully trusted the labels, most people would chase the same link and eventually clog it. But if people trusted the labels only partly (as implied by ɑ = ⅔), some would choose alternative links nearby. This small hesitation thus spreads traffic across several routes.

    Alas, ignorance beyond the helpful range eventually starts to hurt. When people have no idea which links are fast or slow, they spread out almost evenly. That sounds fair but it also ‘wastes’ good options. Even in cities with many quicker routes, some travellers drift to slower side streets or paths that fizzle out, so the average travel time rises. This waste grows further in larger networks. Thus there is a sort of separatrix between “helpful doubt” and “harmful cluelessness”. If fast links are scarce, a planner can tolerate more doubt before the network’s performance drops. If fast routes are plentiful, on the other hand, only near-total cluelessness can cause harm. That is, in very large and well-served networks, things go bad only when people are almost completely in the dark.

    So to be clear, ignorance in the study didn’t mean carelessness or lack of effort. Users still knew the map, could see congestion, and chose the route that looked best to them. What they didn’t know for sure was which roads were actually quicker and which were actually slower. And this specific kind of ignorance had two virtues: first, it prevented users from overreacting to “fast” labels and kept a small subset of links from being overloaded; second, users’ ignorance caused different users to make different routing decisions even when some links looked attractive, thus keeping them from clogging these links in a coordinated way.

    These virtues might sound familiar if you’ve been to other parts of physics. In statistical physics, for example, adding noise to a weak signal can help it cross a valuable threshold, a phenomenon called stochastic resonance. If the amount of this noise is just right, it can improve the system’s response. A familiar example is in hearing assistance. Some hearing aids add a soft, random ‘hiss’ under speech. On its own, this hiss is too weak to notice — but for listeners with mild hearing loss, it helps small sound cues, like faint consonants in Hindi, cross the ear’s detection threshold more reliably. Thus speech becomes somewhat more clear at lower volumes or in quiet rooms.

    In ecology and evolution, some seeds have been known to germinate over multiple seasons because they’re not sure which ones are going to be bad. Similarly, with algorithms and machine learning, a bit of uncertainty can make models work better. During training, a program can turn off some parts of the network at random, so a model doesn’t simply memorise patterns from the data. Small, carefully added noise in the training labels can have a similar effect. In reinforcement learning, letting the program try some actions at random can help keep it from getting stuck on a strategy that looks good early but isn’t actually the best.

    And in behavioural game theory, people don’t always pick the mathematically ‘best’ move. They just pick a pretty good move most of the time. This can help alleviate crowding because as a result not everyone chases the same option at once. A similar idea can help in clinics as well: if a sign or app always says “counter 3 is fastest”, for example, everyone might rush there and block it. If instead the app randomly assigned people to different counters, everyone isn’t steered to the same counter.

    The overall lesson isn’t that ignorance is good in itself but that perfect certainty can produce brittle, crowded choices in systems with congestion or competition. A carefully controlled amount of uncertainty can instead spread the load and pull the system as a whole from a state governed by selfish dynamics to one by social optima.

  • In defence of ignorance

    Wish I may, wish I might
    Have this wish, I wish tonight
    I want that star, I want it now
    I want it all and I don’t care how

    Metallica, King Nothing

    I’m a news editor who frequently uses Twitter to find new stories to work on or follow up. Since the lockdown began, however, I’ve been harbouring a fair amount of FOMO born, ironically, from the fact that the small pool of in-house reporters and the larger pool of freelancers I have access to are all confined to their homes, and there’s much less opportunity than usual to step out, track down leads and assimilate ground reports. And Twitter – the steady stream of new information from different sources – has simply accentuated this feeling, instead of ameliorating it by indicating that other publications are covering what I’m not. No, Twitter makes me feel like I want it all.

    I’m sure this sensation is the non-straightforward product of human psychology and how social media companies have developed algorithms to take advantage of it, but I’m fairly certain (despite the absence of a personal memory to corroborate this opinion) that individual minds of the pre-social-media era weren’t marked by FOMO, and more certain that they were marked less so. I also believe one of the foremost offshoots of the prevalence of such FOMO is the idea that one can be expected to have an opinion on everything.

    FOMO – the ‘fear of missing out’ – is essentially defined by a desire to participate in activities that, sometimes, we really needn’t participate in, but we think we need to simply by dint of knowing about those activities. Almost as if the brains of humans had become habituated to making decisions about social participation based solely on whether or not we knew of them, which if you ask me wouldn’t be such a bad hypothesis to apply to the pre-information era, when you found out about a party only if you were the intended recipient of the message that ‘there is a party’.

    However, most of us today are not the intended recipients of lots of information. This seems especially great for news but it also continuously undermines our ability to stay in control of what we know or, more importantly, don’t know. And when you know, you need to participate. As a result, I sometimes devolve into a semi-nervous wreck reading about the many great things other people are doing, and sharing their experiences on Twitter, and almost involuntarily develop a desire to do the same things. Now and then, I even sense the seedling of regret when I look at a story that another news outlet has published, but which I thought I knew about before but simply couldn’t pursue, aided ably by the negative reinforcement of the demands on me as a news editor.

    Recently, as an antidote to this tendency – and drawing upon my very successful, and quite popular, resistance to speaking Hindi simply because a misguided interlocutor presumes I know the language – I decided I would actively ignore something I’m expected to have an opinion on but there being otherwise no reason that I should. Such a public attitude exists, though it’s often unspoken, because FOMO has successfully replaced curiosity or even civic duty as the prime impetus to seek new information on the web. (Obviously, this has complicated implications, such as we see in the dichotomy of empowering more people to speak truth to power versus further tightening the definitions of ‘expert’ and ‘expertise’; I’m choosing to focus on the downsides here.)

    As a result, the world seems to be filled with gas-bags, some so bloated I wonder why they don’t just float up and fuck off. And I’ve learnt that the hardest part of the antidote is to utter the words that FOMO has rendered most difficult to say: “I don’t know”.

    A few days ago, I was chatting with The Soufflé when he invited me to participate in a discussion about The German Ideology that he was preparing for. You need to know that The Soufflé is a versatile being, a physicist as well as a pluripotent scholar, but more importantly The Soufflé knows what most pluripotent scholars don’t: that no matter how much one is naturally gifted to learn this or that, knowing something needs not just work but also proof of work. I refused The Soufflé’s invitation, of course; my words were almost reflexive, eager to set some distance between myself and the temptation to dabble in something just because it was there to dabble. The Soufflé replied,

    I think it was in a story by Borges, one of the characters says “Every man should be capable of all ideas, and I believe that in the future he will be.” 🙂

    To which I said,

    That was when the world was simpler. Now there’s a perverse expectation that everyone should have opinions on everything. I don’t like it, and sometimes I actively stay away from some things just to be able to say I don’t want to have an opinion on it. Historical materialism may or may not be one of those things, just saying.

    Please bear with me, this is leading up to something I’d like to include here. The Soufflé then said,

    I’m just in it for the sick burns. 😛 But OK, I get it. Why do you think that expectation exists, though? I mean, I see it too. Just curious.

    Here I set out my FOMO hypothesis. Then he said,

    I guess this is really a topic for a cultural critic, I’m just thinking out loud… but perhaps it is because ignorance no longer finds its antipode in understanding, but awareness? To be aware is to be engaged, to be ‘caught up’ is to be active. This kind of activity is low-investment, and its performance aided by social media?

    If you walked up to people today and asked “What do you think about factory-farmed poultry?” I’m pretty sure they’d find it hard to not mention that it’s cruel and wrong, even if they know squat about it. So they’re aware, they have possibly a progressive view on the issue as well, but there’s no substance underneath it.

    Bingo.

    We’ve become surrounded by socio-cultural forces that require us to know, know, know, often sans purpose or context. But ignorance today is not such a terrible thing. There are so many people who set out to know, know, know so many of the wrong ideas and lessons that conspiracy theories that once languished on the fringes of society have moved to the centre, and for hundreds of millions of people around the world stupid ideas have become part of political ideology.

    Then there are others who know but don’t understand – which is a vital difference, of the sort that The Soufflé pointed out, that noted scientist-philosophers have sensibly caricatured as the difference between the thing and the name of the thing. Knowing what the four laws of thermodynamics or the 100+ cognitive biases are called doesn’t mean you understand them – but it’s an extrapolation that social-media messaging’s mandated brevity often pushes us to make. Heck, I know of quite a few people who are entirely blind to this act of extrapolation, conflating the label with the thing itself and confidently penning articles for public consumption that betrays a deep ignorance (perhaps as a consequence of the Dunning-Kruger effect) of the subject matter – strong signals that they don’t know it in their bones but are simply bouncing off of it like light off the innards of a fractured crystal.

    I even suspect the importance and value of good reporting is lost on too many people because those people don’t understand what it takes to really know something (pardon the polemic). These are the corners the push to know more, all the time, often even coupled to capitalist drives to produce and consume, has backed us to. And to break free, we really need to embrace that old virtue that has been painted a vice: ignorance. Not the ignorance of conflation nor the ignorance of the lazy but the cultivated ignorance of those who recognise where knowledge ends and faff begins. Ignorance that’s the anti-thing of faff.

  • Two sides of the road and the gutter next to it

    I have a mid-October deadline for an essay so obviously when I started reading up on the topic this morning, I ended up on a different part of the web – where I found this: a piece by a journalist talking about the problems with displaying one’s biases. Its headline:

    It’s a straightforward statement until you start thinking about what bias is, and according to whom. On 99% of occasions when a speaker uses the word, she means it as a deviation from the view from nowhere. But the view from nowhere seldom exists. It’s almost always a view from somewhere even if many of us don’t care to acknowledge that, especially in stories where people are involved.

    It’s very easy to say Richard Feynman and Kary Mullis deserved to win their Nobel Prizes in 1965 and 1993, resp., and stake your claim to being objective, but the natural universe is little like the anthropological one. For example, it’s nearly impossible to separate your opinion of Feynman’s or Mullis’s greatness from your opinions about how they treated women, which leads to the question whether the prizes Feynman and Mullis won might have been awarded to others – perhaps to women who would’ve stayed in science if not for these men and made the discoveries they did.

    One way or another, we are all biased. Those of us who are journalists writing articles involving people and their peopleness are required to be aware of these biases and eliminate them according to the requirements of each story. Only those of us who are monks are getting rid of biases entirely (if at all).

    It’s important to note here that the Poynter article makes a simpler mistake. It narrates the story of two reporters: one, Omar Kelly, doubted an alleged rape victim’s story because the woman in question had reported the incident many months after it happened; the other, the author herself, didn’t express such biases publicly, allowing her to be approached by another victim (from a different incident) to have her allegations brought to a wider audience.

    Do you see the problem here? Doubting the victim or blaming the victim for what happened to her in the event of a sexual crime is not bias. It’s stupid and insensitive. Poynter’s headline should’ve been “Reporters who are stupid and insensitive fail their sources – and their profession”. The author of the piece further writes about Kelly:

    He took sides. He acted like a fan, not a journalist. He attacked the victim instead of seeking out the facts as a journalist should do.

    Doubting the victim is not a side; if it is, then seeking the facts would be a form of bias. It’s like saying a road has two sides: the road itself and the gutter next to it. Elevating unreason and treating it at par with reasonable positions on a common issue is what has brought large chunks of our entire industry to its current moment – when, for example, the New York Times looks at Trump and sees just another American president or when Swarajya looks at Surjit Bhalla and sees just another economist.

    Indeed, many people have demonised the idea of a bias by synonymising it with untenable positions better described (courteously) as ignorant. So when the moment comes for us to admit our biases, we become wary, maybe even feel ashamed, when in fact they are simply preferences that we engender as we go about our lives.

    Ultimately, if the expectation is that bias – as in its opposition to objectivity, a.k.a. the view from nowhere – shouldn’t exist, then the optimal course of action is to eliminate our specious preference for objectivity (different from factuality) itself, and replace it with honesty and a commitment to reason. I, for example, don’t blame people for their victimisation; I also subject an article exhorting agricultural workers to switch to organic farming to more scrutiny than I would an article about programmes to sensitise farmers about issues with pesticide overuse.