Science, culture, complexity

Tag: deepfake

  • Spotting fakes by looking at them

    On March 10, the Supreme Court said a balance has to be struck between warding against misinformation online and protecting citizens’ right to free speech. The context was the Centre’s attempts to defend the 2023 IT Rules: when the comedian Kunal Kamra asked who would decide if online content is “fake or misleading”, the Centre said, “When we see it, we know it is fake”.

    The case has been led by Kamra, the Editors Guild of India, and other petitioners and in its course the Bombay High Court and the Supreme Court have been asked to weigh the constitutionality of a “Fact-Check Unit” (FCU) mandated by the national government. The petitioners have argued that giving the government the power to flag “fake, false or misleading” information will have a chilling effect on free speech and that the provisions turn the state into a judge in its own cause. The Centre’s defence — “known it when we see it” — is, as both history and data science show, a recipe for disaster.

    Solicitor general Tushar Mehta, who offered the defence on the Centre’s behalf, is likely to know that the line echoes a chaotic chapter in American legal history. In the 1964 case Jacobellis v. Ohio, US Supreme Court Justice Potter Stewart had to define obscenity. But frustrated by the lack of a precise legal definition, he famously wrote in his concurrence: “I shall not today attempt further to define the kinds of material I understand to be embraced within that shorthand description; and perhaps I could never succeed in intelligibly doing so. But I know it when I see it.”

    The legacy of this little statement was a big mess. If a Supreme Court Justice couldn’t articulate a clear standard, it was clearly folly to expect local police and juries to do so — more so since what passed for art in Manhattan could lead to a prison sentence in rural Georgia. The immediate result was different federal circuit courts applying different tests, creating a patchwork of legal outcomes across the country. Inevitably, the situation descended into the absurd: throughout the late 1960s, US Supreme Court Justices regularly screened films to determine if they were “obscene” in a projection room at the Court, literally deciding the law based on their own instincts and physiological reactions.

    By 1973, the Court realised this was unworkable. In Miller v. California, it established a three-part framework known since as the Miller test. It asked whether the average person, applying “contemporary community standards”, would find the work to be prurient; whether it depicts sexual conduct in a “patently offensive” way; and whether it lacks serious literary, artistic, political, or scientific value (a.k.a. the SLAPS test).

    But since even this framework lacked a universal standard, the potential for harm persisted. For instance, because “community standards” were local, federal prosecutors in the 1980s and 1990s began a practice called jurisdiction shopping: they would carefully prosecute distributors in the most conservative parts of the country for material that was actually sold nationwide. The practice then forced businesses to calibrate their content to the most restrictive local market in the country in order to avoid jail time — a sort of regression to the most conservative position.

    The “know it when I see it” heuristic ultimately became meaningless with the coming of the internet, which allowed content producers to be located in California even as their content is served in Alabama, thus confusing the notion of ‘community’ and the resulting community standards. Federal prosecutors were eventually forced to abandon most obscenity cases altogether and shift their focus to child exploitation, which is prohibited regardless of location or community.

    India’s proposed FCU threatens to play through this same history of failures. And it will begin as a patchwork of censorship that will depend on who’s looking at a screen when a certain clip is playing.

    But the fact is nobody has to know it just by seeing it. Data science and international regulations today offer testable ways to identify misinformation.

    One option is automated fact-checking that uses large databases of verified information. Instead of an official simply declaring a claim false, a system can check whether the statement connects to any documented policy decisions or records. If a viral post claims that “the government has banned P”, the system can scan policy documents, gazette notifications, and other reliable databases to check whether such a decision appears anywhere. If no record exists, the claim can be flagged and labelled as unsupported. The machine need not be all that intelligent as the bigger point here is to ensure the verdict can be traced to evidence available in the public domain.

    For instance, if a viral post claims “the government has banned P”, the algorithm will calculate the shortest ‘logical path’ between the nodes for “government”, “banned”, and “P” across all known policy documents. If no logical path exists, the system flags the information with a low truth-value score. This provides a quantifiable metric, moves the conversation from “I think this is fake” to “the data shows no factual connection for this claim”, and could even spare the people staffing censorship teams at social media companies considerable psychological harm. The government making the algorithm open-source — as it should be considering it will be in service of the public — will also add another layer of integrity.

    Another option is to look at the European Union’s Digital Services Act, which — instead of deciding whether every individual post is true or false — has regulators ask whether a stream of information poses a systemic risk to public health, security or democratic debate. Platforms are then required to monitor patterns like how quickly a claim spreads and whether coordinated networks of accounts (e.g. bots) are pushing the same message. So the focus here is not on the content of a single post but to examine the behaviour of the information as it moves through the network itself.

    The Centre’s current argument, however, ignores these tools and doubles down on a standard that has failed every time it has been applied, chiefly because it creates a legal landscape in which no one knows the rules until they have already broken them. Unless of course this is the Centre’s aim.

  • Why everyone should pay attention to Stable Diffusion

    Many of the people in my circles hadn’t heard of Stable Diffusion until I told them, and I was already two days late. Heralds of new technologies have a tendency to play up every new thing, however incremental, as the dawn of a new revolution – but in this case, their cries of wolf may be real for once.

    Stable Diffusion is an AI tool produced by Stability.ai with help from researchers at the Ludwig Maximilian University of Munich and the Large-scale AI Open Network (LAION). It accepts text or image prompts and converts them into artwork based on, but not necessarily understand, what it ‘sees’ in the input. It created the image below with my prompt “desk in the middle of the ocean vaporwave”. You can create your own here.

    But it strayed into gross territory with a different prompt: “beautiful person floating through a colourful nebula”.

    Stable Diffusion is like OpenAI’s DALL-E 1/2 and Google’s Imagen and Parti but with two crucial differences: it’s capable of image-to-image (img2img) generation as well and it’s open source.

    The img2img feature is particularly mind-blowing because it allows users to describe the scene using text and then guide the Stable Diffusion AI by using a little bit of their own art. Even a drawing on MS Paint with a few colours will do. And while OpenAI and Google hold their cards very close to their chests, with the latter even refusing to release Imagen or Parti in private betas, Stability.ai has – in keeping with its vision to democratise AI – opened Stable Diffusion for tinkering and augmentation by developers en masse. Even the ways in which Stable Diffusion has been released are important: trained developers can work directly with the code while untrained users can access the model in their browsers, without any code, and start producing images. In fact, you can download and run the underlying model on your system, requiring some slightly higher-end specs. Users have already created ways to plug it into photo-editing software like Photoshop.

    Stable Diffusion uses a diffusion model: a filter (essentially an algorithm) that takes noisy data and progressively de-noises it. In incredibly simple terms, researchers take an image and in a step-wise process add more and more noise to it. Next they feed this noisy image to the filter, which then removes the noise from the image in a similar step-wise process. You can think of the image as a signal, like the images you see on your TV, which receives broadcast signals from a transmitter located somewhere else. These broadcast signals are basically bundles of electromagnetic waves with information encoded into the waves’ properties, like their frequency, amplitude and phase. Sometimes the visuals aren’t clear because some other undesirable signal has become mixed up with the broadcast signal, leading to grainy images on your TV screen. This undesirable information is called noise.

    When the noise waveform resembles that of a bell curve, a.k.a. a Gaussian function, it’s called Gaussian noise. Now, if we know the manner in which noise has been added to the image in each step, we can figure out what the filter needs to do to de-noise the image. Every Gaussian function can be characterised by two parameters, the mean and the variance. Put another way, you can generate different bell-curve-shaped signals by changing the mean and the variance in each case. So the filter effectively only needs to figure out what the mean and the variance in the noise of the input image are, and once it does, it can start de-noising. That is, Stable Diffusion is (partly) the filter here. The input you provide is the noisy image. Its output is the de-noised image. So when you supply a text prompt and/or an accompanying ‘seed’ image, Stable Diffusion just shows off how well it has learnt to de-noise your inputs.

    Obviously, when millions of people use Stable Diffusion, the filter is going to be confronted with too many mean-variance combinations for it to be able to directly predict them. This is where an artificial neural network (ANN) helps. ANNs are data-processing systems set up to mimic the way neurons work in our brain, by combining different pieces of information and manipulating them according to their knowledge of older information. The team that built Stable Diffusion trained its model on 5.8 billion image-text pairs found around the internet. An ANN is then programmed to learn from this dataset as to how texts and images correlate as well as how images and images correlate.

    To keep this exercise from getting out of hand, each image and text input is broken down into certain components, and the machine is instructed to learn correlations only between these components. Further, the researchers used an ANN model called an autoencoder. Here, the ANN encodes the input in its own representation, using only the information that it has been taught to consider important. This intermediate is called the bottleneck layer. The network then decodes only the information present in this layer to produce the de-noised output. This way, the network also learns what about the input is most important. Finally, researchers also guide the ANN by attaching weights to different pieces of information: that is, the system is informed that some pieces are to be emphasised more than others, so that it acquires a ‘sense’ of less and more desirable.

    By snacking on all those text-image pairs, the ANN effectively acquires its own basis to decide when it’s presented a new bit of text and/or image what the mean and variance might be. Combine this with the filter and you get Stable Diffusion. (I should point out again that this is a very simple explanation and that parts of it may well be simplistic.)

    Stable Diffusion also comes with an NSFW filter built-in, a component called Safety Classifier, which will stop the model from producing an output that it deems harmful in some way. Will it suffice? Probably not, given the ingenuity of trolls, goblins and other bad-faith actors on the internet. More importantly, it can be turned off, meaning Stable Diffusion can be run without the Safety Classifier to produce deepfakes that are various degrees of disturbing.

    Recommended here: Deepfakes for all: Uncensored AI art model prompts ethics questions.

    But the problems with Stable Diffusion don’t lie only in the future, immediate or otherwise. As I mentioned earlier, to create the model, Stability.ai & co. fed their machine 5.8 billion text-image pairs scraped from the internet – without the consent of the people who created those texts and images. Because Stability.ai released Stable Diffusion in toto into the public domain, it has been experimented with by tens of thousands of people, at least, and developers have plugged it into a rapidly growing number of applications. This is to say that even if Stability.ai is forced to pull the software because it didn’t have the license to those text-image pairs, the cat is out of the bag. There’s no going back. A blog post by LAION only says that the pairs were publicly available and that models built on the dataset should thus be restricted to research. Do you think the creeps on 4chan care? Worse yet, the jobs of the very people who created those text-image pairs are now threatened by Stable Diffusion, which can – with some practice to get your prompts right – produce exactly what you need, no illustrator or photographer required.

    Recommended here: Stable Diffusion is a really big deal.

    The third interesting thing about Stable Diffusion, after its img2img feature + “deepfakes for all” promise and the questionable legality of its input data, is the license under which Stability.ai has released it. AI analyst Alberto Romero wrote that “a state-of-the-art AI model” like Stable Diffusion “available for everyone through a safety-centric open-source license is unheard of”. This is the CreativeML Open RAIL-M license. Its preamble says, “We believe in the intersection between open and responsible AI development; thus, this License aims to strike a balance between both in order to enable responsible open-science in the field of AI.” Attachment A of the license spells out the restrictions – that is, what you can’t do if you agree to use Stable Diffusion according to the terms of the license (quoted verbatim):

    “You agree not to use the Model or Derivatives of the Model:

    • In any way that violates any applicable national, federal, state, local or international law or regulation;
    • For the purpose of exploiting, harming or attempting to exploit or harm minors in any way;
    • To generate or disseminate verifiably false information and/or content with the purpose of harming others;
    • To generate or disseminate personal identifiable information that can be used to harm an individual;
    • To defame, disparage or otherwise harass others;
    • For fully automated decision making that adversely impacts an individual’s legal rights or otherwise creates or modifies a binding, enforceable obligation;
    • For any use intended to or which has the effect of discriminating against or harming individuals or groups based on online or offline social behavior or known or predicted personal or personality characteristics;
    • To exploit any of the vulnerabilities of a specific group of persons based on their age, social, physical or mental characteristics, in order to materially distort the behavior of a person pertaining to that group in a manner that causes or is likely to cause that person or another person physical or psychological harm;
    • For any use intended to or which has the effect of discriminating against individuals or groups based on legally protected characteristics or categories;
    • To provide medical advice and medical results interpretation;
    • To generate or disseminate information for the purpose to be used for administration of justice, law enforcement, immigration or asylum processes, such as predicting an individual will commit fraud/crime commitment (e.g. by text profiling, drawing causal relationships between assertions made in documents, indiscriminate and arbitrarily-targeted use).”

    As a result of these restrictions, law enforcement around the world has incurred a heavy burden, and I don’t think Stability.ai took the corresponding stakeholders into confidence before releasing Stable Diffusion. It should also go without saying that the license choosing to colour within the lines of the laws of respective countries means, say, a country that doesn’t recognise X as a crime will also fail to recognise harm in the harrassment of victims of X – now with the help of Stable Diffusion. And the vast majority of these victims are women and children, already disempowered by economic, social and political inequities. Is Stability.ai going to deal with these people and their problems? I think not. But as I said, the cat’s already out of the bag.