how google is ruining the web, and what comes after the webpocalypse
From bankrolling disinformation, to stealing content and traffic for error-prone AI, to hiding news stories, Google seems determined to kill the web as we know it.
We already talked about the fact that the web is dying, and not of natural causes. And if there’s a victim suffering exsanguination as onlookers shudder in horror, there must also be a perpetrator, a Colonel Mustard who ambushed said victim in the library with an old machete. Or rather, in this case, Google, with an army of fake news bots, an AI that steals content from hundreds of millions of indexed sites to spit it right back out with error after error, removing links to news to avoid paying for what it scrapes, and SEO suggestions that ruined many a site with absurd and arbitrary demands.
When it first started out, Google was just a simple, clean search box that led to decent results, a far, far cry from the eyesores like Yahoo and AltaVista. This is why it utterly dominated the search game and performed around 9 in 10 searches since 2008. But as it had to find ways to keep growing after basically hitting market saturation, things started to get a little weird, as they always do in parasitic VC capitalism.
You see, its ad network could pay top dollar for certain search terms, so an entire industry got to work making sure certain sites landed at the top of the search results to monetize on them. This is the number one reason so many fake news sites exist. They want gullible people to click, or see ads, and they do it by lying about current events, or creating viral hyper-partisan hoaxes to get the right high paying ads and affiliate marketing deals. That’s especially true with many fake news sites which are effectively just bait for GOP Boomers without ad blockers or critical thinking skills.
But as fake news sites proliferate through the web and are crawled by Google, they pollute search results so much that even if you are a completely neutral observer who has absolutely no skin in the game and just want to verify something you saw, there’s a one in five chance that you’ll come away convinced of a lie. After all, in those 200+ factors which the company loves to mention in its PR pieces, veracity of facts is very, very unlikely to figure highly, if at all.
There are, quite literally, over a billion websites on the web. Fact checking every single last one of them is an impossible task, especially if you’re starting this far behind. And if Google tried to pay real experts to verify links, the shareholders would have Sundar Pichai spit-roasting over an open file fueled by hundred dollar notes about 15 minutes after the meeting was over as a warning. Green line go brrr, or else. Got it?
And if you think this is an exaggeration, consider that the company spent a year laying off thousands of employees by shareholder demand, paring down and moving a team working on a critical language for ML, endangering the future of its cross-platform UX tool Flutter, and scaring programmers from even trying to apply to the company, all to justify a massive $70 billion stock buyback and dividend. So yes, blood for the Blood God, with banks, hedge funds, and wealth managers filling in for Chaos.
But perhaps the cardinal sin Google is now in the process of committing undermines its entire purpose as a search engine and its implicit contract with the rest of the web. Starting with modifying search to show scraped answers to relevant queries, it began to effectively deny traffic to sites in a bid to keep users on their pages longer. These efforts are now intensifying with its AI search results, which summarize the content its crawlers scrape, regurgitating them as paragraphs with a few reference links here and there, maybe, if you’re lucky.
The end product can range between decently accurate and a complete dumpster fire of fact ad libs, so users end up learning that Earth weighs 5.97 septillion kilograms as far as we measured, and that Russian novelist Leo Tolstoyevskiy’s magnum opus was War And Peace And Love And Sex And Robots, written in 2865. So, yeah, Kwalitee™ approved knowledge on tap without ever needing to use anything other than Google. Unless you care about the search results actually making sense and being correct all the time, not just when the AI is having a good day after its latest training batch.
how to get the absolute worst of all worlds
So, let’s summarize the situation. We have a search engine that demands sites write everything at a 6th grade reading level, make titles match potential questions or short news headlines as much as possible, has no regard for truth or accuracy in the results it presents, funds purveyors of disinformation, often leads people to lies, conspiracy theories, scams, and propaganda, and is now literally stealing content from countless sites it insists on scrapping daily, and barfing back bits of it at users.
Oh, and if it tries to fix any if this while affecting one penny of revenue or profit, its executive team will be fired. Out of a cannon, into the sun. Again, green line go brrr. End of discussion.
But this state of affairs is unsustainable. After enough bad experiences, users will no longer trust Google for anything other than very basic facts and figures, and certainly not for research or fact checking. At the same time, the search giant gobbling up ever more websites to vomit them back out while denying traffic to those sites is going to make the content creators quite upset. The whole point was that we let the crawlers index and read us, then Google sends interested readers our way. Now, its AI abusing the crawl is the equivalent of saying “your content is ours now, fuck off.”
If that’s the case, why bother creating new content to be crawl-able? Why not just deny the bots and either catch evidence of Google ignoring the exemption request and put them on blast for stealing, and do what we can on social media to spread the word? Let the search results give users a concert by the Tinfoil Choir in Batshit Major, or spew nonsense from dismembered articles until they get fed up and start looking for alternatives.
Except what’s the alternative? Where do you go when the platform so synonymous with the act of searching the web that its name was a verb for almost a generation seems completely inescapable?
A big part of the problem is that any competitor to Google is going to run into more or less the same problem. They have to index a billion sites, literally, and then find ways to pay for collecting, storing, and parsing the immense amounts of data collected by the crawlers, as well as the constant re-calculations to keep search result quality high. They’ll have to figure out how make sure this data is factual. And they may have to do it all with the same revenue model, which is the business equivalent fighting a zoo of 900 pound gorillas with only your fists and wooden sticks.
DuckDuckGo, which advertises itself as Google without the constant monitoring and intrusive ads, started out strong but has been more or less been turning into a ripoff of the search giant, complete with shadowy data leaks to Microsoft in its browser, and an AI answer bot which also barfs out paraphrased chunks of websites, though in this case, it’s focused on online encyclopedias. To its credit, it does exclude your typical SEO hyper-optimized content mills and corrects the follies on which it’s called rather than ignoring criticism and just doubling down. But it still makes money through ads or affiliate marketing deals with Yahoo and Bing, which also aren’t doing too well.
Yahoo has been neglected for well over a decade now, more a living fossil which sort of just exists by inertia. Bing is effectively Microsoft Google and gives absolutely zero shits about the actual content it serves — as a steady stream of articles have pointed out over the years — and hosts an utterly unhinged, creepy and manipulative chat bot, as well as even more disinformation. And it should be noted that it’s an executive who worked at Yahoo currently being blamed by the tech press for choosing revenue over quality at Google in the first place.
if you want quality, you have to pay for it yourself
And all this brings us to an experiment called Neeva, a search engine for which you’d pay starting at $4.95 per month. It sounds absurd, right? Pay for a certain number of searches when you can do unlimited ones for free? But as its co-founders point out, you get what you pay for. If you don’t want ads, if you want verified links scanned for accuracy by humans, and an algorithm tweaked for quality rather than clicks, where playing SEO whack-a-mole doesn’t get you anywhere, you have to flip the script and change the incentives.
Now, while I’d love to link you to this engine so you can try it for yourself, Snowflake, the data company, ate it last year. Its closest analog is Kagi, which charges $5 per month for 300 queries and $10 per month for unlimited searches. The searches are snappy and pretty decent, warning about paywalls, and offering archived articles as viable options, but there’s still a fair bit of work to do in properly ranking sites in the result set, and it can benefit from adopting Neeva’s idea for “a sort of nutrition label for news sources” to help people avoid clicking on lies and hoaxes which we know can work when done assertively, according to a growing body of research.
It currently has just over 26,000 members doing half a million searches per day, which is only about 20 searches per user on average. But that stat compares quite favorably to Google’s numbers. The search giant does between 3.5 and 8.5 billion searches per day according to numerous estimates, but with a user base between 2.5 and 3 billion, meaning that its average per user search volume is way lower. (In reality though, the search habits of each person vary greatly by both person and day.)
Sure, $60 to $120 per year is not trivial, but companies responsible for search results have to be sustainable, and selling you access and tools is the only way they will make money, or, at least, should make money. Otherwise, they would need to rely on ads of some sort, or VCs, which reintroduces the same same exact terrible incentives which are turning search into a burning trash pile of fake news, spam, and hoaxes, all of it for growth, glorious growth, even when you’re a monstrously profitable monopoly.
In the meantime, Google is apparently considering charging users to inaccurately and haphazardly parse through the aforementioned pile of garbage with an AI bot. While still showing ads. So, if you’re already paying for a way to make sense of your search, why not pay not to see ads? And finance fact checks for media outlets? And validate top sources by human experts? That sounds kind of… nice? And useful?
Sadly, what we're seeing is par for the course in the end stages of what sci-fi author and blogger Cory Doctorow calls “enshittification,” and describes as the endpoint at which unchecked greed destroys businesses by turning them from providers of what people actually want, to extractive, parasitic entities which don’t care about the user or business partner experience, only revenue. As a result, neither users or business partners like the platform anymore and start looking for alternatives where possible. Or as Doctorow puts it more succinctly…
First, they are good to their users; then they abuse their users to make things better for their business customers; finally, they abuse those business customers to claw back all the value for themselves. Then, they die.
It was suicidal to ask people to pay for basic tools on the web when it was still young and had tens of thousands of sites at most. Hell, you could successfully argue that it may have never taken off if the early web was behind a paywall and you had to pay a monthly fee to an ISP and subscriptions to every website. But now, it’s the ubiquitous lifeblood of the modern world, and overly aggressive advertising across hundreds of millions of sites turned it from a wonderful place to learn new things and talk to new people, to a time-wasting, depressing chore more often than not.
We’re spied on every nanosecond of every day, have to be on constant lookout for cybercriminals, and every day, one of these tools either fails as a business, or wants more money for the same or worse product, or gets way too in our face with ads, or fills up more and more with scams, influencers, hoaxes, lies, snake oil, and AI porn, which may just include you if you annoy the wrong troll.
It seems like the only way to reset this increasingly toxic, dangerous, and dishonest ecosystem, and make it genuinely fun, easy, and pleasant again, is to stop being the products packaged for advertisers, and become the customers by paying for an ad-free, fact-checked, human supervised service worried about minimizing churn and maximizing quality rather than infinite growth by any means necessary. And the only way we can do that is to make these business models sustainable with our wallets, rather than leaving the web in the hands of increasingly unhinged VC tech bros.