[ad_1]
\n<\/p>\n
When Reddit said last month that it would block unauthorized data scraping from its site, everyone\u2019s (rightful) first reaction was \u201cAI, AI, AI.\u201d However, now that the change has taken effect, chatbot makers may not be the only ones being locked out. The widely used forum also appears to be blocking major search engines other than Brave and Google, the latter of which reportedly inked a deal earlier this year with Reddit worth $60 million annually.<\/a> However, a Reddit spokesperson told Engadget that the empty search results are about Google\u2019s rivals not agreeing to the company\u2019s requirements for AI training. It says it\u2019s it\u2019s in discussions with several of them.<\/p>\n 404 Media<\/em> reported on Wednesday<\/a> (and Engadget confirmed in our queries) that searching for Reddit results from the past week on rival engine Bing (using \u201csite:reddit.com\u201d) returns empty results. The publication reported that DuckDuckGo produced seven links without any descriptions, only providing the note, \u201cWe would like to show you a description here but the site won\u2019t allow us.\u201d The engine now appears to have removed even those, as our test only produced an empty page, reading, \u201cno results found.\u201d<\/p>\n When Reddit said last month<\/a> that it would update its Robots Exclusion Protocol (robots.txt) to block automated data scraping, it\u2019s now apparent that it wasn\u2019t only meant to thwart AI companies like Perplexity and its controversial \u201canswer engine.\u201d Currently, Google appears to be the only search engine allowed to crawl Reddit and produce results from \u201cthe front page of the internet.\u201d<\/p>\n A Reddit spokesperson told Engadget on Wednesday it isn\u2019t accurate to say the missing search results are a result of its Google deal. \u201cWe block all crawlers that are unwilling to commit to not using crawl data for AI training, which is in line with enforcing our Public Content Policy and updated robots.txt file,\u201d the company said. \u201cAnyone accessing Reddit content must abide by our policies, including those in place to protect redditors. We are selective about who we work with and trust with large-scale access to Reddit content.\u201d<\/p>\n Meanwhile, a source familiar with Reddit\u2019s thinking told Engadget on Wednesday that Bing\u2019s omission is due to Microsoft refusing to agree to Reddit\u2019s terms regarding AI crawling. Instead, the Bing maker allegedly claimed its standard web controls were sufficient. The source claims Microsoft\u2019s stance conflicts with Reddit\u2019s data privacy policy, leading to the impasse and empty search results.<\/p>\n The ubiquitous robots.txt is the web standard that communicates which parts of a site can be crawled. Although many crawlers are known to ignore its instructions, Google\u2019s standard procedure is to respect it. So, on the technical side, the companies in cahoots on the lucrative deal appear to have deployed some manual override.<\/p>\n The saga could be seen as a trickle-down effect of AI chatbots scraping the live web for results<\/a>. With courts slow to determine how much of the open web is fair use to train chatbots on<\/a>, companies like Reddit, whose bottom lines now depend on safeguarding their data from those who don\u2019t pay, are building walls at the expense of the open web. (Although, given the integral role Microsoft has played in this AI era, cozying up with OpenAI<\/a> early on, it seems ironic that Bing finds itself on the losing end of at least one aspect of the fallout.)<\/p>\n Colin Hayhurst, CEO of lesser-known \u201cno-tracking\u201d search engine Mojeek, told 404 Media<\/em> that Reddit is \u201ckilling everything for search but Google.\u201d In addition, the executive said his attempts to contact Reddit were ignored. \u201cIt\u2019s never happened to us before,\u201d he said. \u201cBecause this happens to us, we get blocked, usually because of ignorance or stupidity or whatever, and when we contact the site you certainly can get that resolved, but we\u2019ve never had no reply from anybody before.\u201d<\/p>\n Reddit has made no secret of its desire to block AI companies from scraping its treasure trove of data in this burgeoning age of AI. Last year, CEO Steve Huffman risked alienating large portions of its user base by blocking third-party API requests<\/a>, leading to the demise<\/a> of beloved apps like Christian Selig\u2019s Apollo<\/a>. Despite widespread protests among moderators and forum-goers<\/a>, the company only temporarily lost negligible numbers of users.<\/p>\n The gamble appeared to pay off, and Reddit recovered. It went public in March<\/a>.<\/p>\n