[ad_1]
\n<\/p>\n
Reddit has a warning for AI companies and other scrapers: play by our rules or get blocked. The company said in <\/a> that it plans to update its Robots Exclusion Protocol (robots.txt file), which allows it to block automated scraping of its platform.<\/p>\n The company said it will also continue to block and rate-limit crawlers and other bots that don\u2019t have a prior agreement with the company. The changes, it said, shouldn\u2019t affect \u201cgood faith actors,\u201d like the Internet Archive and researchers.<\/p>\n Reddit\u2019s notice comes shortly after multiple reports that Perplexity and other AI companies regularly <\/a> websites\u2019 robots.txt protocol, which is used by publishers to tell web crawlers they don\u2019t want their content accessed. Perplexity\u2019s CEO, in a recent <\/a> with Fast Company<\/em>, said that the protocol is \u201cnot a legal framework.\u201d<\/p>\n In a statement, a Reddit spokesperson told Engadget that it wasn\u2019t targeting a particular company. \u201cThis update isn\u2019t meant to single any one entity out; it\u2019s meant to protect Reddit while keeping the internet open,\u201d the spokesperson said. \u201cIn the next few weeks, we\u2019ll be updating our robots.txt instructions to be as clear as possible: if you are using an automated agent to access Reddit, regardless of what type of company you are, you need to abide by our terms and policies, and you need to talk to us. We believe in the open internet, but we do not believe in the misuse of public content.\u201d<\/p>\n It\u2019s not the first time the company has taken a hard line when it comes to data access. The company cited AI companies\u2019 use of its platform when it began charging for <\/a> last year. Since then, it has struck licensing deals with some AI companies, including <\/a> and <\/a>. The agreements allow AI firms to train their models on Reddit\u2019s archive and have been a significant source of revenue for the newly-public Reddit. The \u201ctalk to us\u201d part of that statement is likely a not-so-subtle reminder that the company is no longer in the business of handing out its content for free.<\/p>\n<\/div>\n