[ad_1]
\n<\/p>\n
Perplexity, a company that describes its product as “a free AI search engine,” has been under fire over the past few days. Shortly after Forbes<\/em><\/a> accused it of stealing its story and republishing it across multiple platforms, Wired<\/em><\/a> reported that Perplexity has been ignoring the Robots Exclusion Protocol, or robots.txt, and has been scraping its website and other Cond\u00e9 Nast publications. Technology website The Shortcut<\/em><\/a> also accused the company of scraping its articles. Now, Reuters<\/em><\/a> has reported that Perplexity isn’t the only AI company<\/a> that’s bypassing robots.txt files and scraping websites to get content that’s then used to train their technologies.<\/p>\n Reuters<\/em> said it saw a letter addressed to publishers from TollBit, a startup that pairs them up with AI firms so they can reach licensing deals, warning them that “AI agents from multiple sources (not just one company) are opting to bypass the robots.txt protocol to retrieve content from sites.” The robots.txt file contains instructions for web crawlers on which pages they can and can’t access. Web developers have been using the protocol since 1994, but compliance is completely voluntary.<\/p>\n TollBit’s letter didn’t name any company, but Business Insider<\/em><\/a> says it has learned that OpenAI<\/a> and Anthropic<\/a> \u2014 the creators of the ChatGPT and Claude chatbots, respectively \u2014 are also bypassing robots.txt signals. Both companies previously proclaimed that they respect “do not crawl” instructions websites put in their robots.txt files.<\/p>\n During its investigation, Wired<\/em> discovered that a machine on an Amazon server “certainly operated by Perplexity” was bypassing its website’s robots.txt instructions. To confirm whether Perplexity was scraping its content, Wired<\/em> provided the company’s tool with headlines from its articles or short prompts describing its stories. The tool reportedly came up with results that closely paraphrased its articles “with minimal attribution.” And at times, it even generated inaccurate summaries for its stories \u2014 Wired<\/em> says the chatbot falsely claimed that it reported about a specific California cop committing a crime in one instance.<\/p>\n