\n
![](\"https:\/\/fortune.com\/img-assets\/wp-content\/uploads\/2024\/06\/GettyImages-2031368235-e1718131176974.jpg?w=2048\")
<\/p>\n [ad_1]
\n<\/p>\n
AI\u2019s unstoppable quest for training data is hoovering up growing amounts of increasingly questionable content\u2014including details of children whose use by AI breaks the law, researchers have found.\u00a0<\/p>\n
At least 170 links to photos and personal details of children in Brazil have been scraped from the internet and utilized to train AI systems without parental consent or knowledge, Human Rights Watch said in a report this week<\/a>. Some of those AI-systems have generated explicit and violent images of children, HRW said.\u00a0<\/p>\n Brazilian law prohibits the processing of children\u2019s personal data without the consent of a child\u2019s guardian, Hye Jung Han, a children\u2019s technology rights researcher and the author of the report, told Fortune<\/em>.\u00a0<\/p>\n The links to the photos were scraped from personal blogs and social media sites into a large data set called LAION-5B, which has been used to train popular image generators such as Stable Diffusion<\/a>. The 170 photos of children is likely a \u201csignificant undercount,\u201d HRW said, since the group only reviewed 0.0001 percent of the 5.8 billion images captured in LAION-5B.\u00a0<\/p>\n \u201cMy wider concern is that this is the tip of the iceberg,\u201d Han told Fortune<\/em>. \u201cIt\u2019s likely that there\u2019s many more children and many more Brazilian children\u2019s images in the data set.\u201d<\/p>\n LAION-5B scraped photos of children from as far back as 1994, and which were clearly posted with the expectation of privacy, Han said. One of the photos features a 2-year-old girl meeting her newborn sister, and the photo\u2019s caption includes not only both of the girls\u2019 names, but the name and address of the hospital where the baby was born.\u00a0<\/p>\n That kind of information was available in the URLs or the metadata of many of the photos, Han said. Childrens\u2019 identities are often easily traceable from the photos, either from the caption, or through information about their whereabouts when their photo was taken.\u00a0<\/p>\n Young children dancing in their underwear at home, students giving a presentation at school, and highschoolers at a carnival are only a few examples of the personal photos that were scraped.\u00a0 Many of them were posted from mommy blogs, or screenshots taken from personal family Youtube videos with small view counts, Han said. The photos \u201cspan the entirety of childhood,\u201d the report found.\u00a0<\/p>\n \u201cIt\u2019s very likely that these were personal accounts, and [the people who uploaded the images] just wanted these videos shared with family and friends,\u201d Han added.\u00a0<\/p>\n All publicly available versions of LAION 5B were taken down last Decembe<\/a>r after a Stanford investigation<\/a> found that it had scraped images of child sexual abuse. Nate Tyler, a spokesperson for LAION, the nonprofit that runs the data set,\u00a0 said that the organization is working with the Internet Watch Foundation, the Canadian Centre for Child Protection, Stanford, and Human Rights Watch to remove all known references to illegal content from LAION 5B.\u00a0<\/p>\n \u201cWe are grateful for their support and hope to republish a revised LAION 5B soon,\u201d Tyler said.\u00a0<\/p>\n He added that since LAION 5B is built from URL links, rather than direct photographs, simply removing the URL links from the LAION dataset won\u2019t remove any illegal content from the web.\u00a0<\/p>\n However, there is still identifying information about minors within links, Han said. She told Fortun<\/em>e she\u2019s asked LAION to do two things: first, prevent future ingestion of children\u2019s data, and second, regularly remove their data from the dataset.\u00a0<\/p>\n \u201c[LAION] has not responded or committed to either of those things,\u201d Han said.\u00a0<\/p>\n Tyler did not directly address this criticism, but underscored the nonprofit\u2019s commitment to addressing the issue of illegal material in the database.<\/p>\n \u201cThis is a larger and very concerning issue, and as a nonprofit, volunteer organization, we will do our part to help,\u201d Tyler said.<\/p>\n Much of LAION-5B\u2019s data is sourced from Common Crawl, which is a data repository that copies swaths of the open internet. However, Common Crawl\u2019s executive director, Rich Skrenta, previously told the Associated Press that it is LAION\u2019s responsibility to filter what it takes before making use of it.\u00a0<\/p>\n Once their photos are collected, children face real threats to their privacy, Han said. AI models, including those trained on LAION-5B data, have notoriously regurgitated<\/a> private information\u00a0 \u2013 such as medical records<\/a> or personal photographs \u2013 when prompted.\u00a0\u00a0<\/p>\n AI models can now generate convincing clones <\/a>of a child with just one or two images, the report wrote.\u00a0<\/p>\n \u201cIt is pretty safe to say that the photos that I found absolutely contributed to the model being able to produce realistic images of Brazilian kids, including sexually explicit imagery,\u201d Han said.\u00a0<\/p>\n More maliciously, some users have used text-to-image AI sites to generate child pornography. One such site, called Civiai, trains their data off of LAION-5B and is overrun by requests for explicit content \u2013 60% of images generated on the platform are considered lewd. Some users asked for and were provided with images related to \u201cvery young girl,\u201d and \u201csex with dog,\u201d an investigation<\/a> from 404Media, a tech journalism company, found.\u00a0<\/p>\n Civiai, upon request, even generated lewd images of girls that specifically did not<\/em> look \u201cadult, old\u201d or \u201chave big breasts,\u201d 404Media revealed.\u00a0<\/p>\n After the investigation was released, the cloud computing provider for Civiai, OctoML, dropped its partnership with the company. Now, Civiai includes a NSFW filter, much to the dismay of some users, who said that the platform will now be like \u201cany other,\u201d according to 404Media.\u00a0<\/p>\n A spokesperson from CIviai told Fortune<\/em> that it immediately bans anyone who produces NSFW content involving minors, and has introduced a \u201csemi-permeable membrane,\u201d referring to the filter which blocks inappropriate content.\u00a0<\/p>\n Deepfake technology has already begun to impact young girls, Han said. At least 85 Brazilian girls have faced harassment from classmates who used AI to create sexually explicit deepfakes of them, based on photos taken from their social media profiles, according to the report. Han said she started investigating the topic due to the consistency and realism of these deepfakes.\u00a0<\/p>\n \u201cI started looking at what was it about this technology that was able to produce such realistic imagery, horrific imagery, of Brazilian kids, and that investigation led me to the training data set,\u201d Han added.\u00a0<\/p>\nPotential for harm<\/strong><\/h2>\n