Nonprofit organization Creative Commons, known for its work in content licensing, has announced tentative support for ‘pay-to-crawl’ technology, a system designed to automate compensation for website content accessed by AI webcrawlers. This move follows CC’s earlier initiative to establish an open AI ecosystem framework, aiming to facilitate dataset sharing between data controllers and AI developers.
The pay-to-crawl concept involves charging AI bots each time they scrape a website for data collection and model training. Traditionally, websites allowed webcrawlers to index their content for search engine inclusion. However, with the rise of AI chatbots providing direct answers, user click-through rates to source websites have declined, adversely impacting publishers’ search traffic.
By endorsing pay-to-crawl systems, Creative Commons sees a potential solution for websites to sustain content creation, manage substitutive uses, and maintain public accessibility. This approach could especially benefit smaller publishers without the leverage to negotiate individual content agreements with AI providers, offering a lifeline in an evolving digital landscape.
Source: TechCrunch
Leave a Reply