Tea@programming.dev to Technology@lemmy.worldEnglish · 14 days agoCloudflare announces AI Labyrinth, which uses AI-generated content to confuse and waste the resources of AI Crawlers and bots that ignore “no crawl” directives.blog.cloudflare.comexternal-linkmessage-square107fedilinkarrow-up1600arrow-down16
arrow-up1594arrow-down1external-linkCloudflare announces AI Labyrinth, which uses AI-generated content to confuse and waste the resources of AI Crawlers and bots that ignore “no crawl” directives.blog.cloudflare.comTea@programming.dev to Technology@lemmy.worldEnglish · 14 days agomessage-square107fedilink
minus-squarelily33@lemm.eelinkfedilinkEnglisharrow-up9·14 days ago while allowing legitimate users and verified crawlers to browse normally. What is a “verified crawler” though? What I worry about is, is it only big companies like Google that are allowed to have them now?
minus-squarewingiee@lemm.eelinkfedilinkEnglisharrow-up11·14 days agoI assume a crawler which adheres to robots.txt
minus-squarelily33@lemm.eelinkfedilinkEnglisharrow-up2·14 days agoI would love to think so. But the word “verified” suggests more.
minus-squarekilleronthecorner@lemmy.worldlinkfedilinkEnglisharrow-up1·14 days agoIP verification is a not uncommon method for commercial crawlers
What is a “verified crawler” though? What I worry about is, is it only big companies like Google that are allowed to have them now?
I assume a crawler which adheres to robots.txt
I would love to think so. But the word “verified” suggests more.
IP verification is a not uncommon method for commercial crawlers