count_dongulus@lemmy.worldtoTechnology@lemmy.world•Developer Creates Infinite Maze That Traps AI Training BotsEnglish
422·
8 days agoThis won’t work against commercial crawlers. They check page contents with something similar to a simhash and don’t recrawl these pages. They also have limiters like for depth to avoid getting stuck in circular links.
You could generate random content for each new page, but you’ll still eventually hit the depth limit. There are probably other rules related to content quality to limit crawling too.
Americans and their silly performative outrage