General Don't want to give free content to the AI bots?

[Tracy]

If you use CloudFlare, you can use the below ruleset in your WAF panel and block the major (currently) players.
These AI bot trainers really should allow an opt-in to their services instead of forcing it to be an opt-out. They are hoovering up tons of free data and turning around and making a profit on it for themselves. I really prefer not to support them. They want to use the data created on my site, they can do like one did with Reddit... pay for it.

(http.user_agent contains "claudebot") or (http.user_agent contains "CCBot") or (http.user_agent contains "ChatGPT-User") or (http.user_agent contains "GPTBot") or (http.user_agent contains "Omgili") or (http.user_agent contains "ImagesiftBot ") or (http.user_agent contains "cohere-ai") or (http.user_agent contains "anthropic-ai") or (http.user_agent contains "Google-Extended") or (http.user_agent contains "ByteSpider")

The diffbot one is the default for the package but can be changed by whomever is using it, so checking your logs for bot visits remains a good idea.

If you have shell access on your server and want to check our log out for certain terms quickly, grep is your friend. You do need access to read the logs and if you have that, it is as simple as
cat access.log | grep searchword where access.log is your HTTP server access log name and the searchword is the word you want to search for.

And example using the search word of bot from todays log of my astro site.

 

[Jason]

People helping AI are like those people trained to help the replacement, or is it? Are they being trained to help the destructor of humanity (Terminator Movies)?

 

[Tracy]

People helping AI are like those people trained to help the replacement, or is it? Are they being trained to help the destructor of humanity (Terminator Movies)?

That aspect I really don't care about.
What I do care about is them taking content created on my site and gaining financial benefit from it without even asking or offering compensation, especially considering that I pay for the pipeline they are stealing their data down from my site.
I have a feeling that soon there will be laws placed on the book dealing with that type of action exactly.

 

[Jason]

That aspect I really don't care about.
What I do care about is them taking content created on my site and gaining financial benefit from it without even asking or offering compensation, especially considering that I pay for the pipeline they are stealing their data down from my site.
I have a feeling that soon there will be laws placed on the book dealing with that type of action exactly.

Yeah, definitely it's a thing where people are exploited cause they can't read the fine print, like with the .com thing we were discussing.

 

[Tracy]

Just to give you a hint of how heavy some of these hit your site. This sudden rise was right after I added ClaudeBot into the block list.
By far the majority of those blocks (over about a 7 hour timespan) were from that one AI bot.

 

[Cedric]

Is there any downside to giving them access though?

 

[Tracy]

Is there any downside to giving them access though?

yeah, they usually use your content to create their response with no attribution to your site or way for a user to find your site for more than what they may have used it for.
Why would I want to give them information that is "hard earned" on many sites now so that they can benefit from it and the site it came from doesn't get squat?

Depending on how a couple of cases go I would not be surprised to see more and more sites being paywalled off, even if the paywall is limiting the full reading to signed in members, with guests only getting a small "taste".
Will there be people that run sites that don't care? Very likely. But there are others that feel that the AI bot scraping is infringing on their hard work.

 

[Jason]

yeah, they usually use your content to create their response with no attribution to your site or way for a user to find your site for more than what they may have used it for.
Why would I want to give them information that is "hard earned" on many sites now so that they can benefit from it and the site it came from doesn't get squat?

Depending on how a couple of cases go I would not be surprised to see more and more sites being paywalled off, even if the paywall is limiting the full reading to signed in members, with guests only getting a small "taste".
Will there be people that run sites that don't care? Very likely. But there are others that feel that the AI bot scraping is infringing on their hard work.

Everything is people trying to get your info which is something in the same boat. For instance, app deals giving you $1 burgers, even in this inflated economy, are designed to grab your info, which is incredibly valuable.

 

[cpvr]

Just to give you a hint of how heavy some of these hit your site. This sudden rise was right after I added ClaudeBot into the block list.
By far the majority of those blocks (over about a 7 hour timespan) were from that one AI bot.
View attachment 3767

I’m at the point where I’ma block the claudebot as well. This is how high my bandwidth usage has been all day. It’s never been this high. There’s been at least 20-30 claudebot’s on my forum since 2pm.

 

[Tracy]

I’m at the point where I’ma block the claudebot as well. This is how high my bandwidth usage has been all day. It’s never been this high. There’s been at least 20-30 claudebot’s on my forum since 2pm.

I don't know if claudebot honors it or not (but from reading it appears it does), but apparently they do try to read robots.txt... but they aren't even allowed to get to that point now. And there have been several of these attempts back to back also.



An interesting read for those that were inquiring on why to block stuff like this.

DDoS from Anthropic AI | Linode Questions

Going by Twitter, I'm not the only one who's seeing a huge amount of bot traffic coming from AI company Anthropic. Their bot ("ClaudeBot") doesn't respect robots.txt or ai.txt and they appear t...

www.linode.com

The reddit one has some drift, but there are still some nuggets of knowledge in there.