ChatGPT now crawls websites, how do we stop it?

[Henrywrites]

I read that ChatGPT has launched a new model that enable the software to crawl an active website, scan and download everything that is on the website. This is worrisome and it makes me wonder what ways that can be stopped from happening. Do you have an idea?

 

[Arantor]

Do we know what user-agent it identifies itself as?

 

[Oblivion Knight]

Has all the information needed..

OpenAI API

An API for accessing new AI models developed by OpenAI

platform.openai.com

 

[Tracy]

from what I understand, in robots.txt (for basic search) CCBot in your robots.txt should work to block.
You may also need to put <meta name="CCBot" content="nofollow"> in your META tag.

 

[Shortie]

from what I understand, in robots.txt (for basic search) CCBot in your robots.txt should work to block.
You may also need to put <meta name="CCBot" content="nofollow"> in your META tag.

Thank you so much for sharing this @Tracy as much as I do love seeing where AI and ChatGPT are going, for forum owners it could be a huge issue especially if it is crawling the site and taking information from the site to use within their software.

I would assume this would cause issues with plagiarism and also double content as well on different sites.

 

[Arantor]

CommonCrawl is something different to the bot used by OpenAI (CC is much closer in intent to the Internet Archive, though has a different outlook, you may wish to block it anyway, but it's not obviously connected to ChatGPT). @Oblivion Knight has the correct information to block ChatGPT itself.