Info about web crawlers, spiders and bots
Posted: Tue Apr 30, 2019 9:56 pm
I’ve had a few people ask about these so here’s some info on them.
Have you seen a user on here with a gray username that can’t be clicked?
That “user” is a web crawler, also known as a spider or bot. It’s not a real person, but rather an automated script that goes about the web and indexes information it finds.
In other words, they help improve search engines such as Google.
PhpBB automatically reserves the first 45 IDs or so (excluding the anonymous user and me) for these bots. Their accounts get activated when their scripts comes across Shibaface.
Bots have access to only some forums. They can’t go onto the staff board or the parts of the site that only mods and admins can view.
Some bots aren’t recognized by phpBB, and appear as guests. As of April 30th, the date this post was written, a total of 5 unique bots have visited the site.
You can read more about web crawlers here:
https://en.m.wikipedia.org/wiki/Web_crawler
TL;DR: Web crawlers/bots aren’t bad. They just help improve search engines.
Have you seen a user on here with a gray username that can’t be clicked?
That “user” is a web crawler, also known as a spider or bot. It’s not a real person, but rather an automated script that goes about the web and indexes information it finds.
In other words, they help improve search engines such as Google.
PhpBB automatically reserves the first 45 IDs or so (excluding the anonymous user and me) for these bots. Their accounts get activated when their scripts comes across Shibaface.
Bots have access to only some forums. They can’t go onto the staff board or the parts of the site that only mods and admins can view.
Some bots aren’t recognized by phpBB, and appear as guests. As of April 30th, the date this post was written, a total of 5 unique bots have visited the site.
You can read more about web crawlers here:
https://en.m.wikipedia.org/wiki/Web_crawler
TL;DR: Web crawlers/bots aren’t bad. They just help improve search engines.