Page 1 of 1

Info about web crawlers, spiders and bots

Posted: Tue Apr 30, 2019 9:56 pm
by Blu
I’ve had a few people ask about these so here’s some info on them.

Have you seen a user on here with a gray username that can’t be clicked?

That “user” is a web crawler, also known as a spider or bot. It’s not a real person, but rather an automated script that goes about the web and indexes information it finds.

In other words, they help improve search engines such as Google.

PhpBB automatically reserves the first 45 IDs or so (excluding the anonymous user and me) for these bots. Their accounts get activated when their scripts comes across Shibaface.

Bots have access to only some forums. They can’t go onto the staff board or the parts of the site that only mods and admins can view.

Some bots aren’t recognized by phpBB, and appear as guests. As of April 30th, the date this post was written, a total of 5 unique bots have visited the site.

You can read more about web crawlers here:
https://en.m.wikipedia.org/wiki/Web_crawler

TL;DR: Web crawlers/bots aren’t bad. They just help improve search engines.

Re: Info about web crawlers, spiders and bots

Posted: Tue Apr 30, 2019 11:28 pm
by Lexi
Thanks Blu. I've always wondered about those.

Re: Info about web crawlers, spiders and bots

Posted: Wed May 01, 2019 11:04 am
by Oblivion
ahh ok

Re: Info about web crawlers, spiders and bots

Posted: Wed May 01, 2019 3:21 pm
by purple.
That makes a ton of sense, thanks for explaining!

Re: Info about web crawlers, spiders and bots

Posted: Wed May 01, 2019 4:16 pm
by Milk
Abd then there are real bots
we uh
we don't talk about those
Jk, i actually never knew what those did, i just knew they were fake for some purpose. Thanks for the info!

Re: Info about web crawlers, spiders and bots

Posted: Wed May 01, 2019 8:15 pm
by Coffee
Well, the more you know.

Image

Re: Info about web crawlers, spiders and bots

Posted: Sun May 05, 2019 9:32 pm
by Lilith
I asked about these forever ago lol

Re: Info about web crawlers, spiders and bots

Posted: Tue Jan 28, 2020 5:51 pm
by Derpy Memester
Hmmmmmm good info