Looking for design inspiration?   Browse our curated collections!

Return to Main Discussion Page
Discussion Quote Icon

Discussion

Main Menu | Search Discussions

Search Discussions
 
 

Dan Carmichael

1 Month Ago

Question About Those Bots

I fully understand about the bots... they've been discussed ad-nauseam.
What I don't understand is why (like beverly hills) they crawl the same thing almost every day.
Is this just a poorly-programmed bot, or is there a purpose?

Reply Order

Post Reply
 

Abbie Shores

1 Month Ago

To make sure it is still there and not to be removed. They just do a site, not an image as such. They index the pages...if not there any more then it will be removed from the search

 

Dan Carmichael

1 Month Ago

DOH! Didn't think of that. Simply updating the database. Makes sense.
Thanks Abbie. Closed.

Edit: must be a D--- powerful machine... with the ability to crawl millions of images on multiple sites every day.

 

This discussion is closed.