Brighton SEO Roundup – Searchbots: Lost Children or Hungry Psychopaths? What Do Searchbots Actually Do?


Roland Dunn

Roland Dunn

Google does not tell us when it visits which sections of sites in Webmaster Tools.

Use your webserver log files for more insight. If you can write a script to dig into these files, this can provide some real insight.

You can plot visits to the top level sections of your site on a chart to see how Googlebot is crawling your site.

Insight you can gain from log files:

  • Does Googlebot actually request all URLs seved by a website?
  • How does Googlebot spend its time, what does it request?
  • Does Googlebot spend its time cost-effectively?

Googlebot does not request all content at once, can become distracted, lost. Googlebot does not spend its time efficiently, so we need to point it towards things that we value.If Google doesn’t thing that your site is valuable to users, it won’t crawl it as frequently.

Example Googlebot log:

Conclusions

Searchbot visits are precious, you need to make sure you point in them in the right direction. Embrace your log files to determine behaviour and alter internal navigation and linking.

One thought on “Brighton SEO Roundup – Searchbots: Lost Children or Hungry Psychopaths? What Do Searchbots Actually Do?

  1. Pingback: BrightonSEO Roundup – Top 5 Takeouts | Copify Blog

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Are you human? Click the Pineapple...