Learn how to find if your site is in Google's search results already, and if it's not, how to get it included. To add your content to Google, use our content submission tool: www.google.com You understand more about robots.txt files, check out this … [Read more...]
Errors that Google may find and report while crawling your site
Learn more about errors that Google's system can encounter while trying to discover your content, and where to go to troubleshoot these problems. Read more about crawling errors: support.google.com Find out what is going on with HTTP status code … [Read more...]
Using Sitemaps to help Google find content hosted on your site
Sitemaps help Google and other search engines discover your content so we can serve it to users for relevant queries. Read more about Sitemaps here: support.google.com Check out our resources on Sitemap creation: support.google.com Have a lot of … [Read more...]
What hardware and software powers Googlebot?
Hi Matt, Could you give any insight into the sort of hardware and/or server-side software which power a typical Googlebot (web crawler) server? Tristan Perry, Cardiff, UK Have a question? Ask it in our Webmaster Help Forum: www.google.com Want your … [Read more...]
Should I block Googlebot from crawling a dynamically generated calendar?
Should I deny an access to a dynamically generated vBulletin calendar via robots.txt? Whenever I open my Who's Online?" page, I can always see more than 20 Google spiders indexing it." Uncle, Prague, Czech Republic Have a question? Ask it in our … [Read more...]
How does Google search work?
Hi Matt, could you please explain how Google's ranking and website evaluation process works starting with the crawling and analysis of a site, crawling timelines, frequencies, priorities, indexing and filtering processes within the databases etc. … [Read more...]
Do robots follow links on a page which uses rel=”canonical” to point to a different page?
On a website, if a page B implements a rel="canonical" link tag with URL of page A, do robots follow every links in page B? Ad, Paris Have a question? Ask it in our Webmaster Help Forum: www.google.com Want your question to be answered on a video … [Read more...]
Is Google able to crawl protocols other than http and https?
Is Google able to crawl more protocols than http and https? Daniël Voogsgerd, Veenendaal, Netherlands Have a question? Ask it in our Webmaster Help Forum: www.google.com Want your question to be answered on a video like this? Follow us on Twitter and … [Read more...]
How is Google finding pages which don’t have any links to them?
How can Googlebot crawl and index pages that don't have any links to them on my website? I find each day two or three pages in the index that don't have any links to them on my site. The pages are generated by the search field of my website. Danny, … [Read more...]
Can I use robots.txt to optimize Googlebot’s crawl?
Can I use robots.txt to optimize Googlebot's crawl? For example, can I disallow all but one section of a site (for a week) to ensure it is crawled, and then revert to a 'normal' robots.txt? Blind Five Year Old, SF, CA … [Read more...]