, grasp the depth and breadth of
can be seen from the figure, the number of visits and grabbing quantity of love Shanghai spiders were 98600 and 224896, Google spider visits and grab amount is respectively 31157 and 172790, let us use the primary school math teacher to count the number of arithmetic average page spider visit, love Shanghai: 224896/98600=2.28, Google spider spider 172790/31157=5.54, can be seen love Shanghai crawl breadth than Google superior, while Google spiders crawl depth is significantly higher than the love Shanghai, love Shanghai every two pages to see the average spider, so much the amount of data of older sites often appear in this situation, Google included more, and the love of Shanghai included very little, at this point, Shanghai sex included the amount of time, if a large amount of data, the inside pages of random article calls this block must be Well, as for how to play their own random.
want to do must be to Shanghai dragon, the search engine spiders know enough, to understand the spider, to seriously on the log analysis, I love mules according to web logs and try to do a brief analysis of the spider, a few screenshots below before the web log analysis, we take a look at love Shanghai spiders and Google spiders, other SE ignore the decisive.
according to the previous several new observation log, the spider love Shanghai for a new general first day will be crazy to grab, and then a period of silence, while Google is quite honest, love a little bit to follow the prescribed order, more and more. Pictured above is a new station on June I log on June 15th, up less than an hour after the love of Shanghai included the home page, and crazy to grab 5500 times, second days to start silent in the outbreak, little Google began crawling slowly, more and more (statement: Map > log
three, the new grab
is a few spider random return 404 status code to grab the page, from indicated in the drawings can be seen, even if the site no chain, spiders will still capture a part of the death of love, love of spiders in Shanghai and was grabbing half URL seduce elsewhere, Google also has a HTM suffix catch into HTML. But relatively speaking, Google grab error is very small, but the love of Shanghai many, the reason is estimated that only two engineers to clear, we do not need to pay attention to this reason. At this point, whether you have stood the chain, try to go to add 404 error pages, because whether you have links, spiders will catch.
two, the error link to grab