存档

文章标签 ‘robots.txt’

robots.txt协议中User-agent的匹配

2009年8月3日 没有评论

之前写过一篇关于robots.txt协议(http://hi.baidu.com/wuzsh/blog/item/cef1fc03f6ff54723912bbbe.html)的文章 ,今天收到站长投诉,说是我们的spider不遵守robots.txt协议。把它们的robots.txt下下来测试一下,果然,以前的robots.txt程序有个bug。主要是没把robots.txt中user-agent的匹配弄好,于是再翻出来看看,下面是原文:(摘自http://www.robotstxt.org/norobots-rfc.txt阅读全文…


Warning: mysql_query() [function.mysql-query]: Access denied for user 'root'@'localhost' (using password: NO) in /home/wuzeshengpwmuqzeeas7h4eknug/wwwroot/wp-content/plugins/quickstats/quickstats.php on line 345

Warning: mysql_query() [function.mysql-query]: A link to the server could not be established in /home/wuzeshengpwmuqzeeas7h4eknug/wwwroot/wp-content/plugins/quickstats/quickstats.php on line 345

Warning: mysql_query() [function.mysql-query]: Access denied for user 'root'@'localhost' (using password: NO) in /home/wuzeshengpwmuqzeeas7h4eknug/wwwroot/wp-content/plugins/quickstats/quickstats.php on line 346

Warning: mysql_query() [function.mysql-query]: A link to the server could not be established in /home/wuzeshengpwmuqzeeas7h4eknug/wwwroot/wp-content/plugins/quickstats/quickstats.php on line 346

Warning: mysql_fetch_row(): supplied argument is not a valid MySQL result resource in /home/wuzeshengpwmuqzeeas7h4eknug/wwwroot/wp-content/plugins/quickstats/quickstats.php on line 346