Recent problems with Google search of MrExcel.com

[PaddyD]

Have raised it, Jerry.

 

[Jerry Sullivan]

I'd like to make another plea for some attention to this problem.

MrExcel.com hasn't been the same the last 2 months since the traffic has dropped significantly.

The "Zero Reply Posts" search list has dwindled to a handful of items, presumably because the same number of helpers is responding to fewer helpees.

Doing a little more searching, the cause of problem and its solution seem pretty clear.
The MrExcel.com site has a robots.txt file which appears to be blocking search engines from searching the site.

The file is accessible to view at:
http://www.mrexcel.com/robots.txt

The first few lines of that file as of today are:

User-agent: *
Disallow: /
Disallow: /forum/admincpanel/
Disallow: /forum/announcement.php
Disallow: /forum/faq.php

This site The Web Robots Pages
explains that the Disallow: / line will exclude all robots from the entire server.

This was probably placed unknowingly, because if I'm understanding this correctly, the listing of all the subsites below the Disallow: / line is redundant since the entire site is excluded.

I'm hoping that Tracy or Bill could either remove that line, or provide an explanation if it is there intentionally.

Thanks all.

 

[starl]

actually, it was placed on purpose to put a stop to the theft of board content. Robots have been trawling us.

 

[starl]

Update: we did have a bad robots.txt file. Thank you for catching it. It may take a few days for google and bing to catch up, but the issue should be fixed.
We may continue to tweak the settings to relieve traffic on the board - the bots can and do cause problems, but we don't want to stop legit bots.

 

[Aladin Akyurek]

actually, it was placed on purpose to put a stop to the theft of board content. Robots have been trawling us.

Do they literally steal the content or create a reference to it?

I feel that the point Jerry raises is valid.

Maybe unrelated: all cached references (by Google, etc.) consist of death links as well as the ones I copied and pasted in the workbooks I created while helping.

 

[starl]

theft was the wrong word to use.
The robots are taking up a lot of bandwidth. The text file allows us to limit where they go and the period of time they're allowed to trawl. Without limitations, they can take up 1000 guest positions on the board.

We have no control of how long Google takes to update their links. We're trying to ensure links in various formats work, but may not be able to work around them all.

 

[Jerry Sullivan]

Update: we did have a bad robots.txt file. Thank you for catching it. It may take a few days for google and bing to catch up, but the issue should be fixed.
We may continue to tweak the settings to relieve traffic on the board - the bots can and do cause problems, but we don't want to stop legit bots.

Tracy, Thanks so much for fixing this.