x

Robots.txt error on Google Console

Hi there, I am a beginner with the SEO side of things. 

On the Google webmasters site I am getting this message on the first page of Search Console

Severe health issues are found in your property. Check property health

Is robots.txt blocking important pages?
Some important page is blocked by robots.txt.

Sitemap: http://www.vinylupdate.co.nz/sitemap.xml

User-agent: NerdyBot
Disallow: /

User-agent: *
Disallow: /ajax/
Disallow: /apps/

 Could someone please let me know how to fix this? Many thanks

5,150 Views
Message 1 of 16
Report
15 REPLIES 15

I'm guessing this isnt really and issue and I should just ignore those errors...

5,144 Views
Message 2 of 16
Report
Square

The two folders that are blocked from indexing are resources used by your site that you don't want indexed (scripting and so forth), so that's normal to see. I always find it strange that Google gives a message like that when there are valid reasons to not want something indexed. Thanks for asking, though!

5,107 Views
Message 2 of 16
Report

I have got the same problem. Please help me to solve it. When i google the website, the description of the website does not show and i get a message that says, "A description for this result is not available because of this site's robots.txt"

5,073 Views
Message 2 of 16
Report
Square

Was there a period of time after you pointed your domains to Weebly that your sites weren't yet published? It sounds like Google indexed them prior to having the sites live. Google should update this information automatically, although it's hard to say when they will.

5,044 Views
Message 2 of 16
Report

For anyone who may find this topic when looking for the answer to this problem, if you see your pages in Google's search index with no meta description and with the message from Google saying that the description isn't available because of your site's robots.txt file, here's the fix:

Generally this is because you accientally checked "Hide this page from search engines" in the SEO settings of a specific page, or for your entire site.

On the page level in the Weebly editor click into "SEO Settings" and scroll to the bottom to find the check box:

image

Uncheck it, and then republish  your site.

For your entire site, go into the "Settings" menu, click "SEO" and then scroll down until you see that toggle:

image

These options in the Weebly Editor allow you to block your page or your site from search engine crawling via your site's robots.txt file (yourdomain.com/robots.txt) ... when you have both toggled off, your site is available for crawling by search engine spiders, and all of the information on them is available.

As for the other blocked resources (ajax, etc.), as the moderator said these are back-end scripts that you don't need to worry about, handled by Weebly. In some themes there won't be an issue with blocked resources by Google, so if it bothers you then your best option may be to choose a different theme.

The reasons why you might want to block a webpage from search engines vary, but might include:

  1. It's a template page that you've created with place-holder content, to make it very easy to make a copy and add content for landing pages for different products or guides. There's no real content on the place-holder page, so you don't want it crawled and indexed.
  2. It's a page that you're still working on but have not yet finished, but you need to push changes you've made to another page on your website live by publishin the site.
  3. It's a page that you need to have (for example privacy policy or advertising policy), but that you don't necessarily want people entering your site on (they can find it via site-search or in the site's footer).
  4. It's an old page without much content and it's creating a duplicate content warning with other thin pages on your website.

In addition to using these built-in toggle features to block your webpage from being crawled via robots.txt, if you'd like to make sure that it is not indexed by search engines at all, you can do that by adding a meta tag to the header of the page in the page's SEO Settings. This tag would be:

<meta name="robots" content="noindex, nofollow">

Mos has a good guide about the Meta Robots tag here.

Hope this is helpful to the next person with this issue!

4,040 Views
Message 2 of 16
Report

Hi JoeH

I started off with the Weebly page SEO option of having some pages hidden from Search engines for the reasons you've listed in your post. On the Webmasters page it listed these pages with an orange "Valid with Warnings".

Unsure whether this negatively affects my website, I did a bit of Google research and desided that a No Index meta tag would be the better option (according to Google).

Google is now reindexing these pages, and has reclassified them with a red "Error" message. Is this worse than the orange "Valid with Warnings"?

Should I go back to where I was? Which is the lesser of the two evils?

TIA for looking into this for me.

3,861 Views
Message 2 of 16
Report

Did you also get a message on the google search page that said, "A description for this result is not available because of this site's robots.txt"

5,072 Views
Message 2 of 16
Report

I'm apparently having a similar problem with a robot.txt file - and cannot find it.  It appears to be blocking Google's access to my first page, which is where my Nabble forum is embedded, so Google isn't finding the content of the forum, but does all the other pages of the site.

I've checked the SEO settings and I do not have the page hidden from search engines.  I've checked the header and footer code, to no avail.  But here's what Google's Analytics say.  What am I missing?

image

4,915 Views
Message 2 of 16
Report
Square

Definitely not an expert, but does this help out at all? https://support.google.com/webmasters/answer/6153277?hl=en

4,913 Views
Message 2 of 16
Report

Thanks for the reply.  I've actually read that, albeit subsequent to posting.  It provided some info, as did the Weebly SEO guide.  At this point it looks like the problem is with Nabble, not Weebly as the /robots.txt file shows the page to be open, meaning not blocked.  So, something on the page isn't working, and about all that is on there is a piece of code Nabble gave me to embed.

So, I'm trying to get help from Nabble.  But, if you know what to do please tell me.

3,303 Views
Message 2 of 16
Report

Did you solve this, Gary? We're having EXACTLY the same problem with our Weebly site. It's just a normal business site, and our home page is not getting indexed and we're not even appearing in Google FOR OUR OWN BUSINESS NAME! This happened yesterday (11 July 18). Our internal pages seem to be ranking as normal. Weebly are being NO HELP WHATSOEVER and putting their head in the sand and saying there is no problem.

2,758 Views
Message 2 of 16
Report

The problem actually was with Nabble, not Weebly.  All the other pages of my site were being indexed properly except for the page with the embedded Nabble forum.  They, Nabble, finally realized it was their problem and got it solved.

But, I will say that Google changed something that caused the problem.  Other search engines were still crawling and indexing the forum, but not Google.  Finally Nabble found a way around the problem.

2,502 Views
Message 2 of 16
Report

Well, I now appear to have another problem somewhat along these lines.  And, while I'm not sure this is the place to post it, I'll try here and y'all tell me if not.

In using Google Analytics I've realized that about a month ago I started getting "blocked resources" messages.  The graph is below and it shows that I'm now up to 273 pages with blocked resources.  And, when I delve deeper it always says the problem is this statement in the robots.txt file: Disallow: /ajax/

Is there really a problem?  If not, why did Google start seeing it as a problem a month ago when most of those pages have been static for a year or two?  Who changed what?  And, what can I do about it?

Thanks,

Gary

imageimage

2,479 Views
Message 2 of 16
Report
Square

Those pages are all backend resources so it won't have any impact on the rest of your site (you wouldn't really want those indexed anyway) - totally normal.

2,477 Views
Message 2 of 16
Report

Adam - I thought that might be the case.  Weird that it just started a month ago.  Anyway, thanks.

2,468 Views
Message 2 of 16
Report