google.com search results are linking doc pages without stylesheets

[hayzey]

I'm submitting a ...

• bug report
• feature request
• other

Current behavior:

The CSS on the docs page for angular.element doesn't seem to be loading: https://docs.angularjs.org/partials/api/ng/function/angular.element.html

Expected / new behavior:

The page would have CSS applied as usual.

Minimal reproduction of the problem with instructions:

Go to https://docs.angularjs.org/partials/api/ng/function/angular.element.html

AngularJS version: 1.x.y

stable

Browser: [all | Chrome XX | Firefox XX | Edge XX | IE XX | Safari XX | Mobile Chrome XX | Android X.X Web Browser | iOS XX Safari | iOS XX UIWebView | iOS XX WKWebView ]

Chrome 63 in macOS

Anything else:

These are the only network requests I'm seeing:

[frederikprijck]

You don't want to use partials in the url, but instead point to the page (not the partial) https://docs.angularjs.org/api/ng/function/angular.element

If you search for angular.element on https://docs.angularjs.org/, you'll be redirect to https://docs.angularjs.org/api/ng/function/angular.element as well.

Partials are not supposed to have CSS.
If you'd inspect the source, you'd also see this isn't even a valid html page, but it's a partial instead (as you can tell from looking at the URL).

[Narretz]

@frederikprijck is right. How did you get the partial url?

[hayzey]

@frederikprijck The CSS loads if I access it with that link.

@Narretz I just searched "angular.element docs" on Google and that was that first link that popped up.

[frederikprijck]

@Narretz @hayleytom

Wow that's correct and that's NOT good.

https://www.google.be/search?q=angular.element+docs&oq=angular.element+docs&aqs=chrome..69i57j69i60l2j69i61j0.198j0j7&sourceid=chrome&ie=UTF-8

So despite the fact CSS is not expected to be loaded on that page, it shouldn't be indexed by google! 😟

[Narretz]

I see. Something went wrong with the robots.txt during migration I assume

[petebacondarwin]

Weird the robots.txt is:

User-agent: *

Disallow: /components/
Disallow: /examples/
Disallow: /img/
Disallow: /js/
Disallow: /partials/
Disallow: /ptore2e/
Disallow: /*.js$
Disallow: /*.map$
Disallow: /Error404.html

which looks correct to me. Any ideas?

[petebacondarwin]

Perhaps there was a random crawling during the switchover to Firebase Hosting that missed the robots file? Can we trigger a new crawl @Narretz - I don't have access to the Search console for this site.

[Narretz]

I just pushed an update that added this robots.txt. 😊 Before there was actually none. I'll look if I have access.

[Narretz]

@petebacondarwin I don't have docs.angularjs.org in the search console either. Probably @IgorMinar must trigger the re-crawl.

[IgorMinar]

It's strange that the crawler seems to to be not honoring the #! contract. We are correctly setting the meta tag.

I don't know if this is related, but I was just saw a notification that https://code.angularjs.org/snapshot/docs/js/all-versions-data.js is being banned via robots.txt and this prevents the crawler that indexes the site after the js executed from working.

[IgorMinar]

some other issues I found:

• the sitemap is missing: https://docs.angularjs.org/sitemap.xml
• robots.txt is missing: https://docs.angularjs.org/robots.txt
• the escaped_fragment url doesn't seem to be returning any content either e.g. https://docs.angularjs.org/?_escaped_fragment_=api%2Fng%2Ffunction%2Fangular.bind just redirects to /api. this seems wrong according to https://developers.google.com/webmasters/ajax-crawling/docs/getting-started. but I'm not sure if I got the url to test with right.

@Narretz can you please take a look?

[IgorMinar]

except for robots.txt the other two issues still seem unresolved.

@Narretz I though you said that the sitemap was fixed a few days ago, did I misunderstand you?

[Narretz]

@IgorMinar I did, and when I checked it was there. Not sure what's going on. The sitemap is visible on the docs folders for the snapshots. (Not that we need them there, just saying that the build produces them)