ng add @angular/seo

[alexeagle]

From @IgorMinar on April 18, 2018 10:7

Bug Report or Feature Request (mark with an x)

- [ ] bug report -> please search issues before submitting
- [x] feature request

Area

- [ ] devkit
- [x] schematics

Versions

v6

Desired functionality

I'd like to be able to invoke ng add @angular/seo and have the CLI make my application search engine friendly. Initially I'd like to see the following feature provided by the schematics:

• generate robots.txt with no rules but a simple instructions/link on how to setup blacklist rules
• generate sitemap.xml with similar documentation/links as in robots.txt
• add code that would feature detect and conditionally load polyfills needed by the crawlers
• setup error logging via global error handler with an option to redirect the output to a remote server
• provide facilities to easily add and remove the noindex metadata to emulate the HTTP 404 and 500 status codes

Mention any other details that might be useful

Copied from original issue: angular/devkit#729

[alexeagle]

From @hansl on April 18, 2018 16:15

Why not make this as part of the ng new experience directly?

provide facilities to easily add and remove the noindex metadata to emulate the HTTP 404 and 500 status codes

That implies a router, no? We don't have a router by default (maybe we should).

[alexeagle]

From @clydin on April 18, 2018 16:47

There are classes of applications that do not need/want to worry about SEO: internal applications, applications exclusively behind a login, etc. For those cases the files and logic could manually be removed, I suppose.

However, some of those items (e.g., conditional polyfills) are things that could be elevated beyond SEO though.

[alexeagle]

From @IgorMinar on April 18, 2018 17:6

I think we need to experiment with this stuff more before we roll it out to everyone. This would be a safe way to iterate and find a the right solution for most apps without breaking things.

On Wed, Apr 18, 2018, 10:47 AM clydin [email protected] wrote:

There are classes of applications that do not need/want to worry about SEO: internal applications, applications exclusively behind a login, etc. For those cases the files and logic could manually be removed, I suppose.

However, some of those items (e.g., conditional polyfills) are things that could be elevated beyond SEO though.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/angular/devkit/issues/729#issuecomment-382453516, or mute the thread https://github.com/notifications/unsubscribe-auth/AANM6IGYVfpeP7yBHWm1MYviQ0wdnBHVks5tp24SgaJpZM4TZyPX .

[alexeagle]

From @tobi-or-not-tobi on April 19, 2018 8:36

If it's only a set of relatively static files that can be generated initially, it wouldn't matter to much whether its part of ng new or done in a schematics, besides the note of @IgorMinar. But I'd imagine that the real value goes into some dynamically data that will change over time, i.e. updates to the sitemap based on the router. That would require a repeatable activity and would be used while building and maintaining the app, and then initial creation only wouldn't be enough.

[alexeagle]

From @tobi-or-not-tobi on April 19, 2018 8:48

Another aspect is that part of the generated files might need to be extended by users based on dynamic data, which are not known in the app. This is a separate flow and we should be careful do allow for such manual additions. We could generate a sitemap index file during the initial creation of the app (i.e. ng new --sitemap) and build specific sitemaps while the app grows (i..e ng add @angular/sitemap-routes). The later would generate specific links which are simply not available initially, and write them in a separate sitemap.xml files.

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex>
  <sitemap>
    <loc>http://www.we-need-a-domain.com/static-routes.xml</loc>
    [...]
  </sitemap>
<!-- developers could add specific sitemaps to the sitemapindex file. 
  <sitemap>
    <loc>http://www.we-need-a-domain.com/products.xml</loc>
    [...]
  </sitemap>
</sitemapindex>

Developers can create other manual sub sitemap files and include them into the main sitemap (which should not be generated again).

[alexeagle]

From @ampgular on April 19, 2018 9:41

Great Idea, as i built my site, it would had helped me, if at least, I would have known the which basics I have to take care. My take on sitemap.xml: The Sitemap is very dynamic (last-mod, frequency and prority) for each URL, therefore for sitemaps we could ask what is worth to offer to the devellopper?

1. Create an empty sitemap.xml with xml structure
2. Based upon a json config and a string[] with routes create the sitemap.xml config= { "host":"www.mywebsite.com" "frecuency":"weekfly", ....... }
3. Following the idea from @tobi-or-not-tobi of @schematics/routes we could develop a small crawler that on request craw all internal "routerLink" attributes and create a list of "urls", which merged with a config file will create the string[] with the urls.

Probably the trade off effort/results is to go for the number 2. The number 3 is the interesting one, for ssr I can imagine how could it be done, for client side rendering I dont´t have a clue so far....

[alexeagle]

From @CharlBest on April 19, 2018 11:40

I think this is a really cool idea and didn't know about any of this so adding this with one command potentially with some granularity/options would be awesome!

[alexeagle]

From @kylecordes on April 19, 2018 14:47

It seems to me that in the general case, for a dynamic application, sitemap.xml would be generated by server-side code that accesses the same data as the Angular application.

Therefore - perhaps the sitemap idea fits in well with Universal, but not with client-only Angular?

Universal is becoming more popular; perhaps angular/seo ought depend on /universal.

[alexeagle]

From @tobi-or-not-tobi on April 19, 2018 22:41

@kylecordes agreed that this would make a lot of sense in universal, but as @IgorMinar pointed out at ngconf, he's in favor to see a solution in non-universal driven websites. Ideally we try to achieve the same for both in a generic way.

[alexeagle]

From @hiepxanh on May 7, 2018 14:44

I believe SEO is the most important think. More than speed, comfort, simple code, etc.... Without SEO, my best beautiful fastest ever website is just nothing, it just like a business without marketing... I think we have two problems:

• Problem 1: Developer should create sitemap.xml and robots.txt but he should create an automation way to automatic update that file. ( when?)
• Problem 2: How we can update that without touching that file? That is the client side problem.

[alexeagle]

From @hiepxanh on May 7, 2018 14:57

About the price he said. I think IgorMinar is right. More money, more SEO... It is a truth and it is acceptable. Assume that your website is dynamic content (just like news, or blog or something with auto generated content). About current situation:

1. People need push more money and effort to make angular SEO able.

• money: every website with dynamic content need a server. ( but, as Loki said "we have firebase functions" 😄)
• effort: every website with dynamic content need few setup. (our case is universal). if they choose that. how we can do it painlessly?

2. if using client only. I heard that they have Rendertron but I think we can provide something better. or just integrate Rendertron to universal?

[alexeagle]

From @BenjaminHofstetter on May 25, 2018 10:47

A great idea.... and maybe some kind of support for JSON lD (https://json-ld.org/)

[alexeagle]

From @ampgular on June 3, 2018 12:3

@IgorMinar diving into the angular.io index.html to see which best practies are you using. I came across following script // Dynamically, pre-emptively, addnoindex, which will be removed when the doc is ready and valid tag = document.createElement('meta'); tag.name = 'robots'; tag.content = 'noindex'; document.head.appendChild(tag); which as it is written add a noindex tag whih will be removed once the app is bootstrapped. My question is then, for search bots other than googlebot, is this tactic not penalyzing the index as these bots will only see the tag noindex? Thanks for your awesome job and contribution

[hiepxanh]

thank you, what strategy should we do for this?

[hiepxanh]

In my opinion, this issue should be high property. Angular only can be strong and bigger if it can SEO. An website only can be spread if it can share on facebook, twitter... and crawable by google. maybe we can have more intention on this?

[ranbuch]

I think nothing will help your web-app to SEO better then server-side-rendering but that means angular-universal and maintaining a dedicated server (in additional to the application backend) witch can be expensive and can sure add a lot of work.

Maybe there's a way to use angular-firebase as a smart CDN instead of using the hole serverless approach. Just host the application there and not use the build-in Databases that firebase has to offer.

Needless to say angular/seo will be even more important in that case . . .

[pasevin]

Meanwhile https://github.com/maciejtreder/ng-toolkit (with serverless) does a decent job for me right now. Just in case someone is looking how to approach this.

[KingDarBoja]

Meanwhile https://github.com/maciejtreder/ng-toolkit (with serverless) does a decent job for me right now. Just in case someone is looking how to approach this.

Same for me, the thing is that doesn't generate robots.txtneither sitemap.xml files at the moment.

[samuelfernandez]

Proposal:

• Custom builder to create robots.txt and sitemap.xml
• It can crawl the app and generate those files dynamically
• Result is static and can be re-generated with a CLI command in a pipeline. No need to be coupled with universal at a first stage

This can be enough to cover simple cases, more advanced can be customized by app developers

[angular-robot[bot]]

Just a heads up that we kicked off a community voting process for your feature request. There are 20 days until the voting process ends.

Find more details about Angular's feature request process in our documentation.

[hiepxanh]

UPPPP VOTEEEE, this is necessary feature

[ElecTreeFrying]

UPPPP