improve website SEO

[akosyakov]

It does not look like Google can index open-vsx.org:

If someone looks for VS Code (extensions) marketplace then OpenVSX should be at least on second position.

[brianking]

@chrisguindon Any ideas about this one?

/cc @croundy

[mbarbero]

The edge load balancer serves a robots.txt file.

@agafgfafa, you added this 8 months ago when the site was having a hard time coping with incoming traffic.

Should it be removed now?

/cc @eclipsewebmaster

[eclipsewebmaster]

Googlebot is allowed to crawl. I'll use the Google webmaster tools to see why it's not indexing.

[eclipsewebmaster]

Actually, it is indexing:

[eclipsewebmaster]

"ruby for vs code" leads to open-vsx.org on page 4. Indeed, we need SEO.

[brianking]

Ok thanks @eclipsewebmaster. I'll ask @croundy for assistance with SEO.

[kineticsquid]

Dup of #1240