rehansaeed.github.io
rehansaeed.github.io copied to clipboard
RehanSaeed
[Comment] Dynamically Generating Robots.txt Using ASP.NET MVC
https://rehansaeed.com/dynamically-generating-robots-txt-using-asp-net-mvc/
Out of curiosity, what is the added benefit of dynamically creating a robots.txt file versus creating a static one in terms of maintenance?
I guess my real concern, is having to choose between polluting my project structure with another file versus having to redeploy the application each time I need to add, edit or delete stuff inside the robots.txt.
What's the lesser of two evils?
Out of curiosity, what is the added benefit of dynamically creating a robots.txt file versus creating a static one in terms of maintenance?
I guess my real concern, is having to choose between polluting my project structure with another file versus having to redeploy the application each time I need to add, edit or delete stuff inside the robots.txt. What's the lesser of two evils?
Polluting your project structure is a real problem with MVC 5 e.g. if you want to support favicons on all the major platforms then you need to add lots of images to the root or add some meta tags. Thankfully, this problem is solved in MVC 5 with the separate wwwroot folder.
Another problem is that the site-map URL has to be absolute. You have to manually go in and edit this URL. There are just so many little things like this you have to do to get a site up quickly, that I thought it would be best if I made the computer do the work instead. You have to remember your average developer using the ASP.NET Core Boilerplate project template is not always a web expert.
I don't see robots.txt as a file you need to edit very often but you are right, you can no longer just edit the file and it means a deployment. Some developers may be less comfortable with this than others.
So, how about SitemapXml? it says "I'll talk about this in a later blog post", so...?
So, how about
SitemapXml? it says "I'll talk about this in a later blog post", so...?
I've been massively busy lately with a career move. I'll get on it as soon as I can :)
I have used your sitemap generating algorithms and robotsTXT
they work both fine at localhost:3333/sitemap.xml and /robots.txt
but after publishing and uploading at hosting service - i have to access those at www.hoxat.com/sitemap.xml/ if i don't add the slash then i get HTML 404
i am using shared hosting Waiting for your kind reply
I have used your sitemap generating algorithms and robotsTXT they work both fine at
localhost:3333/sitemap.xmland/robots.txtbut after publishing and uploading at hosting service - i have to access those at www.hoxat.com/sitemap.xml/ if i don't add the slash then i get HTML 404
i am using shared hosting Waiting for your kind reply
Please raise an issue on the GitHub site listing the version of MVC you are using. The RedirectToCanonicalUrlAttribute is doing the redirecting. As long as you have the NoTrailingSlashAttribute on these actions it should be ok.
Hi, I made all you said to make it work , but I had to put a route config rule because always the response was 404.
routes.MapRoute(
"Robots.txt",
"robots.txt",
new { controller = "Home", action = "RobotsText" },
new[] { "VisaCuba.Controllers" });
It is mandatory to get the correct response? I'm using MVC 5. Greetings.
Hi, I made all you said to make it work , but I had to put a route config rule because always the response was 404.
routes.MapRoute( "Robots.txt", "robots.txt", new { controller = "Home", action = "RobotsText" }, new[] { "VisaCuba.Controllers" });It is mandatory to get the correct response? I'm using MVC 5. Greetings.
Looks like you are not using attribute routing, which is why you needed to add that code.
I've been massively busy lately with a career move. I'll get on it as soon as I can :)
Any update on this?
Any update on this?
This was posted some time ago. https://rehansaeed.com/dynamically-generating-sitemap-xml-for-asp-net-mvc/
Super slick trick. I'm working with AWS API Gateway with a .NET Kestrel server living in a Lambda function that's invoked via the incoming web requests. Every day or so, I would end up with a 404 for my robots.txt file. Since it was only an API, I had no file server to return the robots.txt from. Dynamically generating it solved the problem perfectly!
Don't forget to turn on attribute based routing in ASP.NET MVC 5 when you register your routes in RouteConfig.
It's a simple line statement.
public static void RegisterRoutes(RouteCollection routes) {
routes.IgnoreRoute("{resource}.axd/{*pathInfo}");
routes.MapMvcAttributeRoutes();
// ...
}