Feature request: ordering of allow and disallow rules

[michaelsstuff]

Allowing to set the order of allow and disallow rules would be helpful. As far as I know google does not care for the order, but other do, as speicied in the draft for robots.txt: http://www.robotstxt.org/norobots-rfc.txt -> 3.2.2 The Allow and Disallow lines

To evaluate if access to a URL is allowed, a robot must attempt to match the paths in Allow and Disallow lines against the URL, in the order they occur in the record. The first match found is used.

[some1ataplace]

Create a new model in your models.py file extending the robots.models.Rule class. Apply the required ordering logic by adding the ordering option in the Meta class.

from django.db import models
from robots.models import Rule

class CustomRule(Rule):
    order = models.PositiveIntegerField(blank=True, null=True)

    class Meta:
        ordering = ['order']

Create a admin.py file in your app directory, and register the new CustomRule model with the Django admin.

from django.contrib import admin
from robots.admin import RuleAdmin
from .models import CustomRule

@admin.register(CustomRule)
class CustomRuleAdmin(RuleAdmin):
    pass

Manage the ordering of allow and disallow rules in the Django admin site. Remember that you will need to create a new migration for the new CustomRule model and apply it.