[Bug]: RBAC is not stable

[yhmo]

Is there an existing issue for this?

• [X] I have searched the existing issues

Environment

- Milvus version: v2.3.13
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Repeat the following steps for N times:

1. login with Root
2. create a collection
3. create a user (delete the user if it exists)
4. create a role (delete the role if it exists)
5. grant insert/search/load/createindex privileges to the role
6. add the user to the role
7. do something to the collection, insert, index, search, etc

You will succeed at the first time, and get privilege error at the second time or third time.

Expected Behavior

No response

Steps To Reproduce

Enable RBAC to the milvus server by setting authorizationEnabled=true

Repeat the following steps for N times:
1. login with Root
2. create a collection
3. create a user (delete the user if it exists)
4. create a role (delete the role if it exists)
5. grant insert/search/load/createindex privileges to the role
6. add the user to the role
7. do something to the collection, insert, index, search, etc

Milvus Log

No response

Anything else?

No response

[yhmo]

Script to reproduce:

from pymilvus import (
    connections,
    Role,
    FieldSchema, CollectionSchema, DataType,
    Collection,
    utility,
)

for k in range(100):
    print(f"==================================== REPEAT {k} ===============================================")

    connections.connect(
        alias='default',
        host='localhost',
        port='19530',
        user='root',
        password='Milvus',
    )
    print("login with root")

    collection_name = "my_collection"
    if utility.has_collection(collection_name, using="default"):
        utility.drop_collection(collection_name, using="default")

    fields=[
        FieldSchema(name="id", dtype=DataType.INT64, is_primary=True, auto_id=True),
        FieldSchema(name="vector", dtype = DataType.FLOAT_VECTOR, dim=4),
    ]
    schema = CollectionSchema(fields=fields)
    collection = Collection(name=collection_name, schema=schema, using="default")
    print("collection created by root")

    USER_NAME = "new_user"
    USER_PWD = "new_user_pw"

    names = utility.list_usernames()
    if USER_NAME in names:
        utility.delete_user(USER_NAME)
        print(f"{USER_NAME} deleted")

    utility.create_user(USER_NAME, USER_PWD, using="default")
    print(f"{USER_NAME} created")

    ROLE_NAME = "test_role"
    role = Role(ROLE_NAME)
    if role.is_exist():
        grants = role.list_grants()
        for item in grants.groups:
            role.revoke(item.object, item.object_name, item.privilege)
        role.drop()
        print(f"{ROLE_NAME} deleted")

    role.create()
    print(f"{ROLE_NAME} created")

    role.grant("Collection", "my_collection", "Insert")
    role.grant("Collection", "my_collection", "CreateIndex")
    role.grant("Collection", "my_collection", "Load")
    role.grant("Collection", "my_collection", "Search")
    role.grant("Global", "my_collection", "DropCollection")
    role.add_user("new_user")
    print(f"add {USER_NAME} to {ROLE_NAME}")

    connections.connect(
        alias='default',
        host='localhost',
        port='19530',
        user=USER_NAME,
        password=USER_PWD,
    )
    print(f"login with {USER_NAME}")

    data = [
        [[1, 2, 3, 4], [5, 6, 7, 8]],
    ]
    collection = Collection(collection_name)
    collection.insert(data)
    print("data inserted")

    index_params = {
        'metric_type': "L2",
        'index_type': "FLAT",
        'params': {},
    }
    collection.create_index(field_name="vector", index_params=index_params)
    print("index created")
    collection.load()
    print("collection loaded")
    print("searching...")
    vector = [1, 2, 3, 4]
    results = collection.search(
        data=[vector],
        anns_field="vector",
        param={
            'metric_type': "L2",
            "params": {}
        },
        limit=5,
        output_fields=['vector'],
    )
    print(results)

    utility.drop_collection(collection_name)
    print("collection dropped")
    print(f"=============================================================================================")

[SimFG]

This is because the permission information is sent asynchronously from rootcoord to proxy. If it is to take effect immediately, the permission verification part will not be cached, and the rootcoord interface needs to be called every time, which will cause a great loss in performance.

[yanliang567]

/assign @SimFG sounds like a by design? /unassign

[SimFG]

@yanliang567 yes

[yhmo]

From the user's view, it is like an unstable issue: https://discord.com/channels/1160323594396635310/1171824088109547561/threads/1232406435036528712

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[highvelcty]

Is there a recommendation on how to make the RBAC commands blocking for use cases such as the one suggested in the OP?

[xiaofan-luan]

Is there a recommendation on how to make the RBAC commands blocking for use cases such as the one suggested in the OP?

Can you be a little bit more clear? what is OP? we don't think change RBAC frequently makes any sense. And usually operation takes less than 1 seconds

[highvelcty]

By OP I am meaning the original post of bug report.

Programmatically interfacing to the the RBAC system when it is non-blocking / non-deterministic is difficult. Experimentally, I have not found a way to sequence RBAC operations without arbitrarily long delays between each operation.