backend.ai-webui icon indicating copy to clipboard operation
backend.ai-webui copied to clipboard

fix: quota and max size of virtual folder displays NaN, and magic number remains in resource policy setting dialog

Open lizable opened this issue 2 years ago • 11 comments

Description

This PR fixes two minor bugs about setting size of virtual folder(vfolder) that located in storage host that supports directory-based quota and resource policy. Please refer to the following checkbox descriptions below:

  • [x] the max values of corresponding keys in resource policy to match with the max value of DB.
  • [x] Show quota and max size of virtual folder correctly.

Additional Context

As minimizing server-side updates, Let's just clamp the max value to about 2 PiB for now; the maximum value that the corresponding attribute on Vfolders table in manager DB can afford.

According to the DB schema of "resource policy(keypair_resource_policy)", which is described in Backend.AI manager repo, It has both numeric type, "Integer" and "BigInt". For now, we've been dealing with "0(zero)" value as Infinity, but It's not a good approach when it comes to "disabling" certain values by setting them to zero. IMO, It's better to fix not to treat "0(zero)" value as "Infinity".

lizable avatar Jun 13 '22 11:06 lizable

I think this comment is irrelevant to this PR. Could you make a separate issue for this @Sujin-Kim1 ?

If the user inputs an invalid value, the bottom border turns red but it can be updated. image image

lizable avatar Jun 27 '22 15:06 lizable

I think this comment is irrelevant to this PR. Could you make a separate issue for this @Sujin-Kim1 ?

If the user inputs an invalid value, the bottom border turns red but it can be updated. image image

https://github.com/lablup/backend.ai-webui/issues/1336 here it is.

agatha197 avatar Jun 27 '22 15:06 agatha197

@inureyes @adrysn Could you review this PR?

lizable avatar Jul 06 '22 11:07 lizable

Also, the capacity does not seem to be updated correctly.

https://user-images.githubusercontent.com/7539358/177576439-1549e9a1-6b79-4e60-b48c-25ead6c332e2.mov

adrysn avatar Jul 06 '22 14:07 adrysn

I fixed miscellaneous errors, as @adrysn mentioned in the comments above. Also, I polished up codes related to displaying the maximum size of virtual folders in (keypair) resource policy.

Before

Screen Shot 2022-07-21 at 2 53 37 AM

After

Screen Shot 2022-07-21 at 2 46 53 AM

lizable avatar Jul 20 '22 17:07 lizable

Could you check whether this PR works fine? @Sujin-Kim1 ?

lizable avatar Jul 20 '22 18:07 lizable

Additional Comments

Some of the resource configurations in resource policy should treat 0(zero) as is, for certain reasons. For example, If the admin user wants to block session/virtual folder creation for certain users(probably violated rules?), he/she would likely use 0(zero) value as restricting the limit of maximum counts that users can create. In this situation, 0(zero) should be treated as 0, not infinity. But It seems that it may need server-side updates, and more detailed strategies and scenarios to test, so I think it's better to leave it like this and continue this discussion in another issue.

lizable avatar Jul 20 '22 18:07 lizable

I updated the capacity using the arrow button, and the step is 0.1 but on the screen, it is displayed as an integer, not a float. Did you intend to?

Updated

~Nope, but I think it's because the type of "max_vfolder_size" in the keypair resource policy is BigInt.~ ~I think It's better to make it discrete.~ ~What's your thought about this @Sujin-Kim1 ?~

It's a bit complicated but, this is because of mishandling received value from listing keypair resource policy requests. I finally got a clue to solve this.

  • Keep the unit "AS IS". (supports to the first decimal place)
  • Convert the value correctly when receiving the response.

I'll fix it right away.

lizable avatar Jul 21 '22 03:07 lizable

Btw, I need to change the capacity limit of virtual folder (vfolder-capacity-limit) since the type of attribute corresponding in DB schema is BigInteger. But there's one thing that concerns me. I'm not sure about the exact size of BigInteger(In PostgreSQL says, 8bytes) the description in "sqlalchemy". It should be changed, anyway.

lizable avatar Jul 21 '22 03:07 lizable

Additional context

For now, we don't have any criteria for "maximum" value for actual resources, such as CPU, main memory, GPU, etc. I'll leave them as the max integer of 4 bytes as a workaround, but I think we need to set the guide in the future.

lizable avatar Jul 21 '22 06:07 lizable

@inureyes @adrysn @achimnol @agatha197

Sticking Point of this issue.

I found that many attributes that should treat 0(zero) value "as is" in keypair resource policy. The reason why I issued this is because of avoiding the gap between what users expect on value configuration and the actual value works inside Backend.AI components.

Here are all attributes that contain in a single keypair resource policy. For better understanding, I put a "Y" mark in every "Does it correspond to the zero value issue? (Y/N)" column that fits into this condition.

attribute name description Does it correspond to the zero value issue? (Y/N)
name the name of keypair resource policy N
created_at timestamp of when keypair resource policy created N
default_for_unspecified enum value used for configuring unspecified value in total_resource_slots N
total_resource_slots physical resources used for computing (except storage) N
max_session_lifetime the amount of time a session can exist in the 'RUNNING' state from the time it was created. N
max_concurrent_sessions the maximum number of sessions that user can create on person's own Y
max_containers_per_session the maximum number of containers that user can create in a single session Y
max_vfolder_count the maximum number of vfolder that user can create Y
max_vfolder_size the maximum size of vfolder(storage) that user can allocate in a single vfolder Y
idle_timeout the idleness of a session by the elapsed time since last used. N
allowed_vfolder_hosts the type of storage hosts that allowed to mount for corresponding keypairs N

IMO, attributes with Y mark should be handled in the same policy with unlimited and zero value. For example, max_vfolder_count attribute treat zero as not allowing any extra folder creation/clone operation, and treat unlimited with four options:

  • Option 1. Use the maximum number of the type as Unlimited. (⚠️ NOTE: using maximum number !== magic numbers)
    • pros: Relatively Fast implementation. No need to do extra chores on the server side, for now. (May need "some conditions to check unlimited in certain situations, though").
    • cons: Bad for maintainability.
      • If the type changes, we need to change the value on the client side manually.
      • Need to know every type of the corresponding attribute, like whether is Integer(4bytes), BigInteger(8bytes), etc.
  • Option 2. Use specific values such as unlimited, like -1
    • pros: Also Fast implementation. No need to do extra chores as Option 1 does.
    • cons: Need to consider the side-effects of malicious or unintended value manipulation.
  • Option 3. Use null(empty) value as unlimited.
    • pros: No need to think about the type and no concerns for affected by type changes, of course.
    • cons: Same reason for option 2.
  • Option 4: Create a column for unlimited and the type would be Boolean.
    • pros: Code readers/contributors can easily understand it because of the explicit factor of "unlimited", including the benefit that Option 3 does.
    • cons: Extra server-side works.
      • Need DB migrations
      • Need to consider backporting since the consequences of the change may affect many major operations such as session creation, vfolder creation, etc. we might violate our latest backporting policy because of this.

I've been also thinking about which option would be the best way in both time efficiency and maintainability, but It's hard to decide. It would be great to give your opinion about this issue and if none of them meets your satisfaction, then adding your own opinion(s) would be appreciated.

lizable avatar Jul 23 '22 14:07 lizable