[Cloud Block Service] AWS Volume shows status available. On modifying throws error cannot be modified when state is MODIFYING or OPTIMIZING.

[anvithks]

Issue/Feature Description: When a volume that is newly created is modified the status is shown as available. After the status is shown available if another modify request is sent the backend throws an error which has no information (panic error). On checking the docker logs the below error is shown.

On trying to modify a volume which is in MODIFYING or OPTIMIZING state the API throws an error.

These states can be fetched from the cloud backend and returned in the API to give the user correct information

[2020-09-28T21:49:21+05:30] [ERROR] [/root/gopath/src/github.com/sodafoundation/multi-cloud/contrib/datastore/block/aws/aws.go] [aws.(*AwsAdapter).UpdateVolume():318] [PID:1] Error in updating volume: {
  Iops: 0,
  Size: 600,
  VolumeId: "vol-08986585387cd1eff",
  VolumeType: "st1"
}%!(EXTRA *awserr.requestError=IncorrectModificationState: Volume vol-08986585387cd1eff cannot be modified in modification state MODIFYING
	status code: 400, request id: 7e2390e5-7d2c-415b-af3a-9bd49857b5bd)
[2020-09-28T21:49:21+05:30] [ERROR] [/root/gopath/src/github.com/sodafoundation/multi-cloud/contrib/datastore/block/aws/aws.go] [aws.(*AwsAdapter).UpdateVolume():322] [PID:1] IncorrectModificationState: Volume vol-08986585387cd1eff cannot be modified in modification state MODIFYING

Why this issue to fixed / feature is needed(give scenarios or use cases): If the state is shown as available then the user will try to modify. we are already showing creating and updating as intermediate states wherein a user will expect some change. Similarly showing the state as modifying or optimizing as per the cloud vendor will give more accurate information to the user and will also help in preventing making API calls to the backend when certain states do not allow it.

How to reproduce, in case of a bug:

Other Notes / Environment Information: (Please give the env information, log link or any useful information for this issue)

[kumarashit]

Should be fixed in next release. Reason for not fixing in Greenland release Less chances that so frequent modification will be done in actual/production environment

[himanshuvar]

@anvithks @kumarashit This is as expected from AWS Cloud Backend

@anvithks Could you please update the issue with the Response from API

[anvithks]

The issue is wrt the status in the SODA API response not matching the AWS Cloud backend error. The SODA API returns status as available. When the user tries to modify it throws above error.

You can check the scenario in the below screenshots.

1. Created a GP volume with 100GB : available

2. Docker logs

3. API response in the Modify volume page

4. Docker logs in the modify volume page.

5. Modified the size to 500GB. Status is updating

6. API response is updating.

7. Status is now available.

Docker logs

8. Modify request sent. Size increased to 750GB. Status is available but the cannot modify as modification state is OPTIMIZING.

[himanshuvar]

@anvithks Could you please share Response for Step 8

[anvithks]

The API response

[himanshuvar]

@anvithks Is this PUT Response for 750 GB?, Context Deadline exceeded was fixed using go routine. PUT API should return some response in terms of Error.

[himanshuvar]

@anvithks I suggest try same operations on AWS cloud Backend, and notice the behavior of Request/Response from AWS console.

[anvithks]

Yes @himanshuvar. This is for PUT. This setup was made last night.

[anvithks]

@anvithks I suggest try same operations on AWS cloud Backend, and notice the behavior of Request/Response from AWS console.

SODA clients will communicate with the SODA backend. The response we receive from the SODA backend willl be considered the same as that of the cloud vendor.

[himanshuvar]

@anvithks I suggest try same operations on AWS cloud Backend, and notice the behavior of Request/Response from AWS console.

SODA clients will communicate with the SODA backend. The response we receive from the SODA backend willl be considered the same as that of the cloud vendor.

Yes, I understand that SODA APIs & Clients communication should be in sync. If SODA clients expect something on some operations that is not given back from SODA APIs is certainly an issue in the APIs I have suggested the Cloud operation just to make our understanding clear about the AWS cloud operations. What is the expected behavior for their call operations. i.e. On modifying throws error cannot be modified when state is MODIFYING or OPTIMIZING -> How AWS response to this operation.

[anvithks]

This is how AWS console works. Hope this helps.

Modify prompt

Modify Success

Intermediate state. Volume is available but there is a modification state maintained and returned in the API. Denoted with yellow color next to available in the list.

API with modification state

Error when trying to modify the volume when in optimizing state.

Error from API

[himanshuvar]

Thanks @anvithks for confirming the operations from AWS Cloud Backend. I have tested the exact operations with master branch using POSTMAN client, Please find the results below. Hope you are also executing the above results on latest code.

1. Create a 100GB gp2 Volume using SODA multi-cloud Block API

docker logs for creating a 100GB Volume

Volume is created at AWS backend, can be seen in AWS console as below

2. Get Volume, created in Step 1 using SODA multi-cloud Block API

3. Update Volume to 500GB using SODA multi-cloud Block API

docker logs

Volume is Updated at AWS backend, can be seen in AWS console as below

4. Get Volume, updated in Step 3 using SODA multi-cloud Block API

5. Update Volume to 750GB using SODA multi-cloud Block API. THIS IS A FAILED OPERATION

This Calls returns the error with the valid reasons response from backend

docker logs

6. Get Volume using SODA multi-cloud Block API, Volume is in its old STATE which is an expected behavior

Volume is in its old STATE which is an expected behavior at AWS backend, can be seen in AWS console as below Volume Resource is maintained at the SODA Backend as AWS Backend

@kumarashit Please look into above findings. Please let me know if above SODA multi-cloud API's request/response are correct or not. Please let us know if any other API behavior expectated for these API calls.

@kumarashit @anvithks Let me know if there is any gap from my side in understanding the expected SODA multi-cloud backend APIs requirements.

[anvithks]

AWS adds the modification state to the response when the volume is modifying as shown here.

if the modification state is returned in SODA API then we can preempt making any requests for modification when the volume is in any of the intermediate states. AWS uses the colored dot next to the state name to indicate the intermediate state. An available volume is shown in green. An available volume in OPTIMIZING or MODIFYING state is shown with a yellow dot.

If the client receives the modification state object as shown above then making another request for modification can be prevented.

[himanshuvar]

@anvithks This is another approach to leverage the STATE model from a cloud vendor into SODA, but I don't think this Approach is feasible in SODA because of the following reasons:

1. SODA doesn't support Operation Handling. Its just a placeholder for the provisioning & metadata etc.
2. Approach is not scalable for extending the other Cloud vendor support as the other cloud vendor might not support the same STATE models as AWS like OPTIMIZING or MODIFYING. So we can't go with AWS biasing in SODA unless that is standard across multi-cloud vendors.
3. The PUT API calls can't be restricted based on the STATE model only. There are other factors/Errors need to be considered like Operation permitted within time limit that anyways are handled from cloud backend so as by SODA multi-cloud API.

Instead of confusing the Users for different states, the API request/response can be used for managing API calls that is already supported in the API. Client should utilize the API response properly.

[anvithks]

@himanshuvar

SODA doesn't support Operation Handling. Its just a placeholder for the provisioning & metadata etc.

Could you please elaborate a bit on this. Not sure I understand this correctly.

Approach is not scalable for extending the other Cloud vendor support as the other cloud vendor might not support the same STATE models as AWS like OPTIMIZING or MODIFYING. So we can't go with AWS biasing in SODA unless that is standard across multi-cloud vendors.

True. But we need to have a mechanism that would allow cloud vendor specific restrictions, validations or responses to be available to the client. For ex: in the current volume modify issue, a user who may be using SODA APIs will not be able to know that the status is available but in another intermediate state. Even if it is a valid scenario to throw an error it doesn't add up to a clean experience.

The PUT API calls can't be restricted based on the STATE model only. There are other factors/Errors need to be considered like Operation permitted within time limit that anyways are handled from cloud backend so as by SODA multi-cloud API.

With a front end client you do, in fact, get some amount of logic that can control the number of calls that need to be made to the backend. using the APIs directly using any REST client will still work as you explained above. If some of the limits and validations are known to us then we can leverage the same in the SODA APIs to make the usage more inline with e cloud backend. If the SODA APIs will only pass the error from the cloud vendor backend then this may lead to many wasted calls from the client.

Instead of confusing the Users for different states, the API request/response can be used for managing API calls that is already supported in the API. Client should utilize the API response properly.

The users will in fact be in a confused state when they see an error for an available volume. This is my main concern. if the status is available and there is no way for the user to know that the volume may not be modified how do we inform the user abou the same.