Add completion prediction to API actions based on globally monitored stats

[cubiclesoft]

trafficstars

Currently, the /v2/actions API has to be queried to get the status of an action. Since there is no way to know when an action might complete, the API has to be repeatedly queried on a regular basis, which chugs through API calls (5,000 API calls starts looking rather small when this API limitation is discovered). You know your own network conditions, so you can probably build something to reasonably guess when any given action type will complete and can tell the client making the request to effectively "back off until" some time in the future before running the query again. From there, the client could show a progress bar or responsibly wait until that time passes and then run another query at which point the action might have completed or your system returns another future completion prediction. The net result is a much more powerful solution to waiting for actions to complete than some strict percentage-based value. Assuming you don't already monitor this stuff, it would also allow you to detect faults in your system where completion predictions start skyrocketing and/or exceed alert thresholds that you set (e.g. an action that is taking 4-5 times longer to complete than it should have) and then preemptively alert someone that something is possibly broken.

However, you should also consider adding percentage-based values to some actions. For example, transferring a snapshot between regions has hard and fast numbers (i.e. how much of the total data has been copied to the other data center) while rebooting a Droplet is much more vague and therefore percentages don't make a whole lot of sense BUT you can reduce the number of action query API calls with future prediction based on current network conditions. Both approaches have merit depending on the action type.

Also for consideration: WebSocket support for actions. Clients could then wait for an action to complete in near-perfect network silence and the server-side can immediately notify the client when the monitored action changes status.