[ENG-2203] Update Storage Metrics Gathering

[UdayVarkhedkar]

Purpose

Updates storage metrics in automated csv export and keen scripts

Changes

• Adds target_custom_cap to data_storage_raws csv monthly export
• Re-actives run_keen_snapshots to re-activate addon snapshot
• Updates addon snapshot to include data on public/private nodes and public/private nodes affiliated with a non-COS institution for each addon

QA Notes

This should be dev tested

Documentation

N/A

Side Effects

N/A

Ticket

JIRA Ticket

[UdayVarkhedkar]

Sample Output of AddonSnapshot counts

[{'provider': {'name': 'box'},
  'users': {'enabled': 0, 'authorized': 0, 'linked': 0},
  'nodes': {'total': 0,
   'connected': 0,
   'connected_public': 0,
   'connected_public_affiliated': 0,
   'connected_private': 0,
   'connected_private_affiliated': 0,
   'deleted': 0,
   'disconnected': 0}},
 {'provider': {'name': 'dataverse'},
  'users': {'enabled': 0, 'authorized': 0, 'linked': 0},
  'nodes': {'total': 0,
   'connected': 0,
   'connected_public': 0,
   'connected_public_affiliated': 0,
   'connected_private': 0,
   'connected_private_affiliated': 0,
   'deleted': 0,
   'disconnected': 0}},
 {'provider': {'name': 'dropbox'},
  'users': {'enabled': 0, 'authorized': 0, 'linked': 0},
  'nodes': {'total': 0,
   'connected': 0,
   'connected_public': 0,
   'connected_public_affiliated': 0,
   'connected_private': 0,
   'connected_private_affiliated': 0,
   'deleted': 0,
   'disconnected': 0}},
 {'provider': {'name': 'figshare'},
  'users': {'enabled': 0, 'authorized': 0, 'linked': 0},
  'nodes': {'total': 0,
   'connected': 0,
   'connected_public': 0,
   'connected_public_affiliated': 0,
   'connected_private': 0,
   'connected_private_affiliated': 0,
   'deleted': 0,
   'disconnected': 0}},
 {'provider': {'name': 'forward'},
  'users': {'enabled': 0, 'authorized': 0, 'linked': 0},
  'nodes': {'total': 0,
   'connected': 0,
   'connected_public': 0,
   'connected_public_affiliated': 0,
   'connected_private': 0,
   'connected_private_affiliated': 0,
   'deleted': 0,
   'disconnected': 0}},
 {'provider': {'name': 'github'},
  'users': {'enabled': 0, 'authorized': 0, 'linked': 0},
  'nodes': {'total': 0,
   'connected': 0,
   'connected_public': 0,
   'connected_public_affiliated': 0,
   'connected_private': 0,
   'connected_private_affiliated': 0,
   'deleted': 0,
   'disconnected': 0}},
 {'provider': {'name': 'gitlab'},
  'users': {'enabled': 0, 'authorized': 0, 'linked': 0},
  'nodes': {'total': 0,
   'connected': 0,
   'connected_public': 0,
   'connected_public_affiliated': 0,
   'connected_private': 0,
   'connected_private_affiliated': 0,
   'deleted': 0,
   'disconnected': 0}},
 {'provider': {'name': 'mendeley'},
  'users': {'enabled': 0, 'authorized': 0, 'linked': 0},
  'nodes': {'total': 0,
   'connected': 0,
   'connected_public': 0,
   'connected_public_affiliated': 0,
   'connected_private': 0,
   'connected_private_affiliated': 0,
   'deleted': 0,
   'disconnected': 0}},
 {'provider': {'name': 'zotero'},
  'users': {'enabled': 0, 'authorized': 0, 'linked': 0},
  'nodes': {'total': 0,
   'connected': 0,
   'connected_public': 0,
   'connected_public_affiliated': 0,
   'connected_private': 0,
   'connected_private_affiliated': 0,
   'deleted': 0,
   'disconnected': 0}},
 {'provider': {'name': 'osfstorage'},
  'users': {'enabled': 31, 'authorized': 31, 'linked': 31},
  'nodes': {'total': 84,
   'connected': 84,
   'connected_public': 58,
   'connected_public_affiliated': 0,
   'connected_private': 26,
   'connected_private_affiliated': 1,
   'deleted': 0,
   'disconnected': 0}},
 {'provider': {'name': 'owncloud'},
  'users': {'enabled': 0, 'authorized': 0, 'linked': 0},
  'nodes': {'total': 0,
   'connected': 0,
   'connected_public': 0,
   'connected_public_affiliated': 0,
   'connected_private': 0,
   'connected_private_affiliated': 0,
   'deleted': 0,
   'disconnected': 0}},
 {'provider': {'name': 'onedrive'},
  'users': {'enabled': 0, 'authorized': 0, 'linked': 0},
  'nodes': {'total': 0,
   'connected': 0,
   'connected_public': 0,
   'connected_public_affiliated': 0,
   'connected_private': 0,
   'connected_private_affiliated': 0,
   'deleted': 0,
   'disconnected': 0}},
 {'provider': {'name': 's3'},
  'users': {'enabled': 0, 'authorized': 0, 'linked': 0},
  'nodes': {'total': 0,
   'connected': 0,
   'connected_public': 0,
   'connected_public_affiliated': 0,
   'connected_private': 0,
   'connected_private_affiliated': 0,
   'deleted': 0,
   'disconnected': 0}},
 {'provider': {'name': 'twofactor'},
  'users': {'enabled': 0, 'authorized': 0, 'linked': 0},
  'nodes': {'total': 0,
   'connected': 0,
   'connected_public': 0,
   'connected_public_affiliated': 0,
   'connected_private': 0,
   'connected_private_affiliated': 0,
   'deleted': 0,
   'disconnected': 0}},
 {'provider': {'name': 'wiki'},
  'users': {'enabled': 31, 'authorized': 31, 'linked': 31},
  'nodes': {'total': 39,
   'connected': 39,
   'connected_public': 13,
   'connected_public_affiliated': 0,
   'connected_private': 26,
   'connected_private_affiliated': 1,
   'deleted': 0,
   'disconnected': 0}},
 {'provider': {'name': 'googledrive'},
  'users': {'enabled': 1, 'authorized': 1, 'linked': 1},
  'nodes': {'total': 1,
   'connected': 1,
   'connected_public': 1,
   'connected_public_affiliated': 0,
   'connected_private': 0,
   'connected_private_affiliated': 0,
   'deleted': 0,
   'disconnected': 0}},
 {'provider': {'name': 'bitbucket'},
  'users': {'enabled': 0, 'authorized': 0, 'linked': 0},
  'nodes': {'total': 0,
   'connected': 0,
   'connected_public': 0,
   'connected_public_affiliated': 0,
   'connected_private': 0,
   'connected_private_affiliated': 0,
   'deleted': 0,
   'disconnected': 0}}]

[brianjgeiger]

I feel like TestDataStorageUsage should be failing because you added a new key. Could you check that out, and maybe add the new key to the test to make sure it's doing what it's supposed to?

[UdayVarkhedkar]

TestDataStorageUsage tests the output of the osf_storage_metrics.csv as opposed to either of the data_usage_raw_(nodes, preprints).csv files. I can add tests for the raw metrics file.

[brianjgeiger]

Ah, right. Hmmm, probably not worth it, then.