Add the ability to be able to query the number of downloads for each (componentID, version)

[bbondy]

It would be good to know the amount of users for each component that we have.

I think this would look like just a mapping of (componentID, version) => count

I think to do this the URLs returned from the go-update server would be to itself, which would then redirect to the real URL. A map could be updated with a count, and there could be a task that updates the DB with the counts periodically (e.g. once per minute for example). I'd recommend against 1 update to the DB per component download request.

On startup, the count would be taken from dynamodb so that if the component is updated, the counts don't start back at 0.

brave-core-crx-updater could perhaps get an additional small script with command line params that allows you to query a component ID and a version (or omit version to get the latest version) to get a count directly from dynamodb.

The architecture described here is just an idea, I'm open to other ideas. E.g. maybe log based and then an aggregator would be another approach.

See plan below by @jumde 👍

[jumde]

Plan of action:

1. Enable bucket metrics for S3 buckets with crx.
2. Create a cloudwatch namespace to view the downloads for each component by version per day/month/year.

The cloud-watch graph will be accessible via aws console.

[jumde]

@hspencer77 will be adding tags manually for the latest version of all components to get the data. We'll be working with this initial list to see how the data looks like.

In parallel, we are working on adding tags to the new extensions: https://github.com/brave/brave-core-crx-packager/issues/123

Component list:

1. Brave Tor Client Updater (Mac) - cldoidikboihgcjfkhdeidbpclkineef
2. Brave Local Data Files Updater - afalakplffnnnlkncjhbmahjfjhmlkal
3. Brave Ad Block Updater (Default) - cffkpbalmllkdoenhmdmpbkajipdjfam
4. Brave NTP sponsored images - gccbbckogglekeggclmmekihdgdpdgoe
5. Brave SpeedReader Updater - jicbkmdloagakknpihibphagfckhjdih
6. Crypto Wallets - odbfpeeihdkbihmopkbjmoonfanlbfcl
7. Brave HTTPS Everywhere Updater - oofiananboodjbbmdelgdommihjbkfag
8. Brave Tor Client Updater (Windows) - cpoalefficncklhjfpglfiplenlpccdb
9. Brave Tor Client Updater (Linux) - biahpgbdmdkfgndcmfiipgcebobojjkp

[hspencer77]

Completed the following:

• Tagged latest objects associated with the component list above in brave-core-ext bucket. For example:

$ aws s3api get-object-tagging --bucket brave-core-ext --key release/cffkpbalmllkdoenhmdmpbkajipdjfam/extension_1_0_375.crx --profile go-updater-stg-devops-rw --output yaml
TagSet:
- Key: Name
  Value: cffkpbalmllkdoenhmdmpbkajipdjfam/extension_1_0_375.crx

• Tagged latest objects associated to bucket brave-core-ext-tor . For example:

$ aws s3api get-object-tagging --bucket brave-core-ext-tor --key release/biahpgbdmdkfgndcmfiipgcebobojjkp/extension_1_0_11.crx --profile extensions-prod-devops-rw --output yaml
TagSet:
- Key: Name
  Value: biahpgbdmdkfgndcmfiipgcebobojjkp/extension_1_0_11.crx

• Added metric filters for buckets brave-core-ext and brave-core-ext-tor.

$ aws --region us-east-2 s3api list-bucket-metrics-configurations --bucket brave-core-ext --profile go-updater-stg-devops-rw --output yaml
IsTruncated: false
MetricsConfigurationList:
- Filter:
    Tag:
      Key: Name
      Value: cffkpbalmllkdoenhmdmpbkajipdjfam/extension_1_0_375.crx
  Id: Brave-Ad-Block-Updater-Default
- Filter:
    Tag:
      Key: Name
      Value: oofiananboodjbbmdelgdommihjbkfag/extension_1_0_16.crx
  Id: Brave-HTTPS-Everywhere-Updater
- Filter:
    Tag:
      Key: Name
      Value: afalakplffnnnlkncjhbmahjfjhmlkal/extension_1_0_22.crx
  Id: Brave-Local-Data-Files-Updater
- Filter:
    Tag:
      Key: Name
      Value: gccbbckogglekeggclmmekihdgdpdgoe/extension_1_0_60.crx
  Id: Brave-NTP-sponsored-images
- Filter:
    Tag:
      Key: Name
      Value: jicbkmdloagakknpihibphagfckhjdih/extension_1_0_1.crx
  Id: Brave-SpeedReader-Updater
- Filter:
    Tag:
      Key: Name
      Value: biahpgbdmdkfgndcmfiipgcebobojjkp/extension_1_0_6.crx
  Id: Brave-Tor-Client-Updater-Linux
- Filter:
    Tag:
      Key: Name
      Value: cldoidikboihgcjfkhdeidbpclkineef/extension_1_0_6.crx
  Id: Brave-Tor-Client-Updater-Mac
- Filter:
    Tag:
      Key: Name
      Value: cpoalefficncklhjfpglfiplenlpccdb/extension_1_0_6.crx
  Id: Brave-Tor-Client-Updater-Windows
- Filter:
    Tag:
      Key: Name
      Value: odbfpeeihdkbihmopkbjmoonfanlbfcl/extension_1_0_18.crx
  Id: Crypto-Wallets
- Filter:
    Prefix: release
  Id: brave-core-ext

$ aws s3api list-bucket-metrics-configurations --bucket brave-core-ext-tor --profile extensions-prod-devops-rw --output yaml
IsTruncated: false
MetricsConfigurationList:
- Filter:
    Tag:
      Key: Name
      Value: biahpgbdmdkfgndcmfiipgcebobojjkp_extension_1_0_11.crx
  Id: biahpgbdmdkfgndcmfiipgcebobojjkp
- Filter:
    Prefix: release
  Id: brave-core-ext-tor
- Filter:
    Tag:
      Key: Name
      Value: cldoidikboihgcjfkhdeidbpclkineef/extension_1_0_11.crx
  Id: cldoidikboihgcjfkhdeidbpclkineef
- Filter:
    Tag:
      Key: Name
      Value: cpoalefficncklhjfpglfiplenlpccdb/extension_1_0_11.crx
  Id: cpoalefficncklhjfpglfiplenlpccdb

Metrics were just enabled, so we won't see any metrics associated with anything before today.

Please let me know if there are any additional questions.

[hspencer77]

Added metric filters per https://github.com/brave/devops/issues/3167. Filters are in place:

$ aws s3api list-bucket-metrics-configurations --bucket brave-ext --profile extensions-prod-devops-rw --output yaml
IsTruncated: false
MetricsConfigurationList:
- Filter:
    And:
      Prefix: release/cffkpbalmllkdoenhmdmpbkajipdjfam/
      Tags:
      - Key: version
        Value: Brave-Ad-Block-Updater-Default/latest
  Id: Brave-Ad-Block-Updater-Default
- Filter:
    And:
      Prefix: release/oofiananboodjbbmdelgdommihjbkfag/
      Tags:
      - Key: version
        Value: Brave-HTTPS-Everywhere-Updater/latest
  Id: Brave-HTTPS-Everywhere-Updater
- Filter:
    And:
      Prefix: release/afalakplffnnnlkncjhbmahjfjhmlkal/
      Tags:
      - Key: version
        Value: Brave-Local-Data-Files-Updater/latest
  Id: Brave-Local-Data-Files-Updater
- Filter:
    And:
      Prefix: release/gccbbckogglekeggclmmekihdgdpdgoe/
      Tags:
      - Key: version
        Value: Brave-NTP-sponsored-images/latest
  Id: Brave-NTP-sponsored-images
- Filter:
    And:
      Prefix: release/jicbkmdloagakknpihibphagfckhjdih/
      Tags:
      - Key: version
        Value: Brave-SpeedReader-Updater/latest
  Id: Brave-SpeedReader-Updater
- Filter:
    And:
      Prefix: release/odbfpeeihdkbihmopkbjmoonfanlbfcl/
      Tags:
      - Key: version
        Value: Crypto-Wallets/latest
  Id: Crypto-Wallets

$ aws s3api list-bucket-metrics-configurations --bucket brave-ext-tor --profile extensions-prod-devops-rw --output yaml
IsTruncated: false
MetricsConfigurationList:
- Filter:
    And:
      Prefix: release/biahpgbdmdkfgndcmfiipgcebobojjkp/
      Tags:
      - Key: version
        Value: Brave-Tor-Client-Updater-Linux/latest
  Id: Brave-Tor-Client-Updater-Linux
- Filter:
    And:
      Prefix: release/cldoidikboihgcjfkhdeidbpclkineef/
      Tags:
      - Key: version
        Value: Brave-Tor-Client-Updater-Mac/latest
  Id: Brave-Tor-Client-Updater-Mac
- Filter:
    And:
      Prefix: release/cpoalefficncklhjfpglfiplenlpccdb/
      Tags:
      - Key: version
        Value: Brave-Tor-Client-Updater-Windows/latest
  Id: Brave-Tor-Client-Updater-Windows