Track build & release metrics

[aajtodd]

It would be nice to track build and release metrics over time such as:

• artifact size
• benchmark results
• build times
• LOC

This is a placeholder ticket for doing this work. It's possible some of this will be available from Catapult so we should do some investigating. It's also likely this task can be decomposed into smaller tasks.

Below are some ideas I've thrown together for inspiration (not necessarily prescriptive).

Tracking artifact size with gradle task

    tasks.findByName("jvmJar")?.let {
        println("${project.name} has jvmJar")
        tasks.create("metrics") {
            val jvmJar = tasks.getByName<Jar>("jvmJar")
            dependsOn(jvmJar)
            val reportsBuildDir = Paths.get(project.buildDir.absolutePath, "reports", "metrics").toFile()
            val csv = Paths.get(reportsBuildDir.path, "artifacts.csv").toFile()
            outputs.file(csv)

            doLast {
                reportsBuildDir.mkdirs()
                val length = jvmJar.archiveFile.get().asFile.length()
                println("${jvmJar.archiveBaseName.get()} is $length bytes")
                csv.writeText("${jvmJar.archiveBaseName.get()},$length\n")
            }
        }
    }

Top level task to aggregate:

tasks.create("metrics") {
    val rootMetrics = this
    val outputFiles = mutableListOf<File>()
    subprojects {
        afterEvaluate {
            val metricsTask = tasks.findByName("metrics")
            if (metricsTask != null) {
                rootMetrics.dependsOn(metricsTask)
                outputFiles.add(metricsTask.outputs.files.singleFile)
            }
        }
    }

    val reportsBuildDir = Paths.get(project.buildDir.absolutePath, "reports", "metrics").toFile()
    val csv = Paths.get(reportsBuildDir.path, "artifacts.csv").toFile()
    outputs.file(csv)
    doLast {
        val content = outputFiles.joinToString(separator = "") { it.readText() }
        csv.writeText(content)
    }
}

This generates a file:

aws-config,222163
aws-core,9783
aws-endpoint,14982
aws-http,60454
aws-signing,50560
aws-types,17076
crt-util,31746
http-client-engine-crt,56920
testing,6690
aws-event-stream,44084
aws-json-protocols,16550
aws-xml-protocols,44620

NOTE: This is some quick and dirty gradle I through together. More than likely we'll want this to be a plugin to handle multiplatform artifacts, etc.

Tracking in cloudwatch metrics

    aws cloudwatch put-metric-data --namespace "KotlinSdk" --metric-data file://metrics.json

file: metrics.json

[
    {
        "MetricName": "ArtifactSize",
        "Timestamp": "2021-11-12T12:00:00Z",
        "Value": 2.27,
        "Unit": "KiloBytes",
        "Dimensions": [
            {
                "Name": "Artifact",
                "Value": "aws-common"
            }
        ]
    },
    ...
]

• Dimensions probably need to take into account KMP, so aws-common-jvm for instance vs aws-common-linux64
• Alternatively add a dimension KmpTarget=jvm
• We could publish this data as part of the batch service check job or at release time (might be a bit too late to do anything though)