kokkos-kernels icon indicating copy to clipboard operation
kokkos-kernels copied to clipboard
Published 2 months ago • verified publisher icon kokkos

improve context information for benchmarks

Open cz4rs opened this issue 2 years ago • 3 comments

I made a related comment over here: https://github.com/kokkos/kokkos-kernels/pull/1626#issuecomment-1402698018

This looks like a really great set of information. Would it be possible to include the following:

General:

  • compilers and versions used for the build
  • as precise a version as possible of all dependencies
    • Kokkos Core SHA
    • Kokkos Kernels SHA
    • any TPLs
    • Google Benchmark

OpenMP environment variables for OpenMP benchmarks (from querying the environment variables, or querying the OpenMP API programmatically)

  • OMP_NUM_THREADS
  • OMP_DYNAMIC
  • OMP_PROC_BIND
  • OMP_PLACES

For CUDA (from parsing nvidia-smi, or using the CUDA runtime API query functions)

  • Nvidia driver version (probably taken from nvidia-smi)
  • GPU name / model / revision (enough to unambiguously identify the SKU, this is more than just the GPU architecture)

For AMD / Intel GPUs:

  • analogous to CUDA

This information is good to have to reproduce any issues that arise.

Originally posted by @cwpearson in https://github.com/kokkos/kokkos-kernels/issues/1636#issuecomment-1405289099

Include more information in benchmark context.

Relevant part of the current JSON output. 
{
  "context": {
    "date": "2023-03-02T19:19:26+01:00",
    "host_name": "perrinel-MS-7C75",
    "executable": "/home/perrinel/Dev/kokkos/build_kernel_benchmark/perf_test/KokkosKernels_PerformanceTest_Benchmark",
    "num_cpus": 20,
    "mhz_per_cpu": 5300,
    "cpu_scaling_enabled": true,
    "caches": [
      {
        "type": "Data",
        "level": 1,
        "size": 32768,
        "num_sharing": 2
      },
      {
        "type": "Instruction",
        "level": 1,
        "size": 32768,
        "num_sharing": 2
      },
      {
        "type": "Unified",
        "level": 2,
        "size": 262144,
        "num_sharing": 2
      },
      {
        "type": "Unified",
        "level": 3,
        "size": 20971520,
        "num_sharing": 20
      }
    ],
    "load_avg": [3.13,1.83,0.87],
    "library_build_type": "debug",
    "CPU architecture": "none",
    "Default Device": "N6Kokkos6SerialE",
    "GPU architecture": "none",
    "KOKKOSKERNELS_ENABLE_TPL_ARMPL": "no",
    "KOKKOSKERNELS_ENABLE_TPL_BLAS": "no",
    "KOKKOSKERNELS_ENABLE_TPL_CBLAS": "no",
    "KOKKOSKERNELS_ENABLE_TPL_CHOLMOD": "no",
    "KOKKOSKERNELS_ENABLE_TPL_CUBLAS": "no",
    "KOKKOSKERNELS_ENABLE_TPL_CUSPARSE": "no",
    "KOKKOSKERNELS_ENABLE_TPL_LAPACK": "no",
    "KOKKOSKERNELS_ENABLE_TPL_LAPACKE": "no",
    "KOKKOSKERNELS_ENABLE_TPL_MAGMA": "no",
    "KOKKOSKERNELS_ENABLE_TPL_METIS": "no",
    "KOKKOSKERNELS_ENABLE_TPL_MKL": "no",
    "KOKKOSKERNELS_ENABLE_TPL_ROCBLAS": "no",
    "KOKKOSKERNELS_ENABLE_TPL_ROCSPARSE": "no",
    "KOKKOSKERNELS_ENABLE_TPL_SUPERLU": "no",
    "KOKKOS_COMPILER_GNU": "940",
    "KOKKOS_ENABLE_ASM": "yes",
    "KOKKOS_ENABLE_CXX17": "yes",
    "KOKKOS_ENABLE_CXX20": "no",
    "KOKKOS_ENABLE_CXX23": "no",
    "KOKKOS_ENABLE_DEBUG_BOUNDS_CHECK": "no",
    "KOKKOS_ENABLE_GNU_ATOMICS": "no",
    "KOKKOS_ENABLE_HBWSPACE": "no",
    "KOKKOS_ENABLE_HWLOC": "no",
    "KOKKOS_ENABLE_INTEL_ATOMICS": "no",
    "KOKKOS_ENABLE_INTEL_MM_ALLOC": "no",
    "KOKKOS_ENABLE_LIBDL": "yes",
    "KOKKOS_ENABLE_LIBRT": "no",
    "KOKKOS_ENABLE_PRAGMA_IVDEP": "no",
    "KOKKOS_ENABLE_PRAGMA_LOOPCOUNT": "no",
    "KOKKOS_ENABLE_PRAGMA_UNROLL": "no",
    "KOKKOS_ENABLE_PRAGMA_VECTOR": "no",
    "KOKKOS_ENABLE_SERIAL": "yes",
    "KOKKOS_ENABLE_SERIAL_ATOMICS": "no",
    "KOKKOS_ENABLE_WINDOWS_ATOMICS": "no",
    "Kokkos Version": "4.0.99",
    "KokkosKernels Version": "4.0.99"
  },
  "benchmarks": [
    (benchmark results...)
  ]
}
  • [x] 1. Compilers and versions used for the build (already have from core)
    • Example: 
      "KOKKOS_COMPILER_GNU": "940"

      "KOKKOS_COMPILER_CLANG": "1507"
  • [x] 2. Precise versions of all libraries
    • [x] 2.1 Kokkos Core SHA (already available in Kokkos Core benchmarking code, but not exposed in Kokkos::print_configuration)
      • Example:
      "GIT_BRANCH": "HEAD",
"GIT_CLEAN_STATUS": "CLEAN",
"GIT_COMMIT_DATE": "2023-03-03T10:10:33-05:00",
"GIT_COMMIT_DESCRIPTION": "Merge pull request #5912 from ndellingwood/release-candidate-4.0.0",
"GIT_COMMIT_HASH": "aa1f48f31",
(...)
"Kokkos Version": "4.0.0"
    • [x] 2.2 Kokkos Kernels SHA
    • https://github.com/kokkos/kokkos-kernels/pull/1722
    • [x] 2.3 Any TPLs (yes/no already available in KokkosKernels::print_configuration)
      • Example:
      "KOKKOSKERNELS_ENABLE_TPL_ARMPL": "no",
"KOKKOSKERNELS_ENABLE_TPL_BLAS": "no",
"KOKKOSKERNELS_ENABLE_TPL_CBLAS": "no",
"KOKKOSKERNELS_ENABLE_TPL_CHOLMOD": "no",
"KOKKOSKERNELS_ENABLE_TPL_CUBLAS": "no",
"KOKKOSKERNELS_ENABLE_TPL_CUSPARSE": "no",
"KOKKOSKERNELS_ENABLE_TPL_LAPACK": "no",
"KOKKOSKERNELS_ENABLE_TPL_LAPACKE": "no",
"KOKKOSKERNELS_ENABLE_TPL_MAGMA": "no",
"KOKKOSKERNELS_ENABLE_TPL_METIS": "no",
"KOKKOSKERNELS_ENABLE_TPL_MKL": "no",
"KOKKOSKERNELS_ENABLE_TPL_ROCBLAS": "no",
"KOKKOSKERNELS_ENABLE_TPL_ROCSPARSE": "no",
"KOKKOSKERNELS_ENABLE_TPL_SUPERLU": "no",
      Providing precise version information: https://github.com/kokkos/kokkos-kernels/pull/1693
  • [x] 3. Google Benchmark SHA or version
    • https://github.com/kokkos/kokkos-kernels/pull/1750
    • Example: GOOGLE_BENCHMARK_VERSION: 1.6.2
  • [x] 4. OpenMP environment variables for OpenMP benchmarks
    (potentially to be done in Kokkos Core) (from querying the environment variables, or querying the OpenMP API programmatically)
    • [x] OMP_NUM_THREADS
    • [x] OMP_DYNAMIC
    • [x] OMP_PROC_BIND
    • [x] OMP_PLACES
    • #1789
    • information currently displayed in Kokkos::print_configuration: 
      "Kokkos": "OpenMP thread_pool_topology[ 1 x 8 x 1 ]"
    • those are [numa_count x core_per_numa x thread_per_core], see relevant code in Kokkos Core
  • [ ] 5. For CUDA (from parsing nvidia-smi, or using the CUDA runtime API query functions) (potentially to be done in Kokkos Core)
    • [ ] 5.1 Nvidia driver version (probably taken from nvidia-smi)
    • [ ] 5.2 GPU name / model / revision (enough to unambiguously identify the SKU, this is more than just the GPU architecture)
  • [ ] 6. For AMD / Intel GPUs: analogous to CUDA (potentially to be done in Kokkos Core)

cz4rs avatar Mar 06 '23 09:03 cz4rs

My suggestion for priority is:

1. compilers and versions used for the build (already have from core) 

2. Kokkos Core SHA (already avail), Kokkos Kernels SHA (make PR in kernels)

3. Google Benchmark SHA or version  (make PR in kernels)

4. OpenMP environment variables for OpenMP benchmarks  
    (potentially to be done in CORE)
    (from querying the environment variables, or querying the OpenMP API programmatically)

 	OMP_NUM_THREADS, OMP_DYNAMIC, OMP_PROC_BIND, OMP_PLACES

5. For CUDA (from parsing nvidia-smi, or using the CUDA runtime API query functions) 
    (potentially to be done in CORE)
    Nvidia driver version (probably taken from nvidia-smi)
    GPU name / model / revision (enough to unambiguously identify the SKU, this is more than just the GPU architecture)

6. For AMD / Intel GPUs: analogous to CUDA (potentially to be done in CORE)

fnrizzi avatar Mar 06 '23 09:03 fnrizzi

@lucbv @cwpearson are you ok with the priority above?

fnrizzi avatar Mar 06 '23 10:03 fnrizzi

Sure that sounds reasonable. I would like to point out that some of the features listed above are implemented (partially and/or fully) here: https://github.com/kokkos/kokkos-tools/blob/ecp-kpp3/profiling/kpp3-verifier/kp_ecp_kpp3.cpp see this function for example: extract_gpuinfo()

lucbv avatar Mar 07 '23 05:03 lucbv