Performance regression in 0.8.0

[attila-kun]

Compiler version: 0.8.0-dev.2133+ad33e3483 OS: Linux DESKTOP-8AGJOHC 4.19.128-microsoft-standard ziglang/zig#1 SMP Tue Jun 23 12:58:10 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux (running in WSL 2)

Consider the following Zig program:

const std = @import("std");

fn init_array(points: []f32) void {
    var i: u64 = 0;
    while(i < points.len) : (i += 1) {
        points[i] = 5.3;
    }
}

fn sum_points(array: []f32) f64 {
    var sum: f64 = 0;
    for (array) |point| {
        sum = sum + point;
    }
    return sum;
}

pub fn main() anyerror!void {
    var array = try std.heap.page_allocator.alloc(f32, 1000 * 1000 * 30);
    init_array(array);

    var start = std.time.milliTimestamp();
    var sum = sum_points(array);
    var elapsed = std.time.milliTimestamp() - start;

    const stdout = std.io.getStdOut().writer();
    try stdout.print("Elapsed: {any} ms\n", .{elapsed});
    try stdout.print("Sum: {any}\n", .{sum});
}

Running the program with zig build run -Drelease-fast=true reports that executing the sum_points method takes 80ms. Running with zig build run -Drelease-safe=true or -Drelease-small=true reports that execution takes 23ms. The expected behaviour is that execution with the -Drelease-fast=true flag takes 23ms too.

I did a comparison of the LLVM IR generated for the different release modes for sum_points, but the IR was identical between them. This to me suggests that Zig must be invoking LLVM with different parameters when running with the -Drelease-fast=true flag. I suspect that either this mode of invocation, or LLVM executing this invocation has a performance regression now. Version 0.7.1 does not seem to be affected, execution in any of the release modes takes about 23ms.

A question came up in the Reddit thread whether these kind of regressions can be tested against.

[g-w1]

My first guess is https://github.com/ziglang/zig/pull/8390

[matu3ba]

@attila-kun Did you repeat the benchmark?

If one wants to be pedantic, one needs to store 1. the LLVM version and 2. the zig code together and build LLVM from source or use elfshaker. Then one can compare the LLVM diffs.

Point 1 sounds unfeasible to me without some sort of binary hosting solution for elfshaker, ie on the zig build server.

If you want to explore this idea further to estimate when it could make sense storing and comparing these things (likely in the release window of llvm), you can investigate first (and then implement an automatic solution).