Cache feature does not work

[Evilweed]

trafficstars

Describe the bug Running betterer with --cache flag multiple times does not actually make anything calculate faster. Despite code not being changed, all comands run and for example recompile Typescript.

To Reproduce If possible please share a repo with the issue, or another repository which demonstrates the issue. If possible please include the .betterer.ts file, and the .betterer.results file if applicable. At a minimum please share the command you are using to run Betterer.

Expected behavior Running command twice with --cache flag makes second run instant.

Screenshots

Versions (please complete the following information):

System:

• OS: macOS 11.6
• Memory: 35.78 MB / 16.00 GB

Binaries:

• Node: 16.13.0 - ~/.nvm/versions/node/v16.13.0/bin/node
• Yarn: 1.22.17 - /usr/local/bin/yarn
• npm: 8.1.0 - ~/.nvm/versions/node/v16.13.0/bin/npm

Languages:

• Bash: 3.2.57 - /bin/bash

Betterer:

• 5.0.0-alpha.0

Debug log:

[phenomnomnominal]

Yeah I guess this wasn't working properly in 5.0.0-alpha.0, so if you're still on that version, please update to latest and try it again.

[Evilweed]

@phenomnomnominal ok on latest version cache file is being created but there is no content in it:

import { typescript } from '@betterer/typescript'

export default {
    'stricter typescript compilation': () =>
        typescript('./tsconfig.json', {
            strict: true
        }).include('./**/*.ts'),
}

[phenomnomnominal]

Hmm, could you try it without the TypeScript test, but with the other tests?

[Giesch]

I'm not sure if it's the same thing Evilweed ran into, but we ran into a case where 'incremental' has to be disabled for betterer's caching to work as expected. With 'incremental: true', any change appeared to bust the entire cache. As a result, we've got 'incremental: true' in the real tsconfig.json and 'incremental: false' in the .betterer.ts tsconfig.

It looked like this was the culprit: https://github.com/phenomnomnominal/betterer/blob/1add290987b918e59c5202ddc69b3115287cbc81/packages/typescript/src/typescript.ts#L75

Does betterer interact with tsc's caching in a way that makes that line important for correctness?

[phenomnomnominal]

Going to be honest and say I don't entirely remember why that line was important! I believe the intention was that I thought tsc would do a better/more correct job of caching/doing a minimal compile, so if incremental was already enabled, we shouldn't try to be smarter? @Giesch is it actually working faster for you with the betterer cache vs incremental tsc?

[Giesch]

It was faster than tsc incremental, but we ended up removing caching and the precommit hook for unrelated reasons. There were correctness problems, but I don't know if that came from betterer or not - I'd have to dig back through CI records.

[Giesch]

If it was a problem with betterer's file caching, I think it would be either:

1. A change to an export in file A causes the cached file B to no longer compile, but B's cache wasn't busted, or
2. Changing between branches didn't clear the cache, so incorrect data was in the cache

But the more likely explanation in our case is that code that didn't pass the betterer ci check made it into master.

As far as caching goes, it was definitely faster; tsc might be able to more reliably address 1 & 2 above, or betterer might be addressing them well enough, I don't know. My only guess would be that 1 is trickier than it looks because it can involve type inference as well as exports/imports. Like, a change to a type can break files that import related functions but don't import the type itself. But the fact that we had a problem does not mean that 1 or 2 was the problem.

[jimmy-guzman]

Not sure if this is related but were facing something similar with betterer cache when changing branches. If we change to a new branch sometimes betterer cache fails with errors that seem like some types are missing from external libs.

Deleting the cache or bypassing the cache results in no errors.

We ended up not using the cache for typescript for now.