f3probe writes multiple MiB of data

[jowagner]

Adding diagnostic output to write_blocks() and also simply looking at an I/O monitor, there clearly is a lot of data being written to the flash disk while running f3probe. The readme says it is the "fastest" and "only writes what's necessary". I'd think the following procedure would suffice. What would be wrong with this approach, assuming like f3probe that fake flash simply discards one or more top address bits?

1. Write 1 block with unique contents to the last block of the fake disk, i.e. at position fake_size_in_blocks - 1. Assign this position to a.
2. Since we assume that fake flash has at least 1 block of real storage we do not need to test that the block can be read back at position a. It will, unless there is some other damage and we are not interested in testing for this here.
3. Set n = log_2(floor(a)).
4. Read at position a - 2^n, i.e. clearing the top non-zero address bit.
5. If our special block comes back record that address bit n is not connected.
6. If a is not 2^(n+1) - 1, write our unique contents also to position 2^n - 1.
7. Set a = 2^n - 1, i.e. the last block of the next smaller power of 2 fake drive size.
8. Go to step 3 if a is not 0 (or above some other reasonable threshold, e.g. 2048 if assuming there is at least 1 MiB of real storage)
9. Report fake flash if at least 1 bit is not connected
10. Report useable size as block_size * 2^m where m is the lowest address bit not connected

[AltraMayor]

Your proposed algorithm is trying to find the wrapping edge; this is already done in find_wrap(). But it is not a complete algorithm to identify fake drives. For example, it is easily fooled when a fake drive has a cache; this is a common feature. The high-level perspective of what f3probe does is summarized in probe_device().

[jowagner]

Do you mean in case the cache is not empty at the start? I don't see a problem when assuming a fresh cache: The algorithm never reads back from the same location it previously has written to or read from. A cache that uses all address bits therefore should not change the behaviour. If there is a cache that ignores one or more of the top address bits this may lead to additional unconnected address lines to be detected but that's not wrong if we extent the meaning of "storage is wrapped" to "storage may be wrapped when in cache". How would the algorithm fail?

If, on the other hand, one is concerned about cache contents from before running f3probe, e.g. the operating system may have read the backup gpt from the end of the disk or the disk may have a persistent cache that doesn't depend on power from the host computer, then the success of removing cache entries by writing large amounts of data depends on the cache replacement policy, which is not known. Since f3probe only validates 64 blocks (=32 KiB) after each large write, the fake disk could be lucky to have kept exactly these blocks. To protect against this, f3probe should validate a large number of blocks, at least a few MiB exceeding the expected cache size. Do you assume a particular cache replacement policy such as "least recently used"? The output line "Approximate cache size" suggests that f3probe also tries to detect the size of the cache. Is this based on measurements of the read speed for reading recently written data of different sizes, assuming a cache policy that keeps recently written data in cache?

[AltraMayor]

The cache I'm referring to is implemented inside of the drive and the problem I'm pointing to doesn't depend on the content of the cache at the start of the test. To have a concrete example of how problematic this cache can be, search for "The devil drive" in f3probe.c. The associated cases from the unit test will show you the problem in a black-white form. In plain English, your algorithm may only talk to the cache and never test the memory behind it.

You find the algorithm that f3probe uses to estimate the cache size and to not read all it writes in find_cache_size(). I do not assume cache replacement policies, but I do assume that the cache is no larger than 1GB; all drives that I've come across have a cache smaller than this.

[jowagner]

I'll need to look at the devil drive when I've more time. I'm not convinced the cache policy doesn't matter, though I'd agree in practice one will not find a drive with a cache policy that fools your size detector because flash drives tend to have low write speed making it attractive to use all cache memory for caching writes.