fread() doesn't read (everything)?

[coderofsalvation]

lua (in redbean.c) seems to return variable string-lengths for:

popen("somecmd","r")
print( read("*a") )`

After some digging: fread() seems to return partial (buffered?) output when indirectly called by lua-code (redbean.c), I tried to reproduce it here:

https://gist.github.com/coderofsalvation/c60d4aa466d1cb137027e743f18bd924

On IRC RhodiumToad mentioned the possible culprit:

<RhodiumToad> | it does only one readv() call, it makes no attempt to loop if insufficient data was read from the pipe
<RhodiumToad> | lua interprets a short read result from fread() as meaning end-of-file
<RhodiumToad> | fread() is not permitted by spec to return a short result in the absence of error or EOF

I've tried to compare it to the musl-version of fread.c hoping that I could somehow shove in some for-loop, but I'm afraid I'm not the right person to do so (modifying fread is risky).

[pkulchenko]

I think this is a very interesting point and only @jart can shed some light on the implementation details for fread().

fread() is not permitted by spec to return a short result in the absence of error or EOF

It looks like Lua aborts read_all (which is what *a is using) as soon as there is a partial return, so its implementation seems to be consistent with this interpretation.

I don't see what in the fread spec may lead to this interpretation though. There is the following text -- "If an error occurs, or the end-of-file is reached, the return value is a short item count (or zero)." -- but I don't think it should be interpreted as if it's the only valid case when a short item count should be returned.

The musl implementation you referenced uses __toread to check if there is anything available before reading, but cosmo fread doesn't do that.

The readv documenation (which is what fread uses on Linux) explicitly states that "Note that it is not an error for a successful call to transfer fewer bytes than requested (see read(2) and write(2))." Similarly for read: "It is not an error if this number is smaller than the number of bytes requested; this may happen for example because fewer bytes are actually available right now (maybe because we were close to end-of-file, or because we are reading from a pipe, or from a terminal), or because read() was interrupted by a signal." and "On Linux, read() (and similar system calls) will transfer at most 0x7ffff000 (2,147,479,552) bytes, returning the number of bytes actually transferred. (This is true on both 32-bit and 64-bit systems.)", which clearly indicates that a partial read is a valid result.

Is the Lua code in the wrong here? I don't see why fread semantics need to be different from read and readv ones.

[ahgamut]

Is this related to #181?

[pkulchenko]

@ahgamut, doesn't seem to be, as #181 is about output/stdout buffering, but this is about (partial) reading.

[coderofsalvation]

reading buffered stdout could explain a variable (1st) chunksize (in the gist). But i do agree #181 seems different, since in our case an external program is producing the stdout (in #181 its an printf-call)

[coderofsalvation]

@jart maybe musl's fread.c is calling __toread so it can write 0 bytes to the filepointer, just to see whether it's still opened? (in case of a pause during buffered output). Is this something cosmopolitan's fread() should do as well?

[coderofsalvation]

btw. temporary (far from ideal) workarounds:

• just put os.execute('sleep 1s') before the fread call
• OR just redirect to file using os.execute( cmd ... ' > out.txt') and later open as textfile: f = io.open('out.txt','r') ; str = f:fread("*a") ; f:close())

[jart]

Could you share some kind of minimal example code (similar to what we have in the examples/ folder) that reproduces this? That'll help us determine if the issue is with Lua or the underlying libc. Our fread() implementation should only return completed operations. We even have code that continues on the EINTR condition, which other libc implementations tend not to do.

[coderofsalvation]

thank you for your reply, ok I will do this.

[coderofsalvation]

minimal example code which demostrates the issue : see this commit at PR https://github.com/jart/cosmopolitan/pull/325