Files missing during compilation

[GWRon]

I got reports by users - and also saw it pretty often in VMs, think it happens more often on Windows and Mac than on Linux:

• downloading BlitzMax
• trying to compile something
• it compiles the modules
• and then reports a file is missing (inside the modules - eg a just compiled file - or precompilate of a module it just did)
• rerun the compilation and it either fails again - or sometimes just builds the previously failed file (think it depends on the stage)

As it happens more often on Windows and Mac - it means it runs (on my computers) either on an old Mac or inside a VM. This points a bit into a "collision" issue of threads. My computer is speedier and the discs are faster - so stuff just might not "collide".

Will try to find some time to investigate what happens

[GWRon]

I have had it happen a lot on Windows 10, Windows XP, Cent OS VMs and also on my normal Linux Mint machine. I even had it happen on VMs of github (github actions) so not strictly related to my hardware then.

[braxtonrivers]

I wonder if this could be what I have been noticing with my windows machines

I checked out the differences between git bcc, and bmk against my local copy and the only difference was my make.bmk where I added globals.SetOption(“cc_opts”, “optimization”, “-Os”) in default_cc_opts

Side note: I had to add it to make.bmk because for some reason setting anything in core.bmk appears to be ignored on windows, but then again this could be an isolated incident, as no one else has ever reported anything.

I was going to try out -flto but decided against it in case there were any side effects but even so using -Os produces a rather nice compact executable

[GWRon]

I had this happen on default installs. Means: either official releases or official releases updated with more current modules and bcc/bmk builds.

Your reported issue might be a separate one. That ignored stuff in bmk files should be reported individually so people can try to replicate that.

[GWRon]

Can it be that in multithreaded processing files are not finished writing but bmk already continued?

[ 11%] Processing:base.util.event.bmx
[ 27%] Processing:TVTower.bmx
[ 27%] Compiling:TVTower.bmx.debug.linux.x64.incbin2.c
[ 94%] Compiling:TVTower.bmx.console.debug.linux.x64.c
[100%] Linking:TVTower.debug
/usr/bin/ld: /home/ronny/Arbeit/Projekte/TVTower/Current/source/Dig/.bmx/base.util.event.bmx.debug.linux.x64.o kann nicht gefunden werden
collect2: error: ld returned 1 exit status
Build Error: Failed to link /home/ronny/Arbeit/Projekte/TVTower/Current/TVTower.debug
Process complete

Tried again:

Building TVTower
[ 27%] Processing:TVTower.bmx
[ 27%] Compiling:TVTower.bmx.debug.linux.x64.incbin2.c
[ 79%] Compiling:base.util.event.bmx.debug.linux.x64.c
[ 94%] Compiling:TVTower.bmx.console.debug.linux.x64.c
[100%] Linking:TVTower.debug
Executing:TVTower.debug

as if bmk does not wait for some files to be generated properly - it executes the command (eg calls GCC) but does not wait for the results to exist/processes to be finished.

[GWRon]

In commit https://github.com/bmx-ng/bmk/commit/2fdd34d76aede6bcb50f19681996f5a7b49f987b

@woollybah changed something which made something in the "diff" visible I was not aware of before:

		' get version
		If Platform() = "win32" Then
			process = CreateProcess(MinGWBinPath() + "/gcc.exe -dumpversion -dumpfullversion", HIDECONSOLE)
		Else	
			process = CreateProcess("gcc -dumpversion -dumpfullversion")
		End If
		Local s:String

		While True
			Delay 10

			Local line:String = process.pipe.ReadLine()

			If Not process.Status() And Not line Then
				Exit
			End If

After starting the process bmk starts a while loop - and in there it waits 10 milliseconds. it waits before even trying to read the first line. Means even if "start process + availability of first line" is happening in the first millisecond, it will ALWAYS wait 10 milliseconds.

So .. why was it done that way? why not delay "later" ?

OK ... so what made me stumble there? The time. In multithreaded builds time is crucial ... and timing even more. So my thoughts now were moving toward this idea and assumption: BMK checks FileTime() when it comes to "have I to recompile something"?

FileTime() by default returns mtime (last modified file time).

BCC uses SaveText() (from brl.mod/textstream.mod) to save interface files etc. SaveText() either uses SaveString() (Latin1 text) or WriteStream() when it comes to UTF.

Sidenode:

	Local TStream:TTextStream=TTextStream.Create( stream,format )
	TStream.WriteString str
	TStream.Close
	stream.Close
	Return True

BTW SaveText() uses an odd variable naming ("Local TStream" - it is the same as a type ... dunno if that can result in issues, it just does look odd).

While the file is written it can be accessed (regarding "mtime").

So what does that mean? Checks in "bcc" or "bmk" regarding "filetime" can lead to decisions based on "currently being written" files. This means it could eg. think "a file is there" which is in that very moment not complete. Same to say for any external command executed. So "GCC" might incorrectly spit out an error about a "not found file" which was "there" - but for the command it was an "empty file".

Consider this sample code here:

SuperStrict
Framework Brl.StandardIO
Import Brl.Threads
Import Brl.Filesystem
Import Brl.TextStream


Function ThreadFunc:Object( data:Object )
	Local s:TStream = WriteStream("filemtime.txt")
	'write "header"
	s.WriteByte $ef
	s.WriteByte $bb
	s.WriteByte $bf
	
	'wrap in a second stream
	Local s2:TTextStream=TTextStream.Create(s, ETextStreamFormat.UTF8)
	
    'do some work
    For Local i:Int = 1 To 5
		s2.WriteString("Hello~n")
		Delay(500)
    Next
	
	s2.Close()
	s.Close()
    Return "done."
End Function


If FileType("filemtime.txt") = FILETYPE_FILE
	Print "existing file mtime = " + FileTime("filemtime.txt")
EndIf
Local fileThread:TThread = CreateThread(ThreadFunc, "")


For Local i:Int = 1 To 5
	If FileType("filemtime.txt") = FILETYPE_FILE
		Print "Step " + i + ": file mtime = " + FileTime("filemtime.txt")
		
		Print "Content: " + LoadText("filemtime.txt")
	EndIf
	Delay(500)
Next

WaitThread( fileThread )
Print "File thread is done. file mtime = " + FileTime("filemtime.txt")
Print "Content: " + LoadText("filemtime.txt")

Running it results here in this output:

Building filemtime
Executing:filemtime.debug
existing file mtime = 1677831257
Step 1: file mtime = 1677831257
Content: 
Step 2: file mtime = 1677831331
Content: 
Step 3: file mtime = 1677831331
Content: 
Step 4: file mtime = 1677831331
Content: 
Step 5: file mtime = 1677831331
Content: 
File thread is done. file mtime = 1677831334
Content: Hello
Hello
Hello
Hello
Hello

See four important things (run the code twice to have the first part printed):

• if the file was existing, the first "filetime()" reports the modified time of "last run" albeit you might assume "hey I told someone to write a new file" -> this is what that "delay 10" in the BMK code in the top of this message was made me think about it
• the "filetime" changes" in that moment in which the "thread" (in this example) starts to write to the file
• the "content" of the file is empty until the thread closes the stream (btw filesize is 0 too)
• the "filemtime" of the written file is updated once the thread closes the stream

---

TL/DR:

The conclusion of my thought is: The discrepancy between "Filetime()" and an "actually useable file" might lead to the issue we see here: files not being found. We might consider to check the filesize too ... as we know that our files never should be empty.

BUT ... I am not sure how it works out with "append" file modes, yet I think we do not "append" in bmk or bcc.

[GWRon]

Continuing this the thought above - and tinkering about solutions:

Any thread writing "files" could write to a global "currentlyProcessedFiles"-container. Before starting it adds, and after finishing it removes (with mutex protection etc).

This way any thread can check if the file of interest is currently "rewritten" by someone else. External commands can only be started with specific filenames in the params, if the filenames do not exist in the "currentlyProcessedFiles"-container. This avoids to hand out "gcc" currently open files.

BUT ... it might be worth to check first what happens if you pass GCC files which are "currently open". Same to say about files, GCC creates (the o files?). Files might still be written by the OS while BMK or BCC think GCC is done (process ended).

I think Brucey knows best about the order of executed commands, when which file is created and when external processes are run etc. So maybe my "findings" are not suitable here.