Dependency file parsing assumes one file per line

[matthijskooijman]

Apparently, the parsing of .d files assumes there is a single file on each line of the file. However, in some cases, presumably when filenames are short, there might be more than one in a single line. Consider this example from this report.

C:\Users\Graham\AppData\Local\Temp\builde33ce9ddee6346054afe0349d71c85f0.tmp\libraries\UTFT\UTFT.cpp.o: \
 F:\Arduino\libraries\UTFT\UTFT.cpp F:\Arduino\libraries\UTFT\UTFT.h \
 C:\Users\Graham\AppData\Local\Arduino15\packages\arduino\hardware\sam\1.6.7\cores\arduino/Arduino.h \
 (rest of the file removed)

The relevant source is here: https://github.com/arduino/arduino-builder/blob/master/src/arduino.cc/builder/builder_utils/utils.go#L222-L233

I guess the solution here is to do more proper parsing of the file, such as removing any escaped newlines, and then splitting the result into filenames (taking into account escaped spaces). I'm not entirely sure how reliable this can be done, since GNU make is known to be bad in handling spaces in filenames (though I think it works ok as long as no variables are involved).

[ghlawrence2000]

Ok, thanks Matthijs.

[ghlawrence2000]

Time passes by........... and nothing happens.........?

[PaulStoffregen]

This worked properly when I implemented in Java.

But to be fair, I didn't handle spaces in pathnames properly until much later. So many little details to get right...

[ghlawrence2000]

Hi Paul, Is that something I can do?

[cmaglie]

IMHO the difference between the original java version and this one is in the loop here:

    for _, row := range rows {
        depStat, err := os.Stat(row)
        if err != nil && !os.IsNotExist(err) {
            return false, i18n.WrapError(err)
        }
        if os.IsNotExist(err) {
            return false, nil
        }
        if depStat.ModTime().After(objectFileStat.ModTime()) {
            return false, nil
        }
    }

IIRC the java version returned false if the file is not found, this one instead throws an error. Probably the java version will always trigger a full rebuild of the sketch in this case, but probably this is better and more conservative than throwing an error, I'll make a PR for this in a moment.

The long term solution is to properly parse the .d file, that may have more than one file per line, this could be tricky because the file are separated by space " " but a file may contains spaces escaped by backslash "\ ", so we could not simply split the line by spaces.

[cmaglie]

Well, to be precise, there is already the condition if os.IsNotExist(err) { that should handle the "file not found" error, the problem is that in this case the error is different we have a:

GetFileAttributesEx F:\Arduino\libraries\UTFT\UTFT.cpp F:\Arduino\libraries\UTFT\UTFT.h: The filename, directory name, or volume label syntax is incorrect.

so in some way the parsed filename F:\Arduino\libraries\UTFT\UTFT.cpp F:\Arduino\libraries\UTFT\UTFT.h is able to trigger this subtle error instead of "file not found".

[matthijskooijman]

Wouldn't it be better to keep this issue open until it is really properly fixed? Also, I think that a proper fix might share some code with a fix for this comment: https://github.com/arduino/arduino-builder/pull/131#discussion_r61214796

[ghlawrence2000]

Hi mathhijs,

The supplied arduino-builder.exe cures the problem, but as suspected, rebuilds all each time. Are you talking about a fix that doesn't rebuild if there is a problem? That would be better.

[cmaglie]

Wouldn't it be better to keep this issue open until it is really properly fixed

yes, github auto-closed the issue becuase of my comment on the PR.

[ghlawrence2000]

Hello guys, we got a new problem.....

Arduino: 1.6.8 (Windows 7), Board: "Arduino Due (Programming Port)"

X:\MyDocuments\Downloads\arduino-1.6.8\arduino-builder -dump-prefs -logger=machine -hardware "X:\MyDocuments\Downloads\arduino-1.6.8\hardware" -hardware "C:\Users\Graham\AppData\Local\Arduino15\packages" -tools "X:\MyDocuments\Downloads\arduino-1.6.8\tools-builder" -tools "X:\MyDocuments\Downloads\arduino-1.6.8\hardware\tools\avr" -tools "C:\Users\Graham\AppData\Local\Arduino15\packages" -built-in-libraries "X:\MyDocuments\Downloads\arduino-1.6.8\libraries" -libraries "F:\Arduino\libraries" -fqbn=arduino:sam:arduino_due_x_dbg -vid-pid=0X2341_0X003D -ide-version=10608 -build-path "C:\Users\Graham\AppData\Local\Temp\build381310ddf6281a70c5b308691257b81c.tmp" -warnings=all -prefs=build.warn_data_percentage=75 -verbose "C:\Users\Graham\AppData\Local\Temp\arduino_modified_sketch_163023\UTFT_GHL_Demo_Images.ino"
X:\MyDocuments\Downloads\arduino-1.6.8\arduino-builder -compile -logger=machine -hardware "X:\MyDocuments\Downloads\arduino-1.6.8\hardware" -hardware "C:\Users\Graham\AppData\Local\Arduino15\packages" -tools "X:\MyDocuments\Downloads\arduino-1.6.8\tools-builder" -tools "X:\MyDocuments\Downloads\arduino-1.6.8\hardware\tools\avr" -tools "C:\Users\Graham\AppData\Local\Arduino15\packages" -built-in-libraries "X:\MyDocuments\Downloads\arduino-1.6.8\libraries" -libraries "F:\Arduino\libraries" -fqbn=arduino:sam:arduino_due_x_dbg -vid-pid=0X2341_0X003D -ide-version=10608 -build-path "C:\Users\Graham\AppData\Local\Temp\build381310ddf6281a70c5b308691257b81c.tmp" -warnings=all -prefs=build.warn_data_percentage=75 -verbose "C:\Users\Graham\AppData\Local\Temp\arduino_modified_sketch_163023\UTFT_GHL_Demo_Images.ino"
panic: regexp: Compile("(?m)^.*C:\\Users\\Graham\\AppData\\Local\\Temp\\arduino_modified_sketch_163023\\UTFT_GHL_Demo_Images.ino.*$[\r\n]+"): error parsing regexp: invalid escape sequence: `\U`

goroutine 1 [running]:
regexp.MustCompile(0x12256d90, 0x69, 0x12238060)
    /opt/go/src/regexp/regexp.go:221 +0xfc
arduino.cc/builder.(*WipeoutBuildPathIfBuildOptionsChanged).Run(0x63bc00, 0x1220a000, 0x0, 0x0)
    /home/jenkins/jenkins/jobs/arduino-builder-pr-builder/workspace/src/arduino.cc/builder/wipeout_build_path_if_build_options_changed.go:56 +0x1a9
arduino.cc/builder.(*ContainerBuildOptions).Run(0x63bc00, 0x1220a000, 0x0, 0x0)
    /home/jenkins/jenkins/jobs/arduino-builder-pr-builder/workspace/src/arduino.cc/builder/container_build_options.go:49 +0x1ff
arduino.cc/builder.runCommands(0x1220a000, 0x12217d7c, 0x1c, 0x1c, 0x121dc301, 0x0, 0x0)
    /home/jenkins/jenkins/jobs/arduino-builder-pr-builder/workspace/src/arduino.cc/builder/builder.go:181 +0xeb
arduino.cc/builder.(*Builder).Run(0x12217e70, 0x1220a000, 0x0, 0x0)
    /home/jenkins/jenkins/jobs/arduino-builder-pr-builder/workspace/src/arduino.cc/builder/builder.go:116 +0xb67
arduino.cc/builder.RunBuilder(0x1220a000, 0x0, 0x0)
    /home/jenkins/jenkins/jobs/arduino-builder-pr-builder/workspace/src/arduino.cc/builder/builder.go:212 +0x44
main.main()
    /home/jenkins/jenkins/jobs/arduino-builder-pr-builder/workspace/src/arduino.cc/arduino-builder/main.go:316 +0x1318
arduino-builder returned 2
Error compiling for board Arduino Due (Programming Port).

Thanks,

Graham

[facchinm]

Hi @ghlawrence2000 , the problem you are experiencing was solved by https://github.com/arduino/arduino-builder/commit/67245407aa80e4143b555cdf7a2853a5006dd9b9, latest version of the builder (1.3.18) and IDE nighlty both contain the fix!

[ghlawrence2000]

@facchinm , @cmaglie I can confirm that has fixed the problems, and does not do a full rebuild which is much better.

Thanks.