Suggestion: Pack frontend and backend in one file with Appveyor

[Biswa96]

I've seen that the current Appveyor.yml files compile frontend and backend separately. There is a procedure which can pack both files in one tar.gz container. All credit goes to @IlyaFinkelshteyn for sharing this example share-artifacts-between-jobs.yml. Here is an example of modified version of that Appveyor.yml file:

version: 1.0.{build}
skip_non_tags: true
image:
- Ubuntu 
- Visual Studio 2017 
platform: x64 

environment:
  cygwin_ver: '2.10.0-1'
  backendJob: 'image: Ubuntu'
  package: 'wslbridge-0.2.5-dev-cygwin64.tar.gz'

install:
- sh: |
    sudo apt-get update -qq 
    sudo apt-get install gcc g++ make -qq 
    git clone --quiet https://github.com/rprichard/wslbridge 
    make --directory=wslbridge/backend 

- cmd: |
    git clone --quiet https://github.com/rprichard/wslbridge 
    set PATH=C:\cygwin64\bin;%PATH% 
    make --directory=wslbridge\frontend 

before_build:
- ps: |
    if ($env:APPVEYOR_JOB_NAME -eq $env:backendJob) {
        Get-ChildItem .\wslbridge\out\wslbridge-backend | % { Push-AppveyorArtifact $_.FullName -FileName $_.Name } 
    }

    if ($env:APPVEYOR_JOB_NAME -ne $env:backendJob) {
        $headers = @{ "Content-type" = "application/json" }
        $ProjectUri="https://ci.appveyor.com/api/projects/$env:APPVEYOR_ACCOUNT_NAME/$env:APPVEYOR_PROJECT_SLUG" 
        $project = Invoke-RestMethod -Uri $ProjectUri -Headers $headers -Method GET 
        $jobToWaitJson = $project.build.jobs | where {$_.name -eq $env:backendJob} 
        $jobToWaitId = $jobToWaitJson.jobId; 

        Start-FileDownload "https://ci.appveyor.com/api/buildjobs/$jobToWaitId/artifacts/wslbridge-backend" 
        Move-Item -path .\wslbridge-backend .\wslbridge\out\wslbridge-backend 

        $CygwinLink="http://mirrors.kernel.org/sourceware/cygwin/x86_64/release/cygwin/cygwin-$env:cygwin_ver.tar.xz"
        $CygwinTar="cygwin-$env:cygwin_ver.tar.xz" 
        Invoke-WebRequest $CygwinLink -OutFile $CygwinTar 
    }

build_script:
- cmd: |
    7z x cygwin-%cygwin_ver%.tar.xz 
    7z e cygwin-%cygwin_ver%.tar usr\bin\cygwin1.dll -o"%APPVEYOR_BUILD_FOLDER%\wslbridge\out" 
    cd wslbridge\out
    tar.exe -czf %package% * 
    bash.exe -c "shasum %package%"

artifacts:
  - path: wslbridge\out\*.tar.gz 

This configuration does: Build backend in Ubuntu image > Upload it in artifacts > Build frontend in Windows > Download backend > Download cygwin1.dll > pack all three files in tar.gz. This may be converted to python. I shall try that.

[rprichard]

Yeah, having the second job wait until the first one has finished probably works. (It assumes that AppVeyor starts jobs in a consistent order, and it probably does.) I don't expect to work on it until this weekend.

[Biswa96]

Remove .\wslbridge from every line if you add this in wslbridge repository itself. I've not used the matrix building environment. Also you can add the time to wait. I've remove that because I place the Ubuntu image before Windows. So backend will be compiled before frontend.

[Biswa96]

BTW, is it possible to replace bash.exe with wsl.exe (optional) in frontend? Because this article WSL interoperability with Windows, says

bash.exe has been deprecated and replaced with wsl.exe.

I tried this patch in frontend as per this article:

index 6c13ff5..4590647 100644
--- a/frontend/wslbridge.cc
+++ b/frontend/wslbridge.cc
@@ -1120,7 +1120,7 @@ int main(int argc, char *argv[]) {
         errorSocket = std::unique_ptr<Socket>(new Socket);
     }
 
-    const auto bashPath = findSystemProgram(L"bash.exe");
+    const auto bashPath = findSystemProgram(L"wsl.exe");
     const auto backendPathInfo = normalizePath(findBackendProgram(customBackendPath));
     const auto backendPathWin = backendPathInfo.first;
     const auto fsname = backendPathInfo.second;
@@ -1189,7 +1189,6 @@ int main(int argc, char *argv[]) {
         cmdLine.append(L" ");
         cmdLine.append(mbsToWcs(distroGuid));
     }
-    cmdLine.append(L" -c ");
     appendBashArg(cmdLine, bashCmdLine);
 
     const auto outputPipe = createPipe();

But that patch shows this error:

note: backend error output: /bin/bash: No such file or directory

[rprichard]

BTW, is it possible to replace bash.exe with wsl.exe (optional) in frontend? Because this article WSL interoperability with Windows, says

I don't see the need to use wsl.exe instead of bash.exe. I think there's a long-term-support Windows release that only has bash.exe, so it would add two code paths.

note: backend error output: /bin/bash: No such file or directory

I would guess it's an escaping issue. wsl.exe and bash.exe parse their command lines differently:

https://github.com/Microsoft/WSL/issues/2348#issuecomment-318210848

https://github.com/rprichard/wslbridge/commit/8f24626435375a3c2db0841023df0161dabbdbc8#diff-f1b3322d0672e7bc690f3b5246288c17R1155

[fpqc]

@rprichard there is no current LTS release containing WSL at all. It was temporarily enabled in 1607 but has since been patched out (since including a bleeding-edge beta feature in LTS at the time was absurd). The new LTS should release as 1809 later this year and will have wsl.exe.

[Biswa96]

Will 1809 have wsl.exe only? Where did you get that?

[fpqc]

@biswa96 LTS releases every two years. The next Win10 LTS will be 1809. Bash.exe is deprecated, but it will be retained for compatibility. My point was that there is no point in targeting 1607 (which only has bash.exe) since it doesn't support WSL on LTSB, and all currently-supported and future releases in which WSL is enabled contain and will contain wsl.exe.