Option to publish via FTP?

[Nantris]

I wonder if this is already possible and I'm overlooking it, or otherwise if there's any interest in adding it?

PS: I know this would be better suited as a discussion than an issue (as was my last issue.) Maybe it would be worth considering enabling discussions for the repo?

[mmaietta]

Hmm would that be something like this with a different publisher type? https://gist.github.com/josemariagarcia95/250acdc8171c0e7b3d92d73cf361fd00 I know we already allow custom publisher providers, but could be implemented directly into electron-builder

I had discussions enabled previously but wasn't able to actively maintain it since it was only me to support and wasn't getting notifications for it.

[Nantris]

I think something like that would do the trick if it were to be implemented. It would be nice to have it integrated into the process if possible rather than as an additional task. I guess it might be a little more complex since a lot of FTP depends on SSH keys these days?

Our use case is that we serve updates from S3, but we host the installers for new users on our own server since we're already paying for that bandwidth anyway.

---

Definitely understandable about discussions. Honestly it's amazing how much you manage - building and publishing for all these platforms is no small feat. Thank you for your incredible work.

[mmaietta]

I'm swamped with work + another couple of GH issues atm, so I'm not sure when I can get to this tbh.

Would you be willing to be a guinea pig for the project though? I can go a patch-package route for you to test with. I'll also see if I can figure out how to use a local FTP server for testability. Honestly, allowing a local FTP server would probably significantly improve my auto-update debugging iterations, so maybe I'll pick it up sooner than later

[Nantris]

Totally understandable!

Would you be willing to be a guinea pig for the project though?

Certainly!

We can only use the SSH-based connection. Besides that limitation, I'm more than happy to test any possible changes via patch-package. If/when you get a chance to toy with this just ping me and I'll give them a try.

[mmaietta]

Excellent!

Oomph, let me get a standard FTP(S) set up and then I can figure out how a ssh-based implementation can be approached 😅

Already implemented a local FTP Publisher setup last night, with unit tests successfully running with an FTP-based Updater. Implemented it in hopes I can test my other updater-reported issues (higher priority) with faster iterations

[mmaietta]

So it doesn't seem that FTP supports multiple async uploads (or at least the basic-ftp package can't)? I'm able to get it working on 1 payload, but electron-builder uses an AsyncTaskManager for batch uploads (otherwise I'm sure the upload process could take much longer). Unless I can figure out a way to queue the Promises, I'm not sure if it'll be possible to implement? ☹️

[Nantris]

@mmaietta thanks again for looking into this!

I'm not sure I follow entirely (and even less sure I could be of any help) but if you do think it's worthwhile to explain anyway I'm happy to take a look and to consult GPT4.

It sounds like it could theoretically be made to work with your proof of concept method, albeit slowly? I still think that would have value personally but it's a judgement call on your end whether it's worth it.

---

I also came across this package. I'm not sure it's of any help at all, especially since it doesn't seem likely to support FTP (only SFTP.) Even if it's not directly useful, maybe this section of the readme could provide some conceptual inspiration: https://www.npmjs.com/package/ssh2-sftp-client#org2511fe9

[mmaietta]

Okay, maybe this can get you started in the right direction. From what I can tell, you can dynamically load a publisher in, bypassing scheme validation? You'll need to confirm though as I was using rsync to transfer all my electron-builder changes into my test project. Sneaky command for testing electron-builder changes locally (both checked out in ~/Development dir). Executed from the test project root.

alias resync="rsync -upaRv --include='*.js' --include='*.nsi' --include='*.json' --include='*/' --include='*.py*' --include='*.tiff' --exclude='*'  ~/Development/electron-builder/packages/./* node_modules/"

This used basic-ftp though. Maybe you could get something working with ssh, but the publisher would work the same probably

Provide a

publish: [{ provider: "ftp", host, port, user, password }, s3PublishConfig ], //  provider can be anything you want to name the file by

Loaded dynamically via: https://github.com/electron-userland/electron-builder/blob/5681777a808d49756f3a95d18cc589218be44878/packages/app-builder-lib/src/publish/PublishManager.ts#L344-L351

<buildResourcesDir>/electron-publisher-ftp.js

import { log } from "builder-util"
import { PublishContext, Publisher, UploadTask } from "electron-publish"
import { FtpOptions } from "builder-util-runtime/out/publishOptions"
import * as ftp from "basic-ftp"
import { basename, join } from "path"
import { stat, Stats } from "fs-extra"
import { safeStringifyJson } from "builder-util-runtime/out"
import { Readable } from "node:stream"
import { ProgressBar } from "electron-publish/out/progress"

export class FtpPublisher extends Publisher {
  readonly providerName = "ftp"

  private readonly client: ftp.Client
  private readonly username: string | undefined
  private readonly token: string | undefined

  constructor(context: PublishContext, private readonly info: FtpOptions, private readonly useSafeArtifactName = false) {
    super(context)

    this.username = info.user || process.env.FTP_SERVER_USER || undefined
    this.token = info.password || process.env.FTP_SERVER_PASSWORD || undefined

    this.client = new ftp.Client(this.info.timeout || undefined)
    this.client.ftp.verbose = true // log.isDebugEnabled
  }

  async upload(task: UploadTask): Promise<string> {
    const fileName = (this.useSafeArtifactName ? task.safeArtifactName : null) || basename(task.file)
    log.debug(task, "FTP upload task")

    log.debug({ config: safeStringifyJson(this.info) }, "FTP server connecting")
    const { host, secure, port } = this.info
    await this.client.access({ host, user: this.username, password: this.token, secure })
    await this.client.connect(host, port)

    if (task.fileContent != null) {
      await this.doUpload(fileName, task.fileContent, null, null, task.file)
      this.client.close()
      return fileName
    }

    const fileStat = await stat(task.file)
    const progressBar = this.createProgressBar(fileName, fileStat.size)
    await this.doUpload(fileName, null, fileStat, progressBar, task.file)
    this.client.close()
    return fileName
  }

  async doUpload(fileName: string, buffer: Buffer | null, fileStat: Stats | null, progressBar: ProgressBar | null, file: string): Promise<any> {
    const { cancellationToken } = this.context
    return cancellationToken.createPromise<string>((resolve, reject, onCancel) => {
      // Wonky logic, but either fileContent is provided, or a fileStat is provided FOR a progress bar
      let readable: Readable
      if (fileStat) {
        const readStream = this.createReadStreamAndProgressBar(file, fileStat, progressBar, reject)
        readable = new Readable().wrap(readStream)
      } else {
        readable = Readable.from(buffer!)
      }
      onCancel(() => {
        this.client.close()
        readable.destroy()
      })

      this.client
        .uploadFrom(readable, join(this.info.path || "", fileName))
        .then((response: ftp.FTPResponse) => {
          log.info({ code: response.code }, "FTP transfer successful")
          resolve(fileName)
        })
        .catch(reject)
    })
  }

  async deleteRelease(fileName: string): Promise<void> {
    const { host, secure } = this.info

    log.debug(this.info, "FTP server connecting")
    await this.client.access({ host, user: this.username, password: this.token, secure })

    const response = await this.client.remove(join(this.info.path || "", fileName))
    log.info({ code: response.code }, "FTP delete successful")

    this.client.close()
  }

  toString() {
    const { host, port, path, channel } = this.info
    return `FTP(S) (server: ${host}:${port}, path: ${path}, channel: ${channel})`
  }
}

[Nantris]

I'm not really sure if/when I'll have time to do much more than a high level brainstorming conversation with ChatGPT, but I did start to take a look at this and realized I don't fully understand the problem(s).

Some questions:

1. Could multiple async uploads be foregone in favor of one upload at a time to enable an easier, albeit non-optimal, solution to this? The uploads generally aren't terribly large and the number of files is pretty small I feel
2. With regards to multiple async uploads, is the primary issue starting the uploads, or really more about tracking them and providing a progress bar?
3. Could multiple independent FTP connections be opened via invoking basic-ftp into several instances? Again the progress bar seems a problem though.

[mmaietta]

For some reason I can't figure out how to queue the promises such that only one FTP connection is open at a time, I tried overriding the AsyncTaskManager with a traditional for-await loop and it still didn't process it sequentially. Each upload tries to set up a new connection with the FTP server but accesses from the same port. If I persist the FTP connection across uploads, it still errors out saying that a previous upload had not finished, even with the for-await.

The issue is less so the progress bar and I think more so that multiple connections try to hit the same port on the server?

[beyondkmp]

For some reason I can't figure out how to queue the promises such that only one FTP connection is open at a time, I tried overriding the AsyncTaskManager with a traditional for-await loop and it still didn't process it sequentially. Each upload tries to set up a new connection with the FTP server but accesses from the same port. If I persist the FTP connection across uploads, it still errors out saying that a previous upload had not finished, even with the for-await.

The issue is less so the progress bar and I think more so that multiple connections try to hit the same port on the server?

May only need to authorize it once, rather than needing to authorize every time you upload it.

[Nantris]

I didn't forget about this, but I never did really make any headway on it. I could share what I managed to work through with GPT4 some time back, but I'm not sure it would be of any value and I don't want to waste people's time.