Lemmy-ui 19.4 Sharp + VIPS detection issue

[jSML4ThWwBID69YC]

Requirements

• [X] This is a bug report, and if not, please post to https://lemmy.ml/c/lemmy_support instead.
• [X] Please check to see if this issue already exists.
• [X] It's a single bug. Do not report multiple bugs in one issue.
• [X] It's a frontend issue, not a backend issue; Otherwise please create an issue on the backend repo instead.

Summary

I am unable to use lemmy-ui when built on FreeBSD. I believe the underlying issue is that the lemmy-ui sharp build configuration is not auto-detecting the preinstalled VIPS software. Previous versions of lemmy-ui detect and properly include VIPS when building Sharp.

With version 19.4, the code has changed such that the auto detection is no longer working. Here's the relevant output from building lemmy-ui.

warning [email protected]: The engine "libvips" appears to be invalid.
warning @img/[email protected]: The engine "pnpm" appears to be invalid.
warning @img/[email protected]: The engine "macos" appears to be invalid.
warning @img/[email protected]: The engine "pnpm" appears to be invalid.
warning @img/[email protected]: The engine "glibc" appears to be invalid.
warning @img/[email protected]: The engine "pnpm" appears to be invalid.
warning @img/[email protected]: The engine "macos" appears to be invalid.
warning @img/[email protected]: The engine "pnpm" appears to be invalid.
warning @img/[email protected]: The engine "glibc" appears to be invalid.
warning @img/[email protected]: The engine "pnpm" appears to be invalid.
warning @img/[email protected]: The engine "musl" appears to be invalid.
warning @img/[email protected]: The engine "pnpm" appears to be invalid.
warning @img/[email protected]: The engine "musl" appears to be invalid.
warning @img/[email protected]: The engine "pnpm" appears to be invalid.
warning @img/[email protected]: The engine "glibc" appears to be invalid.
warning @img/[email protected]: The engine "pnpm" appears to be invalid.
warning @img/[email protected]: The engine "glibc" appears to be invalid.
warning @img/[email protected]: The engine "pnpm" appears to be invalid.
warning @img/[email protected]: The engine "glibc" appears to be invalid.
warning @img/[email protected]: The engine "pnpm" appears to be invalid.
warning @img/[email protected]: The engine "glibc" appears to be invalid.
warning @img/[email protected]: The engine "pnpm" appears to be invalid.
warning @img/[email protected]: The engine "glibc" appears to be invalid.
warning @img/[email protected]: The engine "pnpm" appears to be invalid.
warning @img/[email protected]: The engine "glibc" appears to be invalid.
warning @img/[email protected]: The engine "pnpm" appears to be invalid.
warning @img/[email protected]: The engine "glibc" appears to be invalid.
warning @img/[email protected]: The engine "pnpm" appears to be invalid.
warning @img/[email protected]: The engine "pnpm" appears to be invalid.
warning @img/[email protected]: The engine "pnpm" appears to be invalid.
warning @img/[email protected]: The engine "musl" appears to be invalid.
warning @img/[email protected]: The engine "pnpm" appears to be invalid.
warning @img/[email protected]: The engine "musl" appears to be invalid.
warning @img/[email protected]: The engine "pnpm" appears to be invalid.
warning @img/[email protected]: The engine "pnpm" appears to be invalid.
warning @img/[email protected]: The engine "glibc" appears to be invalid.

To be more specific, this appears to be the issue: warning [email protected]: The engine "libvips" appears to be invalid.

Note, Sharp installs and auto-detects VIPS fine when installed from NPM, or YARN outside of Lemmy-ui. It appears to be something in the Lemmy-ui configuration that is causing this to break.

Steps to Reproduce

1. Install prerequisites on FreeBSD. (EX: pkg install vips node npm yarn)
2. git clone https://github.com/LemmyNet/lemmy-ui.git --recursive /root/lemmy-ui
3. cd /root/lemmy-ui
4. git checkout 0.19.4
5. yarn install --pure-lockfile
6. yarn build:prod

Now, deploy the resulting Lemmy-ui build.

It errors out when run with this message.

"/usr/local/bin/node dist/js/server.js" (uid: 5203 gid: 5203)
/apps/lemmy-ui/node_modules/sharp/lib/sharp.js:114
  throw new Error(help.join('\n'));
  ^

Error: Could not load the "sharp" module using the freebsd-x64 runtime
Possible solutions:
- Manually install libvips >= 8.15.2
- Add experimental WebAssembly-based dependencies:
    npm install --cpu=wasm32 sharp
    npm install @img/sharp-wasm32
- Consult the installation documentation:
    See https://sharp.pixelplumbing.com/install
    at Object.<anonymous> (/apps/lemmy-ui/node_modules/sharp/lib/sharp.js:114:9)
    at Module._compile (node:internal/modules/cjs/loader:1358:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1416:10)
    at Module.load (node:internal/modules/cjs/loader:1208:32)
    at Module._load (node:internal/modules/cjs/loader:1024:12)
    at Module.require (node:internal/modules/cjs/loader:1233:19)
    at require (node:internal/modules/helpers:179:18)
    at Object.<anonymous> (/apps/lemmy-ui/node_modules/sharp/lib/constructor.js:10:1)
    at Module._compile (node:internal/modules/cjs/loader:1358:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1416:10)

Node.js v20.14.0

Technical Details

Operating system

• FreeBSD 13.3
• FreeBSD 14.0

Installed software

node-20.14.0_1 V8 JavaScript for client and server (meta port) node20-20.14.0 V8 JavaScript for client and server npm-node20-10.8.1 Node package manager yarn-1.22.18_2 Package manager for node, alternative to npm (meta port) yarn-node20-1.22.19 Package manager for node, alternative to npm yarn-1.22.18_2 Package manager for node, alternative to npm (meta port) yarn-node20-1.22.19 Package manager for node, alternative to npm vips-8.15.2_2 Free image processing system

Lemmy Instance Version

19.4

Lemmy Instance URL

NA

[SleeplessOne1917]

I noticed that your reproduction steps are using yarn. lemmy-ui uses pnpm as its package manager now.

Is the build output just running the UI natively (i.e. not in a container)?

[jSML4ThWwBID69YC]

I noticed that your reproduction steps are using yarn. lemmy-ui uses pnpm as its package manager now.

Ahh, yes. I was using the old build method. I'll use the new one and report back.

Is the build output just running the UI natively (i.e. not in a container)?

Lemmy-ui is run using the attach-daemon command of the uWSGi server. It's basically the same as just running it on the command line.

[dessalines]

Can re-open if its still an issue.

[jSML4ThWwBID69YC]

Can re-open if its still an issue.

@dessalines @SleeplessOne1917

I'm sorry to bother you about this, but it is still an issue. I did installations of 19.4 and 19.3 and for some reason the 19.4 version can't detect vips on the same server.

Here's my build instructions and output for both of them.

19.4 lemmy-ui

git clone https://github.com/LemmyNet/lemmy-ui.git --recursive /root/builds/lemmy-ui
cd /root/builds/lemmy-ui 
git checkout 0.19.4
pnpm i
pnpm build:prod

Now try and run it. /usr/local/bin/node dist/js/server.js

Error message:

Inferno is in development mode.
/root/builds/lemmy-ui/node_modules/.pnpm/[email protected]/node_modules/sharp/lib/sharp.js:114
  throw new Error(help.join('\n'));
  ^

Error: Could not load the "sharp" module using the freebsd-x64 runtime
Possible solutions:
- Manually install libvips >= 8.15.2
- Add experimental WebAssembly-based dependencies:
    npm install --cpu=wasm32 sharp
    npm install @img/sharp-wasm32
- Consult the installation documentation:
    See https://sharp.pixelplumbing.com/install
    at Object.<anonymous> (/root/builds/lemmy-ui/node_modules/.pnpm/[email protected]/node_modules/sharp/lib/sharp.js:114:9)
    at Module._compile (node:internal/modules/cjs/loader:1358:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1416:10)
    at Module.load (node:internal/modules/cjs/loader:1208:32)
    at Module._load (node:internal/modules/cjs/loader:1024:12)
    at Module.require (node:internal/modules/cjs/loader:1233:19)
    at require (node:internal/modules/helpers:179:18)
    at Object.<anonymous> (/root/builds/lemmy-ui/node_modules/.pnpm/[email protected]/node_modules/sharp/lib/constructor.js:10:1)
    at Module._compile (node:internal/modules/cjs/loader:1358:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1416:10)

Node.js v20.14.0

19.3 lemmy-ui

This same build system works fine with Lemmy 19.3 using the old yarn method. Here's building it on the same server to demonstrate that VIPS is installed correctly.

git clone https://github.com/LemmyNet/lemmy-ui.git --recursive /root/builds/lemmy-ui-19.3
cd /root/builds/lemmy-ui-19.3
git checkout 0.19.3
yarn install --pure-lockfile
yarn build:prod

Now run it: /usr/local/bin/node dist/js/server.js

 # /usr/local/bin/node dist/js/server.js
Inferno is in development mode.
Inferno is in development mode.
Inferno is in development mode.
Lemmy-ui vunknown version started listening on http://0.0.0.0:1234

Installed software

npm-10.8.1 Node package manager (meta port) npm-node20-10.8.1 Node package manager yarn-1.22.18_2 Package manager for node, alternative to npm (meta port) yarn-node20-1.22.19 Package manager for node, alternative to npm node-20.14.0_1 V8 JavaScript for client and server (meta port) node20-20.14.0 V8 JavaScript for client and server npm-node20-10.8.1 Node package manager yarn-1.22.18_2 Package manager for node, alternative to npm (meta port) yarn-node20-1.22.19 Package manager for node, alternative to npm vips-8.15.2_2 Free image processing system

I can provide a FreeBSD container if needed for testing. Please let me know how else I can help.

[jSML4ThWwBID69YC]

Oops, should I open this as a new issue?

[dessalines]

We can reopen.

It might be worth opening an issue on the sharp repo, or doing a search there, bc this isn't something we directly control.

[jSML4ThWwBID69YC]

Hello,

Sharp seems to install fine outside the lemmy-ui folder. Here are two examples.

Example with npm

npm install --verbose --foreground-scripts sharp
npm verbose cli /usr/local/bin/node /usr/local/bin/npm
npm info using [email protected]
npm verbose title npm install sharp
npm verbose logfile logs-max:10 dir:/root/.npm/_logs/2024-06-17T06_00_06_092Z-
npm verbose shrinkwrap failed to load node_modules/.package-lock.json out of date, updated: node_modules
npm http fetch GET 200 https://registry.npmjs.org/npm 143ms
npm http fetch GET 200 https://registry.npmjs.org/detect-libc 91ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/color 95ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/semver 124ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/@img%2fsharp-libvips-linux-s390x 149ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/@img%2fsharp-linuxmusl-x64 62ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/@img%2fsharp-libvips-linux-arm64 237ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/@img%2fsharp-linux-arm64 237ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/@img%2fsharp-linuxmusl-arm64 120ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/@img%2fsharp-darwin-x64 275ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/@img%2fsharp-wasm32 73ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/@img%2fsharp-libvips-linux-x64 286ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/@img%2fsharp-libvips-linuxmusl-arm64 309ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/@img%2fsharp-darwin-arm64 316ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/@img%2fsharp-win32-ia32 135ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/@img%2fsharp-libvips-linux-arm 402ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/@img%2fsharp-linux-x64 314ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/@img%2fsharp-libvips-darwin-x64 412ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/@img%2fsharp-libvips-darwin-arm64 416ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/@img%2fsharp-linux-s390x 335ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/@img%2fsharp-linux-arm 422ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/@img%2fsharp-win32-x64 214ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/@img%2fsharp-libvips-linuxmusl-x64 537ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/color-convert 34ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/color-string 48ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/simple-swizzle 41ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/color-name 43ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/color-name 46ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/is-arrayish 30ms (cache miss)
npm verbose reify failed optional dependency /root/node_modules/@img/sharp-wasm32
npm verbose reify failed optional dependency /root/node_modules/tslib
npm verbose reify failed optional dependency /root/node_modules/@img/sharp-linuxmusl-x64
npm verbose reify failed optional dependency /root/node_modules/@img/sharp-linuxmusl-arm64
npm verbose reify failed optional dependency /root/node_modules/@img/sharp-linux-x64
npm verbose reify failed optional dependency /root/node_modules/@img/sharp-linux-s390x
npm verbose reify failed optional dependency /root/node_modules/@img/sharp-linux-arm64
npm verbose reify failed optional dependency /root/node_modules/@img/sharp-linux-arm
npm verbose reify failed optional dependency /root/node_modules/@img/sharp-libvips-linuxmusl-x64
npm verbose reify failed optional dependency /root/node_modules/@img/sharp-libvips-linuxmusl-arm64
npm verbose reify failed optional dependency /root/node_modules/@img/sharp-libvips-linux-x64
npm verbose reify failed optional dependency /root/node_modules/@img/sharp-libvips-linux-s390x
npm verbose reify failed optional dependency /root/node_modules/@img/sharp-libvips-linux-arm64
npm verbose reify failed optional dependency /root/node_modules/@img/sharp-libvips-linux-arm
npm verbose reify failed optional dependency /root/node_modules/@img/sharp-libvips-darwin-x64
npm verbose reify failed optional dependency /root/node_modules/@img/sharp-libvips-darwin-arm64
npm verbose reify failed optional dependency /root/node_modules/@img/sharp-darwin-x64
npm verbose reify failed optional dependency /root/node_modules/@img/sharp-darwin-arm64
npm http fetch POST 200 https://registry.npmjs.org/-/npm/v1/security/advisories/bulk 49ms
npm http fetch GET 200 https://registry.npmjs.org/color-convert/-/color-convert-2.0.1.tgz 103ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/color-string/-/color-string-1.9.1.tgz 104ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/color-name/-/color-name-1.1.4.tgz 102ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/simple-swizzle/-/simple-swizzle-0.2.2.tgz 109ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/detect-libc/-/detect-libc-2.0.3.tgz 100ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/color/-/color-4.2.3.tgz 100ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/is-arrayish/-/is-arrayish-0.3.2.tgz 115ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/tslib/-/tslib-2.6.3.tgz 117ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/semver/-/semver-7.6.2.tgz 124ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/sharp/-/sharp-0.33.4.tgz 125ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/@emnapi/runtime/-/runtime-1.2.0.tgz 158ms (cache miss)
npm info run [email protected] install node_modules/sharp node install/check

> [email protected] install
> node install/check

sharp: Detected globally-installed libvips v8.15.2
sharp: Attempting to build from source via node-gyp
sharp: Please add node-addon-api to your dependencies
npm info run [email protected] install { code: 0, signal: null }

added 8 packages, changed 1 package, and audited 10 packages in 2s

1 package is looking for funding
  run `npm fund` for details

found 0 vulnerabilities
npm verbose cwd /root/builds/sharp
npm verbose os FreeBSD 13.3-RELEASE-p1
npm verbose node v20.14.0
npm verbose npm  v10.8.1
npm verbose exit 0
npm info ok

PNPM

pnpm i sharp
Progress: resolved 30, reused 10, downloaded 0, added 0, done
Done in 1.3s
pnpm list
Legend: production dependency, optional only, dev only
/root
dependencies:
sharp 0.33.4

So both NPM and PNPM seem to be able to build Sharp correctly outside of lemmy-ui. Since Sharp builds outside the lemmy-ui environment, I think this is an issue with the lemmy-ui build scripts. I could use some advice or next steps.

[matc-pub]

.npmrc currently sets ignore-scripts=true. Maybe that's the problem.

[SleeplessOne1917]

I made a PR removing ignore-scripts=true.

[jSML4ThWwBID69YC]

Hello,

I deleted the .npmrc file and tried again. Unfortunately that did not resolve it. Oddly, I can't seem to build Sharp within the lemmy-ui folder manually either. Here's what happens.

 pnpm i sharp
 WARN  7 deprecated subdependencies found: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Already up to date
Progress: resolved 1017, reused 997, downloaded 0, added 0, done
 WARN  Issues with peer dependencies found
.
├─┬ @typescript-eslint/parser 7.12.0
│ └── ✕ unmet peer eslint@^8.56.0: found 9.4.0
├─┬ @typescript-eslint/eslint-plugin 7.12.0
│ ├── ✕ unmet peer eslint@^8.56.0: found 9.4.0
│ └─┬ @typescript-eslint/type-utils 7.12.0
│   ├── ✕ unmet peer eslint@^8.56.0: found 9.4.0
│   └─┬ @typescript-eslint/utils 7.12.0
│     └── ✕ unmet peer eslint@^8.56.0: found 9.4.0
├─┬ eslint-plugin-inferno 7.34.0
│ └── ✕ unmet peer eslint@"^6 || ^7 || ^8": found 9.4.0
├─┬ eslint-plugin-jsx-a11y 6.8.0
│ └── ✕ unmet peer eslint@"^3 || ^4 || ^5 || ^6 || ^7 || ^8": found 9.4.0
├─┬ typescript-eslint 7.12.0
│ └── ✕ unmet peer eslint@^8.56.0: found 9.4.0
└─┬ inferno-helmet 5.2.1
  └─┬ inferno-side-effect 1.1.5
    └── ✕ unmet peer inferno@^5.0.3: found 8.2.3

Done in 7.7s

[SleeplessOne1917]

Your most recent logs show warnings, but I don't see any errors.

[jSML4ThWwBID69YC]

You are correct. I misinterpreted that.

I've dug through the Sharp issue queue but nothing stands out. It installs properly outside of Lemmy-ui, so I am at a loss as to what it could be. Any ideas?

[SleeplessOne1917]

The only idea that comes to mind is nuking your node_modules directory and installing fresh. If that fails, you could always run it in docker.

[jSML4ThWwBID69YC]

It's a FreeBSD system, so no Docker. I wipe the entire folder for each test.

I'll keep working on it. Thank you for helping.

[jSML4ThWwBID69YC]

This is an issue on 19.5 too.

git clone https://github.com/LemmyNet/lemmy-ui.git --recursive /root/builds/lemmy-ui
cd /root/builds/lemmy-ui 
git checkout 0.19.5
pnpm i
pnpm build:prod

Now try and run it.

#  /usr/local/bin/node dist/js/server.js
Inferno is in development mode.
/root/builds/lemmy-ui/node_modules/.pnpm/[email protected]/node_modules/sharp/lib/sharp.js:114
  throw new Error(help.join('\n'));
  ^

Error: Could not load the "sharp" module using the freebsd-x64 runtime
Possible solutions:
- Manually install libvips >= 8.15.2
- Add experimental WebAssembly-based dependencies:
    npm install --cpu=wasm32 sharp
    npm install @img/sharp-wasm32
- Consult the installation documentation:
    See https://sharp.pixelplumbing.com/install
    at Object.<anonymous> (/root/builds/lemmy-ui/node_modules/.pnpm/[email protected]/node_modules/sharp/lib/sharp.js:114:9)
    at Module._compile (node:internal/modules/cjs/loader:1358:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1416:10)
    at Module.load (node:internal/modules/cjs/loader:1208:32)
    at Module._load (node:internal/modules/cjs/loader:1024:12)
    at Module.require (node:internal/modules/cjs/loader:1233:19)
    at require (node:internal/modules/helpers:179:18)
    at Object.<anonymous> (/root/builds/lemmy-ui/node_modules/.pnpm/[email protected]/node_modules/sharp/lib/constructor.js:10:1)
    at Module._compile (node:internal/modules/cjs/loader:1358:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1416:10)

Node.js v20.14.0

[dessalines]

@jSML4ThWwBID69YC open up an issue on the sharp repo I linked above, because we don't control how that builds on different systems.

[jSML4ThWwBID69YC]

Hello,

I believe the sharp issue is resolved on 0.19.5. At least I can't generate missing sharp errors anymore. I had to clear the cache out from the 0.19.4 failed builds.

I have run into a secondary issue. None of the dependency modules are detected when running Lemmy-ui. For example, trying to run /usr/local/bin/node dist/js/server.js leads to an error about mdurl missing. I install that manually and run it again and it complains about uc.micro missing. It continues like that for each module. It turns out all the missing modules are being installed in node_modules/.pnpm/node_modules instead of just node_modules. The modules in the sub folder are not being picked up at runtime.

Is there an environment variable, or other setting I need to ensure the modules in the .pnpm/node_modules folder are loaded?

I appreciate your help. If we can figure it out, I'll update the documentation for other from scratch builders.

[jSML4ThWwBID69YC]

I found the issue. When creating a zip file for deployment, the symlinks are stripped out. This breaks it after deployment because pnpm is using symlinks to reference modules in sub folders. Everything is working now.

[dessalines]

Sweet.