AFFiNE
AFFiNE copied to clipboard
toeverything
New Self-hosted Version Progress & issues
Many people are interested in the development of our new self-hosted version, so I created this issue to keep everyone updated on its progress.
The current version of docker image has lagged behind our stable branch for a long time. We need to update the new docker-image.
Since we ship many features like payment and account on the current backend server, it is not suitable for common self-hosted use cases. So we need to make these features optional in the new self-hosted version, which requires some work.
UPDATED: We plan to release the official version of the self-hosted Docker image along with version 0.12 in early February.
### Tasks
- [x] https://github.com/toeverything/AFFiNE/issues/5003
- [ ] https://github.com/toeverything/AFFiNE/issues/5057
- [x] Update Docker Image
- [x] Self-hosted tutorial
- [ ] https://github.com/toeverything/AFFiNE/issues/2646
- [ ] https://github.com/toeverything/AFFiNE/pull/5506
- [ ] https://github.com/toeverything/AFFiNE/pull/5582
- [ ] https://github.com/toeverything/AFFiNE/pull/5577
Archived issues
- https://github.com/toeverything/AFFiNE/issues/4914
- https://github.com/toeverything/OctoBase/issues/457
- https://github.com/toeverything/AFFiNE/discussions/3962#discussioncomment-7382605
- https://github.com/toeverything/AFFiNE-Community/issues/25
@omarcostahamido I think there is endless potential for integration into other managed systems. A few integrations that would enable more non-tech-people (or those who know about local systems, but nothing about application development/deployment) to use AFFiNE:
- CasaOS
- Synology DSM
- QNAP QTS
- Unraid
@PhilippMolitor
Yes synology DSM, that's what I meant. I'm happy to help with it!
Waiting for official Docker image (ARM64). This is required for any server-side projects nowadays :)
Waiting for official Docker image (ARM64). This is required for any server-side projects nowadays :)
The new version drop few hours ago can be run on ARM64 using a raspberry pi 4 :smile:
The new version for self hosted is running but i have this message :
My server not gonna save my note ? or it's a UI bug ?
I tried to run this docker-compose on my server too, but i faced with several issues:
- Problem with docker-compose version - in the compose.yaml version tag didnt mentionted and because of that i faced with error "Unsupported config option for services: 'postgres'", when i added version: "3" at the top of the file, this error gone.
- The next problem is that in the latest docker-compose, depends-on tag have different type and should be described differently
- After all fixes for docker-compose, everything was up, but i found, that only frontend version was launched and didnt save docs to the database and in no option in desktop version to configurate my server url in order to use it as cloud, only one option - activate affine cloud
- The last problem - i didnt find how to setup OAuth to provide login through google, and button from interface just redirect me back on frontend part login form with token, but login was not completed
I can provide any help that you need with self-hosted version, even write instructions on how to run affine on cloud-ocean droplet or drop docker image on TrueNas Scale.
@Exitialis Thank you for sharing your experience and the workarounds that you found!
no option in desktop version to configurate my server url in order to use it as cloud
I guess that's why https://github.com/toeverything/AFFiNE/issues/2646 is open?
The new version for self hosted is running but i have this message :
My server not gonna save my note ? or it's a UI bug ?
The server is not gonna save you note unless the cloud is enabled. All data will be saved in the browser's local storage. It can be easily erased because of cleaning browsing data.
I tried running the self-host docker-compose myself but whatever should run on port 5555 isn't starting
I tried running the self-host docker-compose myself but whatever should run on port 5555 isn't starting
5555 is reserved for Primsa Studio, you can run yarn prisma studio to start Primsa Studio.
I see, thank you @Brooooooklyn!
Then I have some other issue, if I go to http://123.123.123.123:3010 I just see this:
{"compatibility":"0.12.0","message":"AFFiNE 0.12.0 Server","flavor":{"type":"allinone","main":true,"sync":true,"selfhosted":false}}
I can see the logs saying this:
And the public server should be recognized as https://affine.example.com
But If browse to https://affine.example.com I get a cloudflare timeout. What are the steps to troubleshoot here?
@benjick I had the same issue, which stemmed from me using the docker-compose.yaml from the canary-branch instead of 0.12.0. On canary SERVER_FLAVOR is not set, compared to 0.12.0 where the following is added in the docker-compose file:
- SERVER_FLAVOR=selfhosted
Although my service now comes up as expected and my redis and graph db are both up and running and configured, I still get the prompt, that my data is only stored in the browser instead of in the database:
I cannot find an option to active the db storage explicitly and wonder, what may go wrong during the start up of the service...
I see the same issue as above, when I try to sign in - the frontend says "GraphQL empty body"
And the docker log:
[Nest] 113 - 02/23/2024, 10:37:17 AM WARN [NextAuth] NEXTAUTH_URL
[Nest] 113 - 02/23/2024, 10:37:46 AM ERROR [ExceptionLogger] Error: requestId-HajWAl7Na5GhwDcaPdLD-: Cannot POST /graphql
The new version for self hosted is running but i have this message :
My server not gonna save my note ? or it's a UI bug ?
The server is not gonna save you note unless the cloud is enabled. All data will be saved in the browser's local storage. It can be easily erased because of cleaning browsing data.
@fourdim What is the GraphDB used for then? My understanding was, that we'd be able to save the data to the locally run graph db service.
As an addendum, I get the same error listed above after trying out the :stable version too.
@acranglesim @simonkaiser9
Sorry, our self-hosted version is not yet ready, as it seems that the configuration module is not working properly. Since it is already outside of working hours in our engineers' time zone, I believe we will resolve this issue tomorrow.
FYI, just noticed this after the announcement today. There is a docs.affine.pro page that has a self-host tutorial now: https://docs.affine.pro/docs/self-host-affine-
FYI, just noticed this after the announcement today. There is a docs.affine.pro page that has a self-host tutorial now: https://docs.affine.pro/docs/self-host-affine-
Very cool. I'm struggling to get that to work with my docker-compose.yml though. Best I can do is get it to serve json output on port 3010 that reports "selfhosted: false". Sounds like I should wait until tomorrow for that configure module.
FYI, just noticed this after the announcement today. There is a docs.affine.pro page that has a self-host tutorial now: https://docs.affine.pro/docs/self-host-affine-
I get node errors using the provided unmodified self-hosted compose.yaml
affine_selfhosted | node:internal/errors:496
affine_selfhosted | ErrorCaptureStackTrace(err);
affine_selfhosted | ^
affine_selfhosted |
affine_selfhosted | Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/app/scripts/register.js' imported from /app/
affine_selfhosted | at new NodeError (node:internal/errors:405:5)
affine_selfhosted | at finalizeResolution (node:internal/modules/esm/resolve:327:11)
affine_selfhosted | at moduleResolve (node:internal/modules/esm/resolve:980:10)
affine_selfhosted | at defaultResolve (node:internal/modules/esm/resolve:1193:11)
affine_selfhosted | at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:403:12)
affine_selfhosted | at ModuleLoader.resolve (node:internal/modules/esm/loader:372:25)
affine_selfhosted | at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:249:38)
affine_selfhosted | at ModuleLoader.import (node:internal/modules/esm/loader:335:34)
affine_selfhosted | at node:internal/process/esm_loader:26:84
affine_selfhosted | at node:internal/per_context/primordials:544:39 {
affine_selfhosted | url: 'file:///app/scripts/register.js',
affine_selfhosted | code: 'ERR_MODULE_NOT_FOUND'
affine_selfhosted | }
affine_selfhosted |
affine_selfhosted | Node.js v18.19.1
With beta and stable branches I get JSON output and with canary I get:
creating config file [/root/.affine/config/.env].
creating config file [/root/.affine/config/affine.js].
creating config file [/root/.affine/config/affine.env.js].
running predeploy script.
node:internal/modules/esm/resolve:258
throw new ERR_UNSUPPORTED_DIR_IMPORT(path, fileURLToPath(base), String(resolved));
^
Error [ERR_UNSUPPORTED_DIR_IMPORT]: Directory import '/app/dist/fundamentals/config' is not supported resolving ES modules imported from /app/dist/prelude.js
at finalizeResolution (node:internal/modules/esm/resolve:258:11)
at moduleResolve (node:internal/modules/esm/resolve:917:10)
at defaultResolve (node:internal/modules/esm/resolve:1130:11)
at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:396:12)
at ModuleLoader.resolve (node:internal/modules/esm/loader:365:25)
at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:240:38)
at ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:85:39)
at link (node:internal/modules/esm/module_job:84:36) {
code: 'ERR_UNSUPPORTED_DIR_IMPORT',
url: 'file:///app/dist/fundamentals/config'
}
Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/app/scripts/register.js' imported from /app/
> > FYI, just noticed this after the announcement today. There is a docs.affine.pro page that has a self-host tutorial now: https://docs.affine.pro/docs/self-host-affine-
>
> I get node errors using the provided unmodified self-hosted compose.yaml
>
> ```
> affine_selfhosted | node:internal/errors:496
> affine_selfhosted | ErrorCaptureStackTrace(err);
> affine_selfhosted | ^
> affine_selfhosted |
> affine_selfhosted | Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/app/scripts/register.js' imported from /app/
> affine_selfhosted | at new NodeError (node:internal/errors:405:5)
> affine_selfhosted | at finalizeResolution (node:internal/modules/esm/resolve:327:11)
> affine_selfhosted | at moduleResolve (node:internal/modules/esm/resolve:980:10)
> affine_selfhosted | at defaultResolve (node:internal/modules/esm/resolve:1193:11)
> affine_selfhosted | at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:403:12)
> affine_selfhosted | at ModuleLoader.resolve (node:internal/modules/esm/loader:372:25)
> affine_selfhosted | at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:249:38)
> affine_selfhosted | at ModuleLoader.import (node:internal/modules/esm/loader:335:34)
> affine_selfhosted | at node:internal/process/esm_loader:26:84
> affine_selfhosted | at node:internal/per_context/primordials:544:39 {
> affine_selfhosted | url: 'file:///app/scripts/register.js',
> affine_selfhosted | code: 'ERR_MODULE_NOT_FOUND'
> affine_selfhosted | }
> affine_selfhosted |
> affine_selfhosted | Node.js v18.19.1
>
FYI, just noticed this after the announcement today. There is a docs.affine.pro page that has a self-host tutorial now: https://docs.affine.pro/docs/self-host-affine-
I get node errors using the provided unmodified self-hosted compose.yaml
affine_selfhosted | node:internal/errors:496 affine_selfhosted | ErrorCaptureStackTrace(err); affine_selfhosted | ^ affine_selfhosted | affine_selfhosted | Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/app/scripts/register.js' imported from /app/ affine_selfhosted | at new NodeError (node:internal/errors:405:5) affine_selfhosted | at finalizeResolution (node:internal/modules/esm/resolve:327:11) affine_selfhosted | at moduleResolve (node:internal/modules/esm/resolve:980:10) affine_selfhosted | at defaultResolve (node:internal/modules/esm/resolve:1193:11) affine_selfhosted | at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:403:12) affine_selfhosted | at ModuleLoader.resolve (node:internal/modules/esm/loader:372:25) affine_selfhosted | at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:249:38) affine_selfhosted | at ModuleLoader.import (node:internal/modules/esm/loader:335:34) affine_selfhosted | at node:internal/process/esm_loader:26:84 affine_selfhosted | at node:internal/per_context/primordials:544:39 { affine_selfhosted | url: 'file:///app/scripts/register.js', affine_selfhosted | code: 'ERR_MODULE_NOT_FOUND' affine_selfhosted | } affine_selfhosted | affine_selfhosted | Node.js v18.19.1
i can confirm the register.js file is missing. when changing the import in docker-compose env to the dist/prelude.js file, I get the same error as @PovilasID . when fully removing the node options, i get the same error with more stuff to it:
affine_1 | Error [ERR_UNSUPPORTED_DIR_IMPORT]: Directory import '/app/dist/fundamentals/config' is not supported resolving ES modules imported from /app/dist/prelude.js
affine_1 | at new NodeError (node:internal/errors:405:5)
affine_1 | at finalizeResolution (node:internal/modules/esm/resolve:321:11)
affine_1 | at moduleResolve (node:internal/modules/esm/resolve:980:10)
affine_1 | at defaultResolve (node:internal/modules/esm/resolve:1193:11)
affine_1 | at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:403:12)
affine_1 | at ModuleLoader.resolve (node:internal/modules/esm/loader:372:25)
affine_1 | at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:249:38)
affine_1 | at ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:76:39)
affine_1 | at link (node:internal/modules/esm/module_job:75:36) {
affine_1 | url: 'file:///app/dist/fundamentals/config',
affine_1 | code: 'ERR_UNSUPPORTED_DIR_IMPORT'
affine_1 | }
affine_1 |
affine_1 | Node.js v18.19.1
affine_1 |
affine_1 | at checkExecSyncError (node:child_process:890:11)
affine_1 | at execSync (node:child_process:962:15)
affine_1 | at runPredeployScript (file:///app/scripts/self-host-predeploy.js:41:3)
affine_1 | at file:///app/scripts/self-host-predeploy.js:51:1
affine_1 | at ModuleJob.run (node:internal/modules/esm/module_job:195:25)
affine_1 | at async ModuleLoader.import (node:internal/modules/esm/loader:336:24)
affine_1 | at async loadESM (node:internal/process/esm_loader:34:7)
affine_1 | at async handleMainPromise (node:internal/modules/run_main:106:12) {
affine_1 | status: 1,
affine_1 | signal: null,
affine_1 | output: [
affine_1 | null,
affine_1 | Buffer(0) [Uint8Array] [],
affine_1 | Buffer(932) [Uint8Array] [
affine_1 | 110, 111, 100, 101, 58, 105, 110, 116, 101, 114, 110, 97,
affine_1 | 108, 47, 101, 114, 114, 111, 114, 115, 58, 52, 57, 54,
affine_1 | 10, 32, 32, 32, 32, 69, 114, 114, 111, 114, 67, 97,
affine_1 | 112, 116, 117, 114, 101, 83, 116, 97, 99, 107, 84, 114,
affine_1 | 97, 99, 101, 40, 101, 114, 114, 41, 59, 10, 32, 32,
affine_1 | 32, 32, 94, 10, 10, 69, 114, 114, 111, 114, 32, 91,
affine_1 | 69, 82, 82, 95, 85, 78, 83, 85, 80, 80, 79, 82,
affine_1 | 84, 69, 68, 95, 68, 73, 82, 95, 73, 77, 80, 79,
affine_1 | 82, 84, 93, 58,
affine_1 | ... 832 more items
affine_1 | ]
affine_1 | ],
affine_1 | pid: 19,
affine_1 | stdout: Buffer(0) [Uint8Array] [],
affine_1 | stderr: Buffer(932) [Uint8Array] [
affine_1 | 110, 111, 100, 101, 58, 105, 110, 116, 101, 114, 110, 97,
affine_1 | 108, 47, 101, 114, 114, 111, 114, 115, 58, 52, 57, 54,
affine_1 | 10, 32, 32, 32, 32, 69, 114, 114, 111, 114, 67, 97,
affine_1 | 112, 116, 117, 114, 101, 83, 116, 97, 99, 107, 84, 114,
affine_1 | 97, 99, 101, 40, 101, 114, 114, 41, 59, 10, 32, 32,
affine_1 | 32, 32, 94, 10, 10, 69, 114, 114, 111, 114, 32, 91,
affine_1 | 69, 82, 82, 95, 85, 78, 83, 85, 80, 80, 79, 82,
affine_1 | 84, 69, 68, 95, 68, 73, 82, 95, 73, 77, 80, 79,
affine_1 | 82, 84, 93, 58,
affine_1 | ... 832 more items
affine_1 | ]
affine_1 | }
FYI, just noticed this after the announcement today. There is a docs.affine.pro page that has a self-host tutorial now: https://docs.affine.pro/docs/self-host-affine-
I get node errors using the provided unmodified self-hosted compose.yaml
affine_selfhosted | node:internal/errors:496 affine_selfhosted | ErrorCaptureStackTrace(err); affine_selfhosted | ^ affine_selfhosted | affine_selfhosted | Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/app/scripts/register.js' imported from /app/ affine_selfhosted | at new NodeError (node:internal/errors:405:5) affine_selfhosted | at finalizeResolution (node:internal/modules/esm/resolve:327:11) affine_selfhosted | at moduleResolve (node:internal/modules/esm/resolve:980:10) affine_selfhosted | at defaultResolve (node:internal/modules/esm/resolve:1193:11) affine_selfhosted | at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:403:12) affine_selfhosted | at ModuleLoader.resolve (node:internal/modules/esm/loader:372:25) affine_selfhosted | at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:249:38) affine_selfhosted | at ModuleLoader.import (node:internal/modules/esm/loader:335:34) affine_selfhosted | at node:internal/process/esm_loader:26:84 affine_selfhosted | at node:internal/per_context/primordials:544:39 { affine_selfhosted | url: 'file:///app/scripts/register.js', affine_selfhosted | code: 'ERR_MODULE_NOT_FOUND' affine_selfhosted | } affine_selfhosted | affine_selfhosted | Node.js v18.19.1
The PR #5920 was recently merged. The documentation has been updated, but obviously, the new version of stable tag hasn't been published yet.
You just need to restore the environment.
- - NODE_OPTIONS="--import=./scripts/register.js"
+ - NODE_OPTIONS="--es-module-specifier-resolution=node"
Thanks to the tip from the person above, I managed to get Affine running locally. However, please don't have too high expectations as it is still a fair way off from replacing Notion at this point.
- The web UI cannot sync data with self-hosted server yet, it still working with local (browser) data only.
- The newly added GraphQL API seems to haven't been implemented in the self-hosted server yet, it currently returns a 404 error. - It maybe Affine Cloud only.
@xuqingkuang thanks for the reply, I managed to get it up'n'running but there's a bit of inconsistency I'm missing.
by default, it spins up the graphql server on port :3010. using comment above https://github.com/toeverything/AFFiNE/issues/5069#issuecomment-1961018351 it drops the graphql server and spins up the web UI on the same port but when I try to login is looking for :3010/graphql that's not available anymore.
in theory i can spin up 2 instances and route the requests to /graphql to server instance using traefik for example, i'll try it later on and see if this a solution 🤞
@peterbuga looks like graphql cloud be on secondary port... or two containers? But honestly it sounds like classic case of two services trying claiming the same port. Fore somebody with knowledge of the architecture should not be a very hard fix. In mean time how do I prevent the container from trying to star web UI? I can probably work out traefik config from there....
@PovilasID I did some testing, using 2 containers and routing the /graphql requests to backend using traefik as mentioned above.
I know it's a lot of work from the devs but I do believe it's now yet ready for us to play around, for example the migrations are never ran, the admin user it's not getting created (the db is empty), there's more to it and we should just wait for a while until they get sorted out.
thanks for all the hard work! i'll keep an eye on the project