solidcommunity.net
solidcommunity.net copied to clipboard
solid
Migration using NSS2CSS 12/2024
-
Some data :
- 20241211 accounts 62932 email 51834 58326 from 2020-12-31
- size : data + .db/users/users : 66.5 GB
-
test migration from solidcommunity.net backup on 2024/12/15
- step1 on initialization reading NSS accounts 10 mn
- step 2 CSS accounts creation 1s per account => expected 16 hours
- to investigate : Do not fully succeed. only copy 24805 pods (in 14 hours).
- podname's account converted to lower case (NSS url use lower case also, but not the account)
- step 3 : copy users folders/files : 1 hour (need verification)
- step 4 : update password to NSS one (seems OK was able to login to bourgeoa and bourgeoa1)
- step 5 : replace oidcIssuier in WebID seems OK (ending with '/')
- step 6 : replace acl:defaultForNew with acl:default seems OK
- step 7 : replace links : this broke at first account
- issues
- [ ] step 2
- [x] pod with name's including blank or @ sign cannot be created on CSS (not allowed in an email)
- [ ] on account creation uppercase converted to lower case (email request ?)
- [x] only 24805 pods created out of 62932 pods
- [x] account created even if there is no data folder this induces ENOENT error in the steps 4 to 8
- [ ] step 7 : replace links on files
- [x] pod name beginning with '+' (+devient) broke step 7 on code error
- [ ] improve to be able re-run on all situations
- [x] actually only on .ttl .acl and .meta extensions
- [ ] add .json and .jsonld extensions
- [ ] step 2
7️⃣ CSS: Update 24805 pods : replace links on files content
file:///home/solid/NSS2CSS/copy-pods-to-css.mjs:292
const patt = new RegExp(source)
^
SyntaxError: Invalid regular expression: /+deviant.solidcommunity.net/: Nothing to repeat
at new RegExp (<anonymous>)
at file:///home/solid/NSS2CSS/copy-pods-to-css.mjs:292:22
Node.js v20.18.0
- can we reduce the migration duration.
- [x] Done by removing HTTP CRUD request to directly file create accounts
- after initial migration : migrating pods on request can be done on demand with no difficulty
Temporarily Proposals : (do not create when there is an issue)
- do not create account if the data folder do not exist
- blank and @ in podName either do not copy or %encode --> temporarily do not create
- do not create account with uppercase
Preparing end January 2025 migration :
-
- disks
- [x] solidcommunity.net data should be on disk 1
- [x] disk 3 shall be for weekly backup of solidcommunity.net running instance
- [x] disk 4 shall keep a solidcommunity.net data copy before migration
-
- install pivot
- [x] npm install @solid/pivot@latest
# cd /usr/local/lib
# npm install @solid/pivot@latest ## this is a local installation
# ls -al node_modules/@solid/pivot
# ls -al node_modules/css-mashlib
# ls -al node_modules/mashlib
# ls -al node_modules/@solid/community-server
# cd /usr/local/bin
# ln -s ../lib/node_modules/@solid/pivot pivot
# ln -s ../lib/node_modules/css-mashlib css-mashlib
# ln -s ../lib/node_modules/mashlib mashlib
# ln -s ../lib/node_modules/@solid/community-server community-server
- [x] create and store dedicated customise-me.json
# cd /root/pivot-config
# cat /root/pivot-config/customise-solidcommunity.net.json
{
"comment": "Basic overrides parameters for a production server",
"@context": [
"https://linkedsoftwaredependencies.org/bundles/npm/@solid/community-server/^7.0.0/components/context.jsonld",
"https://linkedsoftwaredependencies.org/bundles/npm/@solid/pivot/^1.0.0/components/context.jsonld"
],
"@graph": [
{
"comment": "The settings of your email server.",
"@type": "Override",
"overrideInstance": {
"@id": "urn:solid-server:default:EmailSender"
},
"overrideParameters": {
"@type": "BaseEmailSender",
"senderName": "[email protected]",
"emailConfig_host": "smtp.sendgrid.net",
"emailConfig_port": 465,
"emailConfig_auth_user": "apikey",
"emailConfig_auth_pass": "xxxxxxxxx"
}
},
{
"comment": "The location of the new pod templates folder.",
"@type": "Override",
"overrideInstance": {
"@id": "urn:solid-server:default:PodResourcesGenerator"
},
"overrideParameters": {
"@type": "StaticFolderGenerator",
"templateFolder": "/usr/local/bin/css-mashlib/templates/pod"
}
},
{
"comment": "Sets the maximum size of a single pod to 50MB.",
"@type": "Override",
"overrideInstance": {
"@id": "urn:solid-server:default:QuotaStrategy"
},
"overrideParameters": {
"@type": "PodQuotaStrategy",
"limit_amount": 50000000,
"limit_unit": "bytes"
}
},
{
"comment": "Serve Databrowser as default representation",
"@id": "urn:solid-server:default:DefaultUiConverter",
"@type": "ConstantConverter",
"contentType": "text/html",
"filePath": "/usr/local/bin/mashlib/dist/databrowser.html",
"options_container": true,
"options_document": true,
"options_minQuality": 1,
"options_enabledMediaRanges": [
"text/turtle",
"text/n3",
"text/markdown",
"application/ld+json",
"application/json",
"application/rdf+xml"
]
},
{
"comment": "Serve Mashlib static files.",
"@id": "urn:solid-server:default:StaticAssetHandler",
"@type": "StaticAssetHandler",
"assets": [
{
"@type": "StaticAssetEntry",
"relativeUrl": "/browse.html",
"filePath": "/usr/local/bin/mashlib/dist/browse.html"
},
{
"@type": "StaticAssetEntry",
"relativeUrl": "/mash.css",
"filePath": "/usr/local/bin/mashlib/dist/mash.css"
},
{
"@type": "StaticAssetEntry",
"relativeUrl": "/mashlib.min.js",
"filePath": "usr/local/bin/mashlib/dist/mashlib.min.js"
},
{
"@type": "StaticAssetEntry",
"relativeUrl": "/mashlib.min.js.map",
"filePath": "/usr/local/bin/mashlib/dist/mashlib.min.js.map"
}
]
}
]
}
- [x] create pm2 script to run pivot on disk 1
# cat /root/pivot-config/pivot-solidcommunity.net.sh
npx community-solid-server \
-c /usr/local/bin/pivot/config/prod.json /root/pivot-config/customise-solidcommunity.net.json \
-f /mnt/volume_lon1_01/solidcommunity.net \
-p 443 -b https://solidcommunity.net:443 \
--httpsKey /etc/letsencrypt/live/solidcommunity.net/privkey.pem \
--httpsCert /etc/letsencrypt/live/solidcommunity.net/fullchain.pem \
-m /usr/local/bin/pivot
- [x] create pm2 script to run pivot on disk 4 with port 8443
# cat /root/pivot-config/pivot-solidcommunity.net-8443-mnt04.sh
npx community-solid-server \
-c /usr/local/bin/pivot/config/prod.json /root/pivot-config/customise-solidcommunity.net.json \
-f /mnt/volume_lon1_04/solicommunity.net \
-p 8443 -b https://solidcommunity.net:8443 \
--httpsKey /etc/letsencrypt/live/solidcommunity.net/privkey.pem \
--httpsCert /etc/letsencrypt/live/solidcommunity.net/fullchain.pem \
-m /usr/local/bin/pivot
-
- last migration test from disk 3 to disk 4
- [x] rm data on disk 4
# pm2 stop pivot-8443-mnt04
# rm -r /mnt/volume_lon1_04/test-subdomain-8443
- [ ] start PIVOT on disk 4 with pm2 script
# pm2 ls
# cat /root/pivot-config/pivot-solidcommunity.net-8443-mnt04.sh
# pm2 del pivot-8443-mnt04
# pm2 start --name pivot-8443-mnt04 /root/pivot-config/pivot-solidcommunity.net-8443-mnt04.sh
# pm2 ls
# pm2 logs pivot-8443-mnt04
- [ ] initialize server root
# cp -r /usr/local/bin/pivot/www /mnt/volume_lon1_04/solidcommunity.net
# ls -al /mnt/volume_lon1_04/solidcommunity.net/www/
.acl
index.html
- [ ] from NSS data on disk 3 to CSS on disk 4
# cd /home/solid/NSS2CSS
# cat solidcommunity.net-mnt04.sh
node copy-pods-to-css.mjs /home/solid/config-443-mnt03/config.json /mnt/volume_lon1_04/test-subdomain-8443/ https://solidcommunity.net:8443/ [email protected]
# nohup ./solidcommunity.net-mnt04.sh > migration-results/nohup.out &
# jobs
-
- migrate NSS2CSS with source disk 4 to dest disk 1
- [x] stop test on port 8443 with data on disk 4
# pm2 stop pivot-8443-mnt04
- [x] rm data on disk 4
# rm -r /mnt/volume_lon1_04/test-subdomain-8443
- [x] stop solidcommunity.net
# pm2 ls
# pm2 stop solid
- [x] copy data from disk 1 to disk 4
# rsync -a /mnt/volume_lon1_01/solid.community/ /mnt/volume_lon1_04/solidcommunity.net.bak.nss/data --stats &
# rsync -a /mnt/volume_lon1_01/.db/oidc/users /mnt/volume_lon1_04/solidcommunity.net.bak.nss/.db/oidc/ --stats &
# ls -al /mnt/volume_lon1_04 | wc -l
- [x] rm data on disk 1
# rm -r /mnt/volume_lon1_01/solid.community &
# rm -r /mnt/volume_lon1_01/.db/oidc/users &
# ls -al /mnt/volume_lon1_01/
- [ ] start PIVOT on disk 1 with pm2 script
# cat /root/pivot-config/pivot-solidcommunity.net.sh
# pm2 start --name pivot /root/pivot-config/pivot-solidcommunity.net.sh
- [ ] initialize server root
# cp -r /usr/local/bin/pivot/www /mnt/volume_lon1_01/solidcommunity.net
# ls -al /mnt/volume_lon1_01/solidcommunity.net/www/
.acl
index.html
- [ ] run migration script from source disk 4 to dest disk 1
# cd /home/solid/NSS2CSS
# nohup ./solidcommunity.net-mnt01.sh > migration-results/nohup.out &
- [ ] check migration results
# cat migration-results/nohup.out
- [ ] cleaning pm2
# pm2 del pivot-8443-mnt04
# pm2 del solid
-
- update backup cron job
# crontab -l
# crontab -e
- [x] rm data on disk 3
# rm -r /mnt/*03/solidcommunity.net
- [x] run first backup rsync should exclude /www/.internal/idp
# rsync -a --dry-run --exclude 'www/.internal/idp' /mnt/volume_lon1_01/solidcommunity.net/ /mnt/volume_lon1_03/solidcommunity.net.bak/ --stats > /var/log/solid/$(date +\%Y\%m\%d)-data.log
# rsync -a --exclude=www/.internal/idp --exclude=www/.internal/locks --exclude=www/.internal/accounts/cookies /mnt/volume_lon1_01/solidcommunity.net/ /mnt/volume_lon1_03/solidcommunity.net.bak/ --stats > /var/log/solid/$(date +\%Y\%m\%d)-data.log &
# ls -al /mnt/volume_lon1_03
- [x] edit crontab
# crontab -e
-
- [ ] update wiki
- NSS backup to disk 4
- users
# ls -al /mnt/volume_lon1_04/solidcommunity.net.bak/.db/o*/users/users | wc -l--> 65471 - containers
# ls -d /mnt/volume_lon1_04/solidcommunity.net.bak/data | wc -l--> 63111
- users
root@solidcommunity:/mnt/volume_lon1_01#
Number of files: 2,418,646 (reg: 1,960,404, dir: 458,242)
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 0
Total file size: 62,099,214,639 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 24,243,027
File list generation time: 3.169 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 81,434,137
Total bytes received: 471,469
sent 81,434,137 bytes received 471,469 bytes 111,664.08 bytes/sec
total size is 62,099,214,639 speedup is 758.18
- migration to CSS
NSS userFiles 65468
Invalid NSS config
invalid Json 4
deprecated config filename 3309
invalid config keys 2293
invalid NSS pods
username with dot 1213
no data folder 89
no profile/card 92
only solid.community webId 40652
external webId 6
username with arobase 0
username with blank 0
username with uppercase letter 1097
valid NSS pods 16713
check control (should be zero) 0
CSS pods
already existing CSS pods 2 (doctorbud, cdr)
created CSS pods 16713
failed create CSS pods 0
failed CSS pod fetch 12624
check control (should be zero) -12626
oidcIssuer issue 33
asymmetriciris
bramb
cacao
dietercasier
edwin
espy
ewout
joshuaprout2
kenji
kludgy
liamthys
liuwei
lucasdb99
mrchanmp
nbrooks22
ordina
pferreir
psycop
reza-soltani
ruben-greeter
solid-test-1
solidtestsuite
soltanireza65
tbaly
teamlion
test002
test003
testkraken
testkraken2
tmey
tradstuc
unitest
vanesstalkers
CSS failed pod fetch
littlestone https://liitlestone.solidcommunity.net/card#me
larrygeyer77 https//larrygeyer77.solidcommunity.net/profile/card#me
solid-arne-test https://vandoorslaerarne.solidcommunity.net/profile/card#me