bug: During upgrade i get some errors

[0ktav1us]

What happened?

hello, please help me finish updrad#84 run the migration: get error

root@1a7e78fdf5bb:/cwd# node ./20240929-autumn-rewrite.mjs 
1 7505 {
  _id: 'o4U--LPqcFkm6NpvCufKn1cmSZbMTUxwQX_qZ7gwDo',
  tag: 'attachments',
  filename: '2023-03-28 12-26-05.mkv',
  metadata: { type: 'File' },
  content_type: 'application/x-matroska',
  size: 1760809,
  message_id: '01GWKW2YNZ1M9DQP66414MH8S6'
}
file:///cwd/20240929-autumn-rewrite.mjs:41
    if (!objectLookup[f.message_id]) {
    ^

ReferenceError: objectLookup is not defined
    at determineUploaderIdAndUse (file:///cwd/20240929-autumn-rewrite.mjs:41:5)
    at file:///cwd/20240929-autumn-rewrite.mjs:347:16
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

Node.js v21.7.3

And if I continue and run containers, I don't see attachments, it says that the files don't exist

[0ktav1us]

new images attach and display

[ThomasBaruzier]

Same behavior, here's what allowed me to finish the migration:

diff --git a/migrations/20240929-autumn-rewrite.mjs b/migrations/20240929-autumn-rewrite.mjs
index c8643bc..1ab854a 100644
--- a/migrations/20240929-autumn-rewrite.mjs
+++ b/migrations/20240929-autumn-rewrite.mjs
@@ -26,12 +26,38 @@ const BUCKET_MAP = {
  *
  * TODO: change if necessary
  */
-const CONNECTION_URL = "mongodb://database";
+const CONNECTION_URL = "mongodb://database:27017";
 
 const mongo = new MongoClient(CONNECTION_URL);
 await mongo.connect();
 
+const objectLookup = {};
+
+/**
+ * Decode timestamp from Ulid
+ * @param {string} id Ulid string
+ * @returns {number} Timestamp in milliseconds
+ */
+function decodeTime(id) {
+  // First 10 characters of ULID represent milliseconds precision timestamp
+  const timestamp = id.substring(0, 10);
+  // Convert from base32 to decimal
+  const base32Chars = '0123456789ABCDEFGHJKMNPQRSTVWXYZ';
+  let decoded = 0;
+  for (let i = 0; i < timestamp.length; i++) {
+    const char = timestamp[i];
+    const value = base32Chars.indexOf(char.toUpperCase());
+    if (value === -1) {
+      throw new Error(`Invalid ULID character: ${char}`);
+    }
+    decoded = decoded * 32 + value;
+  }
+  // ULID timestamps are in milliseconds
+  return decoded;
+}
+
 async function determineUploaderIdAndUse(f, v, i) {
+  console.log(`Processing file ${i}:`, f._id);
   if (f.tag === "attachments" && v === "attachments") {
     if (typeof f.message_id !== "string") {
       console.warn(i, "No message id specified.");
@@ -39,6 +65,7 @@ async function determineUploaderIdAndUse(f, v, i) {
     }
 
     if (!objectLookup[f.message_id]) {
+      console.log(`Querying database for message ${f.message_id}`);
       objectLookup[f.message_id] = await mongo
         .db("revolt")
         .collection("messages")
@@ -52,6 +79,7 @@ async function determineUploaderIdAndUse(f, v, i) {
       return null;
     }
 
+    console.log(`Found message ${f.message_id} for file ${f._id}`);
     return {
       uploaded_at: new Date(decodeTime(f.message_id)),
       uploader_id: objectLookup[f.message_id].author,
@@ -368,6 +396,7 @@ const files_pt2 = await mongo
   })
   .toArray();
 
+if (files_pt2.length > 0) {
 await mongo
   .db("revolt")
   .collection("attachment_hashes")
@@ -403,3 +432,10 @@ for (const file of files_pt2) {
       }
     );
 }
+} else {
+  console.log("No files to process in files_pt2.");
+}
+
+await mongo.close();
+console.log("Migration completed successfully.");
+process.exit(0);

And while the autumn server base URL responds with {"autumn":"Hello, I am a file server!","version":"0.8.1"}, my assets are still not found:

[0ktav1us]

It worked, thanks, but I get the same error {"type":"NotFound","location":"crates/services/autumn/src/api.rs:372:20"}

[ThomasBaruzier]

It worked, thanks, but I get the same error {"type":"NotFound","location":"crates/services/autumn/src/api.rs:372:20"}

Well, we're in this together. I'll see what I can do soon enough but i'm pretty sure it's out of my reach.

[0ktav1us]

// Ignore files that haven't been attached, According to this condition we get an error

[0ktav1us]

I think we need to compare this piece of code with the old autumn, see how the attachments worked in the old one

[ThomasBaruzier]

In my opinion, the migration is not doing its job properly because of my manual dirty fix, hence the missing of attachment retrieval and this error

[ThomasBaruzier]

Hello,

I don't want to put any pressure on this, but it would be great if that issue got solved somehow. I'd be happy to cooperate.

Thank you!

[0ktav1us]

Please help us to resolve it

[poggingfish]

I am having the same issue.. Is there any temporary solution?

[ThomasBaruzier]

Not for now, but I haven't tried hard fixing it. I only fixed the migrarion errors

[poggingfish]

I would really love to see this issue get any amount of attention from the devs, I moved to matrix because I didn't want to have to deal with this. The devs really need to start focusing more on the self hosting process.

[insertish]

Hi, sorry, I'm quite behind on issues across the entire project. I've added the missing const to the script but I would need more details to understand the issue at hand.

If someone has a file that appears 'NotFound' and can send the database document here that would be appreciated, I can guide you through exporting the document if needed.

[ThomasBaruzier]

Hi @insertish,

Following up on providing database documents for a "NotFound" attachment

1. Attachment document (from db.attachments): file id: 9MWzftYHh2x8VhcZ0ubm7rJunNeVkz8SOPRn3p0ZqJ

{
  "_id": "9MWzftYHh2x8VhcZ0ubm7rJunNeVkz8SOPRn3p0ZqJ",
  "tag": "attachments",
  "filename": "Screenshot_2024-09-07-19-34-13-389_chat.revolt.jpg",
  "metadata": { "type": "Image", "width": Long("1080"), "height": Long("2340") },
  "content_type": "image/jpeg",
  "size": Long("168327"),
  "message_id": "01J76RT5J8SE9DPGGB74V0EZMN"
}

2. Message document (from db.messages): message id: 01J76RT5J8SE9DPGGB74V0EZMN

{
  "_id": "01J76RT5J8SE9DPGGB74V0EZMN",
  "nonce": "01J76RT5F48TYD01FCD07Z0P03",
  "channel": "01HZYD0FQGKR2S4N5NR8A492K2",
  "author": "01HZXANW0N6WBTSS7NF7EBG2GE",
  "content": "",
  "attachments": [
    {
      "_id": "9MWzftYHh2x8VhcZ0ubm7rJunNeVkz8SOPRn3p0ZqJ",
      "tag": "attachments",
      "filename": "Screenshot_2024-09-07-19-34-13-389_chat.revolt.jpg",
      "metadata": { "type": "Image", "width": 1080, "height": 2340 },
      "content_type": "image/jpeg",
      "size": 168327
    }
  ]
}

3. attachment_hashes collection check: querying for this attachment in attachment_hashes yields no results:

revolt> db.attachment_hashes.findOne({ file_id: "9MWzftYHh2x8VhcZ0ubm7rJunNeVkz8SOPRn3p0ZqJ" })
null
revolt> db.attachment_hashes.findOne({ _id: "9MWzftYHh2x8VhcZ0ubm7rJunNeVkz8SOPRn3p0ZqJ" })
null

Using db.attachment_hashes.find() shows that its first entry is from March 2, 2025, so 3 months ago. There are no older elements to be found.

[poggingfish]

Any plans on fixing this?