dojo Uploading metadatas should be skipped when metadata didn't change

Describe the bug

We should only upload the metadata to IPFS if they have changed from the last upload. For which we can store the IPFS hash of metadata in the manifest file, and upload the metdata only if IPFS hash has changed.

May 01 '24 12:05 notV4l

Good point. One improvement we could is to output the ipfs metadata uris into the manifest files. So unless it changes, we don't upload to ipfs. Another thing we could do, is to embed a local ipfs server in torii, so uploading is super quick. We will still want the contract metadata set during dev, to enable the worlds.dev interface

What do you think @lambda-0x @glihm @Larkooo

May 01 '24 13:05 tarrencev

Good point. One improvement we could is to output the ipfs metadata uris into the manifest files. So unless it changes, we don't upload to ipfs. Another thing we could do, is to embed a local ipfs server in torii, so uploading is super quick. We will still want the contract metadata set during dev, to enable the worlds.dev interface

What do you think @lambda-0x @glihm @Larkooo

its skipped when using offline mode and should remain usable, so embed ipfs server seems the way

May 01 '24 14:05 notV4l

Good point. One improvement we could is to output the ipfs metadata uris into the manifest files. So unless it changes, we don't upload to ipfs. Another thing we could do, is to embed a local ipfs server in torii, so uploading is super quick. We will still want the contract metadata set during dev, to enable the worlds.dev interface

What do you think @lambda-0x @glihm @Larkooo

I think that with the amount of responsibility that torii now has, with the different services under it, like the libp2p server, the different endpoints and everything together; it only makes sense to embed a ipfs server for maximum reliability and to have it standalone. Especially with the libp2p torii-relay as it's basically the same kind of scope as if we were to embed our own ipfs instance. I do think that we should still have the option to specify a specific ipfs server if wanted though.

https://github.com/ipfs-rust/ipfs-embed

The workflow seems quite similar to libp2p-rust in general as it uses it so I think it should be quite straightforward to implement at some point.

May 01 '24 16:05 Larkooo

I would like to give it a try please

May 20 '24 13:05 Yogalholic

@Yogalholic i have updated the title and description of the issue to give more context of what should be done. Feel free to ask if you have any questions!

May 21 '24 06:05 itzlambda

@Yogalholic any updates on the issue? if you have any questions feel free to ask.

May 28 '24 21:05 itzlambda

I am still working on it. I don't have a specific question at the moment, there are a lot of new information to process and the dojo documentation is mostly toward game developers rather than dojo developers.

May 29 '24 03:05 Yogalholic

For which we can store the IPFS hash of metadata in the manifest file

Am I allowed to add a new field in the DeploymentManifest struct in types.rs ? This file is not in the sozo crate but in dojo-world.

May 29 '24 12:05 Yogalholic

For which we can store the IPFS hash of metadata in the manifest file

Am I allowed to add a new field in the DeploymentManifest struct in types.rs ? This file is not in the sozo crate but in dojo-world.

yeah, for sure.

Jun 02 '24 17:06 itzlambda

@Yogalholic how are you doing on that? Any progress or blockers?

Jun 19 '24 21:06 glihm

I will open a draft PR soon

Jun 20 '24 17:06 Yogalholic

I noticed migrate.rs file got updated while I was working on it. The code no longer handles ArtifactMetadata struct but ResourceMetadata struct

Jun 21 '24 12:06 Yogalholic

@Yogalholic ResourceMetadata is a wrapper around ArtifactMetadata with name field, so it shouldn't be an issue you can just merge with main branch.

Jun 21 '24 12:06 itzlambda

I have opened a draft PR, I want to know if I am on the right track. I am not sure which encryption is used to calculate the IPFS hash

Jun 22 '24 09:06 Yogalholic

I am applying to this issue via OnlyDust platform.

My background and how it can be leveraged

I have extensive experience building apps with C#, js, rust and cairo and would like to tackle this challenge

How I plan on tackling this issue

I would study the codebase to get a good hang of the systems responsible for uploading the metadata to IPFS, before creating a method for storing the ipfs hash of the data before subsequently comparing new hashes before uploading if there is a diff

Aug 30 '24 18:08 RichoKD

I am applying to this issue via OnlyDust platform.

My background and how it can be leveraged

Hi I'm a frontend and also a smart contract develop. i will love to work on this please assign

Aug 30 '24 23:08 CollinsC1O

I am applying to this issue via OnlyDust platform.

My background and how it can be leveraged

Hii, Abhilov this side. I worked as a blockchain developer for over an year. I would like resolve this issue.

Aug 31 '24 20:08 abhilov23

I am applying to this issue via OnlyDust platform.

My background and how it can be leveraged

hello

Sep 25 '24 10:09 terabitiah2

I am applying to this issue via OnlyDust platform.

My background and how it can be leveraged

Based on your description, here's how I would approach implementing this solution:

Understand the Manifest Structure The first step is to understand how the manifest file is structured. This file will need to store the metadata’s IPFS hash after each upload to compare against future metadata changes.

Adding the Hash to the Manifest: After a successful upload, we will store the IPFS hash of the uploaded metadata in the manifest file. This allows us to reference it during the next deployment to check if the metadata has changed.

Example: { "world": { "metadata_hash": "QmXyz1234...", "last_deployed": "2024-09-26" } } 2. Generate the IPFS Hash Locally Before Uploading To prevent unnecessary uploads, we will generate the hash locally (without uploading to IPFS) using the metadata content itself. This can be done with a hashing algorithm like SHA-256 or through IPFS's own hash generation methods.

Compare the Local Hash with the Stored Hash: If the local hash matches the hash in the manifest, skip the upload. If not, proceed with the IPFS upload and update the manifest. Example: def should_upload_metadata(new_metadata): stored_hash = get_stored_hash_from_manifest()

# Generate local hash from new metadata
new_metadata_hash = generate_ipfs_hash(new_metadata)

if new_metadata_hash == stored_hash:
    return False  # No need to upload, metadata hasn't changed
else:
    return True   # Metadata has changed, proceed with upload

Updating the Manifest File After successfully uploading the new metadata to IPFS (only if needed), we need to update the manifest file with the new hash and potentially a timestamp for when the update occurred.

Ensure Manifest Integrity: Update the manifest safely, ensuring no data corruption. Use atomic file writes or backups to maintain consistency during the update process. Example: def update_manifest(new_hash): manifest["world"]["metadata_hash"] = new_hash manifest["world"]["last_deployed"] = get_current_date() save_manifest_to_file(manifest) 4. Version Control for Metadata Storing the hash ensures we track when the metadata changes, but it’s also helpful for version control. You could log previous hashes or metadata changes in case of troubleshooting or audit requirements.

Log Previous Hashes: Maintain a history of hashes in the manifest file or a separate log for reference. Example: { "world": { "metadata_history": [ { "hash": "QmXyz1234...", "date": "2024-09-20" }, { "hash": "QmAbc5678...", "date": "2024-09-25" } ] } } 5. Testing the Hash Comparison Logic Ensure you have unit tests that:

Verify the system detects whether metadata has changed correctly. Confirm that unnecessary IPFS uploads are prevented. Ensure manifest file updates correctly after each deployment. My Background and How It Can Be Leveraged: Programming and System Design Experience:

I have experience in Python development, which allows me to design and implement the logic for checking metadata changes, generating hashes, and updating manifests efficiently. I can also help in designing a robust mechanism to handle manifest updates without data corruption or unnecessary writes. Experience with IPFS and Decentralized Storage:

My familiarity with decentralized storage like IPFS can be leveraged to ensure the metadata is handled properly. I can design an efficient solution to compare IPFS hashes locally before uploading, reducing resource consumption and ensuring only necessary uploads occur. Version Control and Optimization:

I have a good understanding of version control concepts and how to track changes over time. This can be applied to ensure metadata is versioned properly, and the system optimally decides when to upload and update metadata. Testing and Debugging Skills:

I can assist in writing comprehensive tests to ensure the entire process works as expected and can identify any edge cases where metadata changes might not be properly detected. By leveraging these skills, I can help ensure that the system only uploads metadata when necessary, improving both efficiency and reliability.

How I plan on tackling this issue

How I Would Approach the Problem:

Understand the Current Process: Current Flow: Review the current deployment flow to understand how and when the metadata is being uploaded to IPFS. Identify Changes: Locate where the IPFS upload happens in the deployment pipeline and identify how the metadata is being generated and uploaded.
Store Metadata IPFS Hash in the Manifest: Purpose of the Manifest File: The manifest file will act as a record for the previously uploaded metadata's IPFS hash. Modify Manifest Structure: Add a field in the manifest to store the IPFS hash. This will allow us to track the last successfully uploaded metadata. Example: { "world": { "metadata_hash": "QmXyz123456...", "last_deployed": "2024-09-26" } }
Generate and Compare IPFS Hash: Generate Local Hash: Before attempting to upload the metadata, generate its IPFS hash locally based on the content (using the same algorithm that IPFS uses). This allows us to compare it with the previous hash stored in the manifest. Hash Comparison Logic: Compare the newly generated hash with the stored hash in the manifest: If the hash matches, skip the upload (since the metadata hasn’t changed). If the hash is different, proceed with uploading the new metadata to IPFS and update the manifest with the new hash. Example: def should_upload_metadata(new_metadata): stored_hash = get_stored_hash_from_manifest()

Generate local hash from new metadata

new_metadata_hash = generate_ipfs_hash(new_metadata)

if new_metadata_hash == stored_hash: return False # No need to upload, metadata hasn't changed else: return True # Metadata has changed, proceed with upload
Upload Metadata to IPFS (if needed): Conditional Upload: Only upload the metadata if the hash has changed. Update the Manifest: After uploading new metadata to IPFS, update the manifest with the new IPFS hash and deployment date. Example: def upload_metadata_if_needed(new_metadata): if should_upload_metadata(new_metadata): ipfs_hash = upload_to_ipfs(new_metadata) # Upload to IPFS update_manifest(ipfs_hash) # Update manifest with new hash else: print("Metadata unchanged, skipping upload.")
Update the Manifest File: Safe Manifest Update: Once the metadata is successfully uploaded, update the manifest file with the new hash and any relevant information (like the last deployment timestamp). Example: def update_manifest(new_hash): manifest["world"]["metadata_hash"] = new_hash manifest["world"]["last_deployed"] = get_current_date() save_manifest_to_file(manifest)
Test the Solution: Unit Tests: Write tests to ensure that: The hash is being correctly generated from metadata. Metadata is uploaded only when necessary (i.e., when there is a change). The manifest is updated correctly after a successful upload. Edge Cases: Test for cases where metadata might change minimally (or involve non-hashable changes) to ensure the comparison is accurate.

Sep 26 '24 08:09 thesledge04

I am applying to this issue via OnlyDust platform.

My background and how it can be leveraged

I'm a developer with experience in several of the languages used in this project, including Python, JavaScript, and TypeScript. While I haven't worked extensively with Cairo or Rust, I'm familiar with their concepts and eager to expand my skills. I've dealt with optimization problems similar to this one in past projects, particularly involving file handling and data comparison. My experience with IPFS in previous projects will be valuable in tackling this issue.

How I plan on tackling this issue

To address this bug, I'd start by examining the current implementation of metadata uploading. Then, I'd modify the process to include these steps:

Generate a hash of the current metadata before uploading. Read the manifest file to retrieve the previously stored IPFS hash. Compare the new hash with the stored one. If they're different, proceed with the IPFS upload and update the manifest with the new hash. If they're the same, skip the upload to save time and resources.

I'd implement this logic, likely in Python or TypeScript depending on the project's structure. I'd ensure to handle edge cases, like a missing manifest file or IPFS hash. After implementation, I'd thoroughly test the changes, including scenarios where metadata changes and where it doesn't. I'd also measure the performance impact to quantify the optimization.

Oct 07 '24 17:10 akintewe

I am applying to this issue via OnlyDust platform.

My background and how it can be leveraged

I am Full-stack Web Developer with over 4+ years of Experience

How I plan on tackling this issue

Steps to Implement Conditional Metadata Upload

Store the IPFS Hash in a Manifest File Create a manifest file that records the IPFS hash of the last uploaded metadata. This file will serve as a reference point for future uploads.

Example : { "metadataHash": "QmExampleHash", "lastUpdated": "2024-10-16T12:00:00Z" }

Calculate the New Metadata Hash Before uploading new metadata, compute its hash using a suitable hashing algorithm (e.g., SHA-256). This will help you determine if the metadata has changed. Example Code: javascript const crypto = require('crypto');

function calculateHash(metadata) { return crypto.createHash('sha256').update(JSON.stringify(metadata)).digest('hex'); }

Compare Hashes Before Uploading When preparing to upload metadata, compare the newly calculated hash with the hash stored in the manifest file.

Comparison Logic: javascript async function uploadMetadataIfChanged(newMetadata) { const manifest = await fetchManifest(); // Fetch current manifest const currentHash = manifest.metadataHash; const newHash = calculateHash(newMetadata);

if (currentHash !== newHash) {
    const ipfsHash = await uploadToIPFS(newMetadata); // Function to handle IPFS upload
    updateManifest(ipfsHash); // Update manifest with new hash
} else {
    console.log('Metadata has not changed; skipping upload.');
}

}

Upload Metadata to IPFS Use an appropriate method or API to upload your new metadata to IPFS only if it has changed. Using Tatum API Example: javascript async function uploadToIPFS(metadata) { const response = await someIPFSAPI.upload(metadata); // Replace with actual API call return response.ipfsHash; // Return the new IPFS hash }
Update the Manifest File After a successful upload, update the manifest file with the new IPFS hash and any relevant timestamps. Updating Logic: javascript async function updateManifest(newIpfsHash) { const manifest = { metadataHash: newIpfsHash, lastUpdated: new Date().toISOString() }; await saveManifest(manifest); // Function to save updated manifest }