Implement Package Tracking for Efficient Syncing #19

[allamiro]

Issue Description:

Currently, the minirepo.py script does not have the capability to track downloaded packages and only sync new or updated packages during subsequent runs. This can lead to unnecessary re-downloads of packages that have already been fetched, resulting in wasted bandwidth, increased runtime, and redundant storage use. Findings:

• Redundant Downloads
• No Version or Update Checking
• Absence of Metadata Storage : There is no local storage of metadata about downloaded packages (such as package name, version, size, and checksum)

Proposed Solution:

• Implement Local Metadata Storage by creating a local metadata file (e.g., a JSON file) to store details about downloaded packages, including package name, version, file size, and MD5 hash.
• Pre-Download Check Against Metadata
• Sync Only New or Updated Packages the script should check the local metadata
• Use the metadata to identify and download only new or updated packages during subsequent runs

Benefits:

By only downloading new or updated packages, the script will significantly reduce unnecessary data transfer, conserving bandwidth.