Feature Request: Support for Mirroring Debian Repositories with S3

[marmik18]

Hey,

Thank you all for maintaining such a great project! I have a feature request regarding repository mirroring in Aptly.

Current Situation:

Aptly currently supports mirroring Debian repositories that are exported via HTTP(S) or FTP. Since Aptly already supports publishing repositories to S3, I believe it should also support mirroring repositories hosted on S3.

What I am Trying to Achieve:

I am setting up a private repository for Debian packages on S3, where all package versions are stored.

• I successfully created a repository, took a snapshot, and published it to an S3 bucket.
• On the client side, I can install packages using apt-transport-s3.

The Problem:

Since I am building my packages in a CI/CD pipeline, I need a way to:

• Pull all the existing packages from my S3 repository.
• Build a new version of the package.
• Create a new repository with the updated .deb files.
• Take a snapshot and publish it.

Currently, Aptly does not support mirroring repositories stored on S3, which makes it difficult to fetch and manage package versions efficiently.

Request:

Would it be possible to add support for mirroring Debian repositories hosted on S3 in Aptly?

I am relatively new to Debian repositories, so please correct me if I am misunderstanding something.

Thank you in advance! 😊

[neolynx]

Hi !

good idea :-) Duplicate of https://github.com/aptly-dev/aptly/issues/455#issuecomment-2665373614 ?

one other way could be: https://github.com/aptly-dev/aptly/issues/843

I would start hosting the pool in S3, like https://github.com/aptly-dev/aptly/pull/1074 and then centralize the db (i.e. like Azure Cosmos DB as suggested in https://github.com/aptly-dev/aptly/issues/230, but for aws).

This would avoid downloading your packages first, and s3 pool is used directly.

What do you think ?

[marmik18]

That makes sense, I could push the directory ~/.aptly to s3 and then sync them with every container to keep the state synced.

Is my understanding correct?

[connoranderson]

We are looking into doing something similar. I don't think syncing ~/.aptly is desirable since that would include .aptly/pool which contains lots of potentially-large files? Are there any potential solutions which could sync only the few packages with updates or is the entire apt repo required on-disk from the publishing machine?