Use external DB

[kumy]

May it be possible to use external database like PostgreSql, MySql.... ?

[smira]

Something like Redis seems more appropriate, but the only requirement is to be able to satisfy this interface Storage: https://github.com/smira/aptly/blob/master/database/leveldb.go#L20

[smira]

And package pool could be reimplemented on top of Amazon S3 or something like that :)

[kumy]

What would be cool,

• is to launch the api server and be able launch aptly cli for readonly features. (bypass database lock, for read only)
• or (in our environment) launch search queries while aptly serve api (access the database on our own(ro))

Here, as of today, we have hundreds of mirrors, searching for packages by name doesn't give us which mirror contain the package. Asking for PostgreSql (or other) support would allow us to search directly in database... And give our users sources.list lines to configure on they server.

[smira]

It won't help in your case. Data is stored as serialized BLOBs which would be hard to unserialize without aptly. It is much easier to add something that you need to aptly API. (I believe you're talking about aptly package search -with-references).

[kumy]

I understand your point of view, & I didn't know about -with-references

but

aptly package search -with-references aptly
flag provided but not defined: -with-references
Usage: aptly package search <package-query>

Options:
  -architectures="": list of architectures to consider during (comma-separated), default to all available
  -config="": location of configuration file (default locations are /etc/aptly.conf, ~/.aptly.conf)
  -dep-follow-all-variants=false: when processing dependencies, follow a & b if depdency is 'a|b'
  -dep-follow-recommends=false: when processing dependencies, follow Recommends
  -dep-follow-source=false: when processing dependencies, follow from binary to Source packages
  -dep-follow-suggests=false: when processing dependencies, follow Suggests

aptly package search - search for packages matching query

hum, maybe you're talking about implementing such a feature ;)

[smira]

Sorry, it should be aptly package show

[kumy]

seems better, thx.

Now I have to bench it for 2 millions packages!!!

# SELECT count(*) from pacmr_pacmr;
  count  
---------
 2283600
(1 row)

;)

[sigmonsays]

I would second a external database like redis for fault tolerance.

When you publish packages to an external storage system like S3, the only thing of real importance is the aptly database. If the aptly database was synchronized to remote storage as a backup, then it may be less of an issue to restore the atply database. However, I would imagine if the database were in redis, performing a backup and making it fault tolerant (in case of node failure) would be much easier.

If this is something people are interested in, perhaps I can take some time to add redis support to implement the Storage interface mentioned.

[smira]

@sigmonsays package pool should be implemented on some external storage as well, e.g. S3. Once again it's a questions of building implementation that satisfies interface (internal package pool, pool/ in aptly directory).

[sigmonsays]

fantastic!

[mgrrx]

It would be lovely to have this feature. Since aptly recently got more traction again ( <3 ) I wanted to ask if there is a chance that this gets implemented or, in case somebody would work on it, has a chance of getting merged (meaning is this still aligned with the goals of the new maintainers)?