Rsync endpoint for http://openresty.org/package/

[taoky]

trafficstars

@zenithalhourlyrate, as the maintainer of TUNA mirrors, reported a syncing issue in #200. Although this issue has been resolved, it would still be very helpful to provide an rsync endpoint for open-source mirror sites, as it could help:

• Reduce redundant requests and file transmission through network, and saves disk space of downstream, as many files (directories) are actually symlinks.
• Keep the integrity of synced packages better. I'm not sure whether openresty packages are behind CDN, but as far as I know the file listing by CDN may not be updated and thus HTTP-based syncing programs (lftp, etc.) cannot get a full list of packages and metadata files and bring users trouble.

I'm also the maintainer of one of the downstream mirrors of openresty (also we're users of openresty actually!). After a discussion with @zenithalhourlyrate, we think that I should open up a new issue to follow up the progress, as the original issue #200 is closed now.

[taoky]

BTW, all mirrors syncing from https://openresty.org/package/ seems not working since 7 days ago:

https://mirrorz.org/list/openresty

[taoky]

@xiaocang Are there any configuration changes in package server recently?

cc @zenithalhourlyrate

[ZenithalHourlyRate]

Hi, we downstream mirrors found that we are unable to lftp from https://openresty.org/package/

steps to reproduce

cd /tmp
lftp https://openresty.org/package/
lftp openresty.org:/package> mirror --verbose --skip-noaccess
Transferring file `[email protected]'
Transferring file `pubkey.gpg'
Making directory `alinux'
Transferring file `alinux/openresty.repo'
Making directory `alinux/2'
alinux/2: Getting files information (50%) [Receiving data]

And it will be stuck there.

We will consider removing this mirror and redirect requests back to the main server if this failure continues to happen for some time.

[ZenithalHourlyRate]

https://mirrors.tuna.tsinghua.edu.cn/news/remove-julia-openresty-mysql-ftp/

[taoky]

Interestingly, it seems that openresty.org/package sometimes returns "502 Bad Gateway" to users and it stucks lftp:

write(4, "HEAD /package/ HTTP/1.1\r\nHost: o"..., 109) = 109
read(4, 0x7ff92b4b1340, 65536)          = -1 EAGAIN (Resource temporarily unavailable)
select(5, [4], [], NULL, {tv_sec=32, tv_usec=898687}) = 1 (in [4], left {tv_sec=32, tv_usec=789220})
read(4, "HTTP/1.1 502 Bad Gateway\r\nDate: "..., 65536) = 11938
close(4)                                = 0
munmap(0x7ff92b4b1000, 69632)           = 0
madvise(0x7ff92b5c8000, 4096, MADV_FREE) = 0
select(0, 0x55d9c97c72e0, 0x55d9c97c7360, NULL, {tv_sec=1, tv_usec=0}) = 0 (Timeout)
select(0, 0x55d9c97c72e0, 0x55d9c97c7360, NULL, {tv_sec=1, tv_usec=0}) = 0 (Timeout)
select(0, 0x55d9c97c72e0, 0x55d9c97c7360, NULL, {tv_sec=1, tv_usec=0}) = 0 (Timeout)
select(0, 0x55d9c97c72e0, 0x55d9c97c7360, NULL, {tv_sec=1, tv_usec=0}) = 0 (Timeout)
select(0, 0x55d9c97c72e0, 0x55d9c97c7360, NULL, {tv_sec=1, tv_usec=0}) = 0 (Timeout)
select(0, 0x55d9c97c72e0, 0x55d9c97c7360, NULL, {tv_sec=1, tv_usec=0}) = 0 (Timeout)
select(0, 0x55d9c97c72e0, 0x55d9c97c7360, NULL, {tv_sec=1, tv_usec=0}) = 0 (Timeout)
select(0, 0x55d9c97c72e0, 0x55d9c97c7360, NULL, {tv_sec=1, tv_usec=0}) = 0 (Timeout)
select(0, 0x55d9c97c72e0, 0x55d9c97c7360, NULL, {tv_sec=1, tv_usec=0}) = 0 (Timeout)
select(0, 0x55d9c97c72e0, 0x55d9c97c7360, NULL, {tv_sec=1, tv_usec=0}) = 0 (Timeout)
select(0, 0x55d9c97c72e0, 0x55d9c97c7360, NULL, {tv_sec=1, tv_usec=0}) = 0 (Timeout)
select(0, 0x55d9c97c72e0, 0x55d9c97c7360, NULL, {tv_sec=1, tv_usec=0}) = 0 (Timeout)
... (endless select() and timeout)

Currently we excluded "alinux", "tlinux" and "oracle" folders to workaround this issue. It seems that we also can do nothing but remove openresty repo and redirect all users' traffic back to openresty.org if this still happens later on.