Mirroring treasuredata fails

[jmellor]

Help! I am unsuccessfully attempting to construct an internal mirror of the Ubuntu treasuredata repository using debmirror on a trusty machine that mirrors many other repositories, and suspecting that there is something wrong with the debian repository site.

The call to debmirror looks like:

debmirror --no-source --no-check-gpg --rsync-extra=none
-a amd64 -s contrib -h td-agent-package-browser.herokuapp.com
-d trusty -r /2/ubuntu/trusty
--debug --progress --nocleanup --method=https
/home/treasuredata-mirror

which results in the following error log:

$ sudo ./update-treasuredata-mirror [sudo] password for esadmin: Warning: --rsync-extra is not configured to mirror the trace files. This configuration is not recommended. Mirroring to /home/treasuredata-mirror from https://[email protected]/2/ubuntu/trusty/ Arches: amd64 Dists: trusty Sections: contrib Pdiff mode: use Not checking Release gpg signatures. Will NOT clean up. Attempting to get lock ... Getting meta files ... https://td-agent-package-browser.herokuapp.com/2/ubuntu/trusty/dists/trusty/Release => [ 0%] Getting: dists/trusty/Release... #* GET https://td-agent-package-browser.herokuapp.com/2/ubuntu/trusty/dists/trusty/Release ==> 200 OK 200 OK ok https://td-agent-package-browser.herokuapp.com/2/ubuntu/trusty/dists/trusty/Release.gpg => [ 0%] Getting: dists/trusty/Release.gpg... #* GET https://td-agent-package-browser.herokuapp.com/2/ubuntu/trusty/dists/trusty/Release.gpg ==> 200 OK 200 OK ok Release file contains neither Codename nor Suite; using trusty. /usr/bin/debmirror: Created directory: dists/trusty/contrib/binary-amd64/ /usr/bin/debmirror: Created directory: .temp/dists/trusty/contrib/binary-amd64/ dists/trusty/contrib/binary-amd64/Packages.gz does not exist locally or in Release, skipping. Parsing Packages and Sources files ... No packages after parsing Packages and Sources files! WARNING: releasing 1 pending lock...

I've narrowed the problem down to:

$ curl https://td-agent-package-browser.herokuapp.com/2/ubuntu/trusty/dists/trusty/contrib/binary-amd64/Release Unable to load bucket: packages.treasuredata.com

This looks like it might be some kind of proxy error of the server-side. Is it possible to identify what the problem is, and how to get around it? Should I be using some other mirror tool that gets around this?

[repeatedly]

We don't have a experience with mirroring tool so we don't have a good answer for this issue... Our repository is hosted on S3 and we use reprepro for creating repository. Does S3 or reprepro have a problem with mirroring tool?

[jmellor]

I think that the repository layout is incorrect. If I look at the distribution website, I can see:

packages.treasuredata.com/2/Ubuntu/trusty/dists/contrib

which is not correct. This layout means that there is prefix of 2/Ubuntu/trusty and a distribution called contrib and no component layer, so this is not a valid repository layout. The dists directory is currently at the wrong layer in the tree.

It should instead be:

packages.treasuredata.com/2/Ubuntu/dists/trusty/contrib

so that the distribution component would then be trusty and the component would be contrib.

Is it possible to fix this?

From: Masahiro Nakagawa [mailto:[email protected]] Sent: July-20-16 03:48 To: fluent/fluentd-docs Cc: John Mellor; Author Subject: Re: [fluent/fluentd-docs] Mirroring treasuredata fails (#235)

We don't have a experience with mirroring tool so we don't have a good answer for this issue... Our repository is hosted on S3 and we use reprepro for creating repository. Does S3 or reprepro have a problem with mirroring tool?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/fluent/fluentd-docs/issues/235#issuecomment-233870528, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AKkBZaBRTO4Vm9_vw-O2aPoxNLsoqr9Iks5qXdKwgaJpZM4JPskY.

[jmellor]

I believe that there is an error either in the S3 bucket configuration, or in the web service that it is running.

I have attached a screenshot showing a firefox query for the Release file, along with the error response that appears to show that the Amazon site is misconfigured.

Is the repository site not public, and therefore only works for yourself or something simple like that?

From: Masahiro Nakagawa [mailto:[email protected]] Sent: July-20-16 03:48 To: fluent/fluentd-docs Cc: John Mellor; Author Subject: Re: [fluent/fluentd-docs] Mirroring treasuredata fails (#235)

We don't have a experience with mirroring tool so we don't have a good answer for this issue... Our repository is hosted on S3 and we use reprepro for creating repository. Does S3 or reprepro have a problem with mirroring tool?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/fluent/fluentd-docs/issues/235#issuecomment-233870528, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AKkBZaBRTO4Vm9_vw-O2aPoxNLsoqr9Iks5qXdKwgaJpZM4JPskY.

[jmellor]

(sorry, it looks like the github mail service is losing images. Here is the error image:

[yuvraj2987]

I am seeing this issue for macOS agent as well.

[repeatedly]

@yuvraj2987 Sorry, we accidentally revoke permission for this browser. We restore our setting and you now see the repository.

[silverbadge]

Hi,

We use Artifactory to proxy/cache remote repositories. We're getting this the same issue:

Unable to load bucket: packages.treasuredata.com

Are you able to help with this?

[fujimotos]

I'm looking into this issue and poking around for few minutes.

Apparently, the web app (S3 Browser) is just a directory viewer and not designed to proxy files. For actual contents of files, we need to access to the S3 endpoint directly.

In other words, we cannot retrieve a file from:

https://td-agent-package-browser.herokuapp.com/2/ubuntu/trusty/dists/trusty/contrib/binary-amd64/Release

Instead, we need to access to the following URL:

http://packages.treasuredata.com.s3.amazonaws.com/2/ubuntu/trusty/dists/trusty/contrib/binary-amd64/Release

This observation leads us to the following script (I modified the command posted by @jmellor):

debmirror --no-source --no-check-gpg --rsync-extra=none --progress --nocleanup \
          --method=https --disable-ssl-verification \
          -h packages.treasuredata.com.s3.amazonaws.com \
          -a amd64 -s contrib -d trusty -r /2/ubuntu/trusty \
          /path/to/designed/dir/

This should properly mirror the remote files to you local directory.

(Beware the "--disable-ssl-verification" option in the script, though. We need this opt because Amazon does not provide us a proper SSL certificate for S3 endpoints; So it is not fully secure)

[chicofranchico]

For those using Aptly, you can use the following URL e.g. (for xenial v3)

http://packages.treasuredata.com.s3.amazonaws.com/3/ubuntu/xenial/ xenial contrib

And then:

aptly mirror create -architectures=amd64 <mirror_name> http://packages.treasuredata.com.s3.amazonaws.com/3/ubuntu/xenial/ xenial contrib