aptly-dev
Error #01: Needed resources are used by other tasks.
Hi! Need help with publishing packages!
Packages are assembled on Jenkins, and then they are further uploaded and published to the repository using the aptly api:
curl -f -X POST -F file=@"libson-1.11.6248_amd64.deb" http://repos.local/deb/api/files/uploaded-files?_async=true -u repouser:repopassword
curl -f -X POST http://repos.local/deb/api/repos/debrepo/file/uploaded-files/libson-1.11.6248_amd64.deb?_async=true -u repouser:repopassword
curl -f -X PUT http://repos.local/deb/api/publish/:./debrepo?_async=true -H 'Content-Type: application/json' --data '{"Signing":{"Batch":true,"GpgKey":"gpgkey"}}' -u repouser:repopassword
curl -f -X PUT http://repos.local/deb/api/publish/:./ubuntu-20.04?_async=true -H 'Content-Type: application/json' --data '{"Signing":{"Batch":true,"GpgKey":"gpgkey"}}' -u repouser:repopassword
curl -f -X PUT http://repos.local/deb/api/publish/:./mint?_async=true -H 'Content-Type: application/json' --data '{"Signing":{"Batch":true,"GpgKey":"gpgkey"}}' -u repouser:repopassword
curl -f -X PUT http://repos.local/deb/api/publish/:./ubuntu-22.04??_async=true -H 'Content-Type: application/json' --data '{"Signing":{"Batch":true,"GpgKey":"gpgkey"}}' -u repouser:repopassword
The problem is that there may be parallel builds, downloads, and publications at the same time, and I get the following error: aptly — "Error #01: Needed resources are used by other tasks." curl — (22) The requested URL returned error: 409 Conflict
{"level":"info","time":"2024-04-18T09:15:42+05:00","message":"Executing task asynchronously"}
{"level":"info","remote":"172.27.0.4","method":"POST","path":"/api/repos/debrepo/file/uploaded-files/libson-1.11.6248_amd64.deb?_async=true","protocol":"HTTP/1.0","code":"202","latency":"166.954µs","agent":"curl/7.58.0","time":"2024-04-18T09:15:42+05:00"}
{"level":"info","time":"2024-04-18T09:15:42+05:00","message":"Executing task asynchronously"}
{"level":"warn","remote":"172.27.0.4","method":"PUT","path":"/api/publish/:./debrepo?_async=true","protocol":"HTTP/1.0","code":"409","latency":"4.423243ms","agent":"curl/7.58.0","time":"2024-04-18T09:15:42+05:00","message":"Error #01: Needed resources are used by other tasks."}
{"level":"info","time":"2024-04-18T09:15:42+05:00","message":"Executing task asynchronously"}
{"level":"warn","remote":"172.27.0.4","method":"PUT","path":"/api/publish/:./ubuntu-20.04?_async=true","protocol":"HTTP/1.0","code":"409","latency":"4.386ms","agent":"curl/7.58.0","time":"2024-04-18T09:15:42+05:00","message":"Error #01: Needed resources are used by other tasks."}
{"level":"info","time":"2024-04-18T09:15:42+05:00","message":"Executing task asynchronously"}
{"level":"warn","remote":"172.27.0.4","method":"PUT","path":"/api/publish/:./mint?_async=true","protocol":"HTTP/1.0","code":"409","latency":"4.258301ms","agent":"curl/7.58.0","time":"2024-04-18T09:15:42+05:00","message":"Error #01: Needed resources are used by other tasks."}
{"level":"info","time":"2024-04-18T09:15:42+05:00","message":"Executing task asynchronously"}
{"level":"warn","remote":"172.27.0.4","method":"PUT","path":"/api/publish/:./ubuntu-22.04?_async=true","protocol":"HTTP/1.0","code":"409","latency":"4.272133ms","agent":"curl/7.58.0","time":"2024-04-18T09:15:42+05:00","message":"Error #01: Needed resources are used by other tasks."}
Have I tried using this version: https://github.com/aptly-dev/aptly/issues/1125#issuecomment-1331843568 and using ?_async=true, but the error persists.
I tried to build aptly from the master branch, errors persist. What else can I do, or maybe I'm doing something wrong?
Hi, thanks for reporting !
yes, this might be fixed with #1125, which is currently being rebased on master.
the ?_async=true will make the API calls return a task_id. the caller should wait for the task to complete, before issuing a next api call. this can be done with GET /tasks/:id/wait maybe you could try this ?
Thanks for the reply!
Yes, I can. But let me clarify: do I understand correctly that I need to write a shell handler that will process each curl request, take the ID and put it into GET /tasks/:ID/wait ?
Hello, @neolynx!
I apologize for the delay in replying, it took a while to check everything out.
I've added /tasks/:ID/wait in my shell script, but I still get this error when building in parallel on Jenkins:
[GIN] 2024/05/21 - 13:41:39 | 200 | 83.949261ms | 172.19.0.2 | GET "/api/tasks/9/wait"
2024/05/21 13:41:39 Executing task asynchronously
[GIN] 2024/05/21 - 13:41:39 | 202 | 2.403509ms | 172.19.0.2 | PUT "/api/publish/:./ubuntu-20.04?_async=true"
[GIN] 2024/05/21 - 13:41:39 | 200 | 87.470772ms | 172.19.0.2 | GET "/api/tasks/10/wait"
[GIN] 2024/05/21 - 13:41:47 | 200 | 310.719834ms | 172.19.0.2 | POST "/api/files/uploaded-files?_async=true"
2024/05/21 13:41:47 Executing task asynchronously
[GIN] 2024/05/21 - 13:41:47 | 202 | 115.25µs | 172.19.0.2 | POST "/api/repos/libs/file/uploaded-files/libs-generator.87.6316_amd64.deb?_async=true"
[GIN] 2024/05/21 - 13:41:47 | 200 | 307.632802ms | 172.19.0.2 | POST "/api/files/uploaded-files?_async=true"
2024/05/21 13:41:47 Executing task asynchronously
[GIN] 2024/05/21 - 13:41:47 | 409 | 116.276µs | 172.19.0.2 | POST "/api/repos/libs/file/uploaded-files/libs-generator.87.6317_amd64.deb?_async=true"
Error #01: Needed resources are used by other tasks.
Aptly version 1.5.0+ds1-1+b4 .
This is stdout of shell script with set -x:
+ echo 'Moving deb package to repository...'
Moving deb package to repository...
++ curl -f -X POST 'http://repos.local/deb/api/repos/libs/file/uploaded-files//libs-generator.87.6317_amd64.deb?_async=true' -u repo-api:aGF#EQLr
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 53 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (22) The requested URL returned error: 409
And do I understand correctly that I can't get ID and STATE on this request?
+ echo 'Uploading deb package: libsdeb///libs-generator.87.6317_amd64.deb'
Uploading deb package: libsdeb///libs-generator.87.6317_amd64.deb
++ curl -s -f -X POST -F file=@libsdeb///libs-generator.87.6317_amd64.deb 'http://repos.local/deb/api/files/uploaded-files?_async=true'
+ upload_response='["uploaded-files//libs-generator.87.6317_amd64.deb"]'
+ '[' 0 -ne 0 ']'
+ echo '["uploaded-files//libs-generator.87.6317_amd64.deb"]'
+ json_pp
[
"uploaded-files//libs-generator.87.6317_amd64.deb"
]
And is it correct that on GET /tasks/:ID/wait I should get STATE=2? Is there any actual documentation on this function somewhere?
Hi !
I looked in the problem and I think aptly does not support parallel operation, i.e. different clients sending requests at the same time. For this reason in the molior project there is a queue before all aptly operations, making in work with parallel request. The background API also does not solve the problem, is is merely allowing to do other things while waiting for aptly to complete, the requests still must be sent sequentially.
There are ideas to introduce a mutex to wait until the resources are available, but I am not sure this is safe and requests will be executed in order (imagine creating, deleting and recreating same repo. those should be executed in order as they arrive). So probably aptly should queue requests and process them in order, which whould be a bigger implementation.
In the meanwhile, I there a way to wait for all jenkins tasks to complete, and then do all aptly operations in one go, not using async ?
I implemented a queue in https://github.com/aptly-dev/aptly/pull/1271, this should allow to publish from different clients instead of returning 409.
Something like this should work for all commands which support async:
task_id=`curl -f -X PUT http://localhost:3142/api/publish/bookworm_12.5-archs-copy_repos_test_1.0-copy2/stable?_async=true -H 'Content-Type: application/json' --data '{"Signing":{"Batch":true,"GpgKey":"reposign@molior", "PassphraseFile": "/var/lib/aptly/gpg-pass"}}' | jq .ID`
curl http://repos.local/api/tasks/$task_id/wait
Could you give this a try?
Regarding your questions:
- I will check if the file upload API also support async
- yes, STATE=2 is SUCCEEDED (see task/task.go:39)
- all of this should be documented indeed, if you would like to help with this, PRs to https://github.com/aptly-dev/www.aptly.info are very welcome :)
The question now is, how to deal with non async requests. Maybe they should go through the queue as well and additionally just wait for the task to finish before returning 200 ?
The normal non-async requests are not also queued in https://github.com/aptly-dev/aptly/pull/1271, and also file uploads should work concurrently now.
Could you please give it a try?
@neolynx Hi! Sorry for the wait and thank you so much for fix! Builds have been falling less frequently.
I checked parallel builds with aptly version: 1.5.0+207+g372ce3c4. Error Needed resources are used by other tasks is no longer present
I keep getting an error when doing parallel builds on Jenkins with new error:
Error #01: open /var/www/html/devrepo/deb/upload/uploaded-files/lib_fstyle_4.4.3.7368_amd64.deb: no such file or directory
Maybe, I must open new issue?
Jenkins log from the broken job:
13:29:29 + echo '++ curl -f -X POST -F file=@/home/deb//lib_fstyle_4.4.3.7368_amd64.deb '\''http://192.168.100.150/deb/api/files/uploaded-files?_async=true'\'' -u api:pass
13:29:29 ++ json_pp
13:29:29 % Total % Received % Xferd Average Speed Time Time Time Current
13:29:29 Dload Upload Total Spent Left Speed
13:29:29
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
11 813M 0 0 11 92.6M 0 95.9M 0:00:08 --:--:-- 0:00:08 95.9M
23 813M 0 0 23 192M 0 97.8M 0:00:08 0:00:01 0:00:07 97.8M
35 813M 0 0 35 290M 0 98.0M 0:00:08 0:00:02 0:00:06 98.0M
47 813M 0 0 47 389M 0 98.1M 0:00:08 0:00:03 0:00:05 98.1M
58 813M 0 0 58 478M 0 96.2M 0:00:08 0:00:04 0:00:04 96.2M
71 813M 0 0 71 578M 0 96.9M 0:00:08 0:00:05 0:00:03 97.1M
83 813M 0 0 83 679M 0 97.5M 0:00:08 0:00:06 0:00:02 97.4M
95 813M 0 0 95 780M 0 98.0M 0:00:08 0:00:07 0:00:01 98.0M
100 813M 0 0 100 813M 0 87.5M 0:00:09 0:00:09 --:--:-- 79.6M
100 813M 0 0 100 813M 0 79.2M 0:00:10 0:00:10 --:--:-- 63.2M
13:29:29 curl: (22) The requested URL returned error: 500 Internal Server Error
aptly json log:
{
"level": "info",
"remote": "192.168.100.113",
"method": "POST",
"path": "/api/files/uploaded-files?_async=true",
"protocol": "HTTP/1.0",
"code": "200",
"latency": "26.75636ms",
"agent": "curl/7.58.0",
"time": "2024-09-14T13:32:04+05:00"
},
{
"level": "debug",
"time": "2024-09-14T13:32:04+05:00",
"message": "Executing task asynchronously"
},
{
"level": "info",
"remote": "192.168.100.113",
"method": "POST",
"path": "/api/repos/devrepo/file/uploaded-files/lib_masterstyle_4.4.3.7369_amd64.deb?_async=true",
"protocol": "HTTP/1.0",
"code": "202",
"latency": "232.443µs",
"agent": "curl/7.58.0",
"time": "2024-09-14T13:32:04+05:00"
},
{
"level": "debug",
"time": "2024-09-14T13:32:04+05:00",
"message": "Executing task asynchronously"
},
{
"level": "info",
"remote": "192.168.100.113",
"method": "PUT",
"path": "/api/publish/:./devrepo?_async=true",
"protocol": "HTTP/1.0",
"code": "202",
"latency": "5.076542ms",
"agent": "curl/7.58.0",
"time": "2024-09-14T13:32:04+05:00"
},
{
"level": "debug",
"time": "2024-09-14T13:32:04+05:00",
"message": "Executing task asynchronously"
},
{
"level": "info",
"remote": "192.168.100.113",
"method": "PUT",
"path": "/api/publish/:./ubuntu-20.04?_async=true",
"protocol": "HTTP/1.0",
"code": "202",
"latency": "4.331976ms",
"agent": "curl/7.58.0",
"time": "2024-09-14T13:32:04+05:00"
},
{
"level": "debug",
"time": "2024-09-14T13:32:04+05:00",
"message": "Executing task asynchronously"
},
{
"level": "info",
"remote": "192.168.100.113",
"method": "PUT",
"path": "/api/publish/:./ubuntu-22?_async=true",
"protocol": "HTTP/1.0",
"code": "202",
"latency": "4.326821ms",
"agent": "curl/7.58.0",
"time": "2024-09-14T13:32:04+05:00"
},
{
"level": "debug",
"time": "2024-09-14T13:32:04+05:00",
"message": "Executing task asynchronously"
},
{
"level": "info",
"remote": "192.168.100.113",
"method": "PUT",
"path": "/api/publish/:./ubuntu-24?_async=true",
"protocol": "HTTP/1.0",
"code": "202",
"latency": "4.303508ms",
"agent": "curl/7.58.0",
"time": "2024-09-14T13:32:04+05:00"
},
{
"level": "debug",
"time": "2024-09-14T13:32:04+05:00",
"message": "Executing task asynchronously"
},
{
"level": "info",
"remote": "192.168.100.113",
"method": "PUT",
"path": "/api/publish/:./astra-ce-2.12?_async=true",
"protocol": "HTTP/1.0",
"code": "202",
"latency": "4.404118ms",
"agent": "curl/7.58.0",
"time": "2024-09-14T13:32:04+05:00"
},
{
"level": "error",
"remote": "192.168.100.113",
"method": "POST",
"path": "/api/files/uploaded-files?_async=true",
"protocol": "HTTP/1.0",
"code": "500",
"latency": "1.978734261s",
"agent": "curl/7.58.0",
"time": "2024-09-14T13:32:04+05:00",
"message": "Error #01: open /var/www/html/devrepo/deb/upload/uploaded-files/lib_fstyle_4.4.3.7368_amd64.deb: no such file or directory"
},
{
"level": "debug",
"time": "2024-09-14T13:32:04+05:00",
"message": "Executing task asynchronously"
},
{
"level": "info",
"remote": "192.168.100.113",
"method": "POST",
"path": "/api/repos/devrepo/file/uploaded-files/lib_fstyle_4.4.3.7368_amd64.deb?_async=true",
"protocol": "HTTP/1.0",
"code": "202",
"latency": "229.132µs",
"agent": "curl/7.58.0",
"time": "2024-09-14T13:32:04+05:00"
},
{
"level": "debug",
"time": "2024-09-14T13:32:04+05:00",
"message": "Executing task asynchronously"
},
{
"level": "info",
"remote": "192.168.100.113",
"method": "PUT",
"path": "/api/publish/:./devrepo?_async=true",
"protocol": "HTTP/1.0",
"code": "202",
"latency": "21.189441ms",
"agent": "curl/7.58.0",
"time": "2024-09-14T13:32:04+05:00"
},
{
"level": "debug",
"time": "2024-09-14T13:32:04+05:00",
"message": "Executing task asynchronously"
},
{
"level": "info",
"remote": "192.168.100.113",
"method": "PUT",
"path": "/api/publish/:./ubuntu-20.04?_async=true",
"protocol": "HTTP/1.0",
"code": "202",
"latency": "4.111851ms",
"agent": "curl/7.58.0",
"time": "2024-09-14T13:32:04+05:00"
},
{
"level": "debug",
"time": "2024-09-14T13:32:04+05:00",
"message": "Executing task asynchronously"
},
{
"level": "info",
"remote": "192.168.100.113",
"method": "PUT",
"path": "/api/publish/:./ubuntu-22?_async=true",
"protocol": "HTTP/1.0",
"code": "202",
"latency": "3.787162ms",
"agent": "curl/7.58.0",
"time": "2024-09-14T13:32:04+05:00"
},
{
"level": "debug",
"time": "2024-09-14T13:32:04+05:00",
"message": "Executing task asynchronously"
},
{
"level": "info",
"remote": "192.168.100.113",
"method": "PUT",
"path": "/api/publish/:./ubuntu-24?_async=true",
"protocol": "HTTP/1.0",
"code": "202",
"latency": "3.901205ms",
"agent": "curl/7.58.0",
"time": "2024-09-14T13:32:04+05:00"
},
{
"level": "debug",
"time": "2024-09-14T13:32:04+05:00",
"message": "Executing task asynchronously"
},
{
"level": "info",
"remote": "192.168.100.113",
"method": "PUT",
"path": "/api/publish/:./astra-ce-2.12?_async=true",
"protocol": "HTTP/1.0",
"code": "202",
"latency": "3.703295ms",
"agent": "curl/7.58.0",
"time": "2024-09-14T13:32:04+05:00"
}
And I want to clarify: when I query curl
curl -f -X POST -F file=@/home/deb/lib_mastertest_amd64.deb 'http://192.168.100.150/deb/api/files/uploaded-files?_async=true -u api:pass
I get json
[
"uploaded-files/lib_mastertest_amd64.deb"
]
Do I understand correctly that this query does not need to reverse ?_async=true, since I can't get the state?
if you are using parallel uploads / add files, make sure a unique upload directory is used:
folder=`mktemp -u tmp.XXXXXXXXXXXXXXX`
curl -f -X POST -F file=@/home/deb/lib_mastertest_amd64.deb 'http://192.168.100.150/deb/api/files/$folder -u api:pass
...
if the same folder is used from multiple sources, you might get the no such file or directory error message because when adding the uploaded files to a repo, the upload folder is deleted.
the background operation via ?_async=true is only implemented for a few API calls which take a long time, in order to avoid API timeouts. so for uploading a file, the call will block anyway until the file is uploaded.
for the subsequent publishing of the repo, you could use the async operating and poll the corresponsing task until successful (State: 2).
@neolynx thanks! 😃
The doc says that the directory will be created if it doesn't exist, and a successfully downloaded package will be deleted. So the directory will be deleted too.... Got it. I'll check it out right now.
That is, in parallel builds, when one has already uploaded the package and is publishing, the file and directory are deleted. At the same time a file from another build was added to this directory...
@neolynx it's worked! Your support and advice have been incredible helpful 😃