"Read timed out" when downloading from Artifactory

[docdawning]

Describe the bug With newer versions of the Artifactory plugin installed, our Jenkins servers will routinely fail to complete downloads from our Artifactory server.

In our pipelines, this comes back with the not-so-descriptive report:

Searching for artifacts... 
Error occurred for request POST /artifactory/api/search/aql HTTP/1.1: Read timed out. 
Attempting retry #1 
Resume after a restart not supported for non-blocking synchronous steps

Here's what it looks like in the logs:

Feb 17, 2022 11:15:16 AM org.jfrog.hudson.util.JenkinsBuildInfoLog info
INFO: Searching for artifacts...
Feb 17, 2022 11:20:17 AM org.jfrog.hudson.util.JenkinsBuildInfoLog warn
WARNING: Error occurred for request POST /artifactory/api/search/aql HTTP/1.1: Read timed out.
Feb 17, 2022 11:20:17 AM org.jfrog.hudson.util.JenkinsBuildInfoLog warn
WARNING: Error occurred for request POST /artifactory/api/search/aql HTTP/1.1: Read timed out.
Feb 17, 2022 11:20:17 AM org.jfrog.hudson.util.JenkinsBuildInfoLog warn
WARNING: Error occurred for request POST /artifactory/api/search/aql HTTP/1.1: Read timed out.
Feb 17, 2022 11:20:17 AM org.jfrog.hudson.util.JenkinsBuildInfoLog warn

Our team had several people spend multiple days investigating our stack before we finally pin-pointed the issue as something specifically in this plugin. We're not sure what the root cause is, but we found that older versions of this plugin do not fail for us, and more recent versions do fail.

Here's a few versions and their status for this issue: 3.15.4: Download Time out problem 3.14.2: Download Time out problem 3.13.2: No problem

Interestingly, when we reverted to 3.13.2 we also observed the plugin to be radically faster too.

To Reproduce

1. Install version 3.15.4 of the Artifactory plugin in to your Jenkins server
2. Setup your Artifactory server so you can fetch some files from it
3. Configure a Jenkins job with the below pipeline script (update to align with your environment)
4. Launch the job

artifactoryServer = Artifactory.server 'artifactory_server'
lightweight_node_label = 'Name_of_some_specific_test_build_server'

stage("Download Artifacts"){
    node(lightweight_node_label){
        arr = ['1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '18', '19', '20', '21', '22', '23', '24']
        def tasks = [:]
        
        arr.each { num ->
            tasks["$num"] = {
                ws('${num}_test'){
                    cleanWs()        
                    echo num
                    def downloadSpecRelease = """{
              "files": [
                {
                    "pattern": "my_repo/$BRANCH/*/release/bin/warp_engine.bin",
                    "target": "local_directory/release/",
                    "flat": "true",
                    "props": "LatestDaily=true"
                }
                ]
            }"""
            
                artifactoryServer.download spec: downloadSpecRelease
                }
            }
        }
    parallel tasks
        
    }
}

Expected behavior The files are fetched successfully. But instead, we get the time out error mentioned above.

Versions

• Jenkins Artifactory plugin version: 3.15.4
• Jenkins operating system: Windows 10 - Version 10.0.17763 Build 17763
• Artifactory Version: 7.33.12

Workaround We ultimately worked around this by simply reverting to version 3.13.2

[yahavi]

@docdawning, Thanks for reporting this issue.

We ran the pipeline you provided many times with multiple versions with a 20 MB file. Unfortunately, it wasn't reproduced. Additionally, there was no "suspicious" change between 3.13.2 and 3.14.2.

I suggest that if the issue persists, please reach out to the JFrog support.

[docdawning]

@yahavi, thank you for getting back. I think our current license doesn't get us support, but we are looking at possibly stepping up a tier soon.

I guess my steps to reproduce this aren't enough to capture it. Is it possible this is a windows-only issue? We're running Jenkins on Windows (I'm sorry to have to say).

[gillima]

This could probably be related to #623. We're running Jenkins on Windows as well and observer issues with downloads since 3.15.0

[stephanvantienen-keen]

We are experiencing the same issue where we have a lot of read timeouts that often lead to a failing Jenkins job. The timeouts always appear when searching for artifacts:

Finished resolving Build Info published dependencies. Retrieving build artifacts report from: /api/build/patternArtifacts Searching for artifacts... Error occurred for request POST /artifactory/api/search/aql HTTP/1.1: Read timed out. Attempting retry #1 Error occurred for request POST /artifactory/api/search/aql HTTP/1.1: Read timed out. Attempting retry #2 Error occurred for request POST /artifactory/api/search/aql HTTP/1.1: Read timed out. Attempting retry #3 Error occurred for request POST /artifactory/api/search/aql HTTP/1.1: Read timed out. FATAL: Read timed out

The download spec that is used pulls in a LATEST build and contains wildcards in the pattern:

{
        "pattern": "repo/path1/*/path2/*",
        "target": "./path2/",
        "flat": "true",
        "build": "buildname/LATEST"
}

We then tried to use the JFrog CLI tool and experienced the same issue, getting the build info works fine but retrieving the specific artifacts from the pattern takes a very long time - easily ten minutes or longer:

jf rt search repo/path1/specificbuild/path2/*

We then got a tip that it helps to pass the --limit (which will limit the amount of files downloaded but it does not seem to matter that much how high you set this): jf rt search repo/path1/specificbuild/path2/* --limit=400

Doing that will have the search query return immediately. Also if we download using the spec that way it works as expected: jf rt dl --spec download.spec --limit=400

If we set JFROG_CLI_LOG_LEVEL to DEBUG we can see where this limit is used:

08:22:37 [Debug] Searching Artifactory using AQL query: items.find({"path":{"$ne":"."},"$or":[{"$and":[{"repo":"repo","path":"path1/specificbuild/path2","name":{"$match":""}}]},{"$and":[{"repo":"repo","path":{"$match":"path1/specificbuild/path2/"},"name":{"$match":"*"}}]}]}).include("name","repo","path","actual_md5","actual_sha1","sha256","size","type","modified","created").limit(400)

If you do not pass the limit option the AQL query does not have the .limit part at the end. You can then also see that the delay comes from this step, the AQL query. There seems to be a fundamental issue in going from the AQL query to the SQL query into the database (doing a direct SQL query works fast as expected).

The real fix has to come from Artifactory - the delay is always experienced server-side - but I was wondering if the Jenkins plugin would have a way of using the limit option - which could possibly be a configurable item. It very much seems like that could prevent the long delays.

The only other alternative is to not use wildcards in the patterns, but that would severely hamper the usage in our view.

[eyalbe4]

We stringly recommend migrating to the newer JFrog Jenkins Plugin, rather than using the older Artifactory Jenkins Plugin. Since the new plugin is a wrapper for JFrog CLI, it is more robust, provies better performance, and the code of JFrog CLI is continuasly optimized and improved. Both plugins can be used side-by-side in the same pipeline jobs, to allow easier and gradual migration. There's a good chance you will not be experoencing the discussed issue with the JFrog Plugin.

[stephanvantienen-keen]

Thank you very much for the tip @eyalbe4 , we will give that a try - were not aware the JFrog Jenkins Plugin existed.

[github-actions[bot]]

This issue has been marked as stale due to 6 months of inactivity. As part of our effort to address every issue properly, please feel free to remove the stale label or keep this issue active by leaving a comment. Otherwise, it will be closed in 7 days

[github-actions[bot]]

This issue was closed due to 7 days of inactivity after being marked as stale. Feel free to reopen it if it remains relevant.