Exit status "4" when using download with continue and file is already present

[mike-pt]

% sudo http --debug --check-status --follow -bdco file.tar.bz2 URL.file.tar.bz2
HTTPie 1.0.3
Requests 2.22.0
Pygments 2.4.1
Python 3.6.9 (default, Dec 24 2019, 01:17:36) 
[GCC 4.2.1 Compatible FreeBSD Clang 6.0.1 (tags/RELEASE_601/final 335540)]
/usr/local/bin/python3.6
FreeBSD 12.1-RELEASE-p1


    "colors": 8,
    "config": {
        "__meta__": {
            "about": "HTTPie configuration file",
            "help": "https://httpie.org/doc#config",
            "httpie": "1.0.3"
        },
        "default_options": "[]"
    },
    "config_dir": "/root/.httpie",
    "is_windows": false,
    "stderr": "<_io.TextIOWrapper name='<stderr>' mode='w' encoding='US-ASCII'>",
    "stderr_isatty": true,
    "stdin": "<_io.TextIOWrapper name='<stdin>' mode='r' encoding='US-ASCII'>",
    "stdin_encoding": "US-ASCII",
    "stdin_isatty": true,
    "stdout": "<_io.TextIOWrapper name='<stdout>' mode='w' encoding='US-ASCII'>",
    "stdout_encoding": "US-ASCII",
    "stdout_isatty": true
}>

    "allow_redirects": true,
    "auth": "None",
    "cert": "None",
    "data": {},
    "files": {},
    "headers": {
        "Accept-Encoding": "identity",
        "Range": "bytes=142370763-",
        "User-Agent": "HTTPie/1.0.3"
    },
    "method": "get",
    "params": {},
    "proxies": {},
    "stream": true,
    "timeout": 30,
    "url": "https://URL/file.tar.bz2",
    "verify": true
})

% echo $?
4

This makes it a hard to test for 40x http errors. Is this by design or a bug?

[mike-pt]

FWIW this is my way of handling it for now, but it seems wrong to show "4" in such cases.

http --follow -bdco "$file" $download_url 
  exit_code=$?
  if [ "$exit_code" != "0" ]; then
    if [ "$exit_code" == "4" ] && [ -f "$file" ]; then
      echo -e "We already have this file on disk, moving on...\n\n"
    else
      echo -e "Something failed attempting to download package!\nURL: $download_url\n Exit Code: $exit_code\n"
      exit 1
    fi
  fi

[jkbrzt]

Can you include the entire output of the first example?

[mike-pt]

that's the entire output when the file already exists actually

[mike-pt]

or do you mean with headers too?

[jkbrzt]

Just copy the entire output.

[mike-pt]

In that case that is it, I just replaced the real url for privacy reasons, all else is there.

In other words, there is no output to stdout when the file already exists!

[jkbrzt]

I don’t need the exact url. But without the compete output (headers, progress, etc.) I can’t help.

[mike-pt]

But how can I provide any output when none exists!? Or are ou saying it should!? What happens your side when you try to do the same? I mean download a url with -co when the file was already downloaded... do you see output!? I see nothing (no progress even) and the exit code is 4.

I can get the headers if I send -hfor sure, (that's what I was asking above, and it seems to be the culprit actually:

(EDITED: my fat finger typed enter too fast)

HTTP/1.1 416 Requested Range Not Satisfiable
Age: 102
CF-Cache-Status: HIT
CF-RAY: 55b3de858b8ada76-LIS
Cache-Control: public, max-age=259200
Connection: keep-alive
Content-Disposition: attachment; filename="file.tar.bz2"; filename*=UTF-8''file.tar.bz2
Content-Range: bytes */142462219
Content-Type: text/html
Date: Sun, 26 Jan 2020 16:30:03 GMT
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Expires: Wed, 29 Jan 2020 16:30:03 GMT
Server: cloudflare
Set-Cookie: __cfduid=d3093fe2d895aa6d99e2ca6ead65a6a811580056203; expires=Tue, 25-Feb-20 16:30:03 GMT; path=/; domain=.example.org; HttpOnly; SameSite=Lax
Transfer-Encoding: chunked
Vary: Accept-Encoding

So its because of HTTP/1.1 416 Requested Range Not Satisfiable I assume?

What I don't understand now is why, but that might not be httpie issue anyway but the server?

[mike-pt]

Just an update on this I later found the file is also always created (0 byte) if the requests fails!

I.e even on 404 the request would fail but we create a 0 byte file! I noticed cause to work around the 416 I was using test -f in the shell script and it was passing; just curios if that is the intended behavior? (fine by me I'll just change the test to -s but just wanted to be sure that's intended)

[jkbrzt]

@mike-pt could you please provide a minimal test case, and describe what the expected & the actual behavior is? We’re having troubles reproducing it. Thanks!

[mike-pt]

Hum its been some time, the original issue was a odd one, I try to repo again but about that last comment:

miguelc@r2d2:~ % http -h -dco /tmp/file.txt https://domain.com/file.FAKE
HTTP/1.1 404 Not Found
Age: 107
CF-Cache-Status: HIT
CF-RAY: 5a55d0485f78067e-LHR
Cache-Control: public, max-age=259200
Connection: keep-alive
Content-Type: application/json;charset=ISO-8859-1
Date: Thu, 18 Jun 2020 14:48:51 GMT
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Expires: Sun, 21 Jun 2020 14:48:51 GMT
Server: cloudflare
Set-Cookie: __cfduid=df76b0fc75da7cf72600cf683995c29b81592491731; expires=Sat, 18-Jul-20 14:48:51 GMT; path=/; domain=.plex.tv; HttpOnly; SameSite=Lax
Transfer-Encoding: chunked
Vary: Accept-Encoding
X-Artifactory-Id: abc0aca8c8fcf70f:-1015742d:1726eb5d906:-8000
X-Artifactory-Node-Id: artifactory00
cf-request-id: 03698081360000067ee61d0200000001

% file /tmp/file.txt
/tmp/file.txt: empty
% ls -l /tmp/file.txt
-rw-r--r--  1 miguelc  wheel  0 Jun 18 15:48 /tmp/file.txt

note that I do not pass -b here, if I did I would actually get this was the output:

{
  "errors" : [ {
    "status" : 404,
    "message" : "File not found."
  } ]
}

Not sure if that is the intend behavior of if we should end up with no file there, I assume we are creating the file first and then writing the contents to it, or because we could somehow get actually 0 content (download gets canceled at start, unlikely but not impossible); In any case my question there was really if that's the expected case or not.

---

About the original issue I later posted that this happens cause I get a 416 if the file is already on disk with the full content-length (https://github.com/jakubroztocil/httpie/issues/842#issuecomment-578518068) the content-range is */totalsize and so I get a 416.

If so, httpie simple returns a single digit exit code ofc, and so a 416 is still a 4, I was just not noticing the 416 at that at the time.

In other words that is not a bug.

[isidentical]

I think I've missed that it was intentionally a 404 response. So the actual issue here is that us creating an empty file when the server responds something other than 200, right?

[hmeine]

I see this discussion concluded with an empty file from a 404 response, but I have exactly the problem described in the title, with no 404 errors involved. This is also related to the same problem in curl: https://github.com/curl/curl/issues/10521

I tried running https -dco foo.bin ***redacted_url*** X-JFrog-Art-Api:@jfrog-api-key to download a build artifact from an artifactory server, hoping that repeated script runs would not download the large file multiple times.

The server replies with

HTTP/1.1 416 Requested Range Not Satisfiable
Connection: keep-alive
Content-Length: 0
Content-Range: bytes */977880575
Date: Wed, 09 Aug 2023 09:57:07 GMT
Server: nginx/1.18.0 (Ubuntu)
X-Artifactory-Id: ***redacted***
X-Artifactory-Node-Id: ***redacted***
X-Jfrog-Version: Artifactory/7.55.13 75513900

letting httpie exit with code 4.

The corresponding request was

Accept: */*
Accept-Encoding: identity
Connection: keep-alive
Host: artifactory.redacted.lan
Range: bytes=977880575-
User-Agent: HTTPie/3.2.1
X-JFrog-Art-Api: ***redacted***