arangamani
JenkinsApi::Exceptions::ForbiddenWithCrumb: Access denied. Please ensure that Jenkins is set up to allow access to this operation. A crumb was used in attempt to access operation.
I have migrated to jenkins 2.249 and this started happening. (at the bottom I confirm that it is an issue with the library using jenkins_api_client 1.5.3)
E, [2020-10-14T17:46:29.384574 #4] ERROR -- : JenkinsApi::Exceptions::ForbiddenWithCrumb: Access denied. Please ensure that Jenkins is set up to allow access to this operation. A crumb was used in attempt to access operation. Access denied. Please ensure that Jenkins is set up to allow access to this operation.
Traceback (most recent call last):
16: from bin/rails:9:in `require'
15: from railties (6.0.3.3) lib/rails/commands.rb:18:in `<top (required)>'
14: from railties (6.0.3.3) lib/rails/command.rb:46:in `invoke'
13: from railties (6.0.3.3) lib/rails/command/base.rb:69:in `perform'
12: from thor (1.0.1) lib/thor.rb:392:in `dispatch'
11: from thor (1.0.1) lib/thor/invocation.rb:127:in `invoke_command'
10: from thor (1.0.1) lib/thor/command.rb:27:in `run'
9: from railties (6.0.3.3) lib/rails/commands/console/console_command.rb:102:in `perform'
8: from railties (6.0.3.3) lib/rails/commands/console/console_command.rb:19:in `start'
7: from railties (6.0.3.3) lib/rails/commands/console/console_command.rb:70:in `start'
6: from (irb):3
5: from (irb):3:in `rescue in irb_binding'
4: from lib/jenkins_client_factory.rb:11:in `force_job_workoff'
3: from jenkins_api_client (26f0de9cab9b) lib/jenkins_api_client/job.rb:858:in `build'
2: from jenkins_api_client (26f0de9cab9b) lib/jenkins_api_client/client.rb:422:in `api_post_request'
1: from jenkins_api_client (26f0de9cab9b) lib/jenkins_api_client/client.rb:450:in `rescue in api_post_request'
JenkinsApi::Exceptions::ForbiddenWithCrumb (Access denied. Please ensure that Jenkins is set up to allow access to this operation. A crumb was used in attempt to access operation. Access denied. Please ensure that Jenkins is set up to allow access to this operation. )
The client is created with
def self.new_client
ENV["JENKINS_USERNAME"] || raise("no JENKINS_USERNAME provided")
ENV["JENKINS_PASSWORD"] || raise("no JENKINS_PASSWORD provided")
crt = (ENV["JENKINS_CRT"] || File.read("#{Rails.root}/keys/[email protected]"))
crt || raise("no JENKINS_CRT provided")
key = (ENV["JENKINS_KEY"] || File.read("#{Rails.root}/keys/[email protected]"))
key || raise("no JENKINS_KEY provided")
# This crt and key files could be generated with
# openssl pkcs12 -in path.p12 -out newfile.crt.pem -clcerts -nokeys
# openssl pkcs12 -in path.p12 -out newfile.key.pem -nocerts -nodes
# This of course is only if you have p12 file.
JenkinsApi::Client.new(:server_url=>CAST_CONFIG["jenkins"]["server_url"],
crt: crt,
key: key,
ssl: true,
:username=>ENV["JENKINS_USERNAME"],
:password=>ENV["JENKINS_PASSWORD"])
end
and the call is
new_client.job.build(build_name, {"param1"=>value1})
Here is the output in the console
opening connection to examplehost.com:443...
opened
starting SSL for examplehost.com:443...
SSL established, protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384
<- "GET /jenkins/api/json?tree=useCrumbs HTTP/1.1\r\nAccept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3\r\nAccept: */*\r\nUser-Agent: Ruby\r\nAuthorization: Basic xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+\r\nConnection: close\r\nHost: examplehost.com\r\n\r\n"
-> "HTTP/1.1 200 OK\r\n"
-> "Server: nginx/1.18.0 (Ubuntu)\r\n"
-> "Date: Wed, 14 Oct 2020 17:46:23 GMT\r\n"
-> "Content-Type: application/json;charset=utf-8\r\n"
-> "Content-Length: 64\r\n"
-> "Connection: close\r\n"
-> "X-Content-Type-Options: nosniff\r\n"
-> "X-Jenkins: 2.249.2\r\n"
-> "X-Jenkins-Session: XXXXXXXX\r\n"
-> "X-Frame-Options: deny\r\n"
-> "Content-Encoding: gzip\r\n"
-> "\r\n"
reading 64 bytes...
-> "\x1F\x8B\b\x00\x00\x00\x00\x00\x00..."
read 64 bytes
Conn close
I, [2020-10-14T17:46:23.167153 #4] INFO -- : Crumbs turned on. Fetching from the server.
opening connection to examplehost.com:443...
opened
starting SSL for examplehost.com:443...
SSL established, protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384
<- "GET /jenkins/crumbIssuer/api/json HTTP/1.1\r\nAccept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3\r\nAccept: */*\r\nUser-Agent: Ruby\r\nAuthorization: Basic xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+\r\nConnection: close\r\nHost: examplehost.com\r\n\r\n"
-> "HTTP/1.1 200 OK\r\n"
-> "Server: nginx/1.18.0 (Ubuntu)\r\n"
-> "Date: Wed, 14 Oct 2020 17:46:23 GMT\r\n"
-> "Content-Type: application/json;charset=utf-8\r\n"
-> "Content-Length: 155\r\n"
-> "Connection: close\r\n"
-> "X-Content-Type-Options: nosniff\r\n"
-> "X-Jenkins: 2.249.2\r\n"
-> "X-Jenkins-Session: XXXXXXXX\r\n"
-> "X-Frame-Options: deny\r\n"
-> "Content-Encoding: gzip\r\n"
-> "Set-Cookie: JSESSIONID.67fe54fc=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.node0; Path=/jenkins; HttpOnly\r\n"
-> "Expires: Thu, 01 Jan 1970 00:00:00 GMT\r\n"
-> "\r\n"
reading 155 bytes...
-> "\x1F\x8B\b\x00\x00\x00\x00\x00\x00..."
read 155 bytes
Conn close
opening connection to examplehost.com:443...
opened
starting SSL for examplehost.com:443...
SSL established, protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384
<- "POST /jenkins/job/build_name/buildWithParameters HTTP/1.1\r\nAccept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3\r\nAccept: */*\r\nUser-Agent: Ruby\r\nJenkins-Crumb: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx \r\nContent-Type: application/x-www-form-urlencoded\r\nAuthorization: Basic
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+\r\nConnection: close\r\nHost: examplehost.com\r\nContent-Length: 25\r\n\r\n"
<- "params1=value1"
-> "HTTP/1.1 403 Forbidden\r\n"
-> "Server: nginx/1.18.0 (Ubuntu)\r\n"
-> "Date: Wed, 14 Oct 2020 17:46:24 GMT\r\n"
-> "Content-Type: text/html;charset=iso-8859-1\r\n"
-> "Transfer-Encoding: chunked\r\n"
-> "Connection: close\r\n"
-> "X-Content-Type-Options: nosniff\r\n"
-> "Set-Cookie: JSESSIONID.67fe54fc=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.node0; Path=/jenkins; HttpOnly\r\n"
-> "Cache-Control: must-revalidate,no-cache,no-store\r\n"
-> "Content-Encoding: gzip\r\n"
-> "\r\n"
-> "164\r\n"
Update 1: I have seen https://stackoverflow.com/a/62957734/1266681 which mentions
According to Jenkins Directive First you have to check your Jenkins version if the version is < 2.176.2 then per Jenkins guideline CSRF tokens (crumbs) are now only valid for the web session they were created in to limit the impact of attackers obtaining them. Scripts that obtain a crumb using the /crumbIssuer/api URL will now fail to perform actions protected from CSRF unless the scripts retain the web session ID in subsequent requests.
I am not sure if Jenkins_api_client does this because we have just migrated to 2.249.2 and probably that is the cause.
Update 2 It might not be an issue with the library
I just did
$ curl -v http://localhost:8080/jenkins/crumbIssuer/api/json --user theuser
# Than took the crumb
curl -X POST http://localhost:8080/jenkins/job/jobName/buildWithParameters --user theuser -H 'Jenkins-Crumb: cccccccccc'
and jenkins returned
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 403 No valid crumb was included in the request</title>
</head>
<body><h2>HTTP ERROR 403 No valid crumb was included in the request</h2>
<table>
<tr><th>URI:</th><td>/jenkins/job/jobName/buildWithParameters</td></tr>
<tr><th>STATUS:</th><td>403</td></tr>
<tr><th>MESSAGE:</th><td>No valid crumb was included in the request</td></tr>
<tr><th>SERVLET:</th><td>Stapler</td></tr>
</table>
<hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.30.v20200611</a><hr/>
</body>
</html>
Update 3: The trick i think is that after "2.176 a session is required" but the two values send for Set-Cookie are different The first one is -> "Set-Cookie: JSESSIONID.67fe54fc=node01itvfkmabhuf21dsikujbn1il449.node0; Path=/jenkins; HttpOnly\r\n" and the second is -> "Set-Cookie: JSESSIONID.67fe54fc=node02geew3fiibpk1j4w6eciwzipn50.node0; Path=/jenkins; HttpOnly\r\n"
Update 4: I can confirm it is an issue with the library jenkins_api_client
The reason is the following I go to https://support.cloudbees.com/hc/en-us/articles/219257077-CSRF-Protection-Explained and I follow the explanation of how to correctly call jenkins after 2.176 with curl The code is
# Replace with your Jenkins URL and admin credentials
SERVER="http://localhost:8080"
# File where web session cookie is saved
COOKIEJAR="$(mktemp)"
CRUMB=$(curl -u "admin:admin" --cookie-jar "$COOKIEJAR" "$SERVER/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,%22:%22,//crumb)")
curl -X POST -u "admin:admin" --cookie "$COOKIEJAR" -H "$CRUMB" "$SERVER"/job/someJob/build
By using this code I have successfully triggered a build. This call preserves the session between the crumbIssuer and the post request.
I cant see a way to do this with jenkins_api_client
I managed to fix this by adding
end
http.open_timeout = @http_open_timeout
http.read_timeout = @http_read_timeout
response = http.request(request)
+ @cookies = response["set-cookie"]
case response
when Net::HTTPRedirection then
# If we got a redirect request, follow it (if flag set), but don't
# go any deeper (only one redirect supported - don't want to follow
# our tail)
if follow_redirect
redir_uri = URI.parse(response['location'])
response = make_http_request(
Net::HTTP::Get.new(redir_uri.path, false)
)
end
end
I will prepare a pull request for this.
This didn't fix my issue, but https://github.com/arangamani/jenkins_api_client/pull/292 did.
This is a wholly inadequate way to handle cookies. There are multiple places in the file where cookies need to be read--including another place in the above snippet. But beyond that, look at https://stackoverflow.com/questions/1486703/how-to-implement-cookie-support-in-ruby-net-http . Not the selected response, of course. Look further in. Cookies are actually complex beasts. Dropping their attributes is an error. As I understand the standards, cookies with certain attribute settings MUST not be accepted. I could see a server deliberately sending a cookie with invalid attributes & reading it back to trap bots.
@NathanZook by "this is a wholly inadequate" do you mean https://github.com/arangamani/jenkins_api_client/pull/292 or you mean
response = http.request(request)
+ @cookies = response["set-cookie"]
case response
when Net::HTTPRedirection then
If it is the second one, then yes, this is a workaround for my case that gives more information about the issue, but it is not a solution to the issue.