jamf-upload
jamf-upload copied to clipboard
grahampugh
JamfPolicyUploader processor failing with errors when updating policies
I'm seeing a problem with the JamfPolicyUploader processor when running various policies which use it. An example is posted below:
JamfPolicyUploader: No value supplied for icon, setting default value of:
JamfPolicyUploader: No value supplied for replace_icon, setting default value of: False
JamfPolicyUploader: File found at: /Users/ec2-user/Library/AutoPkg/RecipeRepos/com.github.rtrouton.autopkg_recipes/Jamf_Upload/PolicyTemplate-autoupdate.xml
JamfPolicyUploader: ("Replacing any instances of 'POLICY_NAME' with", "'Auto Update CiscoUmbrella'")
JamfPolicyUploader: ("Replacing any instances of 'POLICY_CATEGORY' with", "'Apps-AutoUpdate'")
JamfPolicyUploader: ("Replacing any instances of 'POLICY_TRIGGER' with", "'autoupdate-CiscoUmbrella'")
JamfPolicyUploader: ("Replacing any instances of 'pkg_name' with", "'Cisco_Umbrella_3.0.9.pkg'")
JamfPolicyUploader: Policy data:
JamfPolicyUploader: <?xml version="1.0" encoding="UTF-8"?>
<policy>
<general>
<name>Auto Update CiscoUmbrella</name>
<enabled>true</enabled>
<frequency>Ongoing</frequency>
<category>
<name>Apps-AutoUpdate</name>
</category>
<trigger_other>autoupdate-CiscoUmbrella</trigger_other>
</general>
<scope>
<all_computers>true</all_computers>
</scope>
<package_configuration>
<packages>
<size>1</size>
<package>
<name>Cisco_Umbrella_3.0.9.pkg</name>
<action>Install</action>
</package>
</packages>
</package_configuration>
<scripts>
<size>0</size>
</scripts>
<maintenance>
<recon>true</recon>
</maintenance>
</policy>
JamfPolicyUploader: Checking for existing 'Auto Update CiscoUmbrella' on https://jamf.pro.server.here
JamfPolicyUploader: Checking for existing authentication token
JamfPolicyUploader: Checking https://jamf.pro.server.here against https://jamf.pro.server.here
JamfPolicyUploader: URL for token matches current request
JamfPolicyUploader: Existing token is valid
JamfPolicyUploader: No existing cookie found - starting new session
JamfPolicyUploader: curl command: /usr/bin/curl --silent --show-error -D /tmp/jamf_upload/curl_headers_from_jamf_upload.txt --output /tmp/jamf_upload/curl_output_from_jamf_upload.txt https://jamf.pro.server.here/api/v1/jamf-pro-version --request GET --header authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJhdXRoZW50aWNhdGVkLWFwcCI6IkdFTkVSSUMiLCJhdXRoZW50aWNhdGlvbi10eXBlIjoiSlNTIiwiZ3JvdXBzIjpbXSwic3ViamVjdC10eXBlIjoiSlNTX1VTRVJfSUQiLCJ0b2tlbi11dWlkIjoiMzE3ZTcyMWQtMzBjYi00ZjM3LWE3MDEtMDhhN2QyMWJhYzM1IiwibGRhcC1zZXJ2ZXItaWQiOi0xLCJzdWIiOiIyMCIsImV4cCI6MTY0Njk0MDc2M30.g4Cv9b7TOPyZP1zWPNK1SMO5c-9LxnUGTMBGnuuyY0k --header Accept: application/json
JamfPolicyUploader: Jamf Pro Version: 10.37.0-b.2.t1645716981
JamfPolicyUploader: Token auth will be used,
JamfPolicyUploader: No existing cookie found - starting new session
JamfPolicyUploader: curl command: /usr/bin/curl --silent --show-error -D /tmp/jamf_upload/curl_headers_from_jamf_upload.txt --output /tmp/jamf_upload/curl_output_from_jamf_upload.txt https://jamf.pro.server.here/JSSResource/policies --request GET --header authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJhdXRoZW50aWNhdGVkLWFwcCI6IkdFTkVSSUMiLCJhdXRoZW50aWNhdGlvbi10eXBlIjoiSlNTIiwiZ3JvdXBzIjpbXSwic3ViamVjdC10eXBlIjoiSlNTX1VTRVJfSUQiLCJ0b2tlbi11dWlkIjoiMzE3ZTcyMWQtMzBjYi00ZjM3LWE3MDEtMDhhN2QyMWJhYzM1IiwibGRhcC1zZXJ2ZXItaWQiOi0xLCJzdWIiOiIyMCIsImV4cCI6MTY0Njk0MDc2M30.g4Cv9b7TOPyZP1zWPNK1SMO5c-9LxnUGTMBGnuuyY0k --header Accept: application/json
JamfPolicyUploader: Policy 'Auto Update CiscoUmbrella' already exists: ID 2459
JamfPolicyUploader: Replacing existing policy as 'replace_policy' is set to True
JamfPolicyUploader: Uploading Policy...
JamfPolicyUploader: Policy upload attempt 1
JamfPolicyUploader: No existing cookie found - starting new session
JamfPolicyUploader: curl command: /usr/bin/curl --silent --show-error -D /tmp/jamf_upload/curl_headers_from_jamf_upload.txt --output /tmp/jamf_upload/curl_output_from_jamf_upload.txt https://jamf.pro.server.here/JSSResource/policies/id/2459 --request PUT --header authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJhdXRoZW50aWNhdGVkLWFwcCI6IkdFTkVSSUMiLCJhdXRoZW50aWNhdGlvbi10eXBlIjoiSlNTIiwiZ3JvdXBzIjpbXSwic3ViamVjdC10eXBlIjoiSlNTX1VTRVJfSUQiLCJ0b2tlbi11dWlkIjoiMzE3ZTcyMWQtMzBjYi00ZjM3LWE3MDEtMDhhN2QyMWJhYzM1IiwibGRhcC1zZXJ2ZXItaWQiOi0xLCJzdWIiOiIyMCIsImV4cCI6MTY0Njk0MDc2M30.g4Cv9b7TOPyZP1zWPNK1SMO5c-9LxnUGTMBGnuuyY0k --upload-file /tmp/jamf_upload/jamf_upload_d7e5facc-6b51-4a4a-8b17-25a51698654a.txt --header Content-type: application/xml
JamfPolicyUploader: b'<html>\n<head>\n <title>Status page</title>\n</head>\n<body style="font-family: sans-serif;">\n<p style="font-size: 1.2em;font-weight: bold;margin: 1em 0px;">Conflict</p>\n<p>Error: Problem with package</p>\n<p>You can get technical details <a href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.10">here</a>.<br>\nPlease continue your visit at our <a href="/">home page</a>.\n</p>\n</body>\n</html>\n'
Traceback (most recent call last):
File "/Library/AutoPkg/autopkglib/__init__.py", line 840, in process
self.env = processor.process()
File "/Library/AutoPkg/autopkglib/__init__.py", line 626, in process
self.main()
File "/Users/ec2-user/Library/AutoPkg/RecipeRepos/com.github.autopkg.grahampugh-recipes/JamfUploaderProcessors/JamfPolicyUploader.py", line 304, in main
token=token,
File "/Users/ec2-user/Library/AutoPkg/RecipeRepos/com.github.autopkg.grahampugh-recipes/JamfUploaderProcessors/JamfPolicyUploader.py", line 134, in upload_policy
if self.status_check(r, "Policy", policy_name, request) == "break":
File "/Users/ec2-user/Library/AutoPkg/RecipeRepos/com.github.autopkg.grahampugh-recipes/JamfUploaderProcessors/JamfUploaderLib/JamfUploaderBase.py", line 396, in status_check
f"WARNING: {endpoint_type} '{obj_name}' {action} failed due to a conflict"
autopkglib.ProcessorError: WARNING: Policy 'Auto Update CiscoUmbrella' update failed due to a conflict
File "/Library/AutoPkg/autopkglib/__init__.py", line 840, in process
self.env = processor.process()
Receipt written to /Users/ec2-user/Library/AutoPkg/Cache/local.jamf.CiscoUmbrella-App-AutoUpdate/receipts/local.jamf-receipt-20220310-190539.plist
As far as I can tell, the XML for the policy template is formatted correctly (please see below):
<?xml version="1.0" encoding="UTF-8"?>
<policy>
<general>
<name>%POLICY_NAME%</name>
<enabled>true</enabled>
<frequency>Ongoing</frequency>
<category>
<name>%POLICY_CATEGORY%</name>
</category>
<trigger_other>%POLICY_TRIGGER%</trigger_other>
</general>
<scope>
<all_computers>true</all_computers>
</scope>
<package_configuration>
<packages>
<size>1</size>
<package>
<name>%pkg_name%</name>
<action>Install</action>
</package>
</packages>
</package_configuration>
<scripts>
<size>0</size>
</scripts>
<maintenance>
<recon>true</recon>
</maintenance>
</policy>
Full sanitized verbose output of the AutoPkg run is attached as "JamfPolicyUploader_error_processor.txt." JamfPolicyUploader_error_processor.txt
Hi Rich. The error says "problem with package". That usually means the package that's been added to the policy doesn't exist. Can we look earlier in the run to see if the package and uploads were successful?
Edit: never mind, I now noticed the full output trace link!
So, it did upload successfully. I can only imagine that it's a timing issue. I notice that your trace does not recognise any cookies. I don't know if that's something specific to your set up, but it certainly means that each curl request is going to a random cluster node, which makes it entirely possible that node that's being hit to receive the policy is a different one to the one that received the package, and synchronisation hasn't happened yet.
All the cookie stuff is based on information I got regarding standard Jamf Cloud servers and the cookies they produce. I wonder if your server is different? We might need to analyse the curl headers.
It appears that it was. Please see below:
JamfPackageUploader: No value supplied for pkg_name, setting default value of:
JamfPackageUploader: No value supplied for pkg_info, setting default value of:
JamfPackageUploader: No value supplied for reboot_required, setting default value of:
JamfPackageUploader: No value supplied for os_requirements, setting default value of:
JamfPackageUploader: No value supplied for required_processor, setting default value of: None
JamfPackageUploader: No value supplied for send_notification, setting default value of:
JamfPackageUploader: No value supplied for replace_pkg, setting default value of: False
JamfPackageUploader: No value supplied for replace_pkg_metadata, setting default value of: False
JamfPackageUploader: No value supplied for SMB_URL, setting default value of:
JamfPackageUploader: No value supplied for SMB_USERNAME, setting default value of:
JamfPackageUploader: No value supplied for SMB_PASSWORD, setting default value of:
JamfPackageUploader: Checking for existing 'Cisco_Umbrella_3.0.9.pkg' on https://jamf.pro.server.here
JamfPackageUploader: Checking for existing authentication token
JamfPackageUploader: Checking https://jamf.pro.server.here against https://jamf.pro.server.here
JamfPackageUploader: URL for token matches current request
JamfPackageUploader: Existing token is valid
JamfPackageUploader: No existing cookie found - starting new session
JamfPackageUploader: curl command: /usr/bin/curl --silent --show-error -D /tmp/jamf_upload/curl_headers_from_jamf_upload.txt --output /tmp/jamf_upload/curl_output_from_jamf_upload.txt https://jamf.pro.server.here/api/v1/jamf-pro-version --request GET --header authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJhdXRoZW50aWNhdGVkLWFwcCI6IkdFTkVSSUMiLCJhdXRoZW50aWNhdGlvbi10eXBlIjoiSlNTIiwiZ3JvdXBzIjpbXSwic3ViamVjdC10eXBlIjoiSlNTX1VTRVJfSUQiLCJ0b2tlbi11dWlkIjoiMzE3ZTcyMWQtMzBjYi00ZjM3LWE3MDEtMDhhN2QyMWJhYzM1IiwibGRhcC1zZXJ2ZXItaWQiOi0xLCJzdWIiOiIyMCIsImV4cCI6MTY0Njk0MDc2M30.g4Cv9b7TOPyZP1zWPNK1SMO5c-9LxnUGTMBGnuuyY0k --header Accept: application/json
JamfPackageUploader: Jamf Pro Version: 10.37.0-b.2.t1645716981
JamfPackageUploader: Token auth will be used,
JamfPackageUploader: No existing cookie found - starting new session
JamfPackageUploader: curl command: /usr/bin/curl --silent --show-error -D /tmp/jamf_upload/curl_headers_from_jamf_upload.txt --output /tmp/jamf_upload/curl_output_from_jamf_upload.txt https://jamf.pro.server.here/JSSResource/packages/name/Cisco_Umbrella_3.0.9.pkg --request GET --header authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJhdXRoZW50aWNhdGVkLWFwcCI6IkdFTkVSSUMiLCJhdXRoZW50aWNhdGlvbi10eXBlIjoiSlNTIiwiZ3JvdXBzIjpbXSwic3ViamVjdC10eXBlIjoiSlNTX1VTRVJfSUQiLCJ0b2tlbi11dWlkIjoiMzE3ZTcyMWQtMzBjYi00ZjM3LWE3MDEtMDhhN2QyMWJhYzM1IiwibGRhcC1zZXJ2ZXItaWQiOi0xLCJzdWIiOiIyMCIsImV4cCI6MTY0Njk0MDc2M30.g4Cv9b7TOPyZP1zWPNK1SMO5c-9LxnUGTMBGnuuyY0k --header Accept: application/json
JamfPackageUploader: ID: -1
JamfPackageUploader: No existing cookie found - starting new session
JamfPackageUploader: curl command: /usr/bin/curl --silent --show-error -D /tmp/jamf_upload/curl_headers_from_jamf_upload.txt --output /tmp/jamf_upload/curl_output_from_jamf_upload.txt https://jamf.pro.server.here/dbfileupload --request POST --header authorization: Basic nopenothereZQ== --upload-file /Users/ec2-user/Library/AutoPkg/Cache/local.jamf.CiscoUmbrella-App-AutoUpdate/Cisco_Umbrella_3.0.9.pkg --header Accept: application/xml --header DESTINATION: 0 --header OBJECT_ID: -1 --header FILE_TYPE: 0 --header FILE_NAME: Cisco_Umbrella_3.0.9.pkg --max-time 3600
JamfPackageUploader: HTTP response: 200
JamfPackageUploader: Package uploaded successfully, ID=2371
JamfPackageUploader: Package 'Cisco_Umbrella_3.0.9.pkg' upload successful
JamfPackageUploader: Updating package metadata for 2371
JamfPackageUploader: <package><name>Cisco_Umbrella_3.0.9.pkg</name><filename>Cisco_Umbrella_3.0.9.pkg</filename><category>Apps</category><info></info><notes>Cisco Umbrella 3.0.9 installer uploaded via AutoPkg.</notes><priority>6</priority><reboot_required>False</reboot_required><required_processor>None</required_processor><os_requirements></os_requirements><hash_type>SHA_512</hash_type><hash_value>0627a2160d74b923e46bbc1a5c1cc1c5dc0b51a4b5171377841d3a3287186fc6a936c4a78eaeee237603a3fded8177619948959383833898cf5986480d29f183</hash_value><send_notification>False</send_notification></package>
JamfPackageUploader: Package metadata upload attempt 1
JamfPackageUploader: No existing cookie found - starting new session
JamfPackageUploader: curl command: /usr/bin/curl --silent --show-error -D /tmp/jamf_upload/curl_headers_from_jamf_upload.txt --output /tmp/jamf_upload/curl_output_from_jamf_upload.txt https://jamf.pro.server.here/JSSResource/packages/id/2371 --request PUT --header authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJhdXRoZW50aWNhdGVkLWFwcCI6IkdFTkVSSUMiLCJhdXRoZW50aWNhdGlvbi10eXBlIjoiSlNTIiwiZ3JvdXBzIjpbXSwic3ViamVjdC10eXBlIjoiSlNTX1VTRVJfSUQiLCJ0b2tlbi11dWlkIjoiMzE3ZTcyMWQtMzBjYi00ZjM3LWE3MDEtMDhhN2QyMWJhYzM1IiwibGRhcC1zZXJ2ZXItaWQiOi0xLCJzdWIiOiIyMCIsImV4cCI6MTY0Njk0MDc2M30.g4Cv9b7TOPyZP1zWPNK1SMO5c-9LxnUGTMBGnuuyY0k --upload-file /tmp/jamf_upload/jamf_upload_2658b3a6-899b-43e7-bff5-c97f313fd4d4.txt --header Content-type: application/xml
JamfPackageUploader: Package metadata 'Cisco_Umbrella_3.0.9.pkg' update successful
{'Output': {'jamfpackageuploader_summary_result': {'data': {'category': 'Apps',
'pkg_name': 'Cisco_Umbrella_3.0.9.pkg',
'pkg_path': '/Users/ec2-user/Library/AutoPkg/Cache/local.jamf.CiscoUmbrella-App-AutoUpdate/Cisco_Umbrella_3.0.9.pkg',
'version': '3.0.9'},
'report_fields': ['pkg_path',
'pkg_name',
'version',
'category'],
'summary_text': 'The '
'following '
'packages '
'were '
'uploaded '
'to or '
'updated in '
'Jamf Pro:'},
'pkg_name': 'Cisco_Umbrella_3.0.9.pkg',
'pkg_path': '/Users/ec2-user/Library/AutoPkg/Cache/local.jamf.CiscoUmbrella-App-AutoUpdate/Cisco_Umbrella_3.0.9.pkg',
'pkg_uploaded': True}}
StopProcessingIf
Assuming that is a timing issue, is there some way I can force a pause between running the JamfPackageUploader and JamfPolicyUploader processors? I’m on a Jamf Cloud Premium instance and definitely have multiple cluster nodes in play here.
Ah, just thought of a way to force a step in between. I can move the category upload for the policy category from before the package upload to after the package upload.
From what I understand it can take up to a minute for cluster sync to happen, so not working with cookies has a major impact. Sure, we could write a simple "sleep" processor but getting the correct cookies is the best way to solve this.
On the other hand, I don't remember you complaining about this with JSSImporter runs, which does the same thing with cookies, so I might be barking up the wrong tree. Let's see if the short amount of time spent processing the category is enough.
As far as I can tell, this timing issue doesn’t happen with JSSImporter. I’m actually running Jamf Upload side by side with JSSImporter, where I have one AutoPkg instance with Jamf Upload feeding my test environment and a second AutoPkg instance with JSSImporter feeding my production environment, so I’ve been able to make that assessment between the two solutions.
Could you post the contents of the file /tmp/jamf_upload/curl_cookies_from_jamf_upload.txt after a run?. Hopefully it will give some clues about the cookies. Also, running with -vvv shows cookie information for the package upload (not the other processes).
The /tmp/jamf_upload/curl_cookies_from_jamf_upload.txt file doesn't appear to exist on my AutoPkg instance running Jamf Upload. I have the following files in /tmp/jamf_upload (see attached):
- curl_headers_from_jamf_upload.txt
- curl_output_from_jamf_upload.txt
curl_headers_from_jamf_upload.txt curl_output_from_jamf_upload.txt
I've updated my recipes to move the category upload for the policy category from before the package upload to after the package upload. We'll see if that makes a difference over the weekend.
So far, it looks like moving the policy category upload to after the package upload has addressed the issue. I haven’t seen the issue re-appear after four days.
Would it be possible to add a “wait” Input argument to the processor? Allow the value to be an arbitrary time in seconds and set the default to zero.
That would allow folks with this problem to set their recipe to however many seconds are generally needed (in case the cookie isn’t there) for the package to register before trying to upload the policy.
I noticed I hadn't upped the verbosity of logging on the AutoPkg instance where I saw the behavior. I've now done that and we'll see what happens.
Interestingly enough, I have not seen the policy update failure behavior on my test AutoPkg instance again, where I already had the logging verbosity turned up. Might be a coincidence, but something to add to the store of information about this issue.
I also have similar issues. I used to get the same issues on JSSImporter. I have also been occasionally getting them on the new JCDS upload method.
I was thinking that building a sleep processor might be simple enough, but I like the idea of building a wait into the processor as it is a simpler fix closer to the actual issue.
@rtrouton have you tested if a wait fixes the issue? My assumption is that it would but I haven't tested it.
@matthew-sec,
The best way I could find to implement a "wait" was to move the category upload for the policy category from before the package upload to after the package upload. That may have improved matters somewhat by introducing a small delay at that point, but I'm still seeing the issue periodically.
I saw the problem today on my production AutoPkg setup when Microsoft Edge updated.
I'm attaching verbose output from the run as JamfPolicyUploader_error_processor_20220317.txt and the following files:
curl_headers_from_jamf_upload.txtcurl_output_from_jamf_upload.txt
It looks like the package uploaded by this section of the verbose output:
com.github.grahampugh.jamf-upload.processors/JamfPackageUploader
{'Input': {'API_PASSWORD': 'api.password.here',
'API_USERNAME': 'api.username.here',
'JSS_URL': 'https://jamf.pro.server.here',
'pkg_category': 'Apps',
'pkg_notes': 'Microsoft Edge 99.0.1150.46 installer uploaded via '
'AutoPkg.',
'pkg_path': '/Users/ec2-user/Library/AutoPkg/Cache/local.jamf.MicrosoftEdge-App-AutoUpdate/Microsoft_Edge_99.0.1150.46.pkg',
'version': '99.0.1150.46'}}
JamfPackageUploader: No value supplied for pkg_name, setting default value of:
JamfPackageUploader: No value supplied for pkg_info, setting default value of:
JamfPackageUploader: No value supplied for pkg_priority, setting default value of: 10
JamfPackageUploader: No value supplied for reboot_required, setting default value of:
JamfPackageUploader: No value supplied for os_requirements, setting default value of:
JamfPackageUploader: No value supplied for required_processor, setting default value of: None
JamfPackageUploader: No value supplied for send_notification, setting default value of:
JamfPackageUploader: No value supplied for replace_pkg, setting default value of: False
JamfPackageUploader: No value supplied for replace_pkg_metadata, setting default value of: False
JamfPackageUploader: No value supplied for SMB_URL, setting default value of:
JamfPackageUploader: No value supplied for SMB_USERNAME, setting default value of:
JamfPackageUploader: No value supplied for SMB_PASSWORD, setting default value of:
JamfPackageUploader: Checking for existing 'Microsoft_Edge_99.0.1150.46.pkg' on https://jamf.pro.server.here
JamfPackageUploader: Checking for existing authentication token
JamfPackageUploader: Checking https://jamf.pro.server.here against https://jamf.pro.server.here
JamfPackageUploader: URL for token matches current request
JamfPackageUploader: Existing token is valid
JamfPackageUploader: No existing cookie found - starting new session
JamfPackageUploader: curl command: /usr/bin/curl --silent --show-error -D /tmp/jamf_upload/curl_headers_from_jamf_upload.txt --output /tmp/jamf_upload/curl_output_from_jamf_upload.txt https://jamf.pro.server.here/api/v1/jamf-pro-version --request GET --header authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJhdXRoZW50aWNhdGVkLWFwcCI6IkdFTkVSSUMiLCJhdXRoZW50aWNhdGlvbi10eXBlIjoiSlNTIiwiZ3JvdXBzIjpbXSwic3ViamVjdC10eXBlIjoiSlNTX1VTRVJfSUQiLCJ0b2tlbi11dWlkIjoiM2ZhMTdhNGItZjhkYy00NzQ4LWJkMzAtOTNiNDNmMDM3NWE4IiwibGRhcC1zZXJ2ZXItaWQiOi0xLCJzdWIiOiIyMCIsImV4cCI6MTY0NzU0OTE1Mn0.gqqkIDi4f6D1518GGqhZA9mZT94bMV2yj96sZqhuQHQ --header Accept: application/json
JamfPackageUploader: Jamf Pro Version: 10.36.0-t1644331466
JamfPackageUploader: Token auth will be used,
JamfPackageUploader: No existing cookie found - starting new session
JamfPackageUploader: curl command: /usr/bin/curl --silent --show-error -D /tmp/jamf_upload/curl_headers_from_jamf_upload.txt --output /tmp/jamf_upload/curl_output_from_jamf_upload.txt https://jamf.pro.server.here/JSSResource/packages/name/Microsoft_Edge_99.0.1150.46.pkg --request GET --header authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJhdXRoZW50aWNhdGVkLWFwcCI6IkdFTkVSSUMiLCJhdXRoZW50aWNhdGlvbi10eXBlIjoiSlNTIiwiZ3JvdXBzIjpbXSwic3ViamVjdC10eXBlIjoiSlNTX1VTRVJfSUQiLCJ0b2tlbi11dWlkIjoiM2ZhMTdhNGItZjhkYy00NzQ4LWJkMzAtOTNiNDNmMDM3NWE4IiwibGRhcC1zZXJ2ZXItaWQiOi0xLCJzdWIiOiIyMCIsImV4cCI6MTY0NzU0OTE1Mn0.gqqkIDi4f6D1518GGqhZA9mZT94bMV2yj96sZqhuQHQ --header Accept: application/json
JamfPackageUploader: ID: -1
JamfPackageUploader: No existing cookie found - starting new session
JamfPackageUploader: curl command: /usr/bin/curl --silent --show-error -D /tmp/jamf_upload/curl_headers_from_jamf_upload.txt --output /tmp/jamf_upload/curl_output_from_jamf_upload.txt https://jamf.pro.server.here/dbfileupload --request POST --header authorization: Basic amFtZnByby1hdXRvcGtnOkY4aWFyS016Y1p6MlJ2VA== --upload-file /Users/ec2-user/Library/AutoPkg/Cache/local.jamf.MicrosoftEdge-App-AutoUpdate/Microsoft_Edge_99.0.1150.46.pkg --header Accept: application/xml --header DESTINATION: 0 --header OBJECT_ID: -1 --header FILE_TYPE: 0 --header FILE_NAME: Microsoft_Edge_99.0.1150.46.pkg --max-time 3600
JamfPackageUploader: HTTP response: 200
JamfPackageUploader: Package uploaded successfully, ID=2407
JamfPackageUploader: Package 'Microsoft_Edge_99.0.1150.46.pkg' upload successful
JamfPackageUploader: Updating package metadata for 2407
JamfPackageUploader: <package><name>Microsoft_Edge_99.0.1150.46.pkg</name><filename>Microsoft_Edge_99.0.1150.46.pkg</filename><category>Apps</category><info></info><notes>Microsoft Edge 99.0.1150.46 installer uploaded via AutoPkg.</notes><priority>10</priority><reboot_required>False</reboot_required><required_processor>None</required_processor><os_requirements></os_requirements><hash_type>SHA_512</hash_type><hash_value>604aab6c5876940cf3584d5dbb3cd99cef15934dc16921868bb92fcb6d15446d24937ee58840a4935cb24b0d5149cd532fb5dc89ed0a236e4afe32d895bbac09</hash_value><send_notification>False</send_notification></package>
JamfPackageUploader: Package metadata upload attempt 1
JamfPackageUploader: No existing cookie found - starting new session
JamfPackageUploader: curl command: /usr/bin/curl --silent --show-error -D /tmp/jamf_upload/curl_headers_from_jamf_upload.txt --output /tmp/jamf_upload/curl_output_from_jamf_upload.txt https://jamf.pro.server.here/JSSResource/packages/id/2407 --request PUT --header authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJhdXRoZW50aWNhdGVkLWFwcCI6IkdFTkVSSUMiLCJhdXRoZW50aWNhdGlvbi10eXBlIjoiSlNTIiwiZ3JvdXBzIjpbXSwic3ViamVjdC10eXBlIjoiSlNTX1VTRVJfSUQiLCJ0b2tlbi11dWlkIjoiM2ZhMTdhNGItZjhkYy00NzQ4LWJkMzAtOTNiNDNmMDM3NWE4IiwibGRhcC1zZXJ2ZXItaWQiOi0xLCJzdWIiOiIyMCIsImV4cCI6MTY0NzU0OTE1Mn0.gqqkIDi4f6D1518GGqhZA9mZT94bMV2yj96sZqhuQHQ --upload-file /tmp/jamf_upload/jamf_upload_c388d626-b091-4e1a-8062-d4b404a0ef78.txt --header Content-type: application/xml
JamfPackageUploader: Package metadata 'Microsoft_Edge_99.0.1150.46.pkg' update successful
{'Output': {'jamfpackageuploader_summary_result': {'data': {'category': 'Apps',
'pkg_name': 'Microsoft_Edge_99.0.1150.46.pkg',
'pkg_path': '/Users/ec2-user/Library/AutoPkg/Cache/local.jamf.MicrosoftEdge-App-AutoUpdate/Microsoft_Edge_99.0.1150.46.pkg',
'version': '99.0.1150.46'},
'report_fields': ['pkg_path',
'pkg_name',
'version',
'category'],
'summary_text': 'The '
'following '
'packages '
'were '
'uploaded '
'to or '
'updated in '
'Jamf Pro:'},
'pkg_name': 'Microsoft_Edge_99.0.1150.46.pkg',
'pkg_path': '/Users/ec2-user/Library/AutoPkg/Cache/local.jamf.MicrosoftEdge-App-AutoUpdate/Microsoft_Edge_99.0.1150.46.pkg',
'pkg_uploaded': True}}
StopProcessingIf
{'Input': {'predicate': 'pkg_uploaded == FALSE'}}
StopProcessingIf: (pkg_uploaded == FALSE) is False
However the policy update fails when the package can't be found (presumably because now Jamf Upload is talking to a new cluster node.):
com.github.grahampugh.jamf-upload.processors/JamfPolicyUploader
{'Input': {'API_PASSWORD': 'api.password.here',
'API_USERNAME': 'api.username.here',
'JSS_URL': 'https://jamf.pro.server.here',
'policy_name': 'Auto Update MicrosoftEdge',
'policy_template': 'PolicyTemplate-autoupdate.xml',
'replace_policy': 'True'}}
JamfPolicyUploader: No value supplied for icon, setting default value of:
JamfPolicyUploader: No value supplied for replace_icon, setting default value of: False
JamfPolicyUploader: File found at: /Users/ec2-user/Library/AutoPkg/RecipeRepos/com.github.rtrouton.autopkg_recipes/Jamf_Upload/PolicyTemplate-autoupdate.xml
JamfPolicyUploader: ("Replacing any instances of 'POLICY_NAME' with", "'Auto Update MicrosoftEdge'")
JamfPolicyUploader: ("Replacing any instances of 'POLICY_CATEGORY' with", "'Apps-AutoUpdate'")
JamfPolicyUploader: ("Replacing any instances of 'POLICY_TRIGGER' with", "'autoupdate-MicrosoftEdge'")
JamfPolicyUploader: ("Replacing any instances of 'pkg_name' with", "'Microsoft_Edge_99.0.1150.46.pkg'")
JamfPolicyUploader: Policy data:
JamfPolicyUploader: <?xml version="1.0" encoding="UTF-8"?>
<policy>
<general>
<name>Auto Update MicrosoftEdge</name>
<enabled>true</enabled>
<frequency>Ongoing</frequency>
<category>
<name>Apps-AutoUpdate</name>
</category>
<trigger_other>autoupdate-MicrosoftEdge</trigger_other>
</general>
<scope>
<all_computers>true</all_computers>
</scope>
<package_configuration>
<packages>
<size>1</size>
<package>
<name>Microsoft_Edge_99.0.1150.46.pkg</name>
<action>Install</action>
</package>
</packages>
</package_configuration>
<scripts>
<size>0</size>
</scripts>
<maintenance>
<recon>true</recon>
</maintenance>
</policy>
JamfPolicyUploader: Checking for existing 'Auto Update MicrosoftEdge' on https://jamf.pro.server.here
JamfPolicyUploader: Checking for existing authentication token
JamfPolicyUploader: Checking https://jamf.pro.server.here against https://jamf.pro.server.here
JamfPolicyUploader: URL for token matches current request
JamfPolicyUploader: Existing token is valid
JamfPolicyUploader: No existing cookie found - starting new session
JamfPolicyUploader: curl command: /usr/bin/curl --silent --show-error -D /tmp/jamf_upload/curl_headers_from_jamf_upload.txt --output /tmp/jamf_upload/curl_output_from_jamf_upload.txt https://jamf.pro.server.here/api/v1/jamf-pro-version --request GET --header authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJhdXRoZW50aWNhdGVkLWFwcCI6IkdFTkVSSUMiLCJhdXRoZW50aWNhdGlvbi10eXBlIjoiSlNTIiwiZ3JvdXBzIjpbXSwic3ViamVjdC10eXBlIjoiSlNTX1VTRVJfSUQiLCJ0b2tlbi11dWlkIjoiM2ZhMTdhNGItZjhkYy00NzQ4LWJkMzAtOTNiNDNmMDM3NWE4IiwibGRhcC1zZXJ2ZXItaWQiOi0xLCJzdWIiOiIyMCIsImV4cCI6MTY0NzU0OTE1Mn0.gqqkIDi4f6D1518GGqhZA9mZT94bMV2yj96sZqhuQHQ --header Accept: application/json
JamfPolicyUploader: Jamf Pro Version: 10.36.0-t1644331466
JamfPolicyUploader: Token auth will be used,
JamfPolicyUploader: No existing cookie found - starting new session
JamfPolicyUploader: curl command: /usr/bin/curl --silent --show-error -D /tmp/jamf_upload/curl_headers_from_jamf_upload.txt --output /tmp/jamf_upload/curl_output_from_jamf_upload.txt https://jamf.pro.server.here/JSSResource/policies --request GET --header authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJhdXRoZW50aWNhdGVkLWFwcCI6IkdFTkVSSUMiLCJhdXRoZW50aWNhdGlvbi10eXBlIjoiSlNTIiwiZ3JvdXBzIjpbXSwic3ViamVjdC10eXBlIjoiSlNTX1VTRVJfSUQiLCJ0b2tlbi11dWlkIjoiM2ZhMTdhNGItZjhkYy00NzQ4LWJkMzAtOTNiNDNmMDM3NWE4IiwibGRhcC1zZXJ2ZXItaWQiOi0xLCJzdWIiOiIyMCIsImV4cCI6MTY0NzU0OTE1Mn0.gqqkIDi4f6D1518GGqhZA9mZT94bMV2yj96sZqhuQHQ --header Accept: application/json
JamfPolicyUploader: Policy 'Auto Update MicrosoftEdge' already exists: ID 1664
JamfPolicyUploader: Replacing existing policy as 'replace_policy' is set to True
JamfPolicyUploader: Uploading Policy...
JamfPolicyUploader: Policy upload attempt 1
JamfPolicyUploader: No existing cookie found - starting new session
JamfPolicyUploader: curl command: /usr/bin/curl --silent --show-error -D /tmp/jamf_upload/curl_headers_from_jamf_upload.txt --output /tmp/jamf_upload/curl_output_from_jamf_upload.txt https://jamf.pro.server.here/JSSResource/policies/id/1664 --request PUT --header authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJhdXRoZW50aWNhdGVkLWFwcCI6IkdFTkVSSUMiLCJhdXRoZW50aWNhdGlvbi10eXBlIjoiSlNTIiwiZ3JvdXBzIjpbXSwic3ViamVjdC10eXBlIjoiSlNTX1VTRVJfSUQiLCJ0b2tlbi11dWlkIjoiM2ZhMTdhNGItZjhkYy00NzQ4LWJkMzAtOTNiNDNmMDM3NWE4IiwibGRhcC1zZXJ2ZXItaWQiOi0xLCJzdWIiOiIyMCIsImV4cCI6MTY0NzU0OTE1Mn0.gqqkIDi4f6D1518GGqhZA9mZT94bMV2yj96sZqhuQHQ --upload-file /tmp/jamf_upload/jamf_upload_633d3dc9-7dc5-43a5-9300-4481252f5cd4.txt --header Content-type: application/xml
JamfPolicyUploader: b'<html>\n<head>\n <title>Status page</title>\n</head>\n<body style="font-family: sans-serif;">\n<p style="font-size: 1.2em;font-weight: bold;margin: 1em 0px;">Conflict</p>\n<p>Error: Problem with package</p>\n<p>You can get technical details <a href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.10">here</a>.<br>\nPlease continue your visit at our <a href="/">home page</a>.\n</p>\n</body>\n</html>\n'
Traceback (most recent call last):
File "/Library/AutoPkg/autopkglib/__init__.py", line 840, in process
self.env = processor.process()
File "/Library/AutoPkg/autopkglib/__init__.py", line 626, in process
self.main()
File "/Users/ec2-user/Library/AutoPkg/RecipeRepos/com.github.autopkg.grahampugh-recipes/JamfUploaderProcessors/JamfPolicyUploader.py", line 304, in main
token=token,
File "/Users/ec2-user/Library/AutoPkg/RecipeRepos/com.github.autopkg.grahampugh-recipes/JamfUploaderProcessors/JamfPolicyUploader.py", line 134, in upload_policy
if self.status_check(r, "Policy", policy_name, request) == "break":
File "/Users/ec2-user/Library/AutoPkg/RecipeRepos/com.github.autopkg.grahampugh-recipes/JamfUploaderProcessors/JamfUploaderLib/JamfUploaderBase.py", line 396, in status_check
f"WARNING: {endpoint_type} '{obj_name}' {action} failed due to a conflict"
autopkglib.ProcessorError: WARNING: Policy 'Auto Update MicrosoftEdge' update failed due to a conflict
File "/Library/AutoPkg/autopkglib/__init__.py", line 840, in process
self.env = processor.process()
Receipt written to /Users/ec2-user/Library/AutoPkg/Cache/local.jamf.MicrosoftEdge-App-AutoUpdate/receipts/local.jamf-receipt-20220317-201330.plist
curl_headers_from_jamf_upload.txt curl_output_from_jamf_upload.txt
JamfPolicyUploader_error_processor_20220317.txt
If the policy update hits a failure, could the JamfPolicyUploader processor wait a certain amount of time (say 30 seconds) and then try a second time?
Just to let you know, I'm on vacation for a fe more days. I'll look into this next week. the headers file looks helpful, at least some cookies are being generated so I can work with this.
I think the solution is getting the cookies to work rather than building workarounds like "sleep" processors. Jamf should be able to help with this.
I have a ticket open with Jamf and I've pointed them at this GitHub issue report, as well as mentioning the cookie issue.
@rtrouton just taking another look at this and specifically the repeated No existing cookie found - starting new session messages. The only circumstance that should generate that output is if the cookie_jar file does not exist. The cookie jar file is written for any curl request to any jamf API endpoint (api, uapi, JSSResource, dbfileupload, legacy/packages).
Can you check if no such file is being written to /tmp/jamf_upload/curl_cookies_from_jamf_upload.txt, and if not, could there be something on your system preventing this being written?
I'm seeing the following files in /tmp/jamf_upload:
- curl_cookies_from_jamf_upload.txt
- curl_headers_from_jamf_upload.txt
- curl_output_from_jamf_upload.txt
Please see below for output:
username@computername ~ % cd /tmp/jamf_upload
username@computername jamf_upload % ls -al
total 24
drwxr-xr-x 5 username wheel 160 May 10 06:21 .
drwxrwxrwt 114 root wheel 3648 May 10 13:35 ..
-rw-r--r-- 1 username wheel 509 May 10 13:24 curl_cookies_from_jamf_upload.txt
-rw-r--r-- 1 username wheel 703 May 10 13:24 curl_headers_from_jamf_upload.txt
-rw-r--r-- 1 username wheel 105 May 10 13:24 curl_output_from_jamf_upload.txt
username@computername jamf_upload %
When I checked the contents of curl_cookies_from_jamf_upload.txt, here's what I saw (Jamf Pro server's DNS address removed from output):
username@computername jamf_upload % cat curl_cookies_from_jamf_upload.txt
# Netscape HTTP Cookie File
# https://curl.haxx.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.
jamf.pro.server.address.here FALSE / TRUE 1652793886 AWSALBCORS opiems8RhKYXk5HYMg4DpWe+xp0mFuMz09YLOQwtf6MaYWT9gt5YKIgemqB/HkrIbTDCNkCAnNXkNcwpsgvlDL1j1pXCUi02vRwUbd2SKQEPdWYUghsusSb7FOb3
jamf.pro.server.address.here FALSE / FALSE 1652793886 AWSALB opiems8RhKYXk5HYMg4DpWe+xp0mFuMz09YLOQwtf6MaYWT9gt5YKIgemqB/HkrIbTDCNkCAnNXkNcwpsgvlDL1j1pXCUi02vRwUbd2SKQEPdWYUghsusSb7FOb3
username@computername jamf_upload %
Hmm this is very strange. I have now noticed in your output that no cookie is being set. There should be --cookie-jar /tmp/jamf_upload/curl_cookies_from_jamf_upload.txt and --cookie /tmp/jamf_upload/curl_cookies_from_jamf_upload.txt in each curl output.
Would you please run a verbose trace again and see if this is still the case? I think I have made changes since you ran this in March which might have fixed it.
It looks like the curl command now includes the --cookie-jar and --cookie values:
JamfPackageUploader: curl command: /usr/bin/curl --dump-header /tmp/jamf_upload/curl_headers_from_jamf_upload.txt https://jamf.pro.server.goes.here/api/v1/jamf-pro-version --request GET --silent --show-error --header authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJhdXRoZW50aWNhdGVkLWFwcCI6IkdFTkVSSUMiLCJhdXRoZW50aWNhdGlvbi10eXBlIjoiSlNTIiwiZ3JvdXBzIjpbXSwic3ViamVjdC10eXBlIjoiSlNTX1VTRVJfSUQiLCJ0b2tlbi11dWlkIjoiYTM1NjZkMDUtZjMzOC00OTk4LTkyYWEtMGE0MmY2YmI4ZTYyIiwibGRhcC1zZXJ2ZXItaWQiOi0xLCJzdWIiOiIyMCIsImV4cCI6MTY1MjQ0ODgxMn0.rpv7DPGARSdH0HnmN-qTdH2k-ysGI1bvSWT-hUYLhjE --output /tmp/jamf_upload/curl_output_from_jamf_upload.txt --output /tmp/jamf_upload/curl_output_from_jamf_upload.txt --header Accept: application/json --cookie-jar /tmp/jamf_upload/curl_cookies_from_jamf_upload.txt --cookie /tmp/jamf_upload/curl_cookies_from_jamf_upload.txt
I'm attaching the full output as a separate file (please see attached "autopkg_verbose_output.txt" file.)
Thanks, Rich
Hi @rtrouton Thanks for the output. It looks like the cookies are working as intended now. I've also had confirmation from Jamf that the processors "do the right thing" in terms of use of the cookie jar. They did also say that the unofficial endpoints might not respect the cookies, especially not with third-party fileshare repos, which if the case could explain the problems you have been experiencing.
I see you have added a "sleep" processor, which is presumably solving your problem. Given the above information, it seems a sensible workaround if you otherwise have problems. I would be, however, interested to see if it's still necessary.
But no stress on that if it's working for you with the sleep step.
This is basically resolved, but I'll leave the issue open until the Sleep processor is released, as a reminder to put that in the wiki once it's done. (Hint: if it's done before July 14 I can also mention the solution in our MacAdmins Campfire Talk....)
FYI, I'm integrating sleep into JamfUploader processors. I've tested with JamfComputerGroupUploader and it's working well for me - I needed to add a 2-second sleep since upgrading to Jamf Pro 10.39.1 to prevent failures. I guess it will also need to be added to some of the other processors that may contain dependencies - Policy, Patch and Profile spring to mind. I'll release this soon, but this is just to let you know that a Sleep processor will be probably superfluous (although I released one anyway while testing this stuff out).
Hi All, I am having some issues with JamfUploader since I installed and configured it yesterday and was hoping for some advice.
My apologies I don't have a lot of (if any!) experience in Mac scripting, I was tasked with managing our companies Macs with very little knowledge and found Autopkg, AutoPkgr and JSS Importer. I don't create my own recipes but have used other JSS recipes and have found these worked well for us. I saw that JSS was being deprecated so I went and configured JamfUploader.
A bit of background on my setup, I have a Jamf Pro Cloud instance, and use AutoPkgr with AutoPkg, JSSImporter and now JamfUploader. I use the same Jamf login details for JSS Importer for Jamf Uploader. I configured JamfUploader with my AutoPkgr instance yesterday and found that only a few of the recipes are working for me.
GoogleChrome.Jamf successfully uploaded a package, created a smart group, created a category and then finally made a policy.
Firefox.Jamf however will not, and only uploads the package file, it doesn't create a category, smart group or policy.
I have the same behaviour with the other following recipes:
DropBox.Jamf GotoMeeting.Jamf MicrosoftAutoUpdate.Jamf Miro.Jamf Skype.Jamf Spotify.Jamf VLC.Jamf zoom.us.jamf
I created an override for each of these recipes and still have the same issue.
As a further piece of information which might be useful, the Firefox recipe seemed to progress further when I manually created the category it was trying to create. The recipe then progressed and created a smart group, but still did not create a policy. I am afraid I don't have the exact error message to hand but I recall the error after running this recipe stated something along the lines of 'unable to use substitute key' -(apologies for the vagueness!). This I believe was when it was trying to create the policy.
.JSS files are still uploading without issue however my MS Teams notification integration is posting blank messages after it runs since updating autoPkgr though that is a separate issue, and not as important.
I've attached the log of a Google Chrome.jamf run and 2 runs where they only a package file was uploaded to my Jamf Pro and no other steps were taken.
MiroJamf.txt GoogleChromeJamf.txt FireFoxJamf.txt
I'm struggling to troubleshoot so my apologies for the lack of other technical information that may be required. If there is any advice you able to offer it would be greatly appreciated!
Ben
Hi Ben, it would be better if you start a new issue as your issue does not relate to this one.
Basically it looks like some of my example recipes don't have the JamfCategoryUploader processor in them, so they can't create a category that doesn't exist. I should have added that processor but overlooked it.
Current workaround is to create the category manually, or override the category with one you already have.