Windows configuration profile that uses CDATA stucks in verifying

[ddribeiro]

Fleet version: 4.66.0 and possibly earlier versions

Web browser and operating system: Windows 11 Pro 24H2 10.0.26100.3775

---

💥  Actual behavior

I uploaded the following profile to Fleet. This purpose of this profile is to use variables to inject employee specific information to the devices. For the purposes of this test, we replaced the variables with hard coded strings.

<Replace><CmdID>0</CmdID><Item><Meta><Format>chr</Format><Type>text/plain</Type></Meta><Target><LocURI>./Vendor/MSFT/Policy/ConfigOperations/ADMXInstall/employee/Policy/employeeAdmxFilename</LocURI></Target><Data><![CDATA[<?xml version="1.0" encoding="UTF-8"?>
<policyDefinitions revision="1.0" schemaVersion="1.0">
   <policyNamespaces>
      <using prefix="employee" namespace="employee.Policies" />
      <using prefix="windows" namespace="Microsoft.Policies.Windows" />
   </policyNamespaces>
   <resources minRequiredRevision="1.0" />
   <supportedOn>
      <definitions>
         <definition displayName="SUPPORTED_WIN10" name="SUPPORTED_WIN10" />
      </definitions>
   </supportedOn>
   <categories>
      <category displayName="DefaultCategory" name="DefaultCategory">
      </category>
   </categories>
   <policies>
      <policy name="Subteam" class="Both" displayName="Subteam" key="Software\Policies\employee\Attributes" explainText="Subteam" presentation="String">
         <parentCategory ref="DefaultCategory" />
         <supportedOn ref="SUPPORTED_WIN10" />
         <elements>
            <text id="Subteam" valueName="Subteam" />
         </elements>
      </policy>
      <policy name="Team" class="Both" displayName="Team" key="Software\Policies\employee\Attributes" explainText="Team" presentation="String">
         <parentCategory ref="DefaultCategory" />
         <supportedOn ref="SUPPORTED_WIN10" />
         <elements>
            <text id="Team" valueName="Team" />
         </elements>
      </policy>
      <policy name="company" class="Both" displayName="company" key="Software\Policies\employee\Attributes" explainText="company" presentation="String">
         <parentCategory ref="DefaultCategory" />
         <supportedOn ref="SUPPORTED_WIN10" />
         <elements>
            <text id="company" valueName="company" />
         </elements>
      </policy>
      <policy name="physicalDeliveryOffice" class="Both" displayName="physicalDeliveryOffice" key="Software\Policies\employee\Attributes" explainText="physicalDeliveryOffice" presentation="String">
         <parentCategory ref="DefaultCategory" />
         <supportedOn ref="SUPPORTED_WIN10" />
         <elements>
            <text id="physicalDeliveryOffice" valueName="physicalDeliveryOffice" />
         </elements>
      </policy>
   </policies>
</policyDefinitions>]]></Data></Item></Replace>

<Replace>
  <CmdID>1</CmdID>
  <Item>
    <Target>
      <LocURI>./Device/Vendor/MSFT/Policy/Config/employee~Policy~DefaultCategory/company</LocURI>
    </Target>
    <Data>
      <![CDATA[<enabled/> <data id="company" value="foocorp"/>]]>
    </Data>
  </Item>
</Replace>

<Replace>
  <CmdID>2</CmdID>
  <Item>
    <Target>
      <LocURI>./Device/Vendor/MSFT/Policy/Config/employee~Policy~DefaultCategory/Team</LocURI>
    </Target>
    <Data>
      <![CDATA[<enabled/> <data id="Team" value="footeam"/>]]>
    </Data>
  </Item>
</Replace>

<Replace>
  <CmdID>3</CmdID>
  <Item>
    <Target>
      <LocURI>./Device/Vendor/MSFT/Policy/Config/employee~Policy~DefaultCategory/Subteam</LocURI>
    </Target>
    <Data>
      <![CDATA[<enabled/> <data id="Subteam" value="foosubteam"/>]]>
    </Data>
  </Item>
</Replace>

In Fleet this, profile gets sent to the host successfully and goes to a Verifying state. The state never switches to Verified, even after refetching vitals and waiting several days.

🧑‍💻  Steps to reproduce

1. Deploy the Windows profile provided in the section above to a team that has Windows hosts.
2. Observe that the profile delivers successfully to the Windows host and Fleet shows it Verifying
3. Refetch vitals manually. Observe the profile status never flips to Verified.
4. Wait several days. Observe the profile status never flips to Verified.

🕯️ More info (optional)

I was able to confirm the profile is successfully being delivered to the host by exporting management log files in Settings > Accounts > Access Work or School. This produces a log file in C:\Users\Public\Documents\MDMDiagnostics. Upon opening the MDMDiagReport file in that folder, I am able to see this profile on the host.

🛠️ To fix

Update Fleet's verification to skip straight to "Verified" for these complex data types:

• CDATA
• XML: Here's an example: https://learn.microsoft.com/en-us/windows/client-management/mdm/vpnv2-csp#examples

Also add a note to this guide that tells users that Fleet skips to "Verified" for these data types on Windows. Here's how we explain a similar behavior for iOS/iPadOS: https://fleetdm.com/guides/custom-os-settings#basic-article:~:text=Currently%2C%20iOS%20and%20iPadOS%20hosts%20are%20%22Verified%22%20after%20they%20acknowledge%20all%20MDM%20commands%20to%20apply%20OS%20settings.

[Sampfluger88]

Linked to Unthread ticket:

Question about Windows MDM profile verification time #5565

[getvictor]

@marko-lisica @noahtalerman We currently only support ADMX and WlanXML custom schemas. From my research, I found this one that also has a custom XML schema:

• VPN

We can treat them separately or combine them into one story.

[noahtalerman]

🛠️ To fix

Update Fleet's verification to handle configuration profiles that use CDATA to embed XML.

While we're at it, this bug will also handle updating verification for VPN configuration profiles: https://learn.microsoft.com/en-us/windows/client-management/mdm/vpnv2-csp#examples

FYI @marko-lisica here's where we landed.

[zayhanlon]

hey @lukeheath ! was discussing this issue with the customer-numa team today, and it's a blocker for the Windows testing that they're doing. requesting permission to p2 this one.

[lukeheath]

@zayhanlon P2 sounds appropriate. Thanks!

[lukeheath]

@georgekarrv FYI looks like this may need to come into the current sprint.

[georgekarrv]

@noahtalerman are we ok with down-scoping what 'verified' means here? The complication occurs because CDATA can be almost anything but we can timebox trying to fix this explicit case.

[georgekarrv]

5 to timebox what this entails. Could be much larger based on detailed findings.

[lukeheath]

@georgekarrv Maybe a good one to sync with the new Windows consultant on?

[TsekNet]

Another example of a custom admx CSP that is returning 500 when trying to apply:

https://letsconfigmgr.com/deploy-firefox-bookmarks-intune/#Ingesting_the_Firefox_ADMX_File referencing https://support.mozilla.org/en-US/kb/managing-firefox-intune

[JordanMontgomery]

@ddribeiro where do you expect these values to take effect on the system? I ask because I picked up this story this morning expecting it to be as easy as "upload the profile, see the verification failure" but when I look at what we're getting back from verification we're getting 404 errors for all the policies in the original profile

When I then took a closer look at the policy I realized it's meant to inject data into an app called "employee" which I don't have and as such the 404(not found) is expected. Do we have an example for reproing this with an app we can access? I suspect the customer is seeing something different than I am and we probably do want to fail verification in this specific case

[ddribeiro]

@JordanMontgomery I might not be understanding the question, but the ADMXInstall profiles seem to manifest in the registry.

For example, when I deploy the profile at the top of the ticket, I see an entry HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\PolicyManager\AdmxInstalled\E2D3908D-AFBA-4108-A8D5-F36A64D9FE60\employee\Policy\employeeAdmxFilename

The MS Office ADMX templates are another example of a similar profile that does this.

[JordanMontgomery]

@ddribeiro That is interesting. For some reason when we query for these policies we get a 404 on the system as if they don't exist. Obviously they do in the registry as you've pointed out. I was initially assuming maybe without some related app installed they weren't being properly installed on the system but it seems like we ought to be able to verify them given what you've shown. I'll keep investigating

[JordanMontgomery]

@noahtalerman and @marko-lisica

I've been investigating this ticket this week and I think I have a proposed solution which is slightly different than the criteria mentioned in "to fix" but i think is correct, so I'll bring it to the next design review.

For context we also have this issue which we want to fix: https://github.com/fleetdm/fleet/issues/29769 which is another CDATA one.

When I look at the results we're getting back from the system we're sending a SyncML GET command to get the results of the CSP we're sending down. We're getting back a 404("not found" error, similar to what you see in HTTP) for the ADMX backed policies seen in this ticket

In 29769 we're getting back empty data as opposed to what we likely expected which was just some complex but ultimately slightly different representation of the data like we saw with some ADMX policies as well as WLAN XML and expect to see for VPN.

Here's what I suggest we do:

1. If we detect the CSP is an ADMX ingestion operations as one of the commands is in this ticket, if we get a 404 back, we still mark it as verified(as long as we got a good response when we sent the command)
2. Likewise if we detect the CSP is setting some configuration based on such a policy(there is a specific path used and it contains tilde characters, so we can) and we get a 404 back, we still mark it as verified
3. For anything else with a non-2XX response we mark it failed
4. If we detect that a CSP is an ADMX policy we apply the current verification logic which Victor added
5. If we detect that a CSP is a Wireless XML profile we apply the recently added verification logic
6. Anything else(which at this point must have had a 2XX response which, like in HTTP is success) we mark as verified

There are some good suggestions in #29769 but the scope is likely larger than this story and would likely require exploring what we can and can't verify via the registry then rewriting the query creation logic to account for that.

An alternative for 6 is that we choose a subset of Location URIs to mark as verified rather than everything and we could allowlist ones that we know are problematic or difficult to verify such as the one in #29679 and VPN profiles

Some more information on ADMX Policy Ingestion can be found here. THese are the policies that return a 404: https://learn.microsoft.com/en-us/windows/client-management/win32-and-centennial-app-policy-configuration

Thoughts?

[JordanMontgomery]

Another example of a custom admx CSP that is returning 500 when trying to apply: https://letsconfigmgr.com/deploy-firefox-bookmarks-intune/#Ingesting_the_Firefox_ADMX_File referencing https://support.mozilla.org/en-US/kb/managing-firefox-intune

@TsekNet how are you ingesting the policy? 500 indicates generally that the system rejected it for some reason like improper formatting. It should likely be encoded as CDATA or you could XMLEncode it. This seems to work for me though it does not verify properly yet

windows-firefox-en-us.xml.zip

[JordanMontgomery]

Filed Feedback for Microsoft at http://aka.ms/AAwt9bl - note that this link can only be opened on a Windows device with Feedback Hub

[JordanMontgomery]

After discussion in Design Review with @marko-lisica

For the specific CSPs discussed in the root issue here we're going to add some code that marks them as verified if we get a 404 back while attempting to verify them. I have filed feedback with Microsoft regarding this however no idea if we will get a response or under what timeline we will get a response.

We will document this behavior and, in the meantime, provide some guidance on how customers can manually verify these settings. For the "employee" policy in the root issue the following registry query will return all of the settings it sets:

SELECT * FROM registry WHERE path LIKE 'HKEY_LOCAL_MACHINE\Software\Policies\employee\Attributes\%%';

One open question we ran out of time for in design review @marko-lisica is do we want to do anything about VPN policies? They were originally listed in this ticket not due to customer complaints but due to the fact they are another complex XML based policy like Wireless XML. I am able to repro this behavior - among other things during verification the profile comes back formatted differently and, I suspect, there could be underlying cases where we would see this profile coming back with slightly different data as well but there are so many settings it's difficult to be sure. This profile is able to repro the verification issue on my system, however:

      <Add>
        <CmdID>10001</CmdID>
        <Item>
          <Target>
            <LocURI>./Vendor/MSFT/VPNv2/TestVpnProfile/ProfileXML</LocURI>
          </Target>
          <Data>
&lt;VPNProfile&gt;
  &lt;ProfileName&gt;TestVpnProfile&lt;/ProfileName&gt;
  &lt;NativeProfile&gt;
    &lt;Servers&gt;jordanmontgomery.com;jordanmontgomery.com&lt;/Servers&gt;
    &lt;NativeProtocolType&gt;Automatic&lt;/NativeProtocolType&gt;
    &lt;Authentication&gt;
      &lt;UserMethod&gt;EAP&lt;/UserMethod&gt;
      &lt;MachineMethod&gt;EAP&lt;/MachineMethod&gt;
    &lt;/Authentication&gt;
  &lt;/NativeProfile&gt;
&lt;/VPNProfile&gt;
    </Data>
        </Item>
      </Add>

If we do want to address these VPN profiles as part of this ticket my suggestion would be that we simply verify we get a 2XX back from the URI on verification. Since the URI contains a profile name and returns a 404 if that profile is not installed that will tell us if it is installed or not, in this case for instance if we query this we get a 404 if "TestVpnProfile" is not installed or the install failed ./Vendor/MSFT/VPNv2/TestVpnProfile/ProfileXML

[marko-lisica]

We decided that we won't address the VPN profile verification in this ticket. We are unclear about the use case and its application in the real world. Currently, there are no related requests.

cc @noahtalerman

[TsekNet]

@JordanMontgomery Here's the example firefox xml I was using https://gist.github.com/TsekNet/966a6a7a096c5493be073203acb779c0

I tried to apply it by uploading it to FleetDM via the UI.

[PezHub]

QA Test Results -

I confirmed the profile mentioned in the body of the ticket is now getting verified on my Windows host

<Replace><CmdID>0</CmdID><Item><Meta><Format>chr</Format><Type>text/plain</Type></Meta><Target><LocURI>./Vendor/MSFT/Policy/ConfigOperations/ADMXInstall/employee/Policy/employeeAdmxFilename</LocURI></Target><Data><![CDATA[<?xml version="1.0" encoding="UTF-8"?>
<policyDefinitions revision="1.0" schemaVersion="1.0">
   <policyNamespaces>
      <using prefix="employee" namespace="employee.Policies" />
      <using prefix="windows" namespace="Microsoft.Policies.Windows" />
   </policyNamespaces>
   <resources minRequiredRevision="1.0" />
   <supportedOn>
      <definitions>
         <definition displayName="SUPPORTED_WIN10" name="SUPPORTED_WIN10" />
      </definitions>
   </supportedOn>
   <categories>
      <category displayName="DefaultCategory" name="DefaultCategory">
      </category>
   </categories>
   <policies>
      <policy name="Subteam" class="Both" displayName="Subteam" key="Software\Policies\employee\Attributes" explainText="Subteam" presentation="String">
         <parentCategory ref="DefaultCategory" />
         <supportedOn ref="SUPPORTED_WIN10" />
         <elements>
            <text id="Subteam" valueName="Subteam" />
         </elements>
      </policy>
      <policy name="Team" class="Both" displayName="Team" key="Software\Policies\employee\Attributes" explainText="Team" presentation="String">
         <parentCategory ref="DefaultCategory" />
         <supportedOn ref="SUPPORTED_WIN10" />
         <elements>
            <text id="Team" valueName="Team" />
         </elements>
      </policy>
      <policy name="company" class="Both" displayName="company" key="Software\Policies\employee\Attributes" explainText="company" presentation="String">
         <parentCategory ref="DefaultCategory" />
         <supportedOn ref="SUPPORTED_WIN10" />
         <elements>
            <text id="company" valueName="company" />
         </elements>
      </policy>
      <policy name="physicalDeliveryOffice" class="Both" displayName="physicalDeliveryOffice" key="Software\Policies\employee\Attributes" explainText="physicalDeliveryOffice" presentation="String">
         <parentCategory ref="DefaultCategory" />
         <supportedOn ref="SUPPORTED_WIN10" />
         <elements>
            <text id="physicalDeliveryOffice" valueName="physicalDeliveryOffice" />
         </elements>
      </policy>
   </policies>
</policyDefinitions>]]></Data></Item></Replace>

<Replace>
  <CmdID>1</CmdID>
  <Item>
    <Target>
      <LocURI>./Device/Vendor/MSFT/Policy/Config/employee~Policy~DefaultCategory/company</LocURI>
    </Target>
    <Data>
      <![CDATA[<enabled/> <data id="company" value="foocorp"/>]]>
    </Data>
  </Item>
</Replace>

<Replace>
  <CmdID>2</CmdID>
  <Item>
    <Target>
      <LocURI>./Device/Vendor/MSFT/Policy/Config/employee~Policy~DefaultCategory/Team</LocURI>
    </Target>
    <Data>
      <![CDATA[<enabled/> <data id="Team" value="footeam"/>]]>
    </Data>
  </Item>
</Replace>

<Replace>
  <CmdID>3</CmdID>
  <Item>
    <Target>
      <LocURI>./Device/Vendor/MSFT/Policy/Config/employee~Policy~DefaultCategory/Subteam</LocURI>
    </Target>
    <Data>
      <![CDATA[<enabled/> <data id="Subteam" value="foosubteam"/>]]>
    </Data>
  </Item>
</Replace>

[fleet-release]

Fleet's gentle fix, Windows profiles now verify, Cloud city breathes free.