[BUG] metadata grains failing since 3006.3

[c42-adam]

Description metadata grain is not returning successfully since 3006.3. This did work in 3006.2.

Setup EC2 host in AWS metadata_server_grains: True in minion config

Steps to Reproduce the behavior

$ sudo salt-call grains.get dynamic:instance-identity --out=json
[CRITICAL] Failed to load grains defined in grain file metadata.metadata in function <LoadedFunc name='metadata.metadata'>, error:
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/__init__.py", line 1150, in grains
    ret = funcs[key](**kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 159, in __call__
    ret = self.loader.run(run_func, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1245, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1260, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/grains/metadata.py", line 87, in metadata
    return _search()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/grains/metadata.py", line 64, in _search
    ret[line] = _search(prefix=os.path.join(prefix, line + "/"))
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/grains/metadata.py", line 53, in _search
    linedata["headers"].get("Content-Type", "text/plain")
KeyError: 'headers'
{
    "local": ""
}

Expected behavior command should return some valid json, along the lines of:

{
  "accountId" : "xxx",
  "architecture" : "x86_64",
  "availabilityZone" : "us-east-1a",
  "billingProducts" : null,
  "devpayProductCodes" : null,
  "marketplaceProductCodes" : null,
  "imageId" : "ami-xxx",
  "instanceId" : "i-xxx",
  "instanceType" : "t2.large",
  "kernelId" : null,
  "pendingTime" : "2023-05-02T13:54:39Z",
  "privateIp" : "xxx.xxx.xxx.xxx",
  "ramdiskId" : null,
  "region" : "us-east-1",
  "version" : "2017-09-30"
}

Curl commands against the metadata API return expected results. curl http://169.254.169.254/latest/dynamic/instance-identity/document/

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
          Salt: 3006.3

Python Version:
        Python: 3.10.13 (main, Sep  6 2023, 02:11:27) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: Not Installed
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.13.10
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: ubuntu 22.04.3 jammy
        locale: utf-8
       machine: x86_64
       release: 6.2.0-1009-aws
        system: Linux
       version: Ubuntu 22.04.3 jammy

[welcome[bot]]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey. Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar. If you have additional questions, email us at [email protected]. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[c42-adam]

I have confirmed downgrading to 3006.2 resolved the issue.

[c42-adam]

Still an issue in 3006.4, though it's no longer throwing a stack trace. It just returns an empty object instead of the correct data.

$ sudo salt-call grains.get dynamic:instance-identity --out=json
{
    "local": ""
}

salt --versions-report

Salt Version:
          Salt: 3006.4

Python Version:
        Python: 3.10.13 (main, Oct  4 2023, 21:54:22) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.13.12
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: ubuntu 22.04.3 jammy
        locale: utf-8
       machine: x86_64
       release: 6.2.0-1009-aws
        system: Linux
       version: Ubuntu 22.04.3 jammy

[c42-adam]

Just wondering if we are going to see some movement on this.

Some additional info: I verified that metadata is enabled on multiple affected instances, and that IMDSv2 is set to Optional.

$ sudo salt-call grains.get dynamic:instance-identity --out=json
[CRITICAL] Failed to load grains defined in grain file metadata.metadata in function <LoadedFunc name='metadata.metadata'>, error:
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/__init__.py", line 1154, in grains
    ret = funcs[key](**kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 159, in __call__
    ret = self.loader.run(run_func, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1245, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1260, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/grains/metadata.py", line 87, in metadata
    return _search()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/grains/metadata.py", line 64, in _search
    ret[line] = _search(prefix=os.path.join(prefix, line + "/"))
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/grains/metadata.py", line 53, in _search
    linedata["headers"].get("Content-Type", "text/plain")
KeyError: 'headers'
{
    "local": ""
}

salt-call --versions-report

Salt Version:
          Salt: 3006.5

Python Version:
        Python: 3.10.13 (main, Nov 15 2023, 04:34:27) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.14.2
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: ubuntu 22.04.2 jammy
        locale: utf-8
       machine: x86_64
       release: 6.2.0-1009-aws
        system: Linux
       version: Ubuntu 22.04.2 jammy

[mdschmitt]

So, perhaps @garethgreenaway can weigh in here. Is it possible that the behavior in the most recent commit was bugged..? https://github.com/saltstack/salt/commit/f9fa9381efb7fb8e37baba2aa4c36cf5348f143b

[houstonj1]

This seems related to changes salt.utils.http

Nothing about metadata.py has changed from 3006.2 ... 3006.3, but the http util has changes.

Just wondering if we are going to see some movement on this.

Some additional info: I verified that metadata is enabled on multiple affected instances, and that IMDSv2 is set to Optional.

$ sudo salt-call grains.get dynamic:instance-identity --out=json
[CRITICAL] Failed to load grains defined in grain file metadata.metadata in function <LoadedFunc name='metadata.metadata'>, error:
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/__init__.py", line 1154, in grains
    ret = funcs[key](**kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 159, in __call__
    ret = self.loader.run(run_func, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1245, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1260, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/grains/metadata.py", line 87, in metadata
    return _search()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/grains/metadata.py", line 64, in _search
    ret[line] = _search(prefix=os.path.join(prefix, line + "/"))
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/grains/metadata.py", line 53, in _search
    linedata["headers"].get("Content-Type", "text/plain")
KeyError: 'headers'
{
    "local": ""
}

salt-call --versions-report

linedata is the result of http.query, looking into changes there is next on the list IMO

[houstonj1]

@s0undt3ch do you think changes to http.query are related here?

[s0undt3ch]

@s0undt3ch do you think changes to http.query are related here?

Without digging more, your guess is as good as mine, probably better since you've done some investigation already.

[s0undt3ch]

I just tried with 3006.6, the VM has metadata enabled, and IMDSv2 is set to optional. It works as expected.

[s0undt3ch]

If however IMDSv2 is set to required, then nothing comes back, but also no traceback.

Please confirm that this is still an issue on your end with the latest release.

[s0undt3ch]

With 3007.0rc1 is works with both IMDSv2 set to optional and required.

[c42-adam]

This doesn't appear to be working for me with 3006.6. Testing with salt-call locally is still the same result as above. If I test from the salt master, it just returns the empty string.

I converted my systems to use a custom grain to pull this info instead. That works perfectly fine (aside from the separate grains sync issue). I insert it under a different name so it doesn't conflict. I also re-confirmed that I can query the metadata API with curl.

Let me know if there's any other data I can gather.

$ sudo salt-call grains.get dynamic:instance-identity --out=json
[CRITICAL] Failed to load grains defined in grain file metadata.metadata in function <LoadedFunc name='metadata.metadata'>, error:
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/__init__.py", line 1154, in grains
    ret = funcs[key](**kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 159, in __call__
    ret = self.loader.run(run_func, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1245, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1260, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/grains/metadata.py", line 87, in metadata
    return _search()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/grains/metadata.py", line 64, in _search
    ret[line] = _search(prefix=os.path.join(prefix, line + "/"))
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/grains/metadata.py", line 53, in _search
    linedata["headers"].get("Content-Type", "text/plain")
KeyError: 'headers'
{
    "local": ""
}

salt minion config (redacted):

master:
  - salt.[xx].com
master_type: failover
retry_dns: 0
backup_mode: minion
master_finger: 'xx:xx:xx'
state_output: changes
log_level: warning
metadata_server_grains: True

salt-call -V

Salt Version: Salt: 3006.6

Python Version: Python: 3.10.13 (main, Nov 15 2023, 04:34:27) [GCC 11.2.0]

Dependency Versions: cffi: 1.14.6 cherrypy: 18.6.1 dateutil: 2.8.1 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed Jinja2: 3.1.3 libgit2: Not Installed looseversion: 1.0.2 M2Crypto: Not Installed Mako: Not Installed msgpack: 1.0.2 msgpack-pure: Not Installed mysql-python: Not Installed packaging: 22.0 pycparser: 2.21 pycrypto: Not Installed pycryptodome: 3.19.1 pygit2: Not Installed python-gnupg: 0.4.8 PyYAML: 6.0.1 PyZMQ: 23.2.0 relenv: 0.14.2 smmap: Not Installed timelib: 0.2.4 Tornado: 4.5.3 ZMQ: 4.3.4

System Versions: dist: ubuntu 22.04.2 jammy locale: utf-8 machine: x86_64 release: 6.2.0-1017-aws system: Linux version: Ubuntu 22.04.2 jammy

[houstonj1]

I was able to confirm the same as @s0undt3ch for each version 3006.3 ... 3006.6. The IMDS is required to be set to V1 with V2 optional until a 3007 release gets cut with IMDSv2 support.

I was running into the same issue as @c42-adam previously, but can't reproduce it again.

EDIT: This was using Ubuntu 22.04 LTS

[s0undt3ch]

This doesn't appear to be working for me with 3006.6. Testing with salt-call locally is still the same result as above. If I test from the salt master, it just returns the empty string.

I converted my systems to use a custom grain to pull this info instead. That works perfectly fine (aside from the separate grains sync issue). I insert it under a different name so it doesn't conflict. I also re-confirmed that I can query the metadata API with curl.

Let me know if there's any other data I can gather.

$ sudo salt-call grains.get dynamic:instance-identity --out=json
[CRITICAL] Failed to load grains defined in grain file metadata.metadata in function <LoadedFunc name='metadata.metadata'>, error:
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/__init__.py", line 1154, in grains
    ret = funcs[key](**kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 159, in __call__
    ret = self.loader.run(run_func, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1245, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1260, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/grains/metadata.py", line 87, in metadata
    return _search()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/grains/metadata.py", line 64, in _search
    ret[line] = _search(prefix=os.path.join(prefix, line + "/"))
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/grains/metadata.py", line 53, in _search
    linedata["headers"].get("Content-Type", "text/plain")
KeyError: 'headers'
{
    "local": ""
}

salt minion config (redacted):

master:
  - salt.[xx].com
master_type: failover
retry_dns: 0
backup_mode: minion
master_finger: 'xx:xx:xx'
state_output: changes
log_level: warning
metadata_server_grains: True

salt-call -V

Salt Version: Salt: 3006.6

Python Version: Python: 3.10.13 (main, Nov 15 2023, 04:34:27) [GCC 11.2.0]

Dependency Versions: cffi: 1.14.6 cherrypy: 18.6.1 dateutil: 2.8.1 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed Jinja2: 3.1.3 libgit2: Not Installed looseversion: 1.0.2 M2Crypto: Not Installed Mako: Not Installed msgpack: 1.0.2 msgpack-pure: Not Installed mysql-python: Not Installed packaging: 22.0 pycparser: 2.21 pycrypto: Not Installed pycryptodome: 3.19.1 pygit2: Not Installed python-gnupg: 0.4.8 PyYAML: 6.0.1 PyZMQ: 23.2.0 relenv: 0.14.2 smmap: Not Installed timelib: 0.2.4 Tornado: 4.5.3 ZMQ: 4.3.4

System Versions: dist: ubuntu 22.04.2 jammy locale: utf-8 machine: x86_64 release: 6.2.0-1017-aws system: Linux version: Ubuntu 22.04.2 jammy

How did you install this minion?

[c42-adam]

How did you install this minion?

I used salt-bootstrap, I don't recall the exact command but it used onedir and it was probably installed to version 3004 or 3005, and then upgraded via apt. I also recently downgraded everything to 3006.4 to work around the grains sync issue, and then upgraded this minion to 3006.6. This was all using apt and the official package repos.

It is possible the minion had a pre-onedir version of salt (some other affected servers did), but those packages would have been removed and the upgrade would have still been via bootstrap and onedir.

I will try an apt purge of the salt packages and reinstall via bootstrap.

[c42-adam]

I used apt purge salt-common salt-minion and then went and manually removed /etc/apt/sources.list.d/salt.list, /var/run/salt /opt/Saltstack and /var/cache/salt. I left my /etc/salt folder alone since it was just minion config and pki. I reinstalled with sudo ./bootstrap-salt.sh onedir 3006 using bootstrap-salt version 2024.01.04. It overwrote my minion config anyway. There was one warning about not being able to write a log file, but no critical errors in the bootstrap.

I ran a minimal state that updates the salt minion config to my defaults, added metadata_server_grains: True and restarted the minion. Results are the same.

I don't see any differences in the error or in the versions-report.

$ date ; sudo salt-call grains.get dynamic:instance-identity --out=json
Tue Feb  6 18:57:58 UTC 2024
[CRITICAL] Failed to load grains defined in grain file metadata.metadata in function <LoadedFunc name='metadata.metadata'>, error:
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/__init__.py", line 1154, in grains
    ret = funcs[key](**kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 159, in __call__
    ret = self.loader.run(run_func, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1245, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1260, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/grains/metadata.py", line 87, in metadata
    return _search()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/grains/metadata.py", line 64, in _search
    ret[line] = _search(prefix=os.path.join(prefix, line + "/"))
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/grains/metadata.py", line 53, in _search
    linedata["headers"].get("Content-Type", "text/plain")
KeyError: 'headers'
{
    "local": ""
}

Details

$ sudo salt-call --versions-report Salt Version: Salt: 3006.6

Python Version: Python: 3.10.13 (main, Nov 15 2023, 04:34:27) [GCC 11.2.0]

Dependency Versions: cffi: 1.14.6 cherrypy: 18.6.1 dateutil: 2.8.1 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed Jinja2: 3.1.3 libgit2: Not Installed looseversion: 1.0.2 M2Crypto: Not Installed Mako: Not Installed msgpack: 1.0.2 msgpack-pure: Not Installed mysql-python: Not Installed packaging: 22.0 pycparser: 2.21 pycrypto: Not Installed pycryptodome: 3.19.1 pygit2: Not Installed python-gnupg: 0.4.8 PyYAML: 6.0.1 PyZMQ: 23.2.0 relenv: 0.14.2 smmap: Not Installed timelib: 0.2.4 Tornado: 4.5.3 ZMQ: 4.3.4

System Versions: dist: ubuntu 22.04.2 jammy locale: utf-8 machine: x86_64 release: 6.2.0-1017-aws system: Linux version: Ubuntu 22.04.2 jammy

[mdschmitt]

I can confirm the same with a yum install on AmazonLinux2.

[s0undt3ch]

@c42-adam and @mdschmitt , these VM's, do they have IMDS set to v1 or v2?

[c42-adam]

@c42-adam and @mdschmitt , these VM's, do they have IMDS set to v1 or v2?

IMDSv2 is set to Optional, and I can successfully query IMDSv1 with curl from the affected boxes.

[mdschmitt]

Same. curl http://169.254.169.254/latest/meta-data returns just fine, (IMDSv2 Optional), grain still busted (on 3007 as well). @s0undt3ch

[nicholasmhughes]

I have a 3007.0 master, one 3007.0 minion, and two 3006.7 minions. The 3007.0 minion does not return metadata grains to the master when invoked with the salt command like so:

[root@ip-10-224-36-138 ~]# salt "*0*" grains.get meta-data:instance-id
minion01:
    i-018778a840b76e848
minion02:
    i-0eacb0a92d7daebdc
nifi01:

Interestingly, the salt-call command is able to pull the metadata grains just fine locally on that same 3007.0 minion.

[root@ip-10-224-36-168 minion]# salt-call grains.get meta-data:instance-id
local:
    i-0d63bdf67424557b6

I saw these event loop errors shown in the minion logs:

2024-03-26 16:43:21,878 [salt.loader.lazy :1155][ERROR   ][9261] Exception raised when processing __virtual__ function for salt.loaded.int.grains.metadata. Module will not be loaded: Cannot run the event loop while another loop is running
2024-03-26 16:43:21,883 [salt.loader.lazy :1155][ERROR   ][9261] Exception raised when processing __virtual__ function for salt.loaded.int.grains.metadata_gce. Module will not be loaded: Cannot run the event loop while another loop is running

This lead me to believe that it was some problem with tornado, the default backend for the http utils module. If you change the minion configuration on the 3007.0 minion to include a different backend:

backend: requests

...then it works just fine and does not show the error messages any longer:

[root@ip-10-224-36-138 ~]# salt "*0*" grains.get meta-data:instance-id
minion01:
    i-018778a840b76e848
minion02:
    i-0eacb0a92d7daebdc
nifi01:
    i-0d63bdf67424557b6

@s0undt3ch, can you think of any problems which may cause this issue since we moved away from the vendored tornado implementation?

ref: https://github.com/saltstack/salt/compare/3006.x...3007.x#diff-16

[mdschmitt]

Any update here? I use these grains pretty heavily so I’m actually blocked on upgrading…

[c42-adam]

@mdschmitt I ended up abandoning the built-in metadata and implemented my own. I based it on this: https://github.com/bluprince13/aws-metadata-json/tree/master It pulls in things slightly differently, for instance numbers instead of numbers in strings, so it forced me to change my logic a little, but the structure was all the same.

[mdschmitt]

Do you happen to have a gist of your custom grain? Perhaps I could piggyback

[c42-adam]

@mdschmitt unfortunately because it was for work policy prevents me from sharing it. Sorry.

[mdschmitt]

No problem, it was worth asking. Thanks for the other link, I’ll manage.