boto
Implement equivalent to boto.utils.get_instance_metadata()
This request seems to come up often: Should there be a utility function in boto to retrieve the instance metadata? See: http://stackoverflow.com/questions/31630822/boto3-equivalent-to-boto-utils-get-instance-metadata
There is not currently support for this. I'll mark this as a feature request and we can track it here.
This feature would be awesome! We've been postponing using boto3 for a lot of our stuff because this functionality is not available :+1:
When this is implemented, it would also be great if there was builtin way of recursively grabbing all the metadata as a dictionary, instead of only support for lazy loads (like boto2). Somewhat similarly to what we do for collections in boto3, but dict instead of list:
metadata = dict(instance.metadata.all())
Thanks! :)
I can not believe this is not in boto3 already. Because AWS' next generation of compute services like Lambda include boto3 natively, having to revert back to boto2 for things like this is a PITA.
@mtdowling What is the recommended way to replace this method? Do we have to roll our own thing or should we include outdated boto?
Update (as below): I've made a library ec2-metadata that provides an easy interface to query the EC2 metadata API, with caching.
On the instance, do e.g.:
In [5]: requests.get('http://169.254.169.254/latest/meta-data/ami-id').text
Out[5]: 'ami-f0928696'
I think the reason this hasn't been copied is because multiple HTTP requests are needed to get all the metadata as a dict, whereas often you only need one item like so.
I would also love it if we could implement this one. I'll see what I can do internally!
Maybe it doesn't have to be this full dict, but an object with some lazy loading properties, and an option to materialize the whole dict if some must have it (for logging/monitoring).
I made a separate mini library for this, with @Bartvds 's suggestion for lazy loading, check it out at https://github.com/adamchainz/ec2-metadata
Additionally, it would be great to see boto3 automatically pickup the current region when run on an EC2 instance.
@rendicott does ec2-metadata not fit your use case? I'd like to make it do everything we need.
@adamchainz I'm sure ec2-metadata will work fine, I plan on using it until it's built into boto
+1 It would be very nice to have this included in the native boto3 lib.
Adding the ec2-metadata lib listed above seems like a very heavy handed approach, if I wanted to add another pip, I would just use requests.
Having this in boto3 or botocore would mean I could run the code natively on Amazon Linux or a Lambda function w/o having to add dependencies. It would also prevent me from needing to use six to be able to use a python native http get method.
Recently noticed /run/cloud-init/instance-data.json on systems with cloud-init.
https://cloudinit.readthedocs.io/en/latest/topics/instancedata.html
@zepplen ec2-metadata is a single 286 line file that builds all the requests calls for you :) Python packaging is better now than it used to be, pip install is not so heavy :)
@adamchainz ec2-metadata looks great, but we're working with a software package (CKAN) that hasn't yet migrated to Python 3.
With the changes in IMDSv2, it seems like this is an even more useful feature.
Announcement: https://aws.amazon.com/blogs/security/defense-in-depth-open-firewalls-reverse-proxies-ssrf-vulnerabilities-ec2-instance-metadata-service/
The local metadata service now has a v2 endpoint, which is differentiated from v1 by the existance of a x-aws-ec2-metadata-token header being passed to the local metadata service.
This token is retrieved by sending a PUT request to a new token API in the local metadata service, which will remain active for up to 6 hours (ttl is set as a header in the put request).
It is possible to disable the v1 endpoint, however to do so, ALL metadata queries must use the token, not only IAM Role Credential request calls.
Given the extra complexity of requesting the token, and keeping track of the token TTL for future requests, this undifferentiated heavy lifting, seems like something that should be solved via the AWS provided SDK.
The ec2-metadata library mentioned by @adamchainz does not implement this solution, and I would argue is not a replacement for a solution provided by AWS. (As some development environments have restrictions on what libraries you can import and use).
ec2-metadata does have an open issue: https://github.com/adamchainz/ec2-metadata/issues/150
@zepplen Incidentally, for those needing a workaround, it's very easy with requests. Eg to retrieve the equivalent of get_instance_metadata()['document']:
token = requests.put('http://169.254.169.254/latest/api/token', headers={'X-aws-ec2-metadata-token-ttl-seconds': '60'}).text
document = requests.get('http://169.254.169.254/latest/dynamic/instance-identity/document', headers={'X-aws-ec2-metadata-token': token}).json()
ec2-metadata 2.2.0 now supports v2, thanks to @ThrawnCA . #ossFasterThanAWS
In [5]: requests.get('http://169.254.169.254/latest/meta-data/ami-id').text Out[5]: 'ami-f0928696'I think the reason this hasn't been copied is because multiple HTTP requests are needed to get all the metadata as a dict, whereas often you only need one item like so.
MMhh.. I've tried with requests and I have to agree with @adamchainz it's quite pointless to spend time on this
@nnsense please try out ec2-metadata, it's a bit more work than one request to use the metadata api v2 ..
At least, it's a bit more work than one request if you want it to be efficient. If you just retrieve a new token every time, you're doubling up all your requests. The implementation in ec2-metadata caches tokens for up to 10 hours and refreshes them when expired, all transparently.
This is crazy that access to instance metadata is not part of boto. Come on guys!
You can access specific metadata easily from botocore if you do not mind calling a private function:
from botocore.utils import IMDSFetcher
IMDSFetcher()._get_request("/latest/meta-data/instance-type", None).text
