jjjake
Setting access key/secret as environment varaible
I am using a docker image (made from a base linux image, not an existing IA docker image) to access Internet Archive. As a part of this I pass through access and secret keys as environment varaibles.
Looking back (almost a decade ago) in v0.6.8 I can see the following changes
AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are now deprecated. IAS3_ACCESS_KEY and IAS3_SECRET_KEY must be used if setting IAS3 keys via environment variables.
Shortly after while testing my docker image I was unable to us the ia cli tool to upload. Errored with no/invalid auth. I confirmed they exist in the docker images env.
Searched the repo and the only issues filed with IAS3_ACCESS_KEY also appear to be since 2014. I figured why talk about it if it isn't broken 😅, but then I realised it does not exsit in code. Downloading v0.9.3 I can see they are used in code but at some point they were removed.
I searched all the releases page for "env" to see something liike "removed environment variables support" but I only found a custom config location.
I assume IAS3_ACCESS_KEY and IAS3_SECRET_KEY no longer exist, does this mean there is no way to set those values as environment variables?
As a workaround for now I've made my source code look for ~/.config/internetarchive/ia.ini and if it doesn't exsit but the environment varaibles do then it will create the ia.ini.
I noticed this as well a while ago. They were removed in a big refactor between versions 0.9.8 and 1.0.0: 2b8248871399a95b8fbca1c71173445e0c7a96d5
Sorry for the trouble and confusion @beeradmoore.
As JustAnotherArchivist mentioned, these were removed about a decade ago as we refactored the configuration workflow (along with a lot of other big changes).
Note that you can also provide get_session() with a config dict or config_file if depending on the default config file location is not an option, for example:
In [1]: from internetarchive import get_session
In [2]: import os
In [3]: s = get_session(config={"s3": {"access": os.environ["IAS3_ACCESS_KEY"], "secret": os.environ["IAS3_SECRET_KEY"]}})
In [4]: s.access_key
Out[4]: 'foo'
In [5]: os.environ["IAS3_ACCESS_KEY"]
Out[5]: 'foo'
I'm not opposed to adding support for setting S3 keys via environment variables, but we should always default to the config file if it exists in any of the checked locations first.
Thanks for the response @jjjake (and sorry @JustAnotherArchivist, I thought I had replied to you much earlier).
For the above stuff, do you mean making changes to the python code itself, or is it possible to provide the ia cli tool a way to configure it?
For now my workaround of having a docker image generate the expected ini file works a treat. Having it use environment variables again would be nice, but it is far from a deal breaker by not having them. I am sure there are more important things you can be doing than adding that, considering it doesn't appear to come up often.
At least with this issue having those keywords hopefully if people need to find info they will land here and may do the same thing.
I went through almost the exact same process as the OP (although I haven’t hacked in something to generate the config file yet).
All my current workflows use the CLI (not importing into Python code), so native env var support would be a really useful improvement for me.
I'm not opposed to adding support for setting S3 keys via environment variables, but we should always default to the config file if it exists
Yeah, the behavior I was expecting was using env vars if present, then falling back to config.
@jjjake @beeradmoore Should this be closed? Looks like it was essentially fixed in #713 and released a few days ago in version 5.6.0. (But note the env var names are slightly different than they were before: IA_ACCESS_KEY_ID and IA_SECRET_ACCESS_KEY.)
I have a crawl currently uploading using these env vars and it seems to be working. 👍 🎉
I no longer have any projects syncing, but if you do and it's working please do.
Glad to see it added, look forward to using it next time I use the API