data-transfer-project
data-transfer-project copied to clipboard
google
Can I run the demo container image with useHttps=false?
Hey there. I'm so glad this project exists. Thanks for working on it and helping me out.
I'm wondering if it's possible to run the datatransferproject/demo container image from docker hub, but in such a way that it listens for http requests, not https. This is because I'd like to access the running container through a reverse proxy that I'd prefer speak http to the application.
Currently, when I follow the instructions for Run Locally, and then issue an http request like this, I get an error:
$ curl http://localhost:3000
<html>
<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
<center>The plain HTTP request was sent to HTTPS port</center>
<hr><center>nginx/1.10.3</center>
</body>
</html>
Of course, curl -k https://localhost:3000 works, but only because of -k. I'd like to be able to run the image such that the above request succeeds with an HTTP 200 response.
I can see in here that it's possible to configure the application such that useHttps=false. But AFAICT, this yaml file has to be built into the jar file that is run by this docker container, and that jar file in the docker hub image is preconfigured to have useHttps=true.
I'm a noob when it comes to Java. Is there any way for me to run the docker hub demo image in a way that useHttps=false? For example, by setting an environment variable and/or mounting a custom configuration file.
If not, would it make sense to add this as a feature to how the app boots up? I could pass an env variable pointing to my own common config yaml file (or alternatively env variables for each config like useHttps).
In lieu of advice or such a feature, the only way I can think of doing this without building/hosting a custom image is to somehow override the container image's command to:
- unzip the /app/demo-server-all.jar contents
- edit config/common.yaml (or api.yaml?) to add
useHttps: false - rezip the jar
- run the jar as the image otherwise would
Thanks for your advice!
In case others find this, I implemented that last hacky strategy and it seems to work. Here it is as a kubernetes deployment.
apiVersion: apps/v1
kind: Deployment
metadata:
name: data-transfer-project
spec:
selector:
matchLabels:
app: data-transfer-project
template:
metadata:
labels:
app: data-transfer-project
spec:
initContainers:
- name: reconfigure-jar
image: index.docker.io/datatransferproject/demo:latest
volumeMounts:
- name: init-container-outputs
mountPath: /init-container-outputs
command:
- sh
- -c
- |
# reconfigure the jar
apt-get install -y zip unzip
cd /app
unzip -q demo-server-all.jar -d demo-server-all
echo "unzipped"
printf "\nuseHttps: false\n" >> demo-server-all/config/api.yaml
echo "edited"
(cd demo-server-all && zip -q -r ../demo-server-all-reconfigured.zip .)
echo "rezipped"
mv demo-server-all-reconfigured.zip /init-container-outputs/demo-server-all.jar
echo "wrote /init-container-outputs/demo-server-all.jar"
# Create derrived nginx.conf
cat /etc/nginx/nginx.conf \
| sed 's/^\([[:space:]]\+\)server_name[[:space:]]\+localhost;$/\0\n\n\1listen 80 default_server;\n\1listen [::]:80 default_server;/' \
| sed 's/proxy_pass https:\/\/localhost:8080/proxy_pass http:\/\/localhost:8080/' \
>> /init-container-outputs/nginx.conf
containers:
- name: demo
image: index.docker.io/datatransferproject/demo:latest
envFrom:
- secretRef:
name: data-transfer-project-env-secrets
ports:
- containerPort: 80
name: app-http
- containerPort: 443
name: app-https
- containerPort: 8080
name: api-http
volumeMounts:
- name: init-container-outputs
mountPath: /app/demo-server-all.jar
subPath: demo-server-all.jar
- name: init-container-outputs
mountPath: /etc/nginx/nginx.conf
subPath: nginx.conf
volumes:
- name: init-container-outputs
emptyDir: {}
A related nice-to-have would be able to override the apiBaseUrl at runtime without having to rebuild the jar or container.
There is an apiBaseUrl config parameter in the java jars config/api.yaml, but also one for the client-side angular app that is in the docker image at /usr/share/nginx/html/main.js, which is an output artifact of building the angular app's environment.ts. Because this is set to 'https://localhost:3000' in the demo image, with (I think) no way to override, it means the demo won't work if you're accessing it through a different URL, such as 'dtp.localhost', or a more public URL. You might be able to get the Angular app to load, but when the app tries to load the /datatypes API, it will try to do so at https://localhost:3000, which will fail.
It would be really really nice if:
- The default value of the 'apiBaseUrl' for the angular app was a relative URL instead of an absolute, e.g. '/' instead of 'https://localhost:3000'. Then it wouldn't have to care what host the app is deployed at.
- or the client-side app's apiBaseUrl could be configured at runtime via an environment variable or config file on the filesystem
The broader desire here is for runtime configuration instead of build-time. Right now almost all configuration (other than transfer source/dest service API Keys) are provided at build time (configured in gradle files) and not overridable at runtime.
Thanks @gobengo for the tips and suggestions! The demo server probably has a lot of improvements that could be made.
As far as the host, a runtime config could be nice, however, please note the host will be used to generate redirect urls for the oauth handshake and that url has to be registered at the developer API configuration page at each service endpoint, e.g. GOogle, Microsoft, Flickr,etc.
There is a, possibly older synced, example of starting up with http here: https://github.com/Metaform/data-transfer-starter as well.
@holachuy
As far as the host, a runtime config could be nice, however, please note the host will be used to generate redirect urls for the oauth handshake and that url has to be registered at the developer API configuration page at each service endpoint, e.g. Google, Microsoft, Flickr,etc.
Noted, thanks!
If you and other developers are open to runtime config of server-side values like useHttps, can you advise on what sort of implementation you'd support?
I see this YamlSettingsExtension.java.
Would you be supportive of that class' getSetting method checking environment variables before looking in the settings Map parsed from the yaml files on disk? Or is this straying to far from the point of the class, and you'd want another implementation of SettingsExtension that reads from env? And then maybe compose those two SettingsExtensions (Yaml, EnvironmentVariable) into a single one that checks each in a cascade?
@holachuy Similarly, in search of a path forward for a more flexible solution for the client-side apiBaseUrl that is currently set to to 'https://localhost:3000' and only overridable via a fileReplacement at build time, what do you think of the following short-term solution:
First, I hear from you last comment that just setting it to '/' wouldn't work because the client-side app does need to build an absolute URL to use to initiate the oauth flows with a an absolute redirect_uri.
Instead of the default value being hardcoded as a string with the localhost:3000 origin, build it at runtime using the DOM, e.g. document.baseURI. When the app accessed at 'https://localhost:3000', this JS expression would still evaluate to 'https://localhost:3000', but when the app is accessed at 'https://my-preferred-host', that would also be supported.
Correct me if I'm wrong, but it seems like a no-risk change? If you're supportive I will PR.
Hi @gobengo
Regarding the base URL, another option would be to create an Angular service that abstracts calculation of the base URL. For example:
EnvironmentService#getApiUrl(): string
The implementation would use an injected PlatformLocation.location.origin value. All services that make HTTP requests would then use the EnvironmentService to calculate a proper URL. This would avoid app code having to muck around with the DOM and centralize this process.
What do you think?
Hi @gobengo
That's a good idea to add better support for sourcing config from environment variables.
Note that the existing YAML implementation does this via ENV_COMMON_SETTINGS_PATH/ENV_API_SETTINGS_PATH/ENV_TRANSFER_SETTINGS_PATH, however, that requires infrastructure like K8S to map environment variables to a file.
One approach would be to create a delegating SettingsExtension implementation that first sources from YAML (or another format) and then defaults to an environment variable via System.getEnv(). People that want this behavior can then substitute this extension for the YAML one. The extension could also be limited to only sourcing from environment variables.
Keeping this default behavior in an extension will make it easier for people running multiple instances outside of a container in a shared environment, for example, in local testing.
Do you think that approach would help your use case?
What do you think?
@jimmarino tbh I am not very experienced with Angular or best practices, so I'd defer to someone with more experience there.
Since you're asking, it sounds like that path would add code/abstraction, a small cost to be fair, but I'm not sure what it would gain over the alternative of a one-line change I had in mind.
This would avoid app code having to muck around with the DOM and centralize this process.
Would it? It sounds like EnvironmentService would still be using the DOM API? If anything it sounds like this would move DOM specifics into the app and it's services, instead of it being used just to get a configuration value to boot up the app (but the app wouldn't care/know that the value had come from document.baseURI.
Neither of these solutions is probably ideal in the really long run. For example, neither really helps for the hypothetical scenario where you'd want to really inject a custom apiBaseUrl that is different from the document's baseUri. I don't need this, but someone might: e.g. a whole separate domain, or maybe a baseUri with an extra path at the end ('http://my-host/dtp/'). To facilitate this, you probably still would want to allow an operator to pass a whole custom apiBaseUri value, whether that ends up going in the environment object we that's there now, or some config that configures the defaults for the EnvironmentService you're imagning, it's all kind of the same to me.
What you're proposing doesn't sound bad by any means, I'm just not sure it's the most parsimonious. I know I can make a PR for the one-line change, but I'd probably make more mistakes implementing what you describe, and it would definitely take me longer. :)
@jimmarino
Note that the existing YAML implementation does this via ENV_COMMON_SETTINGS_PATH/ENV_API_SETTINGS_PATH/ENV_TRANSFER_SETTINGS_PATH, however, that requires infrastructure like K8S to map environment variables to a file.
I noticed this in the code too, but I wasn't sure how to use it with the demo docker image, which has a built jar at /app/demo-server-all.jar, and all those file paths zipped inside of it. It seemed like the paths indicated by vars like ENV_COMMON_SETTINGS_PATH had to be paths inside the jar.
I know very little about java, so this could just be me not knowing things.
- Is it possible to use those vars to point to files outside of the jar, e.g. docker-mounted paths?
Since you're asking, it sounds like that path would add code/abstraction, a small cost to be fair, but I'm not sure what it would gain over the alternative of a one-line change I had in mind.
This would avoid app code having to muck around with the DOM and centralize this process.
Would it? It sounds like EnvironmentService would still be using the DOM API? If anything it sounds like this would move DOM specifics into the app and it's services, instead of it being used just to get a configuration value to boot up the app (but the app wouldn't care/know that the value had come from
document.baseURI.
Hi @gobengo
The Angular way is to source this from environment, which is replaced at build time and then referenced by services in the application to construct URLs.
My initial read of what you are you proposing was to replace this mechanism with each service making a request to construct its URL by accessing the DOM. Reading your latest response, it seems that you are proposing to modify the default environment.ts file to create apiBaseUrl from the DOM. I'm OK with this as long as it is not used for the production config (environment.prod.ts). You're right that approach will be less invasive than the injectable service I was proposing :-)
- Is it possible to use those vars to point to files outside of the jar, e.g. docker-mounted paths?
I quickly re-checked the code and it looks like there is some inconsistency in the codebase to where configuration is being sourced. If you want to just replace the YAML settings, I think the easiest way will be to create an extension I outlined above.
A while back I created an example of how to write custom extensions (and a custom build for the UI) here:
https://github.com/Metaform/data-transfer-starter
I think the custom config extension should be fairly straightforward to build. I see a couple of ways this could go:
-
Create an extension that only sources for env settings. This would completely replace the YAML variant.
-
Create an extension that sources from a YAML and if not found sources from env.
Option #2 would be a little more complicated and I can think of two approaches. The first approach would involve creating an extension that replaces the YAML one. This is not ideal because it would lead to duplicate functionality. The second approach would be to enhance the core runtime to allow for multiple sourcing strategies, e.g. try the YAML extension first, then the env one. This could be done by introducing an ordering mechanism in the ServiceExtension signature.
This may seem more complicated than just hardcoding the env sourcing option. However, I think it is important we give people the flexibility to remove that behavior entirely from the runtime.
If you would like to work on this, I'm happy to help out and collaborate.