Pricing Configmap watcher unable to open file

[aminatamoo]

Describe the bug Whilst attempting to customise pricing configurations by adding the pricing-configs configmap, I ran into the following error:

ERROR UPDATING pricing-configs CONFIG: Failed to write file: open /models/gcp.json: permission denied

To Reproduce Steps to reproduce the behavior:

1. Create and deploy a pricing configs configmap. E.g:

apiVersion: v1
kind: ConfigMap
metadata:
  name: "pricing-configs"
  namespace: opencost
data:
  description: "Default prices used to compute allocation between RAM and CPU. GCP pricing API data still used for total node cost."
  CPU: "0.031611"
  spotCPU: "0.006655"
  RAM: "0.004237"
  spotRAM: "0.000892"
  storage: "0.00005479452"
  zoneNetworkEgress: "0.01"
  regionNetworkEgress: "0.01"
  internetNetworkEgress: "0.12"
  projectID: "$GCP_PROJECT_ID"
  billingDataDataset: "$BQ_BILLING_DATASET"

2. Redeploy opencost deployment
3. See error above in the logs

Expected behavior I expected the configmap watcher to populate /models/gcp.json.

Which version of OpenCost are you using? latest (1.101.1)

Additional context Kubernetes versions - 1.23 Public cloud - GCP (GKE)

[AjayTripathy]

@kirbsauce can we get this assigned for a repro?

[kirbsauce]

will do

[mkilchhofer]

Also affected.

Maybe it would be worth to invest some more time and split runtime config from config which is shipped inside the container image. In my environment we enforce every Pod to have a readonly filesystem and make only well-defined directories like /tmp (and some other) writeable via a Kubernetes emptyDir volume.

I first thought that this issue happens in my environment due to that restriction. I then excluded the opencost Pod from the clusterwide policy, but the issue still exists.

[keithhand]

@aminatamoo and @mkilchhofer, how are you configuring the deployment to read your pricing-configs configmap? Based on this doc and this related issue, I get the impression the idea is to apply the configmap data as a JSON file and manually mount that file onto the pod. If I follow a similar format described in the issue I linked, I can get the pricing data mounted onto the pod.

Another method to mount pricing-configs, however, is via the ConfigMapWatcher. When I try and use this method, I see the same error you are seeing.

---

@AjayTripathy, would it be best to advise users to use the ConfigMapWatcher to generate a pricing config rather than having users mount the configmap via the deployment? I tested this on my cluster and could get it working if I passed the KUBECOST_NAMESPACE variable to correspond with my deployed "opencost" namespace. However, as mentioned before, I see the same:

ERROR UPDATING pricing-configs CONFIG: Failed to write file: open /models/gcp.json: permission denied

when the container tries to update the deployed JSON file with the pricing-configs ConfigMap data. Is this a bug or not intended to be used within OpenCost?

[AjayTripathy]

Thanks for investigating @keithhand .

1. file an issue to replace KUBECOST_NAMESPACE with OPENCOST_NAMESPACE
2. document that this needs to be set to use the configmap watcher

Meanwhile, @aminatamoo @mkilchhofer have you tried to follow the docs listed by @keithhand ?

[toscott]

Pretty sure the underlying issue is the baked in json files are being added as root prior to swapping to user 1001. So file mounts at that location fail since user 1001 doesn't have write access to them.

It looks like /models can be overwritten with the env variable: CONFIG_PATH

I suspect we can mount a configmap to a different spot and swap out the environment variable to point to them.

I'll give it a quick shot and if that does the trick I can throw up a PR for the website.

[mkilchhofer]

@aminatamoo and @mkilchhofer, how are you configuring the deployment to read your pricing-configs configmap?

Just by putting a ConfigMap with the name pricing-configs in the namespace where OpenCost is running. The watcher then will pick it up automatically, but is unable to overwrite the preshipped files. Overwriting bundled files is not good practice IMHO, although there are some software pieces in the wild which also does this, eg the official nginx container images do this. But it prevents readOnlyRootFilesystem option which is required in my env ;-)

I currently compare wether we need KubeCost or if OpenCost cover all our usecases. I dont want to have custom prices, but want to test the options available in KubeCost https://github.com/kubecost/cost-analyzer-helm-chart/blob/develop/cost-analyzer/templates/cost-analyzer-pricing-configmap.yaml Most likely the sharedNamespaces config.

[aminatamoo]

Hey @AjayTripathy, I originally took the approach that @mkilchhofer took and ran into this error so I pivoted to mounting the file onto the pod. I tried mounting to the models directory to overwrite the gcp.json config there but that didn't pick up any of the custom prices.

I've now followed the instructions in the docs linked (mounting to /tmp/custom-config) but ran into the following errors:

Custom Pricing file at path '/tmp/custom-config/gcp.json' read error: 'stat /tmp/custom-config/gcp.json: not a directory'

Error downloading default pricing data: stat /tmp/custom-config/gcp.json: not a directory

Failed to download pricing data: stat /tmp/custom-config/gcp.json: not a directory

Custom Pricing file at path '/tmp/custom-config/gcp.json' read error: 'stat /tmp/custom-config/gcp.json: not a directory'

error accessing cloud provider configuration: stat /tmp/custom-config/gcp.json: not a directory

So I switched the file name to gcp.json. I no longer get the above errors but I'm still not getting custom pricing data.

[mattray]

@aminatamoo I was able to override AWS prices following https://www.opencost.io/docs/on-prem#custom-pricing-using-the-opencost-helm-chart (substituting aws.json for default.json). I don't currently have a GCP cluster, but could you check this again?

[KipsasJaujoj]

Same situation with metrics-config: 2023-04-12T08:08:04.894176745Z INF ERROR UPDATING metrics-config CONFIG: error writing to metrics config file: open /tmp/pricing-configs/metrics.json: read-only file system

[r2k1]

I suspect it can be solved by changing opencost deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: opencost
  namespace: opencost
  # ...
spec:
  # ...
  template:
    # ...
    spec:
      # ...
      securityContext:
        fsGroup: 1001 # OpenCost is running as a non-root user, this gives container permission to write to the volume

I'm not sure if opencost actually writes anything to the config file. Is it necessary to open the file in write mode for kubecost?

[KipsasJaujoj]

@r2k1 It watches configmap and tries to update in file config if it changes. Your suggested change does not work. Exactly same error Edit: let me be clear: workaround using CONFIG_PATH works here and one is able to modify behaviour for prices and metrics. You are just not able to update values live by changing configmap values (configmap watcher is suppose to watch these and update config in files). And that's a minor inconvenience.

[mattray]

Could you please verify this is still an issue? We've had several updates to the Helm chart to support pricing via ConfigMap and I think this is fixed.

[kaitimmer]

I think that this still needs to be fixed. Even if you are not using custom pricing, but only a config like this:

       - name: AZURE_BILLING_ACCOUNT
              value: "xxx"
            - name: AZURE_OFFER_ID
              value: "xxx"

Opencost tries to update the config file in models but gets an error like:

ERROR UPDATING pricing-configs CONFIG: error writing config file for provider config: Failed to write file: open /models/azure.json: permission denied

So, IMHO it would still be good not to edit the shipped files directly but copy them first and then edit those. Kind of the same way I did it in https://github.com/opencost/opencost/pull/2521

The reason for this would be the same: Allow read-only root filesystems.

See: https://github.com/opencost/opencost/pull/2599 which should fix this.

[mattray]

I verified this works on GCP with the following in my local-gke.yml when using the OpenCost Helm chart.

opencost:
  customPricing:
    enabled: true
    provider: gcp
    costModel:
      description: Crazy high prices based on internal pricing
      CPU: 100.25
      RAM: 10.50
      storage: 10.25

and I loaded it with

helm install opencost --repo https://opencost.github.io/opencost-helm-chart opencost --namespace opencost -f local-gke.yml --set opencost.exporter.defaultClusterId='gke-opencost'

I logged into the container and see

~ $ cd /tmp/custom-config/
/tmp/custom-config $ ls
gcp.json
/tmp/custom-config $ cat gcp.json 
{
  "CPU": "100.25",
  "GPU": "0.95",
  "RAM": "10.5",
  "description": "Crazy high prices based on internal pricing",
  "internetNetworkEgress": "0.12",
  "regionNetworkEgress": "0.01",
  "spotCPU": "0.006655",
  "spotRAM": "0.000892",
  "storage": "10.25",
  "zoneNetworkEgress": "0.01",
  "provider" : "gcp"
}/tmp/custom-config $ 

I verified this in the UI and see those crazy high prices

I'm going to close this as fixed, but I'll go add this to the docs. I'm working on the read-only containers and will make sure this still works.