ccloud-chargeback-helper icon indicating copy to clipboard operation
ccloud-chargeback-helper copied to clipboard

Published 20 hours ago • verified publisher icon waliaabhishek

Dockerfile issue-

Open davinder26 opened this issue 7 months ago • 18 comments

@waliaabhishek When I use Dockerfile to create image,Image is created but at run time got error-

File "/app/main.py", line 4, in from workflow_runner import execute_workflow File "/app/workflow_runner.py", line 13, in from ccloud.org import CCloudOrgList File "/app/ccloud/org.py", line 7, in import pandas as pd File "/usr/local/lib/python3.11/site-packages/pandas/init.py", line 22, in from pandas.compat import is_numpy_dev as _is_numpy_dev # pyright: ignore # noqa:F401 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/pandas/compat/init.py", line 25, in from pandas.compat.numpy import ( File "/usr/local/lib/python3.11/site-packages/pandas/compat/numpy/init.py", line 4, in from pandas.util.version import Version File "/usr/local/lib/python3.11/site-packages/pandas/util/init.py", line 2, in from pandas.util._decorators import ( # noqa:F401 File "/usr/local/lib/python3.11/site-packages/pandas/util/_decorators.py", line 14, in from pandas._libs.properties import cache_readonly File "/usr/local/lib/python3.11/site-packages/pandas/_libs/init.py", line 13, in from pandas._libs.interval import Interval File "pandas/_libs/interval.pyx", line 1, in init pandas._libs.interval ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

davinder26 avatar Apr 17 '25 19:04 davinder26

Which branch are you using the Dockerfile from ? They might be a little different between the releases and main/WIP branches, etc.

waliaabhishek avatar Apr 17 '25 20:04 waliaabhishek

@waliaabhishek from branch 1.0.4

davinder26 avatar Apr 18 '25 14:04 davinder26

I have seen this happen when the numpy version is not pinned but pandas picks up an incompatible version for that. Unfortunately the current requirements.txt does not pint he numpy version at all.

Can you add the following line to the top of the requirements.txt file? Run the compile and share the latest result please. The line is:

numpy==1.25.2

Let me know if that solves your issue. I might have to add this as a patch fix.

waliaabhishek avatar Apr 18 '25 17:04 waliaabhishek

@waliaabhishek Now got this error- Traceback (most recent call last): File "/app/main.py", line 23, in execute_workflow(arg_flags) File "/app/helpers.py", line 35, in add_entry_exit_logs ret = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/app/workflow_runner.py", line 100, in execute_workflow core_config = try_parse_config_file(config_yaml_path=arg_flags.config_file) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/helpers.py", line 35, in add_entry_exit_logs ret = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/app/workflow_runner.py", line 74, in try_parse_config_file with open(config_yaml_path, "r") as config_file: ^^^^^^^^^^^^^^^^^^^^^^^^^^^ FileNotFoundError: [Errno 2] No such file or directory: './config/config_internal.yaml'

davinder26 avatar Apr 18 '25 18:04 davinder26

Clarification on Configuration File Availability

The config/config_internal.yaml file is present in the repository and should be accessible during standard operations. If you're encountering a FileNotFoundError, it might be due to one of the following reasons:​

  1. Repository Cloning Issues: Ensure that the repository was cloned correctly and that the config directory and its contents are intact.​

  2. Modifications to Docker Configuration: Alterations to the Dockerfile or docker-compose.yml can affect the file paths and working directories, leading to such errors.​

  3. Execution Outside Docker Compose: Running the application outside the prescribed Docker Compose setup might result in the application not locating the configuration file as expected.​

Regarding Container Rebuilding

The prebuilt container is available on Docker Hub and is designed to work out-of-the-box. Rebuilding the container isn't necessary unless you have specific customization requirements. If you choose to rebuild:​

  1. Verify the Repository Structure: Ensure that the directory structure remains unchanged, especially the location of the config directory.​

  2. Avoid Unintended Modifications: Refrain from altering the Dockerfile or docker-compose.yml unless necessary, as changes can impact the application's behavior.​

Scope of Support

Please note that support for issues arising from personal forks or modifications falls outside the scope of this project's support. While I can offer guidance, troubleshooting such issues is the responsibility of the individual maintainer.​

waliaabhishek avatar Apr 18 '25 21:04 waliaabhishek

@waliaabhishek FileNotFoundError: [Errno 2] No such file or directory: './config/config_internal.yaml' But file location is -deployables/assets/chargeback_handler/config/config_internal.yaml

davinder26 avatar Apr 21 '25 01:04 davinder26

@waliaabhishek I run without any modification.

davinder26 avatar Apr 21 '25 01:04 davinder26

@waliaabhishek I changed default="./config/config.yaml" now getting below image 2025-04-21 16:30:34,677 workflow_runner INFO Starting Workflow Runner 2025-04-21 16:30:34,678 workflow_runner INFO File path 2025-04-21 16:30:34,678 workflow_runner INFO ./config/config.yaml Traceback (most recent call last): File "/app/main.py", line 23, in execute_workflow(arg_flags) File "/app/helpers.py", line 35, in add_entry_exit_logs <built-in method read of _io.TextIOWrapper object at 0xfc8956c28d40> ret = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/app/workflow_runner.py", line 104, in execute_workflow core_config = try_parse_config_file(config_yaml_path=arg_flags.config_file) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/helpers.py", line 35, in add_entry_exit_logs ret = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/app/workflow_runner.py", line 82, in try_parse_config_file env_parse_replace(core_config) File "/app/helpers.py", line 64, in env_parse_replace env_parse_replace(v) File "/app/helpers.py", line 64, in env_parse_replace env_parse_replace(v) File "/app/helpers.py", line 66, in env_parse_replace input[k] = find_replace_env_vars(v) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/helpers.py", line 35, in add_entry_exit_logs ret = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/app/helpers.py", line 55, in find_replace_env_vars return get_env_var(input) ^^^^^^^^^^^^^^^^^^ File "/app/helpers.py", line 35, in add_entry_exit_logs ret = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/app/helpers.py", line 46, in get_env_var raise Exception("Cannot find environment variable " + var_name) Exception: Cannot find environment variable LOG_LEVEL

davinder26 avatar Apr 21 '25 16:04 davinder26

@waliaabhishek If I use below command - docker run -d abhiwalia/ccloud_chargeback_handler:1.0.4

I got same error-

Traceback (most recent call last): File "/app/main.py", line 23, in execute_workflow(arg_flags) File "/app/helpers.py", line 35, in add_entry_exit_logs ret = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/app/workflow_runner.py", line 100, in execute_workflow core_config = try_parse_config_file(config_yaml_path=arg_flags.config_file) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/helpers.py", line 35, in add_entry_exit_logs ret = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/app/workflow_runner.py", line 74, in try_parse_config_file with open(config_yaml_path, "r") as config_file: ^^^^^^^^^^^^^^^^^^^^^^^^^^^ FileNotFoundError: [Errno 2] No such file or directory: './config/config_internal.yaml'

davinder26 avatar Apr 22 '25 19:04 davinder26

I see what you mean. The Docker Compose actually overrides the config file flag with a volume mounted config file. Is the Docker compose not working for you ?

if you want to make the container run as standalone you will need to add the following argument to the docker run command --config-file "/app/config/config.yaml" or if you add a volume mount you can change it to what your mount path is. I will fix this in a patch release soon.

waliaabhishek avatar Apr 28 '25 21:04 waliaabhishek

@waliaabhishek I tried this also-

docker run -d abhiwalia/ccloud_chargeback_handler:1.0.4 --config-file "/app/config/config.yaml"

got error-

2025-04-29 13:32:37,734 workflow_runner INFO Starting Workflow Runner Traceback (most recent call last): File "/app/main.py", line 23, in execute_workflow(arg_flags) File "/app/helpers.py", line 35, in add_entry_exit_logs ret = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/app/workflow_runner.py", line 100, in execute_workflow core_config = try_parse_config_file(config_yaml_path=arg_flags.config_file) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/helpers.py", line 35, in add_entry_exit_logs ret = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/app/workflow_runner.py", line 78, in try_parse_config_file env_parse_replace(core_config) File "/app/helpers.py", line 64, in env_parse_replace env_parse_replace(v) File "/app/helpers.py", line 64, in env_parse_replace env_parse_replace(v) File "/app/helpers.py", line 66, in env_parse_replace input[k] = find_replace_env_vars(v) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/helpers.py", line 35, in add_entry_exit_logs ret = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/app/helpers.py", line 55, in find_replace_env_vars return get_env_var(input) ^^^^^^^^^^^^^^^^^^ File "/app/helpers.py", line 35, in add_entry_exit_logs ret = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/app/helpers.py", line 46, in get_env_var raise Exception("Cannot find environment variable " + var_name) Exception: Cannot find environment variable LOG_LEVEL

davinder26 avatar Apr 29 '25 13:04 davinder26

This is expected. You are not providing a value for environment variables which the code needs. The Docker Compose shares what all variables are needed. https://github.com/waliaabhishek/ccloud-chargeback-helper/blob/bbf66450368aeae8e6c83ac9bd9184645c3c1d9b/docker-compose.yml#L17

waliaabhishek avatar Apr 29 '25 17:04 waliaabhishek

@waliaabhishek Its working now .Thanks

davinder26 avatar Apr 30 '25 19:04 davinder26

@waliaabhishek I have one question regarding CCLOUD_LOOKBACK_DAYS.If I specify 200 days,After 200 days do we need to delete the all files from ccloud-chargeback-helper/deployables/datastore.

davinder26 avatar Apr 30 '25 19:04 davinder26

Nope. The datastore is the location where all the prometheus data set is persisted. You delete that, you delete the Prometheus data and the chargeback + billing data long with it.

waliaabhishek avatar Apr 30 '25 21:04 waliaabhishek

@waliaabhishek So why it stops working after CCLOUD_LOOKBACK_DAYS passed.

davinder26 avatar May 01 '25 12:05 davinder26

I am not sure what do you mean by it stops working ? Does it not do anything? Does it slow down ? Does it error out? Does it just wait ? What do the logs say is going on ? I need some more details before i can help with the debug

waliaabhishek avatar May 01 '25 17:05 waliaabhishek

@waliaabhishek I again got this error -

When I run docker image

requests.exceptions.ConnectionError: HTTPConnectionPool(host='prometheus_for_chargeback', port=9090): Max retries exceeded with url: /api/v1/query (Caused by NameResolutionError("<urllib3.connection.HTTPConnection object at 0xf526d5cd72d0>: Failed to resolve 'prometheus_for_chargeback' ([Errno -5] No address associated with hostname)"))

davinder26 avatar May 12 '25 18:05 davinder26