Message: PermissionError: [Errno 13] Permission denied: '/nonexistent' | Message: os.makedirs(directory)

[tarran-durham]

When running in docker container on aws, the following error happens (does not happen locally when running docker container)

Container: main Message: weather_data = fetch_weather_data(location, start, end) Pod: e3afad71767117f8-codeblocktorun-4109565371 Timestamp: 2024-09-18T07:38:22.515052828+00:00

Container: main Message: File "/app/src/weather_history.py", line 25, in fetch_weather_data Pod: e3afad71767117f8-codeblocktorun-4109565371 Timestamp: 2024-09-18T07:38:22.515059845+00:00

Container: main Message: return processing_handler( Pod: e3afad71767117f8-codeblocktorun-4109565371 Timestamp: 2024-09-18T07:38:22.516757064+00:00

Container: main Message: self._data = self._get_data() Pod: e3afad71767117f8-codeblocktorun-4109565371 Timestamp: 2024-09-18T07:38:22.516210391+00:00

Container: main Message: File "/usr/local/lib/python3.12/site-packages/meteostat/core/loader.py", line 59, in processing_handler Pod: e3afad71767117f8-codeblocktorun-4109565371 Timestamp: 2024-09-18T07:38:22.516766748+00:00

Container: main Message: File "/usr/local/lib/python3.12/site-packages/meteostat/interface/meteodata.py", line 127, in _get_data Pod: e3afad71767117f8-codeblocktorun-4109565371 Timestamp: 2024-09-18T07:38:22.516280088+00:00

Container: main Message: ^^^^^^^^^^^^^^^^^^^ Pod: e3afad71767117f8-codeblocktorun-4109565371 Timestamp: 2024-09-18T07:38:22.516763378+00:00

Container: main Message: ^^^^^^^^^^^^^^^^ Pod: e3afad71767117f8-codeblocktorun-4109565371 Timestamp: 2024-09-18T07:38:22.516274939+00:00

Container: main Message: File "/usr/local/lib/python3.12/site-packages/meteostat/interface/meteodata.py", line 51, in _load_data Pod: e3afad71767117f8-codeblocktorun-4109565371 Timestamp: 2024-09-18T07:38:22.517392535+00:00

Container: main Message: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Pod: e3afad71767117f8-codeblocktorun-4109565371 Timestamp: 2024-09-18T07:38:22.517565043+00:00

Container: main Message: File "/usr/local/lib/python3.12/site-packages/meteostat/core/cache.py", line 38, in file_in_cache Pod: e3afad71767117f8-codeblocktorun-4109565371 Timestamp: 2024-09-18T07:38:22.517585474+00:00

Container: main Message: if self.max_age > 0 and file_in_cache(path, self.max_age): Pod: e3afad71767117f8-codeblocktorun-4109565371 Timestamp: 2024-09-18T07:38:22.517476954+00:00

Container: main Message: ^^^^^^^^^^^^^^ Pod: e3afad71767117f8-codeblocktorun-4109565371 Timestamp: 2024-09-18T07:38:22.517387411+00:00

Container: main Message: output.append(load(*dataset)) Pod: e3afad71767117f8-codeblocktorun-4109565371 Timestamp: 2024-09-18T07:38:22.517301976+00:00

Container: main Message: os.makedirs(directory) Pod: e3afad71767117f8-codeblocktorun-4109565371 Timestamp: 2024-09-18T07:38:22.518019086+00:00

Container: main Message: File "", line 215, in makedirs Pod: e3afad71767117f8-codeblocktorun-4109565371 Timestamp: 2024-09-18T07:38:22.518140672+00:00

Container: main Message: File "", line 215, in makedirs Pod: e3afad71767117f8-codeblocktorun-4109565371 Timestamp: 2024-09-18T07:38:22.518089489+00:00

Container: main Message: File "", line 215, in makedirs Pod: e3afad71767117f8-codeblocktorun-4109565371 Timestamp: 2024-09-18T07:38:22.518136848+00:00

Container: main Message: File "", line 225, in makedirs Pod: e3afad71767117f8-codeblocktorun-4109565371 Timestamp: 2024-09-18T07:38:22.518189218+00:00

Container: main Message: PermissionError: [Errno 13] Permission denied: '/nonexistent' Pod: e3afad71767117f8-codeblocktorun-4109565371 Timestamp: 2024-09-18T07:38:22.518193232+00:00

Container: main Message: time="2024-09-18T07:38:23.372Z" level=info msg="sub-process exited" argo=true error="" Pod: e3afad71767117f8-codeblocktorun-4109565371 Timestamp: 2024-09-18T07:38:23.373246457+00:00

The code:

from datetime import datetime, timedelta
import matplotlib.pyplot as plt
from meteostat import Point, Hourly, Stations
import holidays
import os
from utils import setup_connection_to_datasphere, print_public_ip
from hana_ml.dataframe import create_dataframe_from_pandas
import os
import tempfile
print_public_ip()

cache_dir = tempfile.gettempdir()
Stations.cache_dir = os.path.join(cache_dir, 'meteostat_cache')

stations = Stations()

os.environ['MPLCONFIGDIR'] = '/tmp/matplotlib'


local = bool(os.getenv('LOCAL', False) )

def fetch_weather_data(location, start, end):
    data = Hourly(location, start, end)
    return data.fetch()

def enhance_data_with_holidays(data, holiday_region):

    data.reset_index(inplace=True)
    data['hour'] = data['time'].dt.hour
    data['day_of_week'] = data['time'].dt.dayofweek  # Monday=0, Sunday=6
    data['week_of_year'] = data['time'].dt.isocalendar().week
    data['is_weekend'] = data['day_of_week'] >= 5

    swiss_holidays = holidays.Switzerland(subdiv=holiday_region)

    data['is_holiday'] = data['time'].dt.date.apply(lambda x: x in swiss_holidays)

    return data

def save_data_to_csv(data, filename):
    data.to_csv(filename, sep=';', index=False)
    print(f"Data saved to {filename}")


def plot_weather_data(data):
    data.plot(x='time', y=['temp', 'tsun', 'prcp'], subplots=True)
    plt.show()


def main():

    end = datetime.today()
    start = end - timedelta(days=365)


    location = Point(47.6603, 9.1758)


    print("Fetching weather data...")
    weather_data = fetch_weather_data(location, start, end)


    print("Enhancing data with holidays and weekends...")
    enhanced_data = enhance_data_with_holidays(weather_data, 'SH')

    conn_context = setup_connection_to_datasphere()
    create_dataframe_from_pandas(
                conn_context,
                enhanced_data,
                "SM_WEATHER_HISTORY",
                force=True,
                drop_exist_tab=True,
                primary_key=["TIME"]
            )
    
    if local:
        print("Plotting weather data...")
        plot_weather_data(enhanced_data)

if __name__ == "__main__":
    main()