estate icon indicating copy to clipboard operation
estate copied to clipboard

verified publisher icon Proteusiq
→

Metadata

With single command build a beautiful web scraping tool for scheduled scraping and store scraped data in postgres database

Results 13 estate issues
Sort by recently updated
recently updated
newest added

Update Dagster

Dagster has changed a lot. Update knowledge and redo the pipeline

Proteusiq avatar Proteusiq

Update the Project with new Dagster API

Update the project to keep up with the new changes in Dagster, and Docker

Proteusiq avatar Proteusiq

enhancement

User Session to Close DB Connection

See https://docs.dagster.io/concepts/resources

Proteusiq avatar Proteusiq

MindDB

Use [MindDB](https://github.com/mindsdb/mindsdb) to train and score models

Proteusiq avatar Proteusiq

Migrate from Airflow to Dagster

Dagster is simply easier, prettier and model Python.

Proteusiq avatar Proteusiq

enhancement

Predictive models

Introduce Redis to store models Jupyter Lab for modelling Kibana for alerting

Proteusiq avatar Proteusiq

Release

Build intermediate stage for data dumping

Decouple data scraping and sending data to Postgress. Dump data to S3|Minio and have another pipe to pick data from dump to Postgres

Proteusiq avatar Proteusiq

enhancement

Write tests for days and scraper

Use Pytest to write dags tests. # Code without tests is an illusion

Proteusiq avatar Proteusiq

enhancement

Add extra docker-compose-extra.yaml

Extra tools needs there own yaml - Redis UI - Grafana - ELK

Proteusiq avatar Proteusiq

enhancement

Use DagFactory

Use dagfactory [see examples](https://github.com/ajbosco)

Proteusiq avatar Proteusiq

← Metadata

20
Stars
4
Forks
Watchers

Owner

verified publisher icon Proteusiq

Metadata

With single command build a beautiful web scraping tool for scheduled scraping and store scraped data in postgres database

Back