data-migrator

data-migrator copied to clipboard

Data-migrator (version 0.6.3.dev2) is a simple data-migration package for python lovers.

.. image:: https://img.shields.io/pypi/pyversions/data-migrator.svg :target: https://pypi.python.org/pypi/data-migrator

.. image:: https://circleci.com/gh/schubergphilis/data-migrator.svg?style=shield :target: https://circleci.com/gh/schubergphilis/data-migrator

.. image:: https://pyup.io/repos/github/schubergphilis/data-migrator/shield.svg :target: https://pyup.io/repos/github/schubergphilis/data-migrator/ :alt: Updates

.. image:: https://readthedocs.org/projects/data-migrator/badge/?version=latest :target: http://data-migrator.readthedocs.io/en/latest/?badge=latest :alt: Documentation Status

.. image:: https://api.codacy.com/project/badge/Grade/bf6030e9e7e248979607802880336611 :target: https://www.codacy.com/app/schubergphilis/data-migrator?utm_source=github.com&utm_medium=referral&utm_content=schubergphilis/data-migrator&utm_campaign=Badge_Grade

.. image:: https://api.codacy.com/project/badge/Coverage/bf6030e9e7e248979607802880336611 :target: https://www.codacy.com/app/schubergphilis/data-migrator?utm_source=github.com&utm_medium=referral&utm_content=schubergphilis/data-migrator&utm_campaign=Badge_Coverage

.. image:: https://badge.fury.io/py/data-migrator.svg :target: https://badge.fury.io/py/data-migrator

Data-migrator is a declarative DSL for table driven data transformations, set up as an open and extensive system. Use this to create data transformations for changing databases as a result of changing code, initial loads to datalakes (it contains a Kinesis provider)and more.

Example

Data-migrator assumes data is extracted and loaded with client access.

.. code-block:: bash

$ mysql source_db -E 'select id,a,b from table' -B  | python my_filter.py | mysql target_db

It than offers a wide range of primitives with default settings to build complex transformations fast, readable and extendable

.. code-block:: python

from data_migrator import models, transform

class Result(models.Model): id = models.IntField(pos=0) # keep id uuid = models.UUIDField() # generate new uuid4 field a = models.StringField(pos=1, default='NO_NULL', max_length=5, nullable='NULL', replacement=lambda x:x.upper()) b = models.StringField(pos=2, name='my_b')

if name == "main": transform.Transformer(models=[Result]).process()

Installation

Execute the following command to install data-migrator with pip::

pip install data-migrator

See the Installation Instructions <http://data-migrator.readthedocs.io/en/latest/install.html>_ in Documentation for more instructions on installing, upgrading, and uninstalling data-migrator.

The project is maintained at GitHub <https://github.com/schubergphilis/data-migrator>_.

Support and contribute

Questions, comments, bug reports and especially tested patches may be submitted directly to the issue tracker <https://github.com/schubergphilis/data-migrator/issues>_.

Everyone interacting with this codebase, issue trackers, chat rooms, and mailing lists is expected to follow the Code of Conduct <http://data-migrator.readthedocs.io/en/latest/code-of-conduct.html>_.