Fix package structure

[rueberger]

Some major refactoring of the package structure is necessary.

A big issue is that the current import configuration is slow as it loads all modules in the package if you try to import any one module.

I think that something like the numpy package structure is desirable - or maybe with aggressive refactoring we can package this as a barebones plug-and-play module?

What do we want this to be?