tutorial

[schettino72]

doit docs are too hard for a first time reader.

• [x] part 1 - task basics
• [ ] part 2 - task advanced
• [ ] part 3 - re-usable structure
• [ ] part 4 - custom application

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[vlcinsky]

I will think a bit of scenarios for quick start.

My first idea is:

• travel story
• tasks: pack, deliver, unpack

Tutorial steps

(0) Install doit

• aim: create virtualenv, activate it, install doit into it.
• use python -m venv
• activate virtualenv
• then install doit
• try running doit. It will complain about missing dodo.py, but we now know, doit is installed. dodo.py will come soon.

(1) Declare the tasks

Declare the tasks (pack, deliver, unpack).

Use doit list, see the docstring. Planning phase of doit touched for the first time.

(2) Simple command line implementation without dependencies

• Use only CLI, assume commands zip/unzip are present (they shall be available for both Linux and Windows and have consistent use)
• execute just "in order" without any dependency
• ask user to create target directory manually (to prevent problems with "directory already exists")

(3) Add file_dep and targets

• add file_dep and targets
• use dependencies and targets in command call strings
• play with it:
• run repeatedly (not regenerating results which are already done)
• modify datetime of a source, but do not touch content: hey, it keeps track of the content, not the modification time
• modify particular input: it reruns the tasks

(4) Clean and forget

• introduce the "clean" command
• try it: no result
• add "clean" with simple True to the plan, rerun "clean"
• now it works
• ??not sure about forget

(5) Python action

• rewrite the code to use python functions (try the simplest zip/unzip function not to confuse reader)

sixth step: introduce pathlib

• modify the code to use pathlib.Path
• in arguments
• in targets, file_dep

(6) Subtasks

• instead of one "traveller", create three subdirs, each for one. Name adam, bob, cecil
• modify the dodo.py to do all the tasks for these three persons
• list tasks in simple manner - no difference
• list tasks with "all" option, see details
• run all tasks
• clean and run for single person

(7) Parallel execution

• add "sleep" into tasks
• run processing in standard manner, measure execution time
• clean
• run processing with 3 parallel processes, measure execution time

(8) Summarize, point to follow up resources

• doit is great alternative to tools like make. Plus easy to install on any platform.
• the work is done in following phases:
• declare tasks and actions: dodo.py functions task_* returning or yeilding declaration of tasks incl. dependencies and targets.
• plan: first part of processing. Declared tasks and available dependencies are evaluated and tasks to run are decided. In some cases this done iteratively as tasks are executed.
• execute: tasks are actually run
• tasks can be simple command line calls or python functions
• task can declare dependencies and targets.
• tasks can ensure clean action (remove targets)
• There can be one or two levels of tasks (tasks or tasks and subtasks).
• tasks can be run in paralell

There are many more features, e.g.:

• passing command line arguments
• task calculating data usable in other tasks
• customization of task dependency rules
• customization of "is the task uptodate?"
• "auto" execution as soon as file dependency change (only on Linux and MacOS)
• and more

Challenges

The biggest challenge is to find such an example for tasks, which are easy to run on both Windows and Linux boxes. zip and unzip may work, alternative is to find some easy to use python based command line tool, which we can get in place by pip install the_awesome_cli_tool.

[Kwpolska]

python -m venv

virtualenv is better. On Debian, you need to install python3-venv from apt to use venv (and that’s undocumented), and I’ve seen a few other bugs with it.

zip and unzip may work

There is no built-in zip command on Windows. You could just say “Linux/macOS users: use zip; Windows users: use [something else]”.

[vlcinsky]

Thanks @Kwpolska for your hints.

With tutorial it is good to keep things as simple as possible. Any unnecessary conditionals and options may clutter the tutorial, for this reason I consider every sentence which can be removed a success.

For this reason I am not sure about using different commands for zip and unzip.

For installation into virtualenv I personally prefer using tox (I have even cookiecutter receipt for it). I keep tox installed system-wide. With simple tox.ini in style:

[tox]
envlist = py36
skipsdist = True

[testenv]
deps = doit

one can do $ tox and then source .tox/py36/bin/activate

It is repeatable, but it seems too complex for tutorial.

Some python tutorials simply skip that step, it is an option to consider. Or add "install doit into virtualenv" as one article in "How to" section (which is considered distinct from Tutorial).

[schettino72]

My idea for a tutorial was to start with some scripts. Then show the short-comings of it. And improve it step-by-step by using doit and introducing its features.

I was thinking just using some APIs download some data, do some simple calculation, generate a report and publish. These would require any knowledge of other tools.

[schettino72]

I have almost finished "part 1" of tutorial. already live on site: http://pydoit.org/tutorial_1.html

It would be nice to get some early review/feedback :grin: