Break generation out into separate lib

[ghickman]

I'd like to see the code base broken up into two pieces:

• lower level generation lib
• cli app that takes a python class path and returns the flattened output through means (html, sql, etc)
• cli app (for ccbv.co.uk) that looks at CBVs specifically and updates the db
• web app reading from the db

The new flow for updating ccbv.co.uk would use the ccbv specific script to generate a docset for a given django version (number or path) and update the db. The web app then doesn't need to restart for docset updates/additions. e.g.

<ccbv-script> [1.5|[-p|--path]path/to/a/django]

This ccbv-script removes the pain of needing to update the web app's django version for each version of docs it hosts.

The CLI app would be a more generic implementation of the ccbv-script that takes the python path to a class and can output some SQL or HTML. My use case is - I have a class, I want the ccbv style html output for it.

I hope that this could be taken further to generating API style sphinx docs, I'm looking at you Django forms.

[NathanRSmith]

I find the ccbv.co.uk site infinitely helpful for the Django cbv's, but it would be excellent to be able to create this kind of output for ANY cbv. I'm thinking specifically of projects like https://github.com/AndrewIngram/django-extra-views that provide extended cbv's that would be easier to comprehend with ccbv.

[ghickman]

@NathanRSmith this is exactly what I'm going for, except I'd like to able to do it with any class. In my case the Django forms.

[meshy]

It's a great plan, and it's on the cards, but I don't really have all that much time at the moment. I was planning on giving this issue a look at the next http://www.refreshoxford.co.uk/

[kevwilde]

I'm investigating the AST library in python as a viable candidate for this. Reference: http://docs.python.org/2/library/ast.html

Also of interest: (thx @rach !) https://pypi.python.org/pypi/jedi/

Update: I could generate a syntax tree via the parser library. Will investigate later if i can have anything decent from that:

import parser
with open("/home/kevin/.virtualenvs/ccbv/lib/python2.7/site-packages/django/views/generic/edit.py", "r") as pyfile:
    ast = parser.suite("".join(pyfile.readlines()))
ast_list = parser.ast2list(ast)
print ast_list

reference:

• How to use parser app: http://blog.prashanthellina.com/2007/11/14/generating-call-graphs-for-understanding-and-refactoring-python-code/
• Token Identities: https://github.com/python-git/python/blob/master/Include/graminit.h

[ghickman]

@NathanRSmith @kevwilde I'm planning to get some work done on this at the upcoming Refresh Oxford Hack Day if you're interesting in collaborating either in person or online?

[meshy]

@ghickman broken link. Should be refreshoxford.co.uk

[ghickman]

Thanks - fixed.

[NathanRSmith]

Unfortunately (for me working on this project at least) I have a wedding to attend that day, so I won't be able to participate. But I will definitely keep an eye on any progress that is made.

[ghickman]

State of the Union:

I made Classify which attempts to do what was outlined above and mostly manages that - virtualenv path issues aside.

libpydoc is basically the library module from classify but I'm hoping to expand it to cover everything pydoc does. I also need to look into the new cli portion of the inspect lib as of 3.4 which sounds very similar to what I've done here.

The work I've done in Classify used the base template from here so porting that work back should be pretty simple.

As a bonus this should make more regular builds much easier and improve the situation for things like #41

[ghickman]

So far this generates a module list page and (mostly blank) class detail pages from:

ccbv <path/to/django>

It creates a build directory with a root version, like so:

build
 |
 -> 1.5
    |
    -> module.html
    -> module
        |
        -> class.html

This would allow a user/build box to have one Django checkout and switch between the release tags.

[ghickman]

On a deployment note I'd like to put this on s3/Cloudfront where we'll hopefully see snappier page resolutions but if we end up needing more control over Nginx we can probably get a developer account from Rackspace.

[ghickman]

Manic weekend but I'm getting somewhere now.

Usage should now be something like make all to have every version built and uploaded to .

General overview:

• Makefile manually tracks django versions and calls the python script for each version
• Python script shells out to some bash to make sure the django checkout is at the correct tag
• Python script (using jinja) builds the html pages in a build dir

TODO:

• [ ] Update libpydoc/annotate the libpydoc datastructure with extra info
• [ ] Convert to jinja2 templates
• [ ] Add source links
• [ ] Add docs links
• [ ] Add deployment info
• [ ] Document

[alizain]

@ghickman, any plans on making this happen?

[ghickman]

@alizain – the plan is still there, the new work is on static-pages-requiem but I got stalled looking into a nice way to build the current home page and the menu (since they both deal with multiple versions).

I don't have any spare time to work on this atm but @meshy and I hope to plan a weekend to push it through eventually. Unfortunately I'm very tied up with my startup and a newborn right now.

[meshy]

There are some great ideas in here that we should (and intend to) carry forward, but for the moment, given this has stalled, I'm going to close this PR. Maybe one day it will live again :)