The Rewrite: BB2 Redux

[zeffii]

Over the next couple of days (unless something unforeseen happens) I have time to make a start with building the add-on from scratch. Getting it to match the working features of BB2 will take a while. It's difficult to put a time frame on it until the process gets rolling.

The decision to start from scratch is not done in haste. I think some of the responders on this repository have expressed similar feelings about how to proceed. There's lots that can be taken from the current code-base, but a lot that I really want to sanitize and arrange into separate files. The core of this was written when Blender 2.5 was still young and the API not very well defined, and prone to convoluted solutions.

If I can reach feature parity and structure the code better then I will consider it a success, speeding up the import (and avoiding recursive updates) is not my first priority, making it easier for others to contribute is far more important.

Ticks indicates functionality seems to be restored for that panel, certainly report any issues with those sectons. The unticked sections have had no major checks done other than suppressing import errors. The section labelled next is either being worked on or that's where the most recent code changes have taken place.

• [x] gui_panel_ep.py
• [x] gui_panel_mlp.py
• [ ] gui_panel_nma_viz.py
• [ ] gui_panel_output_animation.py
• [x] gui_panel_pdb_import.py
• [ ] gui_panel_pdb_output.py <---next
• [ ] gui_panel_physics_sim.py
• [x] gui_panel_view.py

[zeffii]

The plan. (might change as it progresses, and is not chronologic)

• [x] All new code must be pep8 compliant, with the exception of the following error types --ignore=E402,E702,E501,E731,E241 (Use a pep8 linter!, else thyne code shalt not pass!). If you use tabs instead of 4 spaces for indentation you will be slapped with a fish.
  • pep-20 / Zen of Python helps you decide how to do things.
  • http://docs.quantifiedcode.com/python-anti-patterns/index.html
• [ ] Generate the default scene and objects using a script, unless it really can't be done any other way
• [ ] Generate new documenation folder with layout something like

BioBlender/Manual/images/01_01_Intro.png
BioBlender/Manual/images/01_02_Intro_p2.png
BioBlender/Manual/images/02_01_Install_Guide_step1.png
BioBlender/Manual/images/02_01_Install_Guide_step2.png
BioBlender/Manual/images/02_01_Install_Guide_step3.png
BioBlender/Manual/Introduction.md
BioBlender/Manual/Installation.md
BioBlender/Manual/Feature_01.md  <-- replace number with shortname for feature
BioBlender/Manual/Manual.md  <-- would link all .md files together.

• [x] Separate operators into own files
• [x] Separate panels (ui) into own files
• [x] Separate Tables
• [x] Separate Utility code
• [ ] Employ OS agnostic python tools to deal with the double quote mayhem ( there's a dedicated core python module for it shlex
• [ ] Create disposable non rigged import for fast viewing (already coded..not merged)
• [ ] Parent the EP Curves to mainting outliner sanity. Make sure it works with animated EP

[MonZop]

Great plan! I am certainly willing to take care of the documentation folder (help always welcome, of course, but if you are a coder it might be more useful in other places), which is the only part that I understand, almost. uh

[zeffii]

I like a fresh start, unfortunately there won't be anything to test for a while.. it's like a fetus now

[zeffii]

I made a new branch and split the panels into their own source files. This is no-where near working, but it's necessary until gradually things start to work again. It is my hope that I will have useful progress before the replacement dependency graph (aprox two weeks from now according to the developers meeting).

It has been difficult to find motivation to do this lately, but i'd like to think I can finish what I started.

[zeffii]

panels are now separated, and load in the correct sequence. <still non functional>

now to start linking up the global storage dictionaries again. This is the part I'm not terribly sure what kind of time it will take. But already in the last 2 hours this has gone faster than I anticipated. Back to coding.

[MonZop]

Good to hear things are going! Just let me know whan (and if) I can do something, like testing

[zeffii]

it would help me a lot if you found time to learn how to use GIT from the command-line, or find someone to teach you. Even if you aren't coding anything it helps if i'm confident that the files you are testing are indeed the branch i'm working on.

There are only a few commands that anyone needs to know

git pull --all
git push --all
git add --all
git commit -a -m "commit message here"
git checkout -b <branch_name>  # new_branch_name_based_on_current_branch
git branch -D <branch_name>
git branch

[MonZop]

OK, I'll make an effort, (find someone to help me tomorrow, if available). I already know that once learned, it will be easy, but i've been lazy, and had also much to do. This Sunday (10 of May) I'm going to Munich to the Premiere of the film The Dark Gene, which contains several minutes of an animation made by us using BioBlender! Munich DOK.fest I'm looking forward to see how the poublic will react at those images! (I mean, besides Blender people, and biologists)

[zeffii]

so far so good, but won't know how easy the rest will be until the next push.

Cool, a premiere! Must be great to finally see your work in context, enjoy!

[zeffii]

This is much easier to navigate now. wow.

I think the import panel has reached feature parity with older versions, but it has not been tested extensively yet. The import panel is 99% existing code, and small changes to make things Blender version agnostic. I've replaced boolean and integer global variables with scene specific properties which are by definition global. Added aliasing to avoid 'dot lookups' where possible.

Time to continue to the next panel. MLP. because I know it a little bit.

[zeffii]

mlp surface works now:

still too rough for testing..

[zeffii]

render mlp to surface now too:

[MonZop]

Beautiful! Congratulations! I have also tried (using the redux_s3_FIRE) , and got the MLP on the surface, but not the render. I also have a question: is this MLP the value calculated by MLP.py, or is it the color of pyMOL, simply transformed to grey scale, and rendered?

[zeffii]

I have also tried (using the redux_s3_FIRE) , and got the MLP on the surface, but not the render.

i'll look at it. The code feels a little like this now : screenshot from 2015-05-08 14 46 23

there's a conversion function that strips out the colour, will look carefully at what it does.

[MonZop]

WOW, it makes me feel dizzy with vertigo!

[zeffii]

At present:

I haven't yet traced manually what they do, technically.

[zeffii]

(wrong button pressed closed the thread)

here's the abstract of those two functions, ps: this might take a while to fully write and will edit this post over the course of hours/days. I have made no functional changes to any of this code, except minor formatting (may 2015).

bb2_operator_mlp()

In theory:

• expects user to select an atom of a model, any atom.
• if multiple atoms are selected, its looks to see if the atoms are from one .pdb or multiple
• for every pdb model identified by the selection it creates a tID variable
  • perform the mlp(tID) function
  • todoAndViewpoints()

bb2_operator_mlp_render()

• requires the surface, (or surfaces if multiple) to be selected..
• it finds the selected PDB ids and gives them an ID, tID (the ID is used for internal reference, it doesn't reflect the pdb file name..)
• for every found tID,
  • call mlpRender(tID)
  • todoAndViewpoints()

mlp()

..todo

mlpRender()

..todo

[zeffii]

While reading the operator code, I'm wondering if it wouldn't be tidier if there was a UI list, which shows the currently imported models, and lets the user select them from there, rather than the atoms. I'll read up on how to do a UI list, it's something i've wanted to know for a while anyway.

[MonZop]

Several issues here:

• A list of objects (elements from one or more pdb files) would be great. I thought that this could be done (more) easily in the Outliner, so that every object would be associated with its atoms, its surface, its UV map (from MPL calculation) and its set of lines (from EP).
• BB2_operator_mlp() is supposed to:
  • get the molecular surface
  • run pyMLP.py using data from the pdb file, and fi values stored
  • use the .dx grid generated by pyMLP.py to attribute the value to every vertex of the surface
  • convert those values to grey scale
  • build a .obj file that lists the vertices as x,y,z and mlp
• BB2_operator_mlp_render(), will use the surface, that now has a UV map, and apply the render procedure that makes it rough/dark/dull or smooth/white/shiny according to the mlp value.

Hope this hleps!

[zeffii]

on reflection yes, using the outliner alone is fine for atom selection

i'm not sure about this question:

I also have a question: is this MLP the value calculated by MLP.py, or is it the color of pyMOL, simply transformed to grey scale, and rendered?

My code changes shouldn't have changed any operational features, are you asking me about the internals or has something changed due to my edits?

[zeffii]

mlp()

Quoted text by monzop as response to my (zeffii) code observations.

1. first determines the force formula (dubost,...buckingham)

   Yes, default formula is Testa

2. if posix based (linux, osx), then chmods pyMLP.py (gives it permission to be run)

3. then launches this nightmare of a string concatenation (this is one line) . Which I think means "get from tmp.pdb a tmp.dx"

   command = "%s %s -i %s -m %s -s %f -o %s -v" % (quotedPath(pyPath), quotedPath(homePath + "bin" + os.sep + "pyMLP-1.0" + os.sep + "pyMLP.py"), quotedPath(homePath + "tmp" + os.sep + "tmp.pdb"), method, spacing, quotedPath(homePath + "tmp" + os.sep + "tmp.dx"))
   

   I cannot read (language unknown to me), but the tmp.dx is a file which is generated by pyMLP. One pssiblity is that pyMLP uses a file already prepared, and fills in witht he values. Pls. note that pyMLP needs all the fi values, which are stored as values_fi, by aminoacid and by atom, in the very big BioBlender2.0.py code,(Lines 186 to 224).

4. then it calls surface()

   1. opens tmp.pdb to correct 1+ 1- errors (whatever those are)
   2. generates tmpPath0 a surface.pml which seems to be a script that pyMOL can read
   3. launched pyMOL with python command = "%s -c -u %s" % (quotedPath(pyMolPath), quotedPath(homePath + "tmp" + os.sep + "surface.pml"))
   4. imports a tmp.wrl file as an object and names it "SURFACE"
   5. removes doubles, shades smooth, translates and rotates the surface to match the rotation of the Empty.

   if the wrl file also contains color infomration, this should be removed at this step if the user already generated a surface, it is not necessary to repeat the process here.

5. waits for the asynchronous process (3) to complete

6. then the .dx is opened and read and updates a number of variables dxData, dimension, origin, delta

7. if there was dxData , vertex_paint_mode is entered and again it removes doubles and shades smooth (this might be redundant)

   indeed, it seems redundant. it also seems odd that the removal of doubles and smoothing should be associated with vertex_paint_mode

8. then it reads the vertex colour map (which was generated during the wrl import (4. IV), for each vertex of each polygon in the color map it performs a modification to the vertex colours as imported.

   tmp = (color_map.data[i].color[0] + color_map.data[i].color[1] + color_map.data[i].color[2]) / 3
   color_map.data[i].color = [tmp, tmp, tmp]
   

   definitely forced greyscale

   • this is where we say (in italian) that the donkey falls. In english: error revealed. In step 4.IV all colors must be removed. The new colors are attributed to the verteces by trilinear interpolation of the values in the .dx file, to the corresponding vertex. this step never worked after BB0.6 because (the coder said) vertex identities were not accessible. *

9. then does a vertex_color_smooth on the whole map

10. sets viewport to TEXTURED

11. lastly it hides the atoms in the scene

These last steps do work. I am almost sure that if you do remove the colors from the sutface generated by pyMOL, the part of running pyMLP and attributing colors to verteces has never worked. The way it used to work in early versions was by calling a script called OBJcreator, which was capable of putting together the info from the surface (.wrl) and the MLP (.dx) and created a new file with colors (.obj)

[zeffii]

mlpRender()

todo

[zeffii]

@MonZop i've formatted that last post so it is more clear to me (and readers) who is commenting what.

[zeffii]

OK, i'll check out OBJCreator script

[zeffii]

the .dx file is a massive list of 3 Floats per line (I doubt these are colours?) the .wtl is imported using a import_x3d provided by blender, it contains verts+faces+colours per vertex

In the end where do you want the colour to come from?

[zeffii]

wait... the dx file is a grid ? a representation of values in a 3d vector space? So is the idea to make the polygons on the 'SURFACE' object mesh to the nearest value in the dx file?

[zeffii]

like this?

[zeffii]

object 1 class gridpositions counts 43 40 34
origin -2.213350e+01 -1.778850e+01 -1.658250e+01
delta 1.000000e+00 0.000000e+00 0.000000e+00
delta 0.000000e+00 1.000000e+00 0.000000e+00
delta 0.000000e+00 0.000000e+00 1.000000e+00
object 2 class gridconnections counts 43 40 34
object 3 class array type double rank 0 items 58480 data follows

suddenly makes more sense: 43 * 40 * 34 == 58480

[MonZop]

Great job! Sprry I was offline for some days, or I might have explained better before.

The value to attribute to the vertex in the original plan was obtained by trilinear interpolation. The option of usign the value of the nearest grid point is good, especially if it makes the calculation much faster, but only when the grid spacing is fine (up to 2 A). For coarser grids it is much better to interpolate, in order to have a smooth painting of MLP on the surface.

[zeffii]

That image of the grid was only to communicate the concept, that code is not written yet.

OK. if I can match up the vector space of the dx file with the 3d coordinates of the triangulated surface mesh, mapping to closest or (some function of the nearest n grid points and falloff gradient) is possible using a KDTree.