Only update posts that are marked as modified

[dirkhh]

Github tells us which files were modified. If we find that information in the webhook callback, let's not blindly update all posts but instead only update those posts that we were told were modified.

This adds a new function to the payload class to get the actual head commit included in the payload.

json_decode(json_encode(), true) is used to convert the object to an array - I'm not sure if there is a different way that you would like to use for this (I'm not really a PHP developer - but this works).

Signed-off-by: Dirk Hohndel [email protected]

[mAAdhaTTah]

I haven't forgotten about this. Will take a look this weekend. Do you have this code in place and running on your site? Wondering how it works in the real world.

[dirkhh]

This is NOT ready to pull. I run this (actually, a marginally newer version) on my server. It works perfectly and is extremely fast if you only ever merge or push ONE commit at a time. But it actually gets things wrong if you send more than one commit. I have played with a fix for that but that needs more testing (and crashes in some circumstances with my current code). But I'm convinced that this is the right thing to do and that we have all the pieces to know exactly which files changed and to not waste time on the ones that don't. I should have an update in a day or three.

[mAAdhaTTah]

Sounds good. I await your updates.

[dirkhh]

I think this is worth actually taking a look at. Grab the commits array (you already had an access for this), loop over all of the commits in there, convert them to arrays and grab the modified files from them. I did a medium amount of testing and this seems to work. Unsure if there are situations with adding or deleting files where this would get it wrong.

[litefeel]

My english is so pool.

how to get commits commits is not complate in webhook, it's just only 20 https://developer.github.com/v3/activity/events/types/#pushevent

if commits.length >= 20 then
     // https://developer.github.com/v3/repos/commits/#compare-two-commits
     // get commits with compare commits api, this api can list max is 250, i think is enough.
     GET /repos/:owner/:repo/compare/:commit...master
end

merge commit to find file of update and new.

cache something

Got a packet bigger than 'max_allowed_packet'

max_allowed_packet is 1M by default. so dont overtake it.

Make the cache smaller. Commit only cache github response, no other. Tree only cache sha and path, dont cache commit in tree.

// in tree
function getPostBySha($sha) {
    return $cache->getSha($sha); 
}

function getPostByPath($path) {
    $sha = $this->getSha($path);
    $return $this->getPostBySha($sha);
}

[dirkhh]

Based on @lite3 's comments I am in way over my head. I think I might be able to implement the changes that @mAAdhaTTah requested (actually, sure, I could have done that), but I definitely don't feel comfortable adding calls to a different API. Let's just forget about this - I have it in my version and it does what I need - no one's going to submit more than 20 commits as a pull request to my website.

[mAAdhaTTah]

@dirkhh If you're willing to make the changes suggested, we could resolve the "20 commits issue" by just falling back to the old behavior (get the entire tree), rather than spiking the work you've done already.

[dirkhh]

On Tue, Mar 07, 2017 at 04:47:57AM -0800, James DiGioia wrote:

@dirkhh If you're willing to make the changes suggested, we could resolve the "20 commits issue" by just falling back to the old behavior (get the entire tree), rather than spiking the work you've done already.

Sure, I can do that. It will take me a couple of days (work has me crazy busy this week).

[litefeel]

Now I am doing a new version, do not cache, do not tree. do not too many commits.

[mAAdhaTTah]

@dirkhh @litefeel are either of you guys working on this?

[dirkhh]

I have been completely side tracked. I have a hacked version that does what I need, but I'd need to start from scratch to do this right. So if anyone else wants to look, please don't wait for me.

[litefeel]

@mAAdhaTTah I have been complete it. https://github.com/litefeel/writing-on-github/ don't convert markdown to html.

[litefeel]

@mAAdhaTTah Replace the tree api with the content api

[IeuanWalker]

@dirkhh @mAAdhaTTah I want to implement a similar thing in C#. And I can't really read PHP that well. Could you list the Github API that are in use, and how they are being used

I'm only interested in importing post from Github (not exporting)

[mAAdhaTTah]

@IeuanWalker We used the Git Data API and build up the new tree based on the current tree, then add a new commit pointing to the new tree. This helps us avoid adding multiple commits for things like the initial site export and batch deletion (which is why we don't use the Content API @litefeel).

[IeuanWalker]

@mAAdhaTTah Thanks but that's mostly for exporting isn't it?, do you something different to import the post from GitHub?

All I want to be able to do is add/ update a new post to a repository (not from the web application) and for the web application to pick up the change and add/ update the database. Repository won't need to be updated from the web application.

[mAAdhaTTah]

No, we use it to read the tree as well. You can also use the Content API, which we used initially.

[IeuanWalker]

@mAAdhaTTah Oh ok. So are you using the push webhook to know when something has been added then the git data api to update/ add the new posts?

[mAAdhaTTah]

Yeah, basically.

[IeuanWalker]

@mAAdhaTTah cool makes sense thanks :)