Instacurate Add autorefresh functionality

trafficstars

After data has been loaded, we keep checking in the background if new links are available and display them in a similar way that Twitter says "x new tweets" in its timeline.

The process would be:

Once a request has been executed
Check every x seconds if there are new links available
If so, display a message to the user: "n new links" at the top of the list and "(n)" in the browser tab
When user clicks the message, new teasers are added

Jan 13 '13 13:01 davidbauer

Hi David. There are 2 general ways I know of this can be implemented. The first one, which I have personally implemented on reading.ly, is detailed below. It is not as robust as the more sophisticated approach but it is easy to conceptualize and implement. The second one is using something called Comet. I personally haven't implemented this because at the time when I needed this (for Readingly) it was way over my head. But if you go on stack overflow and just google around, there are many articles and sites about it (http://en.wikipedia.org/wiki/Comet_(programming))

So back to my approach:

in your javascript, create a time function (setInterval()) which runs every X seconds. The more often this runs, the more taxing it is on the browser and your server, so I would be careful about doing something like "every second"
in the time function, create a call to a script on the server (in jQuery this is a simple $.get or $.post method...if you don't use jQuery, it more clunky but doable - basically you need to do an AJAX call)
your script on the server then does the same stuff your server does for the site (i'm assuming you use the twitter API to retrieve the feed which you then parse and organize into a neat HTML) with the exception being that you want to only run this since tweet ID XYZ - you should probably capture this value somewhere in the original HTML, and then read the value when you run the AJAX call; this script should return 3 values: (1) the count of new items, (2) the contents of new items, (3) the most recent tweet ID checked
the AJAX function in the browser receives the callback value and the callback function then processes it:
you alter the title of the page (again using jQuery) to reflect the new item count
you display a notification on the top of the page (with a count) which when clicked --
displays the contents of the new items

Now, I know this is a little dense and I may not be explaining it perfectly but let me know if this makes sense to you conceptually and if you're having trouble with any steps.

Sep 26 '13 20:09 akristofcak

Ok, so this is what I've got so far:

After fetching the links for the first time, I do this to check for new content every minute: setInterval(autorefresh(searchApiMaxId), 60000);
I have an autorefresh function which triggers the link fetching again, but starting from the id of the last tweet I've already fetched.

Since the setInterval function is called inside the fetching function, it will be called again when autorefresh is called after 60 seconds. Is that the point? Or should I rather use .delay instead?

Sep 26 '13 21:09 davidbauer

You could set up the fetchingFunction() to entail the auto-loop mechanism but that's an unnecessary hack; setInterval just does this for you.

It could be as simple as

setInterval(function(){fetchingFunction()},60000);

setInterval will then run fetchingFunction() every 60 seconds after the page loads. Note how you have to wrap the function name in the function(){} expression. I wish I understood why setInterval requires this, but it does.

In your fetchingFunction() you should read the mostRecentTweetID from an existing element in your HTML (I would save this as an attribute in an element somewhere when you're generating the first page), pass that ID onto the server in the AJAX request and then update the mostRecentTweetID based on the data fetched; when setInterval triggers the fetchingFunction() again in 60 seconds, it will read the updated mostRecentTweetID and pass that onto the server in the AJAX request... etc etc etc.

Sep 27 '13 12:09 akristofcak

Thanks, will try that. Looks like I need to pass the tweetId as a parameter to the fetchingFunction, which I currently don't (since it's always starting from the first), so that I can trigger a different behaviour for the autorefresh (display "n new links" etc).

Sep 27 '13 14:09 davidbauer

I've started working on the autorefresh functionality, see commit above. Could anyone of you have a look at it and tell me if that's any good so far? One of the problems I've already noticed is that the setInterval function doesn't know tweetId, which it needs as a parameter to pass along to getLinks()

Sep 29 '13 14:09 davidbauer

I would have fetchingFunction look up tweetID from a place where you "stored" it when you generated the initial load of the page. For example:

var tweetID = $('#wrapper').attr('tweetID');

Here I am assuming that in your div with id wrapper you have an attribute called tweetID where you stored the value of the most recent tweetID.

Then, when you fetch the most recent data, and figure out the most recent tweetID, you need to reset this value to the new tweetID:

$('#wrapper').attr('tweetID',newtweetID);

When fetchingFunction runs next time, it will read this updated value.

Sep 30 '13 10:09 akristofcak

I have tweetId stored it in the script, so I actually don't need to look it up. Just – like always – a scope issue with my functions. @dergraf, @backflip to the rescue!

Sep 30 '13 15:09 davidbauer

tweetId is declared inside process_data() (using var). Instead, you could declare it at the same time as since_id (https://github.com/davidbauer/Instacurate/blob/master/script.js#L225).

Sep 30 '13 18:09 backflip

@backflip Thanks, added var tweetId;. I don't need to set the variable to a value up there, or do I? And do I need to pass it to process_data() or return it from it? This stuff still gives me headaches ;-)

Oct 01 '13 06:10 davidbauer

You don't have to declare a value, by default it is undefined. process_data() has access to this variable, so you don't need to pass it as a parameter.

Oct 01 '13 11:10 backflip

Thanks, @backflip. So I'll proceed with the implementation from here. The alert I've put in as a placeholder seems to work correctly.

Oct 01 '13 15:10 davidbauer

I think I've now implemented everything I need for autorefresh to work as described. However – as always ;-) – it won't work. @backflip, do you have time to have a quick look at what's wrong?

Oct 02 '13 13:10 davidbauer

I guess generateEmbed is not called anymore: https://github.com/davidbauer/Instacurate/blob/master/script.js#L370

EDIT: Nope, not the issue.

Oct 04 '13 05:10 backflip

Hm, it currently works for me. I get a 500 on http://tlinkstimeline.appspot.com/statuses/home_timeline.json every now and then, but apart from that, it works fine. Can you elaborate on "it won't work" a bit? :)

Oct 04 '13 05:10 backflip

All the interactions in the frontend seem to work, but the data processing doesn't seem to be correct.

Once the first autorefresh runs, it shows that a number of new links is found, but this isn't the actual number of new links in the timeline since the first load. Also, if I click on the notification, no new teasers are loaded.

I suspect that the reloading of teasers when scrolling down the page interferes with the autorefresh. I've also noticed that reloaded via scrolling doesn't work once autorefresh has run.

Do you see a different behaviour, @backflip?

Oct 04 '13 12:10 davidbauer

I just cloned the recent version of Instacurate. I’ll look into this. Shouldn’t be that hard to get auto-refresh working.

Dec 18 '13 08:12 oliverwehn

thanks @oliverwehn, I had this on my todos as well, but don't have much time to contribute right now. let me know if you need some help.

Dec 18 '13 13:12 dergraf

I just cloned the recent version of Instacurate. I’ll look into this. Shouldn’t be that hard to get auto-refresh working.

Looking forward to that X-mas present ;-)

Dec 18 '13 13:12 davidbauer

Hmm, I get error 500 responses from the appspot url on almost every request. Any idea why? It’s hard to work on auto-refreshing without having anything to process. ;)

Dec 19 '13 21:12 oliverwehn

@dergraf knows best about this part of Instacurate...

Dec 19 '13 21:12 davidbauer

I had this as well. It is caused by an TwitterAPI request error that we don't handle correctly. So far I couldn't figure out why it isn' working in some cases. Am 19.12.2013 22:50 schrieb "David Bauer" [email protected]:

@dergraf https://github.com/dergraf knows best about this part of Instacurate...

— Reply to this email directly or view it on GitHubhttps://github.com/davidbauer/Instacurate/issues/63#issuecomment-30969913 .

Dec 20 '13 07:12 dergraf

@dergraf, is the appspot thing a Python app you wrote? I somehow missed this part. Is there a separate repo for it?

@davidbauer, as the auto-refresh has to add the new links to the top, it would be great to overcome the fixed three column structure of the document. Otherwise we would have to remove all displayed stories and put them back to the columns in updated order. Would be much easier and faster to just let the prepended content push the other stuff down the timeline.

Dec 20 '13 09:12 oliverwehn

Yes, it is a public repo https://github.com/dergraf/gae-twittertimes

2013/12/20 Oliver Wehn [email protected]

@dergraf https://github.com/dergraf, is the appspot thing a Python app you wrote? I somehow missed this part. Is there a separate repo for it?

@davidbauer https://github.com/davidbauer, as the auto-refresh has to add the new links to the top, it would be great to overcome the fixed three column structure of the document. Otherwise we would have to remove all displayed stories and put them back to the columns in updated order. Would be much easier and faster to just let the prepended content push the other stuff down the timeline.

— Reply to this email directly or view it on GitHubhttps://github.com/davidbauer/Instacurate/issues/63#issuecomment-30998124 .

Dec 20 '13 09:12 dergraf

@oliverwehn Would that have any design implications or are you merely talking about the structure of the html?

Dec 20 '13 11:12 davidbauer

Hmm, I guess there wouldn’t be a way around design adaptations und js-layouting to achieve a similar look. I’ll try to work something out.

Dec 20 '13 11:12 oliverwehn

@oliverwehn, you're the designer, I'm very open to your suggestions.

Dec 20 '13 11:12 davidbauer

Any news here, @oliverwehn?

Jan 06 '14 15:01 davidbauer

Still working on it. Got some issues dealing with the automatic updates. And the Twitter API error from the Python script often keeps me from testing. Stuff I have to solve before adding jquery.isotope. I’ll push the current state to my github repo tonight (if I don’t forget).

Jan 06 '14 15:01 oliverwehn

Ok, it has been hard with all the Python thing errors, but I pushed a first rough version with auto-refresh. The usability has to be optimized by just showing a notice to tell of new tweets with a link to actually prepend them to the current timeline. Haven’t tested it in detail, yet. The HTML structure of index.html has been changed but I haven’t been able to change the styles and to implement Isotope. So it still looks pretty shitty. Nevertheless auto-refreshing is on its way.

Have a look here: https://github.com/oliverwehn/Twitter-Times/tree/dev-autorefresh

Jan 06 '14 23:01 oliverwehn

@dergraf can you have a look at the python errors? i'm totally lost there.

Jan 07 '14 11:01 davidbauer

Instacurate Instacurate copied to clipboard

Add autorefresh functionality

Instacurate
Instacurate copied to clipboard