Limit number of elements | Pagination Support

[sirinath]

Can an option to be added to how many elements that can be added when using:

• beforebegin
• afterbegin
• beforeend
• afterend

with the option to prepend, append and ignore newer times. When prepend and append the oldest item will be removed when the limit is hit.

Also add the ability to:

• specify an ID range or list of item to display based on a state or variable - if returned IDs are outside the list they will not be displayed
• request multiple times - range or the list of items can be specified as variables
• update specific items at particular indices - IDs can be returned in the response header. IDs not in already present can be ignored, prepended or appended.
• insert specific items at particular indices - IDs of adjacent items can be returned in the response header. If items are not adjacent. new items will not be inserted to prevent double insertion.
• delete specific items at particular indices - IDs can be returned in the response header

[1cg]

  <div hx-swap="beforeend limit:10 trim:end" ...>...</div>

[sirinath]

We need to be able to specify what end new items are added or which end items are trimmed. No need to specify both as one implies the other.

[1cg]

Good point:

  <div hx-swap="beforeend limit:10" ...>...</div>

[sirinath]

We need:

• <div hx-swap="beforeend limit:10 trim:end" ...>...</div> - add at the front and trim at the end or
• <div hx-swap="beforeend limit:10 add:end" ...>...</div> - add at the end and trip at the front or
• <div hx-swap="beforeend limit:10 add:start trim:end" ...>...</div> - explicitly add at the front and trim at the end or
• <div hx-swap="beforeend limit:10 add:end trim:end" ...>...</div> - invalid unless something like trim:end:all is not used

Maybe you can also specify how many items to remove like trim:end:5 or trim:end:all or trim:end:hidden (remove items not displayed in the viewport at the end)

[sirinath]

All this might be a overkill so just <div hx-swap="beforeend limit:10 trim:end" ...>...</div> and <div hx-swap="beforeend limit:10 trim:start" ...>...</div> maybe just what is needed. It is implied we are adding new items at the other end and when it fills up we remove the oldest item.

[benpate]

Hey @sirinath -- from this discussion, it looks like you're still just discussing this feature, and that it didn't get built, yet. Is that right? Let me know if you're still waiting on it and I might take a crack at writing a PR.

[sirinath]

I am waiting for these features with:

• pagination
• infinite scrolling support.

Where you can have something like <div hx-get="/example" hx-swap="afterend">Get Some HTML & Append It</div> and also <div hx-get="/example/{num}" hx-swap="afterend">Get Some HTML & Append It</div>. num can be used for pagination and infinite scrolling which would be the item number retrived.

[benpate]

Hi @sirinath,

I've spent a little time on this, and have made some progress. But I'm not sure that this is complete. First, check out my PR #169, and tell me what you think. It should support the following:

<div hx-get="/myResource?page=1" hx-swap="afterend limit:10" hx-trigger="revealed"></div>
<div hx-get="/myResource?page=2" hx-swap="afterend limit:10" hx-trigger="revealed"</div>

In my tests, I was able to scroll down a large list of items, with new items being dynamically added to the bottom, and old items being removed from the top automatically. It's pretty cool 👍

Page limits are interesting. I think we should be able to accomplish this using only a limit:10 tag. The rules could look like this:

if swap==beforebegin then trim would remove the parent.lastChild element. if swap==afterbegin then trim would remove the lastChild element. if swap==beforeend then trim would remove the firstChild element. if swap== afterend then trim would remove the parent.firstChild element.

Next Steps

While this works really nicely when scrolling DOWN, it doesn't let me scroll back UP again. Since older nodes have already been removed, there's no easy way to get them back onto the page.

I'm thinking through some solutions now, and am open to your suggestions. One possibility would be for a node to be able to tell if it's already the parent.firstChild and then take some extra actions in that case. This might even be a job for Hyperscript. I'm just not sure yet.

[benpate]

A Side Note about Updating Nodes

I re-read parts of your earlier note, and I think you're mixing in a lot of things, here. It is super-cool to be able to update nodes in real-time from the server. However, I think that is a job for hx-sse or hx-swap-oob.

I've made a lot of progress on SSEs recently that I hope can make it into the core library soon. With this, you could subscribe to multiple SSE streams, or to a single stream that's separated into individual events. Each record in your infinite scroll could listen to an event name that corresponds to its nodeId. Then, the server could push updates to each node as they happen.

If I'm missing something, please let me know. I'm just trying to be clear on the specific requirements of hx-swap:limit without mixing in features from other parts of the library.

[sirinath]

You can have something like hx-swap="afterend limit:10 indexed". If indexed is used one passes the element ID of the requesting element if present and element index of the required element. The server can send the appropriate item. Also, I don't think revealed is a good trigger as we want to prefetch items before they are revealed. This means we might need hx-swap="afterend limit:10 indexed prefetch:3", where prefetch is a parameter send with the request if present with element ID and index. The server is responsible to send the appropriate response, so no need to track elements in htmx side, other than the index range which is displayed.

[benpate]

Your "indexed" idea is a good step in the right direction. After thinking about it overnight, I think we can give the server even more control over page structures, and we can stick closer to HATEOAS principles. Here's what I propose:

• Add a new swap type hx-swap="infinite" that tells HTMX to manage pages for us automatically.
• Add two new attributes: hx-prev and hx-next that the server includes in page fragments to link to additional pages.
• When a child becomes visible, HTMX loads either the previous or the next record, until the parent node is "full." This means we'll always have an "extra" page preloaded and sitting outside the visible area, waiting to scroll into view. When it does, the next page in the chain is loaded.
• Optionally, you could use hx-swap="infinite limit:10" to fix the maximum number of elements.

Example Code

HTML running on the client would look like this.  We define a new "swap" type
with new rules for tracking child nodes.  On load, this node will load the first page
into the firstChild position.

<div hx-get="/my-url/page=1" hx-swap="infinite">

    <div hx-next="/my-url/page=2">
        <h2>This is page 1.</h2>
        When I load, HTMX will see that it has space for more items, so
        it will use the URL in my "hx-next" tag to load the next page.
    </div>

    <div hx-prev="/my-url/page=1" hx-next="/my-url/page=3">
        <h2>This is page 2</h2>
        I have links to the pages above and below me.  When I become visible
        (either by being loaded onto the page, or scrolling into view)  HTMX uses
        my "hx-prev" and "hx-next" tags to load additional pages as needed
    </div>

    <div hx-prev="/my-url/page=2">
        <h3>This is page 3</h3>
        In this example, I'm the last page to be loaded, because I don't have a
        "hx-next" attribute.  However, this list could go on forever, or event "loop back"
        around to page=1, depending on how the server responses are crafted.
    </div>
</div>

Other Styles of Pagination

The more I experiment with this, the more I really like it, because would be very easy to implement on both the client and server sides. It's also not hardcoded to any single style of pagination. Here's an example of how easy it is to implement indexed pagination, which is a much more efficient way of breaking up pages.

<div hx-get="/page?timeGreaterThan=>0" hx-swap="infinite">
    <div hx-next="/page?timeGreaterThan={{largest timestamp-in-this-page}}" hx-prev="/page?timeLessThan={{smallest-timestamp-in-this-page}}">
        List of records goes here.  Server can identify the lowest timestamp and the highest timestamp, then
        put those parameters in the containing DIV to query the surrounding pages.
    </div>
</div>

Infinite loops, Wierd Stuff

You can even do some pretty crazy things with it. Check out the example below, that maintains a sliding window of 10 DOM nodes inside of virtual, infinite loop, using letter-based pages

<div hx-get="/page/A" hx-swap="infinite limit:10">
    <div hx-prev="/page/Z" hx-next="/page/B">Records beginning with "A"</div>
    <div hx-prev="/page/A" hx-next="/page/C">...</div>
    <div hx-prev="/page/B" hx-next="/page/D">...</div>
    <div hx-prev="/page/C" hx-next="/page/E">...</div>
    <div hx-prev="/page/D" hx-next="/page/F">...</div>
    <div hx-prev="/page/E" hx-next="/page/G">...</div>
    <div hx-prev="/page/F" hx-next="/page/H">...</div>
    <div hx-prev="/page/G" hx-next="/page/I">...</div>
    <div hx-prev="/page/H" hx-next="/page/J">...</div>
    <div hx-prev="/page/I" hx-next="/page/K">...</div>
    <div hx-prev="/page/J" hx-next="/page/L">...</div>
    <div hx-prev="/page/K" hx-next="/page/M">...</div>
    <div hx-prev="/page/L" hx-next="/page/N">...</div>
    <div hx-prev="/page/M" hx-next="/page/O">...</div>
    <div hx-prev="/page/N" hx-next="/page/P">...</div>
    <div hx-prev="/page/O" hx-next="/page/Q">...</div>
    <div hx-prev="/page/P" hx-next="/page/R">...</div>
    <div hx-prev="/page/Q" hx-next="/page/S">...</div>
    <div hx-prev="/page/R" hx-next="/page/T">...</div>
    <div hx-prev="/page/S" hx-next="/page/U">...</div>
    <div hx-prev="/page/T" hx-next="/page/V">...</div>
    <div hx-prev="/page/U" hx-next="/page/W">...</div>
    <div hx-prev="/page/V" hx-next="/page/X">...</div>
    <div hx-prev="/page/W" hx-next="/page/Y">...</div>
    <div hx-prev="/page/X" hx-next="/page/Z">...</div>
    <div hx-prev="/page/Y" hx-next="/page/A">...</div>
</div>

[sirinath]

I think we are making it too complicated. We need something simple like:

<div hx-get="/example" hx-swap="afterend limit:10 prefetch:3" hx-include="[name='pageID'], [name='startID'], [name='endID'], [name='prev'], [name='next']" hx-trigger="submit">...</div>...
<input id="pageID" name="pageID" type="hidden" value="5" hx-swap-oob="true" />
<input id="startID" name="startID" type="hidden" value="0" hx-swap-oob="true" />
<input id="endID" name="endID" type="hidden" value="4" hx-swap-oob="true" />
<input id="prev" type="submit" name="prev" value="prev" />
<input id="next" type="submit" name="next" value="next" />

Here 1 page has 10 items but they are also not received at once. 3 off-screen items are fetched within the page.

[clarkevans]

Pagination, especially automatic scrolling... sounds great I've got a product page with 1000's of products, and I'll need to have a way for users to jump though the pages -- ideally a function of screen size with automatic scrolling. On the server side, I'd expect that I'd want to return a single result with N divs at once? Not N requests. This will make it so that my database query can do a range, say returning products from 30-44 (count=15). What I'd want is a starting index and a size. Does this align with any of the proposals here? The trick is, I guess, to know how many divs per page there is?

[benpate]

Hey @sirinath -- It's rare that I'm the one making things too complicated :)

I think I'm following your code. However, I'm not sure how the "prefetch" would actually function in this example. We're telling HTMX to load three records, but how does HTMX actually determine what URLs to call?

The information about next/previous pages is present in your example, but not in an HTMX-readable form. Unless we define a bunch of "magic" IDs (like pageID, startID, endID, prev, and next) then it's only there for humans to read. I think that's the point of my hx-prev and hx-next recommendation. That would move this meta-data up into a place where HTMX can read it, and knows what to do with it.

From there, prefetch is an interesting idea, but may not be necessary. In my current experimental pages, these pages get loaded automatically in sequence, as each new page becomes visible and triggers a new GET request.

Does this make sense to you?

[benpate]

Hi @clarkevans -- Yes. This proposal would align perfectly with what you're saying. I'm just using "records" as the unit of measure, but many cases would measure by "pages" instead. And, by focusing on which child nodes are visible, it automatically adjusts to your screen size, as well.

Your server would simply group a number of records (let's say 50) into a page, and return that as page 1, along with an hx-next that points to page 2. Server responses could look something like the code below:

<!-- Here's an example for "page 3" of results being returned, 50 records at a time -->
<div hx-prev="server?page=1" hx-next="server?page=3">
    <div id="record101">John</div>
    <div id="record102">Sally</div>
    <div id="record103">Jane</div>
    <div id="record104">Jack</div>
    ....
</div>

I'll try to put together a live demo that others can play with because seeing it work in person makes a huge difference. Last night I scrolled through thousands of auto-generated nodes -- automatically added, then removed from the DOM. That sold it for me.

One More Thing Another bonus is that all of the records in the example above can have their own HTMX behaviors. For example, those individual records can update in realtime by subscribing to an event stream with hx-sse and giving each of these records their own unique event name. I know this last bit is off-topic, but it's a cool example of HTMX's flexibility, and the power we get by combining well-designed primitives

[sirinath]

I think I'm following your code. However, I'm not sure how the "prefetch" would actually function in this example. We're telling HTMX to load three records, but how does HTMX actually determine what URLs to call?

Send it to one sever. Submit the values and the server will determine from the submitted values. Current page number and retrieved number of page items are stored in hidden inputs.

The information about next/previous pages is present in your example, but not in an HTMX-readable form. Unless we define a bunch of "magic" IDs (like pageID, startID, endID, prev, and next) then it's only there for humans to read. I think that's the point of my hx-prev and hx-next recommendation. That would move this meta-data up into a place where HTMX can read it, and knows what to do with it.

This is the ID attribute. But hx-include uses name above.

From there, prefetch is an interesting idea, but may not be necessary. In my current experimental pages, these pages get loaded automatically in sequence, as each new page becomes visible and triggers a new GET request.

The whole page may not be visible at once. So get the number of visible elements + prefetch number of elements for the page. Say a page is 100 items at a time across multiple pages we don't need to get all 100 in the page at once also. As you scroll, get the new items.

[benpate]

I understand how a set of values could be sent to the server, and how you might use swap-oob to push a new set of values back to the client. But I don’t see how your example solves the issue of knowing which values to use when you’re scrolling “up” vs. when you’re scrolling “down”. That’s the fundamental problem, wherever you put the data.

To handle scrolling in both directions, HTMX must receive (somehow) a separate URL (or dataset) for the unloaded pages ABOVE the current window, and the ones BELOW. This means two arguments (somewhere) that can explicitly tell HTMX which is which. I think that putting these two arguments in each returned page is the cleanest way to show what’s going on.

Does this make sense?

[sirinath]

You just need to know if the first, previous, next, last or page n button was clicked. Page navigation are manual clicks and prefetching is automatic as you scroll. Prefetching is used for infinite scrolling.

[sirinath]

Also you can have:

<input id="prev" type="submit" name="prev" value="4" hx-swap-oob="true" />
<input id="next" type="submit" name="next" value="6" hx-swap-oob="true" />

where 4 and 6 are page numbers. hx-swap-oob="true" can be skipped in all elements if they are not in other parts of the page.

[benpate]

This is what I proposed yesterday (https://github.com/bigskysoftware/htmx/issues/130#issuecomment-677694208) with the exception of listening to “magic” IDs of “prev” and “next” instead of attributes labeled specifically for this job “hx-prev” and “hx-next”

Right now, I have a bunch of PRs waiting for approval already, and I’m concerned about getting too far ahead of the master branch. I think this needs to wait a couple of weeks for @1cg to weigh in on the architecture that he wants for HTMX, before we go any further.

Once we do have direction, I think it would take about a day to make the infinite scrolling feature work. I would absolutely love to see it show up on v0.9.

[clarkevans]

I think I grok the suggestion by @benpate, I think hx-next and hx-prev are relatively straight-forward. I'm assuming then HTMX would implement scrolling logic to make it happen; do you picture the scrolling to be smooth, or more like a tabbed layout container? What about options to move forward/back a page with a button (rather than pgup/pgdn and scrollwheel)? How would this approach handle responsive screens, where 2-3 pages might fit on a tablet but only one page would fit on a mobile phone? What's in/out of scope for the feature?

[benpate]

Howdy @clarkevans :) You're right on track, with one small exception.

Listening to Scroll Events, not Triggering Them

The way I'm proposing infinite scrolling, HTMX would handle all of the logic to react to user scroll events; we would not be making the page scroll. When the user scrolls to the end of the list of pages, HTMX would know how to load a new page into the DOM. This would be similar to the hx-trigger="revealed" logic in the library today.

Screen Sizes

This would also be able to accommodate different screen sizes because HTMX could load as many pages as needed to fill the visible space (then remove them when they scroll out of view).

Navigational Buttons

This specific PR is working on infinite scrolling. If you want to navigate through pages with additional buttons on the page, then HTMX already handles this. Just include those buttons your server response and give them hx-get and hx-target attributes to load the prev/next page where you need it to go.

Do these answers make sense?

[benpate]

FWIW, my demo site finally works. Check out http://sseplaceholder.openfollow.org as an example of the SSE and Pagination experiments I'm doing. When I get close to a real solution for infinite scrolling, I'll try to put it there, too.

[benpate]

One more thought on this -- RFC5998 details using Link: headers to specify relationships between documents (or possibly fragments, in our case). This is separate from the <link> tags that I've been used to seeing. Relation types prev, and next seem like they fit perfectly.

So, instead of reinventing the wheel, it might be interesting to simply implement this standard, or even to scan for tags directly in the fragment. It seems like it would fit well with the behaviors specified in other custom response headers already supported by HTMX.

[thewebartisan7]

@benpate to me sound good your solution to handle paginations, can I see the demo? Seem offline. I hope your PR get merged soon by @1cg

[benpate]

Hey @thewebartisan7 Im sorry, the demo just went offline because the free hosting tier from Microsoft just expired. I let it lapse because I thought nobody was using it anymore.

The demo code was still just a proof of concept, and I wouldn't expect it to be merged in quickly.

I was able to remove items from the top of a list once it got too big, but could not scroll back up without knowing where the deleted nodes had come from.

That was why I proposed something like hx-prev and hx-next. They're an interesting idea, but only that -- an idea that's still not ready for prime time.

[thewebartisan7]

I understand the challenge... I see this behaviour in facebook wall, where posts are loaded on scroll bottom and old disapper on top, and when you scroll top, do the opposite. I don't know htmx, I hear about just yesterday, so I am not sure yet how it works, but I will play with it in next days. All your updated code is on this PR? I can check it locally in case. If you have some more update, please share it.

What about how infinte scroll are doing where URL change when new page are loaded? I think here is done: https://github.com/metafizzy/infinite-scroll/blob/master/js/history.js#L204

But instead of appending new items, just replace whole content, and your reference for load new items is just the URL with page number. I think that we just need to know the page number, not next nor prev. Limit and offset can be calculated from just page number on server side.

Maybe I am missins something as I didn't yet think on this enough.

Because keeping only prev and next is fine when you load a set of articles with next / prev simple pagination. But if you want quickly jump to one page or another, you need also pagination with X number of page, next/prev, first and last. I think that was the points someone here try to archive.

[benpate]

Hi @thewebartisan7 :)

Yes, my experimental code for HTMX is all in the PR, but please know that it's almost certainly outdated by now. You can also check out the original sseplaceholder source code for the server component that is no longer hosted. It has some simple API calls for dynamically generating pages of content so that the demos would work.

Thanks for sharing that example. It's really cool, and it raises one more issue around infinite scrolling that we haven't discussed so far -- updating the browser history to point to the current page. I think this is an awesome feature, and want to explore how we could accomplish something similar inside of HTMX. I think it could be done by including an hx-push response header in each page of returned code, but I have not used this feature enough to know for sure.

One more thing about infinite scrolling -- I think there are several overlapping issues.

The first issue is loading new items into the DOM when you reach the bottom of the current page, like Facebook does. HTMX can already accomplish this using the hx-trigger="revealed" -- there's a working example of this on the HTMX website.

The second issue is (optionally) removing items from the top of the list when they scroll off the top of the page. This is harder, because we need to know how to go "back" from where we are right now. It's nice, for example, for keeping an inline iframe small when you have thousands of items to scroll through. It's nice, but not necessary for infinite scrolling to work. This is the feature I tried to build, that will need more work before it's actually a part of the library. Honestly, we're probably better off leaving this for an update in the far future, once the current toolset had had time to settle.

As for jumping to specific pages, HTMX should also handle that. It would look something like this

<div id="contentsGoHere"></div>

<div>
    Go to Page:
    <button hx-get="/content?page=1" hx-trigger="click" hx-target="contentGoesHere">1</button>
    <button hx-get="/content?page=2" hx-trigger="click" hx-target="contentGoesHere">2</button>
    <button hx-get="/content?page=3" hx-trigger="click" hx-target="contentGoesHere">3</button>
    <button hx-get="/content?page=4" hx-trigger="click" hx-target="contentGoesHere">4</button>
    <button hx-get="/content?page=5" hx-trigger="click" hx-target="contentGoesHere">5</button>
</div>

Is any of this helpful to you? I hope so. Please let me know if you still have questions I can help you answer. :)

[paxperscientiam]

Unless I missed it, one thing that's missing is how to indicate which parameter represents a page number.

I would actually use hx-prev and hx-next to store said parameter. It would look something like this:

<div hx-get="/some-path" hx-prev="pageNumber"></div>

htmx would then create a GET request with the parameter pageNumber equal to its current value less one.

This could then be paired with hx-push-url, or maybe could even have a mode switch to determine how to store page state.

EDIT: To cover the case where one might want to advance multiple pages at once, markup could look like this:

<div hx-get="/some-path" hx-prev="pageNumber, 3"></div>

Where 3 is the number of pages to windback (1 being the default).

[benpate]

Hi @paxperscientiam -

My original thought at the time was that hx-prev and hx-next would be full URLs. That would take all of the guessing out of arguments and URL formats. For example, if I've just called /my-rul?page=2, its response might look something like:

<div hx-prev="/my-url?page=1" hx-next="/my-url?page=3">...</div>

This would work just as well as if I'm using a more "rails-ish" syntax. So, if I've just called /page/2, then my responses might be:

<div hx-prev="/page/1" hx-next="/page/3">...</div>

And so on...

With that said, I don't think this concept went anywhere. It's interesting, but it didn't get enough traction to actually make it into code. You could lobby @1cg to advocate for this feature. I think it would be pretty straightforward to do, if it ever does get "blessed."