Page timing out

[stresslimit]

Would be great if you made the page process differently, by ajax or whatever, because I am getting page timeouts on large files and have to split manually into smaller chunks. Otherwise, it's a great, simple, useful plugin.

[sorich87]

I was thinking about doing it with Ajax and jQuery UI Progressbar as the plugin Regenerate Thumbnails does, but I don't have enough time for that now unfortunately.

A pull request would be welcome!

[stresslimit]

Funny, the regenerate thumbnails was exactly what I was thinking of as a model! I also do not have massive amount of time either, but I will put it on my long-term list and see who gets to it first :)

[phy25]

So we are uploading a file. The file size should be limited. Maybe the situation will get better if set-time-limit() (doc) is used.

Here is an example of the uploading page of phpMyAdmin: We can refer to it :).

[stresslimit]

It's a bit of a specific problem, the csv file is only 623K but it includes around 11,000 users; it's the time for WordPress to check if exists and create new user for each one. Not sure if it's a good idea to just set max_execution_time or set_time_limit() to really high in this case, it would be much better to re-architect to post through ajax and ping back the browser periodically with progress reports.

[phy25]

Firstly upload the file and add it to the queue on the server. Then there are two choice:

1. Request the server with ajax to handle the records. Every request handle ten records or more.
2. Make the server work background (by ignore_user_abort() ). Request the server with ajax to get the progress.

[sorich87]

I think solution 2 will not work for some (most?) shared hosts.

[strangerstudios]

Hi there. I like your import plugin and have been using it along with an addon for Paid Memberships Pro to do imports of WP users/members.

I have a client who is funding me to make a custom importer and I'm doing that on top of your plugin. Part of the work will be to update the importer to be able to accept larger files and break them up into background jobs. I can do a pull request with my updates when I'm done.

Let me know if you'd like to chat about it a bit to make sure the work I do aligns with things you've been thinking. Thanks!

[sorich87]

Thanks for reaching out @strangerstudios. Just one question: how are you planning to handle background jobs?

[strangerstudios]

Here's the rough idea:

1. Upload full CSV to server.
2. Split file into chunks of X rows (x is a setting on the upload page)
3. Use AJAX to run imports on chunked up files one at a time showing progress.

[sorich87]

:+1:

[strangerstudios]

FYI a first stab at the AJAX stuff here: https://github.com/strangerstudios/import-users-from-csv

I would very much like some feedback. Basically, it adds an option to "Use AJAX to process the import gradually over time." When that is done, it stores the CSV in the uploads folder and redirects to a page that processes 20 lines at a time over AJAX.

I won't do a pull request yet. I need to:

1. Change it to keep track of positions in the CSVs in an option in the DB (more reliable than transients)
2. Make it better at cleaning up old files.
3. Update the status page to show better stats on progress.
4. Better way to handle/show errors that come up.

And maybe:

1. Way to set how many rows to execute at once (hard coded to 20 for now I think)
2. Add a way to manage the uploaded files. (to resume or delete files)