Bug report, from Cam, via email.

[vaughany]

I'm using your plugin for yourls to upload 60k URL's to have them shortened with a specific short URL tag...and it's working GREAT...except...that I have to split up the CSV files into 2000 rows each and then when I upload them, I get a message that says the page is not available...and to try again later...however...when I go back to the admin interface, I can refresh it to show it IS importing and churning through those 2000 links...and once I see it hit the 2000 mark, I can then upload and repeat the process for the next file of 2000 links.

Do you happen to know of a setting or process to allow me to upload larger csv files...something like 20,000 rows at a time instead of 2000? Is it a php memory limit? Or some server setting?

I currently have YOURLS installed on a shared hosting server...but am able to make some server adjustments for this instance.

Thank you so much and thank you for the great work on this plugin for YOURLS.

[vaughany]

The recent update (tagged v0.2) should hopefully improve things. Would love some feedback tough.

[camblair]

So far, this new update is SO much better. I tried a 20k row csv file, but it failed showing several different errors (attached)...so I split it into two 10k row files and it processed each file and successfully inserted in under 3 minutes each. This is a huge time saver for me from before only being to do around 2k rows at once....and even then, it would take around 30 minutes to finish processing those.

I will keep trying things and update more later.

[vaughany]

Just hacked out a basic Bash script to generate a CSV file with as many rows as I like. Currently my local copy of YOURLS is importing about 6,000 rows from the CSV file per minute. I've not seen the errors you've mentioned, but then I've not done as much testing just yet.

Getting some odd feedback from PHP though:

[Sun Jul 26 10:35:03 2020] 127.0.0.1:59340 Accepted
[Sun Jul 26 10:36:59 2020] PHP Fatal error:  Maximum execution time of 30 seconds exceeded in /home/paulvaughan/projects/yourls/includes/vendor/aura/sql/src/Profiler.php on line 91
[Sun Jul 26 10:36:59 2020] 127.0.0.1:59340 [500]: POST /admin/plugins.php?page=vaughany_bias - Maximum execution time of 30 seconds exceeded in /home/paulvaughan/projects/yourls/includes/vendor/aura/sql/src/Profiler.php on line 91
[Sun Jul 26 10:36:59 2020] 127.0.0.1:59340 Closing

So, the import is running for about 2 minutes (according to the timestamps above) but complaining that it's exceed the max execution time of 30 seconds... Unsure what that's about.

---

Additional: I just imported all 20,000 rows from my (first) CSV file in around two and a half minutes (after bumping up PHP's max_execution_time to 5 mins), according to the timestamps in the yourls_url table in the database. This seems like a vast improvement, as long as you have access to your php.ini file.

[vaughany]

Worth noting that I tried to upload a large-ish file containing 100,000 rows, and I got the errors that @camblair mentioned above. To resolve this I had to edit my php.ini file and increase the upload_max_filesize setting from 2M to something larger. There's a sort-of corresponding setting called something like max_post_size which is 8M by default, so I made the other setting the same.

[vaughany]

99,460 records in 13m30s. Unsure why it didn't process the full 100,000 rows, but it didn't throw any errors, aside the web page itself throwing a ERR_RESPONSE_HEADERS_TOO_BIG error when the import completes processing.