Discord-Scraper icon indicating copy to clipboard operation
Discord-Scraper copied to clipboard

Published 20 hours ago • verified publisher icon Dracovian

Output to text files

Open ch40s opened this issue 5 years ago • 10 comments

It would be nice to have the option to store the messages in text files under each server, channel, YYYY-MM-DD directory instead of just the database.

ch40s avatar Jan 02 '20 04:01 ch40s

I did make an attempt at storing the data in a text document and it ended up being a mess to maintain.

I suppose I could make it so that it just writes the raw JSON data to the hard drive since that's what the Discord site returns.

Dracovian avatar Jan 02 '20 09:01 Dracovian

Yes, I think it would be nice to have it as an option in config.json.

ch40s avatar Jan 02 '20 21:01 ch40s

I won't ping you twice on the same issue, just want to make sure you give the experimental branch a test to see if it is the solution you were looking for.

Otherwise feel free to open this issue again (same title is fine) but with the Experimental tag assigned to it so that I can be assured that the issue is for the experimental branch code and not the master branch code.

Dracovian avatar Jan 22 '20 11:01 Dracovian

@Dracovian thanks! I can't assign labels/tabs though, it might be a permissions issue.

ch40s avatar Jan 22 '20 13:01 ch40s

@ch40s I suppose it looks like Github only allows for moderators and owners of repos to set issue labels... I imagine that was a feature by now but apparently not.

Well now I know, and I'm going to continue working on with the experimental script in the meantime.

Dracovian avatar Jan 22 '20 14:01 Dracovian

@Dracovian the script is not creating any files in the same directory while running in loop.

ch40s avatar Jan 22 '20 15:01 ch40s

@ch40s Is there a Text directory along with a Scrapes directory?

Dracovian avatar Jan 22 '20 23:01 Dracovian

@Dracovian no, they are not being created by the script.

ch40s avatar Jan 23 '20 01:01 ch40s

@ch40s I did find that I shouldn't have added the "&has=image&has=videos" to the query because it skips over the text data. I did that to improve the original script's performance before I added the text grabbing feature.

It's difficult to figure a way to separate queries to improve runtime performance without having to send an excess of requests to the server (which the script already does from poor design).

Hopefully these changes should bring about more results from the text scraping function.

Dracovian avatar Jan 23 '20 02:01 Dracovian

@Dracovian the directories and files are now being created but they are empty, they don't include any messages.

ch40s avatar Jan 23 '20 15:01 ch40s