ComicStreamer
ComicStreamer copied to clipboard
beville
Compiled Package Downloads link is broken
windows https://dl.dropboxusercontent.com/u/12474226/32/ToiletStreamer-0.9.5.exe
mac https://dl.dropboxusercontent.com/u/12474226/ComicStreamer-0.9.51.dmg
latest for mac at this moment... the windows version is out of date...
You can check back later on: https://github.com/Tristan79/ComicStreamer
Using that windows exe right now. Really, great work. One strange bit of behavior, when I'm in the "comiclist" section of the library, I can't open any of the books in reader. I have to replace "comiclist" with "folders" in the URL. Is this intentional behavior? Search results come up on a "comiclist" page.
Oh, another thing, when I click the server restart button next to the back arrow in the /settings page (but not the other restart button that appears under /server), the text is off center. Also, it doesn't seem to be reloading for me after the restart, I end up having to reload the page myself.
Other than that, though, the interface is much improved from the original, the thumbnails are a nice touch, and the whole thing just seems to be faster. Searching by character no longer crashes my server (though it does take a long time, I suppose that's to be expected). I'll report anything else I find, but I see you said that this version is out of date, so I'm probably reporting issues you already know about
I was still running the old original 0.7, and this new mac build certainly looks different.
i think I spotted a bug. I noticed that a most of my chapters of "Ah! My Goddess" were imported with an empty series, but the series was in the title instead. It looks like this has happened in the case were the metadata only had the series and no title (I don't have the title for most).
Given that these archives all have Comic Rack metadata in them, with the series set correctly, it seems like importing the series as a title when there is no title must be a bug.
Reply to both! I have been tinkering with it for a week...
I got a lot of items with no meta data and after trying to detect it from the file name the serie contains the title and the title is empty, so I swapped them, I added that last night :-) I forgot to check for valid metadata first... :-) Will fix that.
The restart window is broken cuz I remove the code while experimenting :-) You have to manually reload the homepage to continue... the whole gui is a tryout at the moment... I have no idea what I am doing... I change stuff and hope it works (I hacked a close button in the comicreader., yeah!, ...wish there was a page slider), could use some help there...
The restart button mentioned under server is the factory reset/server reset button... :-) The menu orders should be different...
New fun stuff for the mac version:
Try installing (by dragging it to the Application folder) Calibre and enabling ebooks in the comicstreamer settings. It then scans and read meta data from ebooks like epubs or from a calibre library. I know it's a comicreader, but it streams books just as good (especially pdf references guides with a lot of pictures :-)
You can always enable pdf in the mac version, I included the conversion tools in the package... (also just ignore the calibre/mudraw/qpdf paths in the settings, they are not used at the moment, I still have to add that logic)
Enable the cache. It will speed up especially chunky if you pull you're comics from a slower medium and cache it on a fast ssd or internal hdd :-)
If you have a static library in which you only add new files (no remove, or modify, it will crash :-) You can try to install and enable the https://dev.mysql.com/downloads/mysql/ server... and run it's install on the command line: mysql_secure_installation
Run these SQL commands... (tailored to you're setup)
CREATE DATABASE comicstreamer
CREATE USER 'comic' IDENTIFIED BY 'comic';
GRANT ALL ON *.* TO 'comic';
flush privileges;
And enable mysql in the settings...
Searches and overviews will be (well on my pc) nearly instant especially with large libraries. SQLite is just too slow with bigger data sets... Even searching for characters... :-)
If someone want to help fixing the sqlalchemy integrity error I get when removing comics from the database, please :-)
Haven't looked at the comiclist/folder thing... I will upload a version for windows soon and give a shout here, I changed to much the last week, it probably is fixed already...
Also it also reads cbt, cb7 comics and webp, bmp and a lot of other picture formats in comparison to the 0.7 version....
:-)
Very exciting stuff Tristan! Thank you for all this work, I'm installing the mysql community server now.
Having trouble getting that MySQL server configured. I'm a dunce...
I only see the MSI installer in that link you gave, is that what I should be using? It doesn't let me run that through CMD. When I just run the regular GUI installer, I end up with a program called MySQL Workbench, which doesn't seem to be able to create databases, only connect to them...
EDIT, found the .zip file. Found mysql_secure_installation.exe inside the bin folder in the zip file. Running that through CMD asks for a root password, nothing works :-(
You are getting the windows installer! If you are on windows, the windows package is created the day I began tinkering. The new stuff is, as stated, for mac only.
I need a couple of hours of sleep. After that I will venture in the postponed adventure of making a windows 64bit package (on my virtual win10 machine) and 32bit package (without webp :-/ on a virtual xp machine) that is up to date and working and tested.
And some proper instructions on the github forks frontpage on both mac and windows about mysql. For windows, hopefully with calibre, qpdf, mudraw support... and mysql. And I will test my linux version on a pi!
If you are using a mac, you can install the mysql osx package, make sure the mysql server is running (click on the mysql icon in the system preferences) and run mysql_secure_installation in the terminal so set things up...
run the sql commands below on the command line or with something like Toad, and tailor then to your own needs... since below gives the user comic access to all databases if you're using mysql, which locally is fine but still a security risk. (check google for right commands).
CREATE DATABASE comicstreamer;
CREATE USER 'comic'@'%' IDENTIFIED BY 'comic';
GRANT ALL PRIVILEGES ON *.* TO 'comic'@'%' WITH GRANT OPTION;
CREATE USER 'comic'@'localhost' IDENTIFIED BY 'comic';
GRANT ALL ON *.* TO 'comic'@'localhost';
FLUSH PRIVILEGES;
Alrighty, for now I'll keep sqlite on. But I eagerly await the updated windows version. Much appreciation from myself and those who use my server
Also, new ideas are welcome!
Are you missing anymore features?
I added a cover view to the table pages (you can switch between, see the button above the table) I am planning to add more cross links, like clicking on series above the random comic, last added/read to a overview of that series... any thoughts? Are you missing more views to the comic data?
Also, next to tagging comics as favourites, one user was overly enthusiastic about the inverse, so due to that user the blacklist of pages is partly completed. You can thank that user also for extracting and sending me over 5000 scanner art images he/she collected for the most popular release groups of american comics for the last 10 years. (hint use comicstreamer -l, fiddle with this piece of code to extract you're own lastpages) Which at the moment are replaced with an image! And will be removed from reading in the future version.
Building of on this blacklist.
I can identify, with some extra code, any unique page as long as you do not change the actual page image by resizing/reformatting/etc... them. Inside any comic file!
That means detecting double pages, detecting double comics!
Even if the archive compression format, filenames of the pages or filename of the archive is changed. Also this means exporting/importing meta data from the database is possible. since even if the archive compression format, filenames of the pages, filename of the archive or the location is changed it still can identify it.
Do you think this is a feature that is useful to have?
Anyway... :-)
I also have a dutch scrapper ready. And almost working with comicstreamer, since it scrapes from a site that is static from 2011 but contains the largest info source on dutch comics, I could find up to this date, I probably will include the resulting cache db on github.
Any thought about other scrapers?
I can see how that might be useful for hiding pages with ads, or, sometimes scanners put their logo at the beginning before the cover of the book, which makes the scanner logo come up as the thumbnail instead of the book cover. I like to hide those. I scrape all metadata using ComicRack's ComicVine Scraper plugin, which is very useful. If you're thinking of adding scrapers to ComicStreamer, a ComicVine scraper would be really popular.
How about a userlist? I have a few dozen people who use my comic database over the internet, it would be nice to give each of them a "most recently read" list rather than having one for everyone. Also, it would be cool to be able to to track read progress for individual users. That sounds like it would be a lot of work to implement, but if you ever have the time, I bet that would be a popular feature. Maybe give everyone their own username and password?
Also, a sorting hierarchy. Sometimes when I sort by date, if two issues of the same series come out in the same month, the latter will appear before the former. Could you make it so that we can have it sort by date, and if the dates are the same, it then sorts by issue number?
And how about uploadable home-button thumbnails and backgrounds? This feature isn't too important to me, but it would be cool to be able to upload our own home button and list backgrounds through the settings page. Just a thought.
I don't know if you have any control over the HTML5 webreader, I know Balaclark made that. But if you know how it works and can make edits, zooming in and out on tablets is kind of inconvenient. If there's a way to make it multi-touch zoomable (like most images and web pages on tablets), that would make the reading process much more pleasant! Ubooquity's web reader works this way, but ubooquity doesn't handle large databases very well... It doesn't do much with the metadata.
Also with the web reader on (in my case, android) tablets, on double page spreads, when I'm zoomed in to match the page height, even when I'm not looking at the rightmost side of the page, tapping the right side of the screen will go to the next page. If there's a way to make a page switch possible only when scrolled all the way to the right (or all the way to the left for left-to-right mode), and otherwise to make tapping the right side of the screen scroll to the right half of the page, that would be awesome. I've accidentally skipped the second half of double page spreads before! The mobile ComicRack app does this, but it's obviously not browser based and has no streaming features :/
Ok, need sleep first... read rest later...
https://dl.dropboxusercontent.com/u/12474226/32/ComicStreamer%200.9.51.exe
windows xp+, please do not use the pdf, ebook function... it probably won't work Cache and mysql support experimental, cache does not like webp on xp...
But, that's all new experimental stuff :-)
What I wonder how many last pages got blacklisted on your setup, those who will disappear in the future :-)
btw, you can blacklist pages yourself in the comicreader, it's the last of those 'x' buttons, the images are saved in your user folder under the folder blacklist, and can be used in any further run/builds (but be sure to make a backup), there will be an option soon to export and import these files to a very small comicstreamer blacklist like the 5000 scanner pages .
you still have to clear the cache if enabled if you click 'x', i forgot to implement to delete it out of the cache... I was thinking about calling blacklist, blocklist and add categories, like scanner grafitti, comic ad, other ad, cover, ...
You are a champion! Was not expecting such a speedy response! Can I buy you a drink or something? Seriously, is there anything I can do for you? Do you want access to my comic server?
And how about uploadable home-button thumbnails and backgrounds? This feature isn't too important to me, but it would be cool to be able to upload our own home button and list backgrounds through the settings page. Just a thought.
You can change and personalise any icon, image or color if you want.
For a package it is located in the install folder on windows in the static folder (or on mac click with other button on the app and select open package in the popup menu and go to the static folder some deep in the pakcage)
Be sure to make a backup such that you do not overwrite them with future installs. Colors are located in comicstreamer.css.
If you have a nice theme send it to me, i will add it ;-)
I truly apologize for interrupting your sleep, but after installing 0.9.51 with the installer you just put up, I'm getting an error message when I try to run ComicStreamer:
"Couldn't open database. Probably the schema has changed."
I made sure to uninstall all other versions of ComicStreamer and restart my computer before installing 0.9.51
I am on Windows 10 x64, by the way.
That's good information about the icons and images! I will work on some themes
Can't think of any features that I'm missing at the moment. I mostly use this app as a library and for putting files into my iPad comic reader (Chunky) every now and then, so my needs are minimal.
The fix for that title/series bug is my main interest at the moment, but I had the following thoughts
-
Is there a rescan button? I couldn't find one.
-
On the UI: Personally I think I'd prefer a more sedate UI, maybe something closer to what Mylar has, but it doesn't really matter as long as it works :). That said, it might be nice if there were captions and tooltips on the buttons at the top of the main screen. It's not immediately obvious what they are (and captions because tooltips won't help on a tablet.
Cheers
Just for the heck of it, I took the comicstreamer.exe from the 0.9.5 folder and put it into the 0.9.51 folder. Guess what? It launched! I bet it won't get far with the wrong exe, but I feel clever for getting past that error message, haha. A lucky try I guess.
EDIT: So, actually it's working pretty well. I'm amazed! So I guess if you're looking for what went wrong with that installer, you should take a look at the exe. I'm happy to be a guinea pig for you to help figure out why I was getting that error message before.
I can't get into the settings menu to enable mysql, so I'm stuck on sqlite. I get a bunch of text, looks like some kind of error log.
Traceback (most recent call last): File "site-packages\tornado\web.py", line 1413, in _execute File "site-packages\tornado\web.py", line 2721, in wrapper File "comicstreamerlib\server.py", line 1109, in get File "comicstreamerlib\server.py", line 1054, in render_config File "site-packages\tornado\web.py", line 704, in render File "site-packages\tornado\web.py", line 811, in render_string File "site-packages\tornado\template.py", line 278, in generate File "default/settings_html.generated.py", line 239, in _tt_execute _tt_tmp = formdata['qpdf'] # default/settings.html:76 KeyError: 'qpdf'
Probably has something to do with me using the wrong exe.
Also, I had an error where the cache was filled up to 7% or so and no comics would load (or I'd get just the top of a page loaded). Clearing the cache fixed that, and it hasn't happened again. I don't know if I can replicate it, but I figured I'd let you know it happened.
Cool, I will look into all that, and make a proper windows version, hopefully :-)
The error you got is because I added some new stuff to the config file related to cache, mysql, etc, (and in the latest version qpdf, to decrypt pdf not read by pypdf2) and new tables to the database related to blacklists.
You have to rescan your db and make a fresh new config. (or copy past the old value's to the fresh config file) Make a backup or delete your config/db and start clean.
On windows start->run
%appdata%\ComicStreamer
it will open your comicstreamer application data folder. The config file is called comicstreamer.ini and not settings anymore (i wanted to double click on it :-)
also on my setup the windows server reset from the gui did not work at all (it does work on my mac) it would not delete the database file. So, for the moment you can manually delete/backup the database, if something goes wrong or you want to reset. If something goes wrong with the experimental mysql database, just run the following SQL commands:
drop database comic
create database comic
As I said the MySQL part is experimental... haven't figured all the code out yet :-)
The ebooks and pdf are disabled on windows since I did not fill in the right tools...
For the mac version hopefully fixed the series/title stuff. :-)
I will slowly, fix old bugs and finish the new features. (Blacklist/Favorites/Cache/PDF&Ebooks/MySQL/DutchScraper)
One thing I will add soon, The original ToiletSitter, requested this feature: In his own words:
"my hdd had to spin up while I was using chunky. But it spun up too late! only after I selected a comic..."
I am going to make an option to wakeup the hdd or media out of sleep after being idle for some time and it receives a request related comic db (like series overview, or something similar) such that the hdd are already woken when you actually pick a comic.
Very good, very good!
I'm having trouble getting the mysql server working now though...
I installed the server with the community installer. Then, I used the commands you gave to create a database called comicstreamer. I created the user account comic (password comic) during the install, and gave it all permissions, so when I tried to create that account through the command line I got an error.
Here's a snippet from my log, this is how I know MySQL isn't working...
2017-02-14 09:09:19,937 - ERROR - [Errno 10048] Only one usage of each socket address (protocol/network address/port) is normally permitted 2017-02-14 09:09:19,888 - INFO - Database: SQLite: (C:\Users\Zack\AppData\Roaming\ComicStreamer\comicstreamer.sqlite) 2017-02-14 09:09:19,887 - DEBUG - Database: SQLite: Started 2017-02-14 09:09:19,887 - WARNING - Database: Switching to SQLite Engine 2017-02-14 09:09:19,885 - ERROR - Database: MySQL: Failed (comicstreamer [localhost:3306]) 2017-02-14 09:09:19,884 - DEBUG - Database: MySQL: Started
the SHOW DATABASE \g command in the MySQL command line shows that there IS a database called comicstreamer now, so at least that part worked.
I can't tell if the user 'comic' has actually been created, but I KNOW that the user 'root' exists... So I set the username and password in the toiletstreamer settings to the information for root. Probably not the best idea, but it still isn't working. I'm still getting the MySQL: Failed message in the log.
I got connected with a osx mysql server from windows, I am installing mysql on windows itself... see what happens... :-) And compile a x64 version... :-)
To see the users use:
select host, user from mysql.user;
show databases;
it takes time to download and install,... report back later ;-)
You probably did everything right, I forgot a module...
https://dl.dropboxusercontent.com/u/12474226/ComicStreamer/ComicStreamer-0.9.51-x64.exe
Still highly experimental, i still have to figure out sqlalchemy... and why I can not delete comic records
EUREKA! It worked!
I wish I knew more about this stuff, but I'll do whatever I can do to help this project along. I love this program, and I love what you're doing with this program.
I'm rescanning my library into the MySQL database now, it's gonna take a while. I have around 36000 comics. But I'll tell you how everything works once everything's all scanned in. It's very speedy now, but with the Sqlite, I noticed it would slow down as the database got bigger.
above 60.000 or so sqlite takes a break, a nap, probably go out for a bite, and then report back... You can toggle the switch between sqlite and mysql in the settings after you scanned mysql (and of course if you not already did, a scan pass for sqlite) Then toggle between them and restart and try the difference.
you could probably help with testing, like windows stuff, blacklisting, userstuff
Ow, one user irc'ed. it run fine with around 200.000 comics. And that the search is broken. It gives all results instead of 50 (default)...
O well another thing to look into ;-)
Okay, I will email you. But also, now that my database is fully scanned, I can't seem to access the server through a browser. I get an endless loading screen... When I stop the MySQL server, I can at least access the settings and switch back to sqlite. Actually, I got it to load the settings page once without stopping the MySQL server. But I can't get to any comic lists, can't get to the page with the random pull, and can't get to the most recently read page. It was working well earlier on in the scan, so I'm guessing it's the size of the database? Maybe I set the MySQL server up wrong?
Oh! As I'm typing this, I'm seeing one of my bookmarked searches load! It seems to be stuck on an empty comic list, "processing."
Here are my logs from after the database finished scanning, maybe they can shed some light on what's going on here?
Ah, this version won't finish scanning in sqlite mode either! Somewhere in the 23000s, the scanning just stops - the monitor on the stats page says it's still scanning, but a new book hasn't been scanned in 2 hours :(
Here's what the log does after it stops scanning in sqlite mode
2017-02-15 08:29:02,799 - DEBUG - STREAM pHYs 57 9 2017-02-15 08:29:02,799 - DEBUG - sBIT 41 4 (unknown) 2017-02-15 08:29:02,799 - DEBUG - STREAM sBIT 41 4 2017-02-15 08:29:02,799 - DEBUG - STREAM IHDR 16 13 2017-02-15 08:29:02,799 - DEBUG - STREAM IDAT 115 8192 2017-02-15 08:29:02,799 - DEBUG - STREAM tEXt 78 25 2017-02-15 08:29:02,799 - DEBUG - STREAM pHYs 57 9 2017-02-15 08:29:02,799 - DEBUG - sBIT 41 4 (unknown) 2017-02-15 08:29:02,799 - DEBUG - STREAM sBIT 41 4 2017-02-15 08:29:02,799 - DEBUG - STREAM IHDR 16 13