fiction-dl
fiction-dl copied to clipboard
A content downloader, capable of retrieving works of (fan)fiction from the web and saving them in a few common file formats.
fiction-dl (1.8.3)
fiction-dl is a command-line utility designed to download works of (fan)fiction from the web and save them in a few portable file formats. It requires a Python interpreter installed, version 3.8 or higher.
The application currently supports almost two dozen websites and six output file formats; the latter being: HTML, ODT, PDF, EPUB and MOBI. It's able to download images found in story content, as well as capable of applying basic typographic corrections to the content itself.
✿ Table of Contents
- Features.
- Installation.
- Usage.
- License.
- Technical.
✿ Features
Downloading stories
fiction-dl can download stories from a multitude of sites - in some cases, it also supports downloading all stories published by any specific user (which is called downloading stories from a channel).
In a select few sites, it allows you to log-in to the site using the command-line; this functionality allows the user to access subforums closed to guest viewers, like the Fandom category on AlternateHistory.com, or quarantined subreddits on Reddit.
# | Category | Site | Authentication support | Channel support |
---|---|---|---|---|
1 | fanfiction | Archive of Our Own | ✓ yes | ✓ yes [1] |
2 | fanfiction | FanFiction.net | ✗ no | ✓ yes [2] |
3 | fanfiction | FicWad.com | ✗ no | ✓ yes |
4 | fanfiction | SpaceBattles.com | ✓ yes | ✗ no |
5 | fanfiction | SufficientVelocity.com | ✓ yes | ✗ no |
6 | fanfiction | AlternateHistory.com | ✓ yes | ✗ no |
7 | fanfiction | QuestionableQuesting.com | ✗ no | ✗ no |
8 | fanfiction | HarryPotterFanFiction.com | ✗ no | ✓ yes |
9 | fanfiction | WhoFic.com | ✗ no | ✓ yes [3] |
10 | fanfiction | SamAndJack.net | ✗ no | ✓ yes |
11 | fanfiction | Ralst.com | ✗ no | ✗ no |
12 | original fiction | Fiction Press | ✗ no | ✓ yes |
13 | original fiction | ✓ yes | ✗ no | |
14 | original fiction | Quotev | ✗ no | ✗ no |
15 | original fiction | Wuxiaworld | ✗ no | ✗ no |
16 | erotica | AdultFanFiction | ✗ no | ✓ yes |
17 | erotica | Literotica | ✗ no | ✓ yes |
18 | erotica | Hentai Foundry | ✗ no | ✓ yes |
19 | erotica | NajlepszaErotyka.com.pl | ✗ no | ✗ no |
20 | erotica | Nifty.org | ✗ no | ✗ no |
21 | erotica | asstr.org/~Kristen | ✗ no | ✗ no |
- [1] fiction-dl is also capable of downloading whole collections and series.
- [2] The application supports also downloading entire collections.
- [3] The application supports downloading whole series as well.
It is also possible to process stories stored in local files. You can put the HTML-formatted story in a raw text file and use fiction-dl to reformat it in all of its output file formats.
Formatting stories
Downloaded stories are auto-formatted and saved in a few file formats:
File Format | Requirements |
---|---|
HTML (.html) | None. |
EPUB (.epub) | None. |
MOBI (.mobi) | Calibre installed on the machine. |
OpenDocument (.odt) | None. |
Portable Document Format (.pdf) | LibreOffice installed on the machine. |
Embedding images
The application can download images found in story content and embed them in output files.
Typographic corrections
fiction-dl automatically applies basic typographic corrections to the content of downloaded stories. For example:
Source Text | Corrected Text |
---|---|
Lorem ipsum... dolor. | Lorem ipsum… dolor. |
Lorem ipsum...dolor. | Lorem ipsum… dolor. |
Lorem ipsum...... dolor. | Lorem ipsum… dolor. |
Lorem ipsum , dolor. | Lorem ipsum, dolor. |
Lorem ipsum????? Dolor. | Lorem ipsum? Dolor. |
Lorem ipsum - dolor. | Lorem ipsum — dolor. |
Packing stories
If you're downloading a lot of short stories, you can pack them into a single file, using the "-pack" command line option. This way you're adding only a single item to your Kindle book list, instead of many.
✿ Installation
You can install fiction-dl using pip:
python3 -m pip install --upgrade fiction-dl
(If the package's already installed, this command will also update it.)
If you're running a Debian-derived Linux distribution, you might also need to install the following packages:
apt-get install libgl1-mesa-glx libglib2.0-0 libmupdf-dev
(fiction-dl uses OpenCV for processing downloaded images, which requires them to be installed. PyMuPDF requires libmupdf-dev when it's being built from the source.)
✿ Usage
To download a story from a URL, simply type:
f-dl URL
In order to download multiple stories, create a text file and place the URLs in it, each one in a separate line. Then type:
f-dl YourFilesName
Options
Option | Result |
---|---|
-h (--help) | shows the overview of command-line options |
-a | authenticates the user to supported sites using interactive mode |
-c | clears the cache before launching the application |
-pack | packs all downloaded stories inside one file (of each type) |
-v | enables the (more) verbose mode |
-f | overwrites output files (in case they already exist) |
-d | enables debug mode (saves some data useful for debugging) |
-no-images | disables downloading images found in story content |
-persistent-cache | preserves the cache after the application quits |
-lo | used to specify the path to the LibreOffice executable (soffice.exe) |
-o | used to specify the output directory path |
Text File Extractor
Create a text file meant to contain the story. In its first lines type:
LOCAL TEXT STORY
*URL*
*The Title of the Story*
*The Author*
*The One-Line Summary*
Follow it by the story's content, wrapped in HTML tags. Then call:
f-dl FilePath.txt
You can insert chapter-breaks in the story by typing "CHAPTER BREAK LINE" in an empty line.
✿ License
GNU GPL 3. The text of the license is provided in the LICENSE file.
✿ Technical
Changelogs
All the changelogs can be found in the Docs/Changelogs.md file.
Documentation
You can generate code documentation using Doxygen; the relevant configuration file is Docs/Docs.doxygen. Generated files will appear in the Docs/Code Documentation directory.
Tests
Launching the Integration Test can be done by entering the Tests directory and executing the following code:
python3 "Integration Test.py"