buku
buku copied to clipboard
jarun
ToDo list
Continued from #343.
Notes
Please check the open issues for an area to contribute. bukuserver has several open features requests.
The list below is a growing one. While suggesting new features please consider contributing to Buku. The code is intentionally kept simple and easy to understand with comments. We'll be happy to assist any new contributor. We need your help!
Some of the fresh-baked features may not have been released yet. Grab the master branch for those.
Cooking
None yet.
Identified tasks
- [ ] support offline bookmark addition
- [ ] support gemini protocol
- [ ] support archive.is
- [x] bukuserver: disable favicon
- [ ] bukuserver: bookmarklet documentation (see #385)
- [ ] bukuserver: thumbnail
- [ ] bukuserver: add login feature (see #309)
- [ ] bukuserver: generate separate package (see #307)
- [ ] Import keywords from browser html (see #311)
- [ ] Android app (with the same schema) (probably a separate project)
- [ ] Add option
--preserveto ignore specific fields during auto-update [see issue #327] - [x] support Vivaldi browser
- [x] better XBEL compatibility
- [x] check for empty search results in piped operations
- [x] Test cases for
exportdb()API (@rachmadaniHaryono) - [x] Purge db file when all records are deleted
- [x] Fix tests, which are taking long time to finish (thanks @rachmadaniHaryono)
- [x] Import firefox exported json (title, url, tags, desc; parent folder if opted) [see API
importdb()] (thanks @ckolumbus) - [x] Disable fetch from web during auto-import, import and merge
- [x] Honor
-nwith-p(thanks @jpdasma) - [x] Support search options with
--export - [x] Import "description" (into description field) and "keywards" (as tags) from
HTML <meta> Tag - [x] Result pagination
- [x] Convert bukuserver README from rst to markdown
- [x] Browse a bookmark (possibly dead URL) on Wayback machine
- [x] Show bookmarks to be deleted before deletion
- [x] Support keyword filtering (records having keywords
aandbbut notcandd) (thanks @SaltyCatFish) - [x] Support filtering by tags with search options (see #250) (thanks @SaltyCatFish)
- [x] Port feature from
googler/ddgr- omniprompt keyO - [x] Show results with most search keyword matches on top (thanks @mosegontar)
- [x] Text-mode user agent for Buku
- [X] Copy search result URL to cipboard
- [x] Read default Firefox profile name from
profiles.ini(see #212, thanks @alex-bender) - [x] Support
--formatin search results (ref, thanks @mosegontar) - [x] API documentation (comments need to be in NumPy format) (thanks @mosegontar)
- [x] Auto-import: optionally add parent folder name as tag, ask for unique tag [like
importdb()] - [x] Support custom colours (thanks @shv-q3)
- [x] Generate packages on Travis-CI using PackageCore (see #189) (thanks @shaggytwodope)
- [x] Search multiple tags, exclusion in tag search (thanks @mosegontar)
- [x] Auto-import Firefox and Google Chrome bookmarks (thanks @alex-bender)
- [x] Suggest tags those go together
- [x] Append/overwrite/remove tags from prompt
- [x] Add more tests (ongoing activity @rachmadaniHaryono)
- [x] A browser plugin (thanks @SamHH for bukubrow)
- [x] Text editor support (thanks @ZwodahS)
- [x] Need a PyPI maintainer (thanks @shaggytwodope)
- [x] Make refreshdb faster using threads (record updates should be synchronized)
- [x] Show usage count in tag list
- [x] Proxy support (thanks @denisfalqueto)
- [x] Continuous search at prompt
- [x] Add prompt help
- [x] Specify custom DB file to class BukuDb (library usage, no exposed option)
- [x] Move to urllib3
- [x] Handle redirects using referrer masking. Example URL. Fixed with urllib3.
- [x] Support URL shortening. This helps to share URLs. (see #92 for limitations)
- [x] Make a bookmark title immutable via refreshdb()
- [x] Markdown import/export
- [x] Regex search
- [x] Ubuntu PPA (thanks @shaggytwodope)
- [x] Export specific tags to HTML
- [x] Exact word match using REGEX. Make substring match optional.
- [x] Delete all records based on a search result
- [x] Delete multiple items, support combination of indices and ranges
- [x] Append tags
- [x] Travis CI integration
- [x] Ubuntu deb package generation on new tag
- [x] Merge bookmark database files (for users who work on multiple systems)
- [x] Export bookmarks in FF or Chrome html format.
- [x] Option to add folder names as tags while importing HTML (thanks @mohammadKhalifa)
- [x] Check and show upstream version
- [ ] Anything else which would add value (please discuss in this thread)
i just noticed that there is possibility to track user with favicon based on this https://github.com/jonasstrehle/supercookie
i may update bukuserver later to set favicon default to false.
The bukuserver is an awesome frontend for Buku!
Some new functionality:
- Add a Dark mode and possibility to turn on & off (create a custom CSS)
- Bootstrap should be updated to minimum "bootstrap 3.4.1" to remove the known vulnerabilities (https://snyk.io/vuln/npm:jquery?lh=3.3.1)
- jQuery should be updated to minimum "jquery 3.5.0" to remove the known vulnerabilities (https://snyk.io/vuln/npm:bootstrap?lh=3.3.5)
- Add group / category to be used together with Tags to create a hierarchy when a lot of bookmarks have been added . That way you can as an example create a group called Newspapers with tags like electronics, legal, sports, economics etc.
- Add year & month when added for both navigation purpose and stats over time (how many bookmarks was created in X)
- When modal opens, should select the first input field by default
- Add mousetrap.js to make keyboard shortcuts available like example CTRL+SHIFT+A to create a new bookmark
@zz5zz
Add a Dark mode and possibility to turn on & off (create a custom CSS) Bootstrap should be updated to minimum "bootstrap 3.4.1" to remove the known vulnerabilities (https://snyk.io/vuln/npm:jquery?lh=3.3.1) jQuery should be updated to minimum "jquery 3.5.0" to remove the known vulnerabilities (https://snyk.io/vuln/npm:bootstrap?lh=3.3.5)
these all depend on flask-admin, but i do hope bukuserver can be updated to use latest flask before updating flask admin
Add group / category to be used together with Tags to create a hierarchy when a lot of bookmarks have been added . That way you can as an example create a group called Newspapers with tags like electronics, legal, sports, economics etc.
bukuserver depend on buku to handle tag so no group and category. but it is actually possible if bukuserver can parser tag and handle part of tag as category/group, for example programming: python tag will be under programming category group.
Add year & month when added for both navigation purpose and stats over time (how many bookmarks was created in X)
buku don't record created date
When modal opens, should select the first input field by default
this depend on flask admin and should be tested when flask admin updated
Add mousetrap.js to make keyboard shortcuts available like example CTRL+SHIFT+A to create a new bookmark
that seem to be nice addition, but it should be noted that bukuserver should handle external javascript dependency before adding new one . see also #307
@zz5zz are you interested in raising PRs?
@rachmadaniHaryono please review.
I'll try to contribute by doing a PR. Very long time since I handled python so need to brush up a bit.
What would it take to integrate the Buku database with Mac Spotlight search ? This strikes me as being a separate project, but its hard to know where to get started with it.
@RichardConroy You can use Alfred with a custom workflow. For the script you can use Python (or what you like) to execute the buku CLI command, the filtering of the bookmarks can be done by Alfred.
i create #546 but after few new bukuserver issue i decided to convert it to draft and merge possible small issue into this pr
unfortunately i can't start weekend but hopefully next few days i can work on it
and maybe if possible upgrade bukusserver to newer flask version
Buku is a great bookmark manager. I use it daily. I wish some of the related projects also work on windows like the bukubrow server and Oil. Search-as-you-type cli front-end. Maybe also an interface to gvim. Anyway, thank you for the good work you've done.
this is current pyproject.toml for buku
[tool.poetry]
name = "buku"
version = "4.6"
description = "Bookmark manager like a text-based mini-web."
authors = ["Arun Prakash Jana <[email protected]>"]
license = "GPLv3"
keywords = ["cli", "bookmarks", "tag", "utility"]
classifiers = [
"Development Status :: 5 - Production/Stable",
"Environment :: Console",
"Intended Audience :: Developers",
"Intended Audience :: End Users/Desktop",
"License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
"Natural Language :: English",
"Operating System :: OS Independent",
"Programming Language :: Python :: 3 :: Only",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.6",
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
"Topic :: Internet :: WWW/HTTP :: Indexing/Search",
"Topic :: Utilities",
]
homepage = "https://github.com/jarun/buku"
readme = "README.md"
[tool.poetry.dependencies]
python = ">=3.6,<4"
beautifulsoup4 = ">=4.4.1"
cryptography = ">=1.2.3"
html5lib = ">=1.0.1"
urllib3 = ">=1.23"
certifi = {version = "^2021.10.8", extras = ["ca-certificates"]}
appdirs = {version = ">=1.4.3", extras = ["server"]}
arrow = {version = ">=0.12.1", extras = ["server"]}
cffi = {version = ">=1.9.1", extras = ["server"]}
click = {version = ">=7.0", extras = ["server"]}
Flask-API = {version = ">=0.6.9", extras = ["server"]}
Flask-Bootstrap = {version = ">=3.3.7.1", extras = ["server"]}
flask-paginate = {version = ">=0.5.1", extras = ["server"]}
flask-reverse-proxy-fix = {version = ">=0.2.1", extras = ["server"]}
Flask-WTF = {version = ">=0.14.2", extras = ["server"]}
Flask = {version = ">=1.0.2,<2.0", extras = ["server"]}
idna = {version = ">=2.5", extras = ["server"]}
itsdangerous = {version = ">=0.24", extras = ["server"]}
Jinja2 = {version = ">=2.10.1", extras = ["server"]}
MarkupSafe = {version = ">=1.0", extras = ["server"]}
packaging = {version = ">=16.8", extras = ["server"]}
pyasn1 = {version = ">=0.2.3", extras = ["server"]}
pycparser = {version = ">=2.17", extras = ["server"]}
requests = {version = ">=2.21.0", extras = ["server"]}
six = {version = ">=1.10.0", extras = ["server"]}
Werkzeug = {version = ">=0.11.15,<0.15.0", extras = ["server"]}
flask-admin = {url = "https://github.com/flask-admin/flask-admin/archive/refs/heads/master.zip", extras=["server"]}
twine = {version = "^3.7.1", extras = ["packaging"]}
[tool.poetry.dev-dependencies]
attrs = ">=17.4.0"
flake8 = ">=3.4.1"
hypothesis = ">=6.0.0"
mypy-extensions = "0.4.1"
py = ">=1.5.0"
pylint = ">=1.7.2"
pytest-cov = "^3.0.0"
pytest-vcr = ">=1.0.2"
pytest = ">=6.2.1"
PyYAML = ">=4.2b1"
setuptools = ">=41.0.1"
vcrpy = ">=1.13.0"
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
[tool.poetry.scripts]
buku = 'buku:main'
bukuserver='bukuserver.server:cli'
the biggest change should be werkzeug for bukuserver
Werkzeug = {version = ">=0.11.15,<0.15.0", extras = ["server"]}
this is because flask_api require werkzeug._compat which only exist before v0.15.0
so therefore bukuserver packages that need to be upgraded
- flask
- flask-api
- flask-admin
- werkzeug
- flask-reverse-proxy-fix
i'm still waiting pr for flask-reverse-proxy-fix
https://github.com/antarctica/flask-reverse-proxy-fix/pull/1
if i finished with current bukuserver bugfix pr, i will fork the project and create new version
related issue
https://github.com/jarun/buku/issues/449
why using poetry?
from their website
- Dependency resolver: Poetry comes with an exhaustive dependency resolver, which will always find a solution if it exists. And get a detailed explanation if no solution exists.
- Isolation: Poetry either uses your configured virtualenvs or creates its own to always be isolated from your system. The behavior is configurable.
while there is guideline, there is no entry about virtualenv and handling package requirements
predicted issue
- buku have to be renamed back to buku.py
- something wrong on pypi after release
to handle pypi release when this is accepted, maybe release alpha/beta/rc version before final version
there are also several tool that i want to add to project
- commitizen: handle program version and commit message format this can be used for contributor, who want commit message format recommendation
- pre-commit: check before commit there is several hook for this program that can be used for easier development
- black: autoformat python code with this we can reduce guideline needed and let black format the code
with any of this tool is accepted, guideline on wiki should be changed as well
python 3.6 just ended on december 2021
https://endoflife.date/python
e:
follow nep 29 https://numpy.org/neps/nep-0029-deprecation_policy.html
idea: if poetry is accepted, post about recruiting new beginner developer for help on doc, test, and program
most of the time, beginner developer can see what is needed or what is not clear on buku
minimal requirement would be just understand poetry and git
e2:
add doc for using pipx to install buku
@jarun, sorry i think i push a commit accidentally. c09ecfbc2745bcc7f2e70942975bfceb0536075a
can you revert it? i think i can do it, but it is better if you do it.
sorry for this
unrelated to above post
but in this issue i thought we could use yaml for issue form
https://github.com/jarun/buku/issues/558#issuecomment-1011482388
imo i think we can update issue template so user have to fill buku version
https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/configuring-issue-templates-for-your-repository#creating-issue-forms
but i miss the notification on top of that page
Note: Issue forms are currently in beta for public repositories on GitHub.com only.
i thought buku should update its template because when you go to https://github.com/jarun/buku/blob/master/.github/ISSUE_TEMPLATE.md there is always this notification
You are using an old version of issue templates. Please update to the new issue template workflow. Learn more
so i create new issue template and feature request template from github
results
https://github.com/rachmadaniHaryono/Buku/issues/new/choose
https://github.com/rachmadaniHaryono/Buku/blob/master/.github/ISSUE_TEMPLATE/bug_report.md
---
name: Bug report
about: Create a report to help us improve
title: ''
labels: bug
assignees: ''
---
**Describe the bug**
A clear and concise description of what the bug is.
Before opening an issue, please try to reproduce on [the latest development version](https://github.com/jarun/Buku#from-source) first. The bug you noticed might have already been fixed.
If we need more information and there is no communication from the bug reporter within 7 days from the date of request, we will close the issue. If you have relevant information, resume discussion any time.
**To Reproduce**
Steps to reproduce the behavior:
1. ...
2. ...
3. ...
4. See error
Debug logs using the `-g` option
**Expected behavior**
A clear and concise description of what you expected to happen.
**Screenshots**
If applicable, add screenshots to help explain your problem.
**Desktop (please complete the following information):**
- Buku version: [e.g. 4.6, or commit hash]
- OS: [e.g. windows, mac, linux, ubuntu]
- Python version [e.g. 3.9, 3.10]
- Terminal emulator and shell [e.g. "alacritty, zsh" or "gnome-terminal, fish"]
- `locale` output, if relevant [e.g. LANG=en_US.UTF-8]
It's a good idea to set your locale to UFT-8. Please refer to [Buku #131](https://github.com/jarun/Buku/issues/30).
**Additional context**
Add any other context about the problem here.
https://github.com/rachmadaniHaryono/Buku/blob/master/.github/ISSUE_TEMPLATE/feature_request.md
---
name: Feature request
about: Suggest an idea for this project
title: ''
labels: ''
assignees: ''
---
**Is your feature request related to a problem? Please describe.**
Please consider contributing the feature back to `Buku` yourself. Feel free to discuss. We are more than happy to help.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.
looking at current template, i don't know if it is worth it to change it
i tried to use poetry for buku not haven't found better configuration yet.
there is still issue to use it without copy buku to buku.py
i also kinda wary(?) about this because this should discussed with other packager
i also tried to create github page with sphinx on current doc, but it is harder
another possibility is to replace it with mkdocs, but that is last option
i am more familiarized with mkdocs for now, so maybe i will try it (and replace sphinx)
pre commit update pr is accepted, so it should be easier for contributor to automate some task
but contributing.md is not yet updated for this
unfortunately i don't give buku time for now, so no fixes for bukuserver for now
Is there a built-in way to add a new tag from the prompt ?
From what I understand, g only works with existing tag.
Thanks !
Use w to edit a bookmark and add a tag. You can add it to more bookmarks after that.
Right, that’s what I’ve been doing. Here are 2 suggestions if other people are interested :
- add a
Goption to add/modify/set tags by name - with this option, if the tag does not exist, create it and add it
buku design doesn't allow having a tag that's not linked to at least one bookmark.
current state for bukuserver before new release
bug (important)
- https://github.com/jarun/buku/issues/603
- https://github.com/jarun/buku/issues/607
- https://github.com/jarun/buku/issues/608
- https://github.com/jarun/buku/issues/618
- https://github.com/jarun/buku/issues/606 ~(this is rejected for now because it is flask-admin default behavior)~
feature (lower priority, should add to todo list)
- https://github.com/jarun/buku/issues/620
- https://github.com/jarun/buku/issues/549
- https://github.com/jarun/buku/issues/564
- https://github.com/jarun/buku/issues/586
- https://github.com/jarun/buku/issues/621 (or maybe already done)
e: 606 already done
…Speaking of running tests locally – one of test_network_handler_with_url runs (corresponding to a Google search query) always fails on my machine due to Google server responding with HTTP 429 Too Many Requests (I get the same response when sending the request with wget).
Not sure why this happens (perhaps it's related to me using a VPN for internet access?), but it certainly results in guaranteed fails every time I run unit tests locally. (…Does it really count as a unit test if the network requests are actually being sent? Normally you'd expect the requests themselves to be mocked or at least extracted to external functionality, in part to avoid such factors from affecting the outcome – after all, I doubt that the goal is to test specifically whether the code can communicate with Google servers.)
Does it really count as a unit test if the network requests are actually being sent?
@LeXofLeviafan
but it kinda also show that this function will not result the same thing on different computer, so it kinda work
but for consistency it is actually bad unit test
several options:
- have second alternative result instead of fail the test immediately
- cache the result, something similar to https://github.com/oprypin/pytest-golden
- mock the result
e: related https://github.com/jarun/buku/pull/443
Unit tests are meant to test the code itself – whether the logic works correctly. And for a test to show that it "will not result in the same thing on different computer", it needs to be purposefully run on multiple machines (…also such a thing normally indicates that either the test or the code is broken :sweat_smile:)
The reason it fails is purely incidental and completely external (i.e. this particular request has been blocked by Google for specific IPs or something like that) which has nothing to do with what the code is being tested for, really (unless the purpose is to test how various servers on the internet handle repeating requests or how their DDoS detection works).
…The caching thing looks like the way to go (provided the cache is actually included in the repo), except what you'd want is to cache the network requests instead of tests outputs. Ideally you'd want to generate such a cache file once (and never really need to modify it afterwards), and then mock the network environment in the test by providing stored response instead of live network feedback (or simply separate I/O from logic in the code and test the logic only). It can even be archived/compressed (a tarball?) to avoid having many small files or a large one in the repo (which seems to be the reason why #443 got rejected).
current issue with no active pr
- docs
- https://github.com/jarun/buku/issues/637
- https://github.com/jarun/buku/issues/649
- ~~https://github.com/jarun/buku/issues/644~~
- bug
- https://github.com/jarun/buku/issues/624 (wait for more info)
- refactor
- https://github.com/jarun/buku/issues/652
- https://github.com/jarun/buku/issues/653
- https://github.com/jarun/buku/issues/654
- https://github.com/jarun/buku/issues/648
e: i'm sorry network cache pr is late, hopefully i can start tomorrow
e2: remove 644
current issue with no active pr
I just realised that #648 appears to be missing from the list.