FTL
FTL copied to clipboard
pi-hole
Add HTTP server and JSON API
By submitting this pull request, I confirm the following (please check boxes, eg [X]) Failure to fill the template will close your PR:
Please submit all pull requests against the development branch. Failure to do so will delay or deny your request
- [X] I have read and understood the contributors guide.
- [X] I have checked that another pull request for this purpose does not exist.
- [X] I have considered, and confirmed that this submission will be valuable to others.
- [X] I accept that this submission may not be used, and the pull request closed at the will of the maintainer.
- [X] I give this submission freely, and claim no ownership to its content.
How familiar are you with the codebase?:
10
This PR adds a fully-fledged HTTP server and JSON API to FTL.
It will replace lighttpd and serve both the API and the next-gen web interface.
More detail is to be added here once this feature advances further.
This pull request has been mentioned on Pi-hole Userspace. There might be relevant details there:
https://discourse.pi-hole.net/t/make-some-advance-api/26786/2
This pull request has been mentioned on Pi-hole Userspace. There might be relevant details there:
https://discourse.pi-hole.net/t/black-whitelisting-does-not-work-on-v5-api/33119/1
We're still following this. I just rebased it onto the latest development branch.
This pull request has been mentioned on Pi-hole Userspace. There might be relevant details there:
https://discourse.pi-hole.net/t/php-webscripts/31016/28
This pull request has been mentioned on Pi-hole Userspace. There might be relevant details there:
https://discourse.pi-hole.net/t/php-webscripts/31016/32
This branch can now host the web interface. Many things will not work as we removed the rusty telnet interface and added a React-based API in this version of FTL. This will have to be adapted for the current API. Eventually, we will be able to get rid of api.php and siblings and have all JS scripts directly talking to FTL.
However, pages not directly depending on the interactive API do already work.
For instance:
This pull request has been mentioned on Pi-hole Userspace. There might be relevant details there:
https://discourse.pi-hole.net/t/php-webscripts/31016/35
This pull request has been mentioned on Pi-hole Userspace. There might be relevant details there:
https://discourse.pi-hole.net/t/how-to-update-dhcp-with-api/34773/1
Is there a rough date when this is going to merge?
When it's ready.
To add to that: will the old API still exist?
Which API are you referring to? There are a few different things that people call an API.
Which API are you referring to? There are a few different things that people call an API.
The current /api.php? summary, lists, topItems, ... endpoints.
/api.php? summary, lists, topItems, ... endpoints.
The goal is to have as little disruption as possible and to open even more opportunity for API use in a true API. The URL endpoint may change, but there will be plenty of notification and discussion prior to that, if that even needs to happen.
The PHP-based API (which is what /api.php?summary, etc. are based on) will be replaced by a proper, more uniform and clear JSON interface. Documentation for this is being written whenever we find the time for it. As @dschaper mentioned, this will by no means be a sudden transition. We will have an extended beta-period preceding the release of the embedded system. However, it brings an enormous amount of benefits and this clearly outweighs the disadvantage of breaking a few scripts. Also, because this allows us to have the API be much more flexible in the future to extend new functionality.
Hi @DL6ER, as a Go developer I would like to start writing a Go SDK for this new API.
I succeeded to install Pi-Hole on a development instance with this branch with the file /etc/pihole/ftlbranch.
I also have the latest Pi-Hole documentation with the API part your are working on.
But, request the /api endpoint always try to redirect me to the "old" /admin and for now, I don't know how to "enable" this new API.
Can you help me to set up Pi-Hole in order to have this API and try to starting playing with ?
Thanks.
I should mention that this API is very much work-in-progress. Even when we made a plan how the API should look like, we do not prevent it from maturing on the way. If we realize only later that a specific path should be changes because this just feels more natural (or similar), we take the liberty to change it. While we're documenting this by updating the documentation itself, it would be too much work to write change logs so tracing our changes may be some substantial amount of work (read as: may or may not, no guarantees in any direction). As such, I have the feeling that it may be too early for even staring an SDK, but I will also not stop you :-)
The redirection to /admin was added only (rather) recently and there is no exception for /api. I'm also not sure if we should use /api when the entire rest of the system is inside /admin so what we currently have is probably even correct. Users may be hosting different stuff at the root-level of the domain and I'm not sure if we should enforce an immutable endpoint on them.
Everything is still open for discussion in this branch. More eyes can often help so any comments are appreciated.
If we realize only later that a specific path should be changes because this just feels more natural (or similar), we take the liberty to change it
I truly understand that your job is work in progress. Let's say I don't have a personal project at the moment, and it could be one :)
So all of your current work is available on /admin instead of /api for now ?
This pull request has been mentioned on Pi-hole Userspace. There might be relevant details there:
https://discourse.pi-hole.net/t/consider-switching-to-using-nginx-unit-instead-of-lighttpd-php-fpm/27379/3
This pull request has been mentioned on Pi-hole Userspace. There might be relevant details there:
https://discourse.pi-hole.net/t/support-for-choice-of-resolver-and-or-web-server/222/15
This pull request has been mentioned on Pi-hole Userspace. There might be relevant details there:
https://discourse.pi-hole.net/t/pihole-blacklist-api-no-netflix-until-your-fitbit-says-you-can/2196/5
This pull request has been mentioned on Pi-hole Userspace. There might be relevant details there:
https://discourse.pi-hole.net/t/domain-regex-enable-disable-command-or-pihole-b-command-with-group-option/43744/4
This pull request has been mentioned on Pi-hole Userspace. There might be relevant details there:
https://discourse.pi-hole.net/t/client-management-only-shows-mac/46236/7
This pull request has been mentioned on Pi-hole Userspace. There might be relevant details there:
https://discourse.pi-hole.net/t/add-api-integration-to-perform-crud-operations-on-local-dns-records/46350/3
Not sure what scope this PR is and where to request new API features so I'll place a comment here. I'd love API functionality for the following:
- Enable/Disable Black/Whitelist entries
- Detect and start an update
- Get instance temp/cpu/mem (same that's displayed on the web interface)
- Manage groups, adlists, clients
- Manage settings (DNS, DHCP)
- Manage local DNS records
Cheers!
Enable/Disable Black/Whitelist entries Get instance temp/cpu/mem (same that's displayed on the web interface) Manage groups, adlists, clients
This is already implemented
@jojost1 The will include everything about Pi-hole that needs to be addressed from the web interface. Because we have a very tight security concept, a few things will not be possible (like restarting the system or running updates) because the FTL daemon never runs itself as root so it cannot do certain things. I do believe that such a thought-through security concept is an essential thing and more other project should follow this. But, as it makes things considerably more difficult in some locations, I don't really expect them to do similar things.
You may say that updating gravity or restarting the system is working now, however, this is only possible because of some dirty privilege tricks which we'll throw out to sandbox the Pi-hole daemon without the overhead of a "real" sandbox.
Isn't this going to increase the database size a lot?
Yes, experiments I ran over the last few days suggest an increase of 30-40% (the majority is storing the client host name).
It is advantageous as it simplifies the user interface because now it doesn't matter if you query on-disk (long-term) or in-memory (fast 24 hours). The database structures are now identical and both loading and storing becomes are easy as
INSERT INTO queries SELECT * FROM disk.queries WHERE timestamp >= ?;
INSERT INTO disk.queries SELECT * FROM queries WHERE id > ? AND timestamp < ?;
Furthermore, the entire content is restored after an FTL restart and there are no artifacts like DNSSEC status always being UNKNOWN after a re-import. This is also what regex_id wants to cover - full restorability.
I did experiment with not storing TTL and REPLY_TIME but these two are only a few bytes (if stored at all, not all queries have TTL and reply time (e.g. no reply arrived at all)) and the logic for excluding them and handling this properly increases the complicity beyond what the gain is.
I currently plan to reduce the duration of the database we typically keep from one year to maybe one month. One could even argue to reduce it to one week or even 24 hours (I'm not planing to be that drastic right now).
This will give us enough room to possibly add more stuff in the future. Users can always decide that they still want a specific number of days using MAXDBDAYS. We'd only change the default value.
I agree with reducing MAXDBDAYS to less than a year. I would go no lower than 30 days as the default. Consider 60 or 90.
I consider 90 days a good range.
Yes, experiments I ran over the last few days suggest an increase of 30-40% (the majority is storing the client host name).
Could this be a place for normalization? DNSSEC and REPLY will be fairly constant, and even client hostname could benefit from a separate table. This could save some space (although space is cheap...). Or would the trade-off, more complex queries, be to high?
Could this be a place for normalization?
DNSSEC and REPLY will be fairly constant
Yes, we only store an ID (like we do already for type, status, etc.) so it will be one byte per entry. A linked table could possibly need more storage, but not less.
client hostname
Yes, for this it would work. However, I kept the IP address in client for backwards compatibility. Because we replicate the table in multiple places (two times in-memory and once on-disk), the additional complexity introduced by a linking table would actually be large. Even when we put the client linking-table only in the on-disk database, the storing/restoring SQL statements would be much more complex than what I shows here: https://github.com/pi-hole/FTL/pull/659#issuecomment-848672429
I'd say "storage is cheap" is a clear winner here.
We've been focusing on the v5.x code lately. The next big step is merging the changes for v5.x into this branch which will happen as early as possible.
This pull request has been mentioned on Pi-hole Userspace. There might be relevant details there:
https://discourse.pi-hole.net/t/secure-web-interface-and-updates/51150/7
Amazing work here! Is the API stable enough to build on top of? I have plans on building an external-dns provider for dynamically modifying DNS records and the legacy API does not support it.
Thanks @adyanth! After spending significant time on the current release branch (v5.x), focus only recently (end of last week) moved back to the development on this branch. We are currently exploring several independent options for the web interface (AdminLTE, Vue-based, etc.) and reverse the right to tweak the individual API endpoint as we see fit. Reason for this is that we designed the REST API implementation by looking at what v5.x has and transforming this to REST best practices. However, it may turn out that it may be more natural to change a few things (like putting objects in an array rather than individual items in an array) as we develop the frontend.
Understood @DL6ER , thanks! This is the place I can follow for updates right? Or is there a discource/any other place where I can keep myself updated on the progress of this?
We're posting noteworthy updates here: https://discourse.pi-hole.net/t/pi-hole-v6-0-development-is-well-on-its-way/43314
This pull request has been mentioned on Pi-hole Userspace. There might be relevant details there:
https://discourse.pi-hole.net/t/support-recursive-resolution-for-non-authoritative-cname-entries/52747/3
We're posting noteworthy updates here: https://discourse.pi-hole.net/t/pi-hole-v6-0-development-is-well-on-its-way/43314
Looks like I either do not have access to it or the link is broken. Will wait for v6.0, thanks!