joelself
Features to implement for 0.2.0 and other tasks
For 0.2.0 I'm looking at adding:
- [ ] Key/value delete
- [ ] Key/value ~~append~~ insert
- [ ] ~~Key modification~~
- [ ] Table ~~and Array of Table add~~ insert
- [ ] Table ~~and Array of Table~~ delete
- [ ] ~~Table and Array of Table modify~~
- [ ] ~~More error reporting~~
- [ ] Remove dependence on regexes to allow tomllib to build on stable
- [x] First tomlkit release
Since I'm moving soon and the prep and actual move will be taking so much time, I'm going to limit 0.2.0 features to just what's needed to get cargo-edit up and running. However I'm adding a change to let tomllib to compile on stable. There's only 4 regexes I need to eliminate for that to happen.
2016-06-14: Adding tomlkit release since there's interest in it and it shouldn't be too bad (there's not a lot of functionality to implement).
I aiming for this to be a really quick release with some key features, but it looks like that's not going to happen because I'm in the middle of moving.
I guess since this won't be as quick of a release as I had planned I could add in tomlkit if people (or even just one person) want it. Do you mind if I ask what you plan to use it for (you probably already know, but the library doesn't have a whole lot of functionality yet.
Edit: Thinking about it more, since there's so little to implement, maybe tomlkit should get a version 0.1.2 release.
Edit: I guess it would be 0.1.2 since there's already a 0.1.1.
I though about https://github.com/killercup/cargo-edit/issues/86#issuecomment-218130010 (doesn't mean I will actually implement it).
Any way, I don't think a git config (which may happen to work on Cargo.toml because of similarity to INI format), but for toml is a useless tool. It may find use outside Rust.
Another use: I was curious which Cargo packages reside inside Rust itself:
rust$ for i in $(find . -name 'Cargo.toml'); do git config -f "$i" -l 2> /dev/null | grep package.name; done
package.name=rustdoc
package.name=fmt_macros
package.name=rustc_save_analysis
package.name=rustc_back
package.name=rustc_lint
package.name=rustc_llvm
package.name=std_shim
package.name=test_shim
package.name=libc
package.name=rustc-main
package.name=error_index_generator
package.name=cargotest
package.name=rustbook
package.name=linkchecker
package.name=getopts
package.name=syntax
package.name=core
package.name=generate-files
package.name=libc-test
package.name=libc
package.name=rand
package.name=rbml
package.name=term
package.name=test
package.name=build_helper
package.name=rustc_const_eval
package.name=rustc_const_math
package.name=rustc_driver
package.name=rustc_platform_intrinsics
package.name=serialize
package.name=syntax_ext
package.name=rustc_passes
package.name=rustc_plugin
package.name=rustc_unicode
package.name=alloc_jemalloc
package.name=log
package.name=std
package.name=rustc_typeck
package.name=rustc_borrowck
package.name=collections
package.name=rustc_front
package.name=rustc_metadata
package.name=rustc_trans
package.name=rustc_bitflags
package.name=alloc
package.name=arena
package.name=flate
package.name=graphviz
package.name=rustc_mir
package.name=rustc
package.name=rustc_data_structures
package.name=rustc_privacy
package.name=alloc_system
package.name=rustc_resolve
package.name=bootstrap
I expect tomlkit (or something like that) be better tool for a job rather than misused git config.
Also such tool (especially for getting and enumerating values) may be useful if somebody wants to create toml config for a Bash script.
Typically such confics are also shell scripts (sourced into the program), and as configs may be edited by non-programmers, it may lead to a trouble. For example, changing DO_SOMETING=no into DO_SOMETHING= yes in a sourced config makes /usr/bin/yes program to be executed instead of setting the value of DO_SOMETHING because of extra space character.
./target/debug/tomlkit --set-value package.keywords[2],'FOO',basic-string --print-doc --file Cargo.toml
Success
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>DOCUMENT<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
[package]
name = "tomllib"
version = "0.1.2"
authors = ["Joel Self <[email protected]>"]
description = "A format-preserving TOML file parser and manipulator"
license = "MIT"
repository = "https://github.com/joelself/tomllib"
readme = "README.md"
documentation = "https://jself.io/rustdocs/tomllib/0.1.1/tomllib/"
keywords = ["toml", "parser", "FOO", "decode", "nom"]
exclude = ["assets/*", "tests/*"]
[dependencies.nom]
version = "^1.2.0"
features = ["regexp"]
[dependencies.regex]
version = "^0.1.48"
# Uncomment these, and comment out the regexp feature to turn on statically compiled regexes
#features = ["regexp_macros"]
#[dependencies.regex_macros]
[dependencies.log]
version = "^0.3.5"
[dependencies.env_logger]
version = "^0.3.2"
[dependencies.pirate]
version = "1.0.0"
[dependencies.csv]
version = "0.14.4"
[[bin]]
name = "tomlkit"
It's basically done. What's left is:
- [x] Set/get/check multiple values in one command
- [x] Option to choose output on one line in CSV format or one value per line.
- [x] Quiet mode option for set-value, right now it prints
Success. End-users might not want that.
I tried installing it with cargo install tomllib and got panicked at 'not yet implemented'.
Will there be a command to list content of entire toml file in line-oriented format (with or without values), like in git config -f ... -l?
It's not completely finished or released yet. When it's released you'll see a version number bump to 0.1.2 or actually it might be a completely different crate. I'm not quite sure how that works right now. If you really want to try it you can check out the 0.1.2 branch and compile it and run it. It has get-value, has-value, get-children, has-children, set-value commands right now, but they each only take a single key to lookup/change. I hadn't planned on adding a command to list all keys and values, but it shouldn't be hard to add.
Almost done. Only thing left is to actually write to the file if you're setting a value.
- [x] Write to file on successful set-value
It's ready for its first release. Here's how you would get the package name, package version, whether or not it has a license and all of the package's dependencies in one short command:
find . -name 'Cargo.toml' | ./toml_parser/target/debug/tomlkit -g package.name,package.version --has-value package.license -c dependencies
"methodtest", "0.1.0", false, [nom]
"mtailtest", "0.1.0", false, [mtaillib]
"multitail", "0.1.0", false, [term, notify, getopts]
"mtaillib", "0.1.0", false, [notify, libc]
"nomplusplus", "0.1.0", true, [radix_trie, regex, lazy_static]
"rproxy", "0.1.0", false, [radix_trie]
"radix_trie", "0.0.8", true, [nibble_vec]
"rustfmt", "0.2.1", true, [toml, rustc-serialize, unicode-segmentation, regex, term, strings, diff, syntex_syntax, log, env_logger, getopts]
"test", "0.1.0", false, []
"tomllib", "0.1.2", true, [nom, regex, log, env_logger, pirate, csv]
"tomllib", "0.1.0", true, [nom, regex, log, env_logger]
"tomllib", "0.1.1", true, [nom, regex, log, env_logger]
"trietest", "0.1.0", false, [radix_trie]
That does not look UNIX-y. Output format looks CSV-ish. It requires further parsing to be usable in shell scripting.
As a minimum there should be a command to dump string value without quotes.
Maybe tomlkit should not try to handle multiple queries in one run, instead providing a good interface to get/set just one value.
That is how I see the fragment above:
for i in $(find . -name 'Cargo.toml'); do
echo "P"
tomlkit -f "$i" -g package.name
tomlkit -f "$i" -g package.version
tomlkit -f "$i" --has-value package.license && echo has license || echo no licence
tomlkit -f "$i" -c dependencies
done
P
methodtest
0.1.0
no licence
nom
P
mtailtest
0.1.0
no lisence
mtaillib
P
multitail
0.1.0
no licence
term
notify
getopts
P
mtaillib
0.1.0
no license
notify
libc
...
This is meant to be a multi-platform tool. It's not just meant for Unix.
The comma+space is just the default output separator, it can be easily changed (as well as the default "true" and "false" values, no need for a script):
find . -name "Cargo.toml" | ./tomllib/target/debug/tomlkit -g package.name,package.version --has-value package.license -c dependencies --set-true "has license" --set-false "no license" --separator $'\n'
"tomllib"
"0.1.2"
has license
[nom, regex, log, env_logger, pirate, csv]
"test"
"0.1.0"
no license
[foo]
"nom"
"1.2.3"
has license
[regex, lazy_static]
"mio"
"0.5.0"
has license
[log, nix, libc, slab, time, bytes, winapi, miow, net2]
There's no way to add arbitrary "P"s before the output of each file and I don't really see the need for the ability.
The quotes are there because they're part of the value: 123 is different from "123" which is different from 'foo' which is different from """foo""" which is different from '''foo''', but it should be very simple to add a "strip-quotes" option.
The output of --get-children/-c looks the way it does because it's an array (it's just like a json array or a TOML array, or a CSV with braces). Without at least some sort of "end-of-array" character there would be no way to tell where an array of values ended and a new value or array began. Well I guess I can make the first value of an array be an integer representing the length the following array. But there is no way around having to parse a variable length list.
The set-value command takes a csv (consider it a tuple) of the key followed by the value followed by the type. If you know of a simpler, easier way to specify multiple values I'd like to hear it. Three separate options for key, value, and type would be much more cumbersome.
I think being able to specify multiple keys for each command makes the tool much easier to use. You don't need to write a script to process multiple files. Not to mention that processing each command separately is very inefficient since each invocation of tomlkit will reparse the document. Imagine needing to parse out a couple dozen values from a large number of TOML documents. One invocation will literally be a couple dozen times faster than one parse per command.
The argument parsing library I chose allowed me to get up and running very quickly, but it ends up lacking a couple important features that I really want to use. For this release I'm just going to stick with it so I can get the first version out the door sooner rather than later, but the next release I'm switching to a different library or fixing up the one I'm using.
One more idea: users specify format string which tomlkit fills in, like in git log --pretty=format:.... And if you allow users to select separators, it's good idea to use zero byte (for xargs -0) sometimes, so there can be tomlkit -0 option.
Your first example may be like this with the format string:
find ... | tomlkit @- --format='"{package.name}", "{package.version}", {package.license} [{dependencies, }]'
Suggestions to file specification CLI parameters:
-- read toml file to stdin, dump changed file to stdout (if there are write commands)- (default) - same as
- MyFile.toml- read specific file, overwrite changes also to this file (maybe with some auto-backup)@ListOfFiles.txt- process each file from the list (separated by\n-s unless-0option given), write changes to the named files as well@-- like above, but read file list from stdin.
I expect most usages of tomlkit may be just getting/setting one/few values or appending/enumerating/removing from some list in one file.
Shall I try to write up a more detailed vision of how I see CLI design of such tool?
Those are all good ideas, but other than the format string, my argument parsing library isn't able to do them. The format string is nice, but would be a lot of work. I think I can add a "strip-quotes" option and options for "array-start", "array-end" and "array-separator" (and maybe an option for "array-length" where instead of "array-start" and "array-end" it prints the length of the array first) very easily and cover most use cases. I can target being as good as commands that have been around for decades in the second release :stuck_out_tongue:.
Hey! Any interest still in this project? It would be extremely helpful for all of the rust community
Yes, I have interest in this project it's just that I've been really busy lately. I'll probably start it up again soon.
@joelself I'm still a rust beginner but I'm interested to help. Where can I start, what are the important issues/features to work on?
