assert_cli icon indicating copy to clipboard operation
assert_cli copied to clipboard

Published 20 hours ago • verified publisher icon assert-rs

Future of assert_cli

Open killercup opened this issue 7 years ago • 19 comments

@epage asked in https://github.com/killercup/assert_cli/pull/40#issuecomment-331887232:

I'm curious, what do you see needing done before moving this to 1.0? While I should get back to my regularly schedule project, I would like to also help work towards more crates hitting 1.0.

Good question! I'd like to make the arg parsing a bit leaner (cf. #25 and #32) and maybe improve the error output (#17, #18, #30).

My general goal is to make a library that makes writing tests like these easy and fun. It'll probably involve generating some test boilerplate code, and I'm not entirely sure what I want that API to look like. I'm not even sure if that's in scope for assert_cli. What do you think?

EDIT

The building blocks for assert_cli 1.0 are

  • https://github.com/assert-rs/assert_fs/
  • https://github.com/assert-rs/assert_cmd/
  • https://github.com/assert-rs/predicates-rs/
  • https://github.com/Freyskeyd/environment ?

killercup avatar Sep 25 '17 14:09 killercup

tl;dr After considering this for the last couple days (and digging into the issues), I feel like PR #43 is sufficient for releasing 1.0.

I feel the main categories of 1.0 blockers

  • API changes
    • ~~#32~~ duct mostly buys us
      • piping, see my comments on waltz.
      • advanced command lines parsing, see comment on #25
    • ~~#25~~, see comment on it
    • ~~#18~~ I feel like this could be done without breaking compatibility
  • Usable.
    • ~~#17~~ from my own testing, this seems like there is enough output
    • ~~#18~~ I feel like a default output policy should be sufficient for release
    • ~~#30~~ Closed, I don't quite remember enough about it to be actionable.
    • I audited the output in PR #43

RE Waltz's tests

While I see a lot of value in what you're doing with waltz, I feel like that has a longer path to stabilization than the core of assert_cli.

assert_cli does a great job of doing one thing very well, testing short lived programs. If we take the approach of trying to have assert_cli cover additional use cases, whether its the side effect testing done in waltz or the orchestrating multiple programs with duct's piping, I feel like it might detract from its primary strengths.

Also, thinking about it some more, side effect testing might have value outside of running programs. Waltz's test harness could be really nice for also wrapping functions.

epage avatar Sep 27 '17 03:09 epage

Thanks for the comprehensive list!

Okay, let's keep this crate small! I'll just have to come up with a name for the crate that does the waltz cli spec thing ;)

I'd like to ping some more of assert_cli's users about feedback on 1.0.

Ed Page [email protected] schrieb am Mi. 27. Sep. 2017 um 05:01:

tl;dr After considering this for the last couple days (and digging into the issues), I feel like PR #43 https://github.com/killercup/assert_cli/pull/43 is sufficient for releasing 1.0.

I feel the main categories of 1.0 blockers

RE Waltz's tests

While I see a lot of value in what you're doing with waltz, I feel like that has a longer path to stabilization than the core of assert_cli.

assert_cli does a great job of doing one thing very well, testing short lived programs. If we take the approach of trying to have assert_cli cover additional use cases, whether its the side effect testing done in waltz or the orchestrating multiple programs with duct's piping, I feel like it might detract from its primary strengths.

Also, thinking about it some more, side effect testing might have value outside of running programs. Waltz's test harness could be really nice for also wrapping functions.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/killercup/assert_cli/issues/41#issuecomment-332395177, or mute the thread https://github.com/notifications/unsubscribe-auth/AABOX8GuxZ_uyShM6clH3PKNWmMVluJoks5smbqRgaJpZM4Pi0Dm .

killercup avatar Sep 27 '17 08:09 killercup

Pinging some people who used assert_cli in their projects (found using Github search):

We are thinking about doing a 1.0 release and would love to hear your opinion on the crate and ideas for its future!

  • @livioribeiro who used it for cargo-readme integration tests
  • @bjgill who updated cargo-edit to assert_cli 0.4
  • @mattgathu who quite recently used assert_cli in rget
  • @jxson who uses it in some tests of to-directory

killercup avatar Sep 27 '17 09:09 killercup

Hello @killercup @epage thank you for building such an amazing crate!! It has been really helpful.

I have been using it to test a rust cli app I'm writing using the assert_cli::Assert::main_binary interface. This is part of my contribution for Increasing Rust's Reach

I was surprised to find assert_cli::Assert::main_binary missing in your docs examples considering how helpful it is!! I had to dig a little deeper into your docs to get it.

I'm planning to:

  • Write a Pull Request to enhance the docs and add example for assert_cli::Assert::main_binary and other useful interfaces.
  • Write a How to test Rust CLI apps blog post, with introduction to writing tests with assert_cli

Let me know what you think!

mattgathu avatar Oct 01 '17 07:10 mattgathu

About future ideas, I have been wondering how feasible it is to test for colored output?

If my CLI binary prints out colored text, can I test this?

What are your thoughts on this.

mattgathu avatar Oct 01 '17 07:10 mattgathu

I was surprised to find assert_cli::Assert::main_binary missing in your docs examples considering how helpful it is!! I had to dig a little deeper into your docs to get it.

Yeah, I suspect thats from it being developed later and that it makes it harder to run the examples as tests. I agree that more attention should be drawn to it.

Let me know what you think!

Sounds great! The main thing to watch out for is not reducing coverage.

epage avatar Oct 02 '17 22:10 epage

About future ideas, I have been wondering how feasible it is to test for colored output? If my CLI binary prints out colored text, can I test this?

Most programs detect if they are outputting to a console and don't print color if they are. Some have a "force color" option.

So either we'd need to figure out how to fake being a console or clients need to have a force color option and set it.

When forced, I assume color just shows up in stdout as weird characters. Can you just manually enter those characters in to test it? Do you have a proposal for what an API might look like for more advanced color testing?

epage avatar Oct 02 '17 22:10 epage

Once #46 goes in, should we also need to wait for Environment to hit 1.0?

epage avatar Oct 12 '17 14:10 epage

Good question. I'm okay with publishing this with a re-export of the 0.1. I also think that Environment could be 1.0 already.

killercup avatar Oct 12 '17 15:10 killercup

I would like to have a way to remove variable from Environment before 1.0. something like:

let env: Environment = vec![("foo", "bar"), ("bar", "baz")].into();
...
env.remove("foo");

I would also like to have a has/check method:

let env = Environment::inherit();

if env.has("foo") && env.check_if("bar", "baz") {
    ...
}

Having a macro style would be a good way to use it:

let e = inherit_env!(
    "foo" -> "bar", // <--- don't know if I can do that
    "bar" -> "baz",
);

let e = empty_env!();

assert_cli::Assert::command(&["printenv"])
     .with_env(inherit_env!( "bar" -> "baz" ))
     .stdout().contain("bar=baz")
     .unwrap();

Freyskeyd avatar Oct 12 '17 17:10 Freyskeyd 

let e = inherit_env!(
    "foo" -> "bar", // <--- don't know if I can do that
    "bar" -> "baz",
);

I think I generally see people use the => syntax when they experiment with these type of things, for example https://github.com/kmcallister/literator

epage avatar Oct 12 '17 17:10 epage

@epage yep that's looking better.

Another question is: Macro style must expend to compile version? or not?

inherit_env! { "bar" => "baz" }

Becomes

Environment::inherit().insert("bar", "baz").compile()
or just
Environment::inherit().insert("bar", "baz")

But it's a discussion on Environment repository I think^^

Freyskeyd avatar Oct 12 '17 17:10 Freyskeyd

@killercup just recently came across https://github.com/emk/subtitles-rs. Looks like some of its building blocks are similar to the type of thing you are trying to do with waltz.

I've not explored it enough to know all of the implications of TestDir, if that concept is even compatible with my projects. I could see the idea of splitting the file system tests out into is own crate so the two can be composed together. I could see using the file system stuff in my lower level integration tests as well.

epage avatar Oct 26 '17 14:10 epage

I did a usability study of assert_cli at RustBeltRust

  • Discoverabiliy: Searching for "test command line" and "assert command line" did not return results on crates.io but "test cli" and "assert cli" did
  • stdout testing: The API looks weird when run through rustfmt because it puts each function on a separate line, so it clouds up what the predicate is operating on
    • Keep it, tools are meant to work for us
    • Go back to having a function per variant
    • Follow Environment and take in a predicate parameter
      • Easier to do multiple predicates (option to take in an array)
      • While it helps in places, I found it annoying in clap to access a bunch of different types
      • More linking is needed to help people find all of the relevant predicates

epage avatar Oct 26 '17 14:10 epage

I did a usability study of assert_cli at RustBeltRust

Cool! Tell all the awesome folks in Columbus who may know me I said hi!

Discoverabiliy

I think we still have a few keywords left we can add to Cargo.toml and "command line" might be a good choice :)

The API looks weird when run through rustfmt

Ha! I didn't even think of that! But it makes sense, as rustfmt can't know the two calls combined have some special meaning.

Go back to having a function per variant

I'm totally fine with that actually. We have 4 methods and two implementors, so in the end it's just 8 methods to cover all output asserts.

Follow Environment and take in a predicate parameter

You mean like .stderr(Output::contains("foo").and_contains("bar").and_is("baz"))?

But! At that point me might as well do .stderr(|x: OutputAssertThingy| -> bool { x.is("bar") || (x.contains("baz") || !x.is("bazinga")) }) to not reinvent boolean expressions. Which I think was something someone was asking for in another issue anyway (cf. #57).

(I'm a bit out of the loop and have only consumed issues on this repo through email for far too long, sorry)

Am 26.10.2017 um 16:34 schrieb Ed Page [email protected]:

I did a usability study of assert_cli at RustBeltRust

• Discoverabiliy: Searching for "test command line" and "assert command line" did not return results on crates.io but "test cli" and "assert cli" did • stdout testing: The API looks weird when run through rustfmt because it puts each function on a separate line, so it clouds up what the predicate is operating on • Keep it, tools are meant to work for us • Go back to having a function per variant • Follow Environment and take in a predicate parameter • Easier to do multiple predicates (option to take in an array) • While it helps in places, I found it annoying in clap to access a bunch of different types • More linking is needed to help people find all of the relevant predicates — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

killercup avatar Oct 26 '17 14:10 killercup

One thing in favor of accepting predicates as parameters is that someone wanted to pass multiple predicates at once which is more of an issue with contains and negated contains/is and not is.

epage avatar Oct 26 '17 15:10 epage

But! At that point me might as well do .stderr(|x: OutputAssertThingy| -> bool { x.is("bar") || (x.contains("baz") || !x.is("bazinga")) }) to not reinvent boolean expressions. Which I think was something someone was asking for in another issue anyway (cf. #57).

I think there is still value in specialized predicates for better error reporting. We could just use From to auto-convert a Fn into one of the predicates.

epage avatar Oct 27 '17 11:10 epage

We've started the major shift to assert_cli 1.0. assert_cmd is the spiritual successor to what assert_cli today. We plan to turn assert_cli into a higher level CLI testing crate like waltz tests.

The building blocks for assert_cli 1.0 are

  • https://github.com/assert-rs/assert_fs/
  • https://github.com/assert-rs/assert_cmd/
  • https://github.com/assert-rs/predicates-rs/
  • https://github.com/Freyskeyd/environment ?

The bad: it will probably take longer to stablize assert_cli because we need to re-evaluate the design and play with ideas

The good: the smaller pieces should be a lot easier to stablize.

epage avatar May 29 '18 21:05 epage

Started looking at how others test CLIs and for python, I only found https://bitheap.org/cram/

epage avatar May 31 '18 16:05 epage