stan-dev
manual testing framework
We aren't testing code that goes into the manual thoroughly and automatically enough. We need a framework where code citations in the manual are automatically extracted from running code and then pasted into the manual.
Extend this to example I/O. The introduction is riddled with references to stan_major=1, etc.
It looks like there are two main packages that are being used now. There's the listings and minted. I tried with minted because it does some useful things.
With minted, I was able to change the existing code blocks to be processed by minted.
Pros:
- full syntax highlighting; I think @jrnold was behind it!
- will be easy to include from files and specify start lines and end lines
- will be able to remove blank spaces up front easily
- there are options for syntax highlighting. Colors for keywords, etc.
Cons:
- relies on
pythonand thePygmentspackage, which is straightforward to install - it's slow. The manual currently builds in ~20 secs. With the minted package, it builds in ~2:20. That's without including files, etc. If we use a simpler lexer (just text instead of Stan), it still builds in ~2:00.
Thoughts?
I'm going to try listings as a comparison.
If anyone wants to try minted, use this branch: feature/issue-319-manual-testing.
To change the style of how the code is rendered, change line 211 of stan-manuals.sty:
\usemintedstyle{friendly}
I tried with \usemintedstyle{bw} and it'll generate the Stan code like it is in the manual now.
What's the output look like?
You mean it builds end to end from clean-all in 2 mins, or that each run of pdflatex takes 2 mins, so the whole thing takes like 10 mins+?
I'm not keen on dependencies, but then I added the fonts, so I can hardly object if someone else thinks we need code highlighting.
I really don't want too much highlighting---otherwise the code looks like an Xmas tree and I find it hard to read. Also, it might not print well if the colors aren't dark and high-contrast (not that I ever print anything). Also, the highlighting has to be correct. Inconsistent or incomplete highlighting is worse than none at all.
With the bw style, it looks like the existing manual.
I checked in a version that'll make it look like a Christmas tree. (I just did that to demonstrate that we could.)
It builds end to end from clean-all in 2 minutes and change.
On Fri, Sep 2, 2016 at 12:57 PM, Bob Carpenter [email protected] wrote:
What's the output look like?
You mean it builds end to end from clean-all in 2 mins, or that each run of pdflatex takes 2 mins, so the whole thing takes like 10 mins+?
I'm not keen on dependencies, but then I added the fonts, so I can hardly object if someone else thinks we need code highlighting.
I really don't want too much highlighting---otherwise the code looks like an Xmas tree and I find it hard to read. Also, it might not print well if the colors aren't dark and high-contrast (not that I ever print anything). Also, the highlighting has to be correct. Inconsistent or incomplete highlighting is worse than none at all.
— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/stan-dev/stan/issues/319#issuecomment-244430457, or mute the thread https://github.com/notifications/unsubscribe-auth/AAZ_F7csHaPLa4JIQdUXC3meBEV7WZnlks5qmFV0gaJpZM4BIN0F .
You committed a verison with \usemintedstyle{friendly}.
This is way too slow. It takes over 1s/page on my machine, and it needs multiple pass, so I estimated over half an hour and got bored waiting and gave up.
Maybe we could do this for release version, but for work-in-progress this is going to be a huge drag on my working on the manual.
I wrote this entire email between pages 110 and 297.
- Bob
On Sep 2, 2016, at 6:54 PM, Daniel Lee [email protected] wrote:
If anyone wants to try minted, use this branch: feature/issue-319-manual-testing.
To change the style of how the code is rendered, change line 211 of stan-manuals.sty: \usemintedstyle{friendly}
I tried with \usemintedstyle{bw} and it'll generate the Stan code like it is in the manual now.
— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or mute the thread.
What are you running on? Did you have to do something to juice Python?
- Bob
On Sep 2, 2016, at 7:00 PM, Daniel Lee [email protected] wrote:
With the
bwstyle, it looks like the existing manual.I checked in a version that'll make it look like a Christmas tree. (I just did that to demonstrate that we could.)
It builds end to end from clean-all in 2 minutes and change.
On Fri, Sep 2, 2016 at 12:57 PM, Bob Carpenter [email protected] wrote:
What's the output look like?
You mean it builds end to end from clean-all in 2 mins, or that each run of pdflatex takes 2 mins, so the whole thing takes like 10 mins+?
I'm not keen on dependencies, but then I added the fonts, so I can hardly object if someone else thinks we need code highlighting.
I really don't want too much highlighting---otherwise the code looks like an Xmas tree and I find it hard to read. Also, it might not print well if the colors aren't dark and high-contrast (not that I ever print anything). Also, the highlighting has to be correct. Inconsistent or incomplete highlighting is worse than none at all.
— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/stan-dev/stan/issues/319#issuecomment-244430457, or mute the thread https://github.com/notifications/unsubscribe-auth/AAZ_F7csHaPLa4JIQdUXC3meBEV7WZnlks5qmFV0gaJpZM4BIN0F .
— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or mute the thread.
No idea. I'm using python 2.7.10 and I just installed Pygments today.
On Sep 2, 2016, at 1:03 PM, Bob Carpenter [email protected] wrote:
What are you running on? Did you have to do something to juice Python?
- Bob
On Sep 2, 2016, at 7:00 PM, Daniel Lee [email protected] wrote:
With the
bwstyle, it looks like the existing manual.I checked in a version that'll make it look like a Christmas tree. (I just did that to demonstrate that we could.)
It builds end to end from clean-all in 2 minutes and change.
On Fri, Sep 2, 2016 at 12:57 PM, Bob Carpenter [email protected] wrote:
What's the output look like?
You mean it builds end to end from clean-all in 2 mins, or that each run of pdflatex takes 2 mins, so the whole thing takes like 10 mins+?
I'm not keen on dependencies, but then I added the fonts, so I can hardly object if someone else thinks we need code highlighting.
I really don't want too much highlighting---otherwise the code looks like an Xmas tree and I find it hard to read. Also, it might not print well if the colors aren't dark and high-contrast (not that I ever print anything). Also, the highlighting has to be correct. Inconsistent or incomplete highlighting is worse than none at all.
— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/stan-dev/stan/issues/319#issuecomment-244430457, or mute the thread https://github.com/notifications/unsubscribe-auth/AAZ_F7csHaPLa4JIQdUXC3meBEV7WZnlks5qmFV0gaJpZM4BIN0F .
— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or mute the thread.
— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or mute the thread.
I really hate that output. Maybe if all the colors were darker and less opposite each other and there was no bold to mess up weight I might be able to learn to accept it if everyone else really really wants highlighting.
Seems to have failed somehow at the end, but the real problem is that it takes a long long time.
> make clean-all
> time make manual
...
make[1]: *** No rule to make target `stan-reference.w18'. Stop.
Latexmk: All targets (stan-reference.pdf) are up-to-date
building src/docs/stan-reference/stan-functions.txt
cp src/docs/stan-reference/stan-reference.pdf doc/stan-reference-2.11.0.pdf; cp src/docs/stan-reference/stan-functions.txt doc/stan-functions-2.11.0.txt;
real 17m16.612s
user 15m28.673s
sys 1m39.578s
Running with make -j4 doesn't seem to be any faster.
- Bob
I saw that error too.
Changing the style to bw helps, but yes, the time is a killer. I'll try the listings package out. Less dependencies is better.
On Sep 2, 2016, at 1:23 PM, Bob Carpenter [email protected] wrote:
I really hate that output. Maybe if all the colors were darker and less opposite each other and there was no bold to mess up weight I might be able to learn to accept it if everyone else really really wants highlighting.
Seems to have failed somehow at the end, but the real problem is that it takes a long long time.
> make clean-all > time make manual ... make[1]: *** No rule to make target `stan-reference.w18'. Stop. Latexmk: All targets (stan-reference.pdf) are up-to-date building src/docs/stan-reference/stan-functions.txt cp src/docs/stan-reference/stan-reference.pdf doc/stan-reference-2.11.0.pdf; cp src/docs/stan-reference/stan-functions.txt doc/stan-functions-2.11.0.txt; real 17m16.612s user 15m28.673s sys 1m39.578sRunning with make -j4 doesn't seem to be any faster.
- Bob — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or mute the thread.
Why do you want to do this? Is it that people want syntax highlighting? If it's not subtle (by which I mean you can barely see it at arm's length), I'm going to object.
- Bob
On Sep 2, 2016, at 7:30 PM, Daniel Lee [email protected] wrote:
I saw that error too.
Changing the style to bw helps, but yes, the time is a killer. I'll try the listings package out. Less dependencies is better.
On Sep 2, 2016, at 1:23 PM, Bob Carpenter [email protected] wrote:
I really hate that output. Maybe if all the colors were darker and less opposite each other and there was no bold to mess up weight I might be able to learn to accept it if everyone else really really wants highlighting.
Seems to have failed somehow at the end, but the real problem is that it takes a long long time.
> make clean-all > time make manual ... make[1]: *** No rule to make target `stan-reference.w18'. Stop. Latexmk: All targets (stan-reference.pdf) are up-to-date building src/docs/stan-reference/stan-functions.txt cp src/docs/stan-reference/stan-reference.pdf doc/stan-reference-2.11.0.pdf; cp src/docs/stan-reference/stan-functions.txt doc/stan-functions-2.11.0.txt; real 17m16.612s user 15m28.673s sys 1m39.578sRunning with make -j4 doesn't seem to be any faster.
- Bob — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or mute the thread.
— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or mute the thread.
I want to split out the Stan code chunks into valid Stan programs. We can include bits of it in the manual. Then, we'd be able to test that the chunks in the manual compile and we can generate output for the manual so we can keep up to date with output.
On Fri, Sep 2, 2016 at 2:00 PM, Bob Carpenter [email protected] wrote:
Why do you want to do this? Is it that people want syntax highlighting? If it's not subtle (by which I mean you can barely see it at arm's length), I'm going to object.
- Bob
On Sep 2, 2016, at 7:30 PM, Daniel Lee [email protected] wrote:
I saw that error too.
Changing the style to bw helps, but yes, the time is a killer. I'll try the listings package out. Less dependencies is better.
On Sep 2, 2016, at 1:23 PM, Bob Carpenter [email protected] wrote:
I really hate that output. Maybe if all the colors were darker and less opposite each other and there was no bold to mess up weight I might be able to learn to accept it if everyone else really really wants highlighting.
Seems to have failed somehow at the end, but the real problem is that it takes a long long time.
> make clean-all > time make manual ... make[1]: *** No rule to make target `stan-reference.w18'. Stop. Latexmk: All targets (stan-reference.pdf) are up-to-date building src/docs/stan-reference/stan-functions.txt cp src/docs/stan-reference/stan-reference.pdf doc/stan-reference-2.11.0.pdf; cp src/docs/stan-reference/stan-functions.txt doc/stan-functions-2.11.0.txt; real 17m16.612s user 15m28.673s sys 1m39.578sRunning with make -j4 doesn't seem to be any faster.
- Bob — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or mute the thread.
— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or mute the thread.
— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/stan-dev/stan/issues/319#issuecomment-244445990, or mute the thread https://github.com/notifications/unsubscribe-auth/AAZ_F7Y2oN8Su82oyWZeUKdq3ZUK2sViks5qmGQzgaJpZM4BIN0F .
OK. I could live with runtime for that functionality. But the style has to be beter.
How would we markup the files that get included?
Also, this seems like a very low priority compared to all the bugfixes and critical refactoring issues we have to do.
- Bob
On Sep 2, 2016, at 8:14 PM, Daniel Lee [email protected] wrote:
I want to split out the Stan code chunks into valid Stan programs. We can include bits of it in the manual. Then, we'd be able to test that the chunks in the manual compile and we can generate output for the manual so we can keep up to date with output.
On Fri, Sep 2, 2016 at 2:00 PM, Bob Carpenter [email protected] wrote:
Why do you want to do this? Is it that people want syntax highlighting? If it's not subtle (by which I mean you can barely see it at arm's length), I'm going to object.
- Bob
On Sep 2, 2016, at 7:30 PM, Daniel Lee [email protected] wrote:
I saw that error too.
Changing the style to bw helps, but yes, the time is a killer. I'll try the listings package out. Less dependencies is better.
On Sep 2, 2016, at 1:23 PM, Bob Carpenter [email protected] wrote:
I really hate that output. Maybe if all the colors were darker and less opposite each other and there was no bold to mess up weight I might be able to learn to accept it if everyone else really really wants highlighting.
Seems to have failed somehow at the end, but the real problem is that it takes a long long time.
> make clean-all > time make manual ... make[1]: *** No rule to make target `stan-reference.w18'. Stop. Latexmk: All targets (stan-reference.pdf) are up-to-date building src/docs/stan-reference/stan-functions.txt cp src/docs/stan-reference/stan-reference.pdf doc/stan-reference-2.11.0.pdf; cp src/docs/stan-reference/stan-functions.txt doc/stan-functions-2.11.0.txt; real 17m16.612s user 15m28.673s sys 1m39.578sRunning with make -j4 doesn't seem to be any faster.
- Bob — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or mute the thread.
— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or mute the thread.
— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/stan-dev/stan/issues/319#issuecomment-244445990, or mute the thread https://github.com/notifications/unsubscribe-auth/AAZ_F7Y2oN8Su82oyWZeUKdq3ZUK2sViks5qmGQzgaJpZM4BIN0F .
— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or mute the thread.