docsh
docsh copied to clipboard
Published
20 hours ago •
erszcz
erszcz
OTP EDoc support analysis
While OTP in general does not use EDoc for documentation, parts of it do so. Since docsh has to be tested against a significant number of documents to verify correctness and accuracy it's natural to use the resources at our fingertips.
Travis job https://travis-ci.org/erszcz/docsh/builds/351337274 shows the results of running docsh on these OTP modules which do have some EDoc documentation (some applications were omitted, but can be easily added to the analysis). The results are as follows:
- 20.2 -
docsh_edocfails for 12 out of 317 modules - 19.3 -
docsh_edocfails for 12 out of 317 modules - 18.3 -
docsh_edocfails for 10 out of 312 modules
The interesting part, though, is that respectively only 2 cases for 20.2 (cerl, erl_syntax), 3 for 19.3 (cerl, erl_syntax, gl) and 3 for 18.3 (cerl, erl_syntax, gl) are caused by bugs in docsh - the rest is due to EDoc, the library, not handling properly EDoc, the comment, or more likely by human mistake in writing the documentation.
Specifically, the error when processing gl seems to be a Unicode character breaking iolist_to_binary. This might have changed due to Unicode handling changes in 20+.
Log for https://github.com/erlang/otp/commit/8313eae72dd170d2f4502d225b74dbf543198313 at https://github.com/erszcz/docsh/commit/a2e0ebde8508a4db853fcf7fab28c807821f7c5a:
$ r3 ct --suite test/completeness_SUITE.erl
===> Verifying dependencies...
===> Compiling docsh
src/docsh_lib.erl:168: Warning: missing specification for function compile_info_source_file/1
src/docsh_lib.erl:229: Warning: erlang:get_stacktrace/0: deprecated; use the new try/catch syntax for retrieving the stack backtrace
src/docsh_lib.erl:251: Warning: missing specification for function get_beam/1
src/docsh_tracer.erl:3: Warning: export_all flag enabled - all functions will be exported
src/docsh_syntax.erl:45: Warning: erlang:get_stacktrace/0: deprecated; use the new try/catch syntax for retrieving the stack backtrace
src/docsh_edoc.erl:26: Warning: missing specification for function to_internal/2
src/docsh_edoc.erl:39: Warning: erlang:get_stacktrace/0: deprecated; use the new try/catch syntax for retrieving the stack backtrace
src/docsh.erl:126: Warning: erlang:get_stacktrace/0: deprecated; use the new try/catch syntax for retrieving the stack backtrace
_build/test/lib/docsh/test/docsh_helpers.erl:2: Warning: export_all flag enabled - all functions will be exported
_build/test/lib/docsh/test/install_SUITE.erl:0: Warning: missing specification for function test/0
_build/test/lib/docsh/test/install_SUITE.erl:2: Warning: export_all flag enabled - all functions will be exported
_build/test/lib/docsh/test/edoc_SUITE.erl:0: Warning: missing specification for function test/0
_build/test/lib/docsh/test/edoc_SUITE.erl:2: Warning: export_all flag enabled - all functions will be exported
_build/test/lib/docsh/test/edoc_example.erl:6: Warning: type l(_) is unused
_build/test/lib/docsh/test/edoc_example.erl:7: Warning: type l() is unused
_build/test/lib/docsh/test/docsh_SUITE.erl:0: Warning: missing specification for function test/0
_build/test/lib/docsh/test/docsh_SUITE.erl:2: Warning: export_all flag enabled - all functions will be exported
_build/test/lib/docsh/test/docsh_docs_v1_SUITE.erl:0: Warning: missing specification for function test/0
_build/test/lib/docsh/test/docsh_docs_v1_SUITE.erl:2: Warning: export_all flag enabled - all functions will be exported
_build/test/lib/docsh/test/completeness_SUITE.erl:2: Warning: export_all flag enabled - all functions will be exported
===> Running Common Test suites...
%%% completeness_SUITE ==> sanity_check: OK
----------------------------------------------------
2018-04-26 18:30:36.824
can't get source for erts erlang: {error,{no_beam_file,preloaded}}
----------------------------------------------------
2018-04-26 18:30:36.824
can't get source for erts erl_prim_loader: {error,{no_beam_file,preloaded}}
----------------------------------------------------
2018-04-26 18:30:36.824
can't get source for erts erts_internal: {error,{no_beam_file,preloaded}}
----------------------------------------------------
2018-04-26 18:30:36.824
can't get source for erts init: {error,{no_beam_file,preloaded}}
----------------------------------------------------
2018-04-26 18:30:36.824
can't get source for erts otp_ring0: {error,{no_beam_file,preloaded}}
----------------------------------------------------
2018-04-26 18:30:36.824
can't get source for erts erts_code_purger: {error,{no_beam_file,preloaded}}
----------------------------------------------------
2018-04-26 18:30:36.825
can't get source for erts prim_buffer: {error,{no_beam_file,preloaded}}
----------------------------------------------------
2018-04-26 18:30:36.825
can't get source for erts prim_eval: {error,{no_beam_file,preloaded}}
----------------------------------------------------
2018-04-26 18:30:36.825
can't get source for erts prim_file: {error,{no_beam_file,preloaded}}
----------------------------------------------------
2018-04-26 18:30:36.825
can't get source for erts prim_inet: {error,{no_beam_file,preloaded}}
----------------------------------------------------
2018-04-26 18:30:36.825
can't get source for erts prim_zip: {error,{no_beam_file,preloaded}}
----------------------------------------------------
2018-04-26 18:30:36.825
can't get source for erts zlib: {error,{no_beam_file,preloaded}}
=ERROR REPORT==== 26-Apr-2018::18:30:47 ===
2550- fatal: {endtag_does_not_match,{was,doc,should_have_been,p}}
=ERROR REPORT==== 26-Apr-2018::18:30:47 ===
2550- fatal: {endtag_does_not_match,{was,p,should_have_been,'N'}}
=ERROR REPORT==== 26-Apr-2018::18:30:48 ===
2550- fatal: {endtag_does_not_match,{was,code,should_have_been,ocde}}
----------------------------------------------------
2018-04-26 18:30:59.658
failed: [{ct_slave,
{error,error,
[{edoc_tags,parse_tag,4,[{file,"edoc_tags.erl"},{line,286}]},
{edoc_tags,parse_tags,5,[{file,"edoc_tags.erl"},{line,269}]},
{edoc_extract,get_tags,6,
[{file,"edoc_extract.erl"},{line,586}]},
{edoc_extract,get_tags,6,
[{file,"edoc_extract.erl"},{line,589}]},
{edoc_extract,source1,5,
[{file,"edoc_extract.erl"},{line,132}]},
{docsh_edoc,to_internal,2,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/src/docsh_edoc.erl"},
{line,32}]},
{completeness_SUITE,
'-docsh_works_for_each_file_with_edoc/1-lc$^3/1-3-',1,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/test/completeness_SUITE.erl"},
{line,37}]},
{completeness_SUITE,
'-docsh_works_for_each_file_with_edoc/1-lc$^4/1-4-',3,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/test/completeness_SUITE.erl"},
{line,37}]}]}},
{unix_telnet,
{error,error,
[{edoc_tags,parse_tag,4,[{file,"edoc_tags.erl"},{line,286}]},
{edoc_tags,parse_tags,5,[{file,"edoc_tags.erl"},{line,272}]},
{edoc_extract,get_tags,6,
[{file,"edoc_extract.erl"},{line,586}]},
{edoc_extract,get_tags,6,
[{file,"edoc_extract.erl"},{line,589}]},
{edoc_extract,source1,5,
[{file,"edoc_extract.erl"},{line,132}]},
{docsh_edoc,to_internal,2,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/src/docsh_edoc.erl"},
{line,32}]},
{completeness_SUITE,
'-docsh_works_for_each_file_with_edoc/1-lc$^3/1-3-',1,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/test/completeness_SUITE.erl"},
{line,37}]},
{completeness_SUITE,
'-docsh_works_for_each_file_with_edoc/1-lc$^4/1-4-',3,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/test/completeness_SUITE.erl"},
{line,37}]}]}},
{ct_util,
{error,error,
[{edoc_tags,parse_tag,4,[{file,"edoc_tags.erl"},{line,286}]},
{edoc_tags,parse_tags,5,[{file,"edoc_tags.erl"},{line,272}]},
{edoc_extract,get_tags,6,
[{file,"edoc_extract.erl"},{line,586}]},
{edoc_extract,get_tags,6,
[{file,"edoc_extract.erl"},{line,589}]},
{edoc_extract,source1,5,
[{file,"edoc_extract.erl"},{line,132}]},
{docsh_edoc,to_internal,2,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/src/docsh_edoc.erl"},
{line,32}]},
{completeness_SUITE,
'-docsh_works_for_each_file_with_edoc/1-lc$^3/1-3-',1,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/test/completeness_SUITE.erl"},
{line,37}]},
{completeness_SUITE,
'-docsh_works_for_each_file_with_edoc/1-lc$^4/1-4-',3,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/test/completeness_SUITE.erl"},
{line,37}]}]}},
{ct_run,
{error,error,
[{edoc_tags,parse_tag,4,[{file,"edoc_tags.erl"},{line,286}]},
{edoc_tags,parse_tags,5,[{file,"edoc_tags.erl"},{line,272}]},
{edoc_extract,get_tags,6,
[{file,"edoc_extract.erl"},{line,586}]},
{edoc_extract,get_tags,6,
[{file,"edoc_extract.erl"},{line,589}]},
{edoc_extract,source1,5,
[{file,"edoc_extract.erl"},{line,132}]},
{docsh_edoc,to_internal,2,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/src/docsh_edoc.erl"},
{line,32}]},
{completeness_SUITE,
'-docsh_works_for_each_file_with_edoc/1-lc$^3/1-3-',1,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/test/completeness_SUITE.erl"},
{line,37}]},
{completeness_SUITE,
'-docsh_works_for_each_file_with_edoc/1-lc$^4/1-4-',3,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/test/completeness_SUITE.erl"},
{line,37}]}]}},
{ct_repeat,
{error,error,
[{edoc_tags,parse_tag,4,[{file,"edoc_tags.erl"},{line,286}]},
{edoc_tags,parse_tags,5,[{file,"edoc_tags.erl"},{line,269}]},
{edoc_extract,get_tags,6,
[{file,"edoc_extract.erl"},{line,586}]},
{edoc_extract,source1,5,
[{file,"edoc_extract.erl"},{line,132}]},
{docsh_edoc,to_internal,2,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/src/docsh_edoc.erl"},
{line,32}]},
{completeness_SUITE,
'-docsh_works_for_each_file_with_edoc/1-lc$^3/1-3-',1,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/test/completeness_SUITE.erl"},
{line,37}]},
{completeness_SUITE,
'-docsh_works_for_each_file_with_edoc/1-lc$^4/1-4-',3,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/test/completeness_SUITE.erl"},
{line,37}]},
{completeness_SUITE,docsh_works_for_each_file_with_edoc,1,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/test/completeness_SUITE.erl"},
{line,36}]}]}},
{ct_logs,
{error,error,
[{edoc_tags,parse_tag,4,[{file,"edoc_tags.erl"},{line,286}]},
{edoc_tags,parse_tags,5,[{file,"edoc_tags.erl"},{line,272}]},
{edoc_extract,get_tags,6,
[{file,"edoc_extract.erl"},{line,586}]},
{edoc_extract,get_tags,6,
[{file,"edoc_extract.erl"},{line,589}]},
{edoc_extract,source1,5,
[{file,"edoc_extract.erl"},{line,132}]},
{docsh_edoc,to_internal,2,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/src/docsh_edoc.erl"},
{line,32}]},
{completeness_SUITE,
'-docsh_works_for_each_file_with_edoc/1-lc$^3/1-3-',1,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/test/completeness_SUITE.erl"},
{line,37}]},
{completeness_SUITE,
'-docsh_works_for_each_file_with_edoc/1-lc$^4/1-4-',3,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/test/completeness_SUITE.erl"},
{line,37}]}]}},
{ct_gen_conn,
{error,error,
[{edoc_tags,parse_tag,4,[{file,"edoc_tags.erl"},{line,286}]},
{edoc_tags,parse_tags,5,[{file,"edoc_tags.erl"},{line,269}]},
{edoc_extract,get_tags,6,
[{file,"edoc_extract.erl"},{line,586}]},
{edoc_extract,get_tags,6,
[{file,"edoc_extract.erl"},{line,589}]},
{edoc_extract,source1,5,
[{file,"edoc_extract.erl"},{line,132}]},
{docsh_edoc,to_internal,2,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/src/docsh_edoc.erl"},
{line,32}]},
{completeness_SUITE,
'-docsh_works_for_each_file_with_edoc/1-lc$^3/1-3-',1,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/test/completeness_SUITE.erl"},
{line,37}]},
{completeness_SUITE,
'-docsh_works_for_each_file_with_edoc/1-lc$^4/1-4-',3,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/test/completeness_SUITE.erl"},
{line,37}]}]}},
{ct_framework,
{error,error,
[{edoc_tags,parse_tag,4,[{file,"edoc_tags.erl"},{line,286}]},
{edoc_tags,parse_tags,5,[{file,"edoc_tags.erl"},{line,272}]},
{edoc_extract,get_tags,6,
[{file,"edoc_extract.erl"},{line,586}]},
{edoc_extract,get_tags,6,
[{file,"edoc_extract.erl"},{line,589}]},
{edoc_extract,source1,5,
[{file,"edoc_extract.erl"},{line,132}]},
{docsh_edoc,to_internal,2,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/src/docsh_edoc.erl"},
{line,32}]},
{completeness_SUITE,
'-docsh_works_for_each_file_with_edoc/1-lc$^3/1-3-',1,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/test/completeness_SUITE.erl"},
{line,37}]},
{completeness_SUITE,
'-docsh_works_for_each_file_with_edoc/1-lc$^4/1-4-',3,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/test/completeness_SUITE.erl"},
{line,37}]}]}},
{ct,{error,error,
[{edoc_tags,parse_tag,4,[{file,"edoc_tags.erl"},{line,286}]},
{edoc_tags,parse_tags,5,[{file,"edoc_tags.erl"},{line,272}]},
{edoc_extract,get_tags,6,
[{file,"edoc_extract.erl"},{line,586}]},
{edoc_extract,get_tags,6,
[{file,"edoc_extract.erl"},{line,589}]},
{edoc_extract,source1,5,
[{file,"edoc_extract.erl"},{line,132}]},
{docsh_edoc,to_internal,2,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/src/docsh_edoc.erl"},
{line,32}]},
{completeness_SUITE,
'-docsh_works_for_each_file_with_edoc/1-lc$^3/1-3-',1,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/test/completeness_SUITE.erl"},
{line,37}]},
{completeness_SUITE,
'-docsh_works_for_each_file_with_edoc/1-lc$^4/1-4-',3,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/test/completeness_SUITE.erl"},
{line,37}]}]}},
{cerl_trees,
{error,error,
[{edoc_tags,parse_tag,4,[{file,"edoc_tags.erl"},{line,286}]},
{edoc_tags,parse_tags,5,[{file,"edoc_tags.erl"},{line,272}]},
{edoc_extract,get_tags,6,
[{file,"edoc_extract.erl"},{line,586}]},
{edoc_extract,get_tags,6,
[{file,"edoc_extract.erl"},{line,589}]},
{edoc_extract,source1,5,
[{file,"edoc_extract.erl"},{line,132}]},
{docsh_edoc,to_internal,2,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/src/docsh_edoc.erl"},
{line,32}]},
{completeness_SUITE,
'-docsh_works_for_each_file_with_edoc/1-lc$^3/1-3-',1,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/test/completeness_SUITE.erl"},
{line,37}]},
{completeness_SUITE,
'-docsh_works_for_each_file_with_edoc/1-lc$^4/1-4-',3,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/test/completeness_SUITE.erl"},
{line,37}]}]}},
{cerl,
{error,function_clause,
[{docsh_edoc_xmerl,unwrap_inline,
[[[]]],
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/src/docsh_edoc_xmerl.erl"},
{line,102}]},
{docsh_edoc_xmerl,'#element#',5,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/src/docsh_edoc_xmerl.erl"},
{line,70}]},
{xmerl,tagdef,5,[{file,"xmerl.erl"},{line,261}]},
{xmerl,export_content,2,[{file,"xmerl.erl"},{line,199}]},
{xmerl,export_content,2,[{file,"xmerl.erl"},{line,191}]},
{xmerl,export_content,2,[{file,"xmerl.erl"},{line,199}]},
{xmerl,export_content,2,[{file,"xmerl.erl"},{line,191}]},
{xmerl,export_content,2,[{file,"xmerl.erl"},{line,199}]}]}},
{edoc_doclet,
{error,error,
[{edoc_tags,parse_tag,4,[{file,"edoc_tags.erl"},{line,286}]},
{edoc_tags,parse_tags,5,[{file,"edoc_tags.erl"},{line,272}]},
{edoc_extract,get_tags,6,
[{file,"edoc_extract.erl"},{line,586}]},
{edoc_extract,get_tags,6,
[{file,"edoc_extract.erl"},{line,589}]},
{edoc_extract,source1,5,
[{file,"edoc_extract.erl"},{line,132}]},
{docsh_edoc,to_internal,2,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/src/docsh_edoc.erl"},
{line,32}]},
{completeness_SUITE,
'-docsh_works_for_each_file_with_edoc/1-lc$^3/1-3-',1,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/test/completeness_SUITE.erl"},
{line,37}]},
{completeness_SUITE,
'-docsh_works_for_each_file_with_edoc/1-lc$^4/1-4-',3,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/test/completeness_SUITE.erl"},
{line,37}]}]}},
{hipe_unified_loader,
{error,error,
[{edoc_tags,parse_tag,4,[{file,"edoc_tags.erl"},{line,286}]},
{edoc_tags,parse_tags,5,[{file,"edoc_tags.erl"},{line,272}]},
{edoc_extract,get_tags,6,
[{file,"edoc_extract.erl"},{line,586}]},
{edoc_extract,get_tags,6,
[{file,"edoc_extract.erl"},{line,589}]},
{edoc_extract,source1,5,
[{file,"edoc_extract.erl"},{line,132}]},
{docsh_edoc,to_internal,2,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/src/docsh_edoc.erl"},
{line,32}]},
{completeness_SUITE,
'-docsh_works_for_each_file_with_edoc/1-lc$^3/1-3-',1,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/test/completeness_SUITE.erl"},
{line,37}]},
{completeness_SUITE,
'-docsh_works_for_each_file_with_edoc/1-lc$^4/1-4-',3,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/test/completeness_SUITE.erl"},
{line,37}]}]}},
{erl_syntax,
{error,badarg,
[{erlang,iolist_to_binary,
[[[["Creates an abstract comment with the given padding and text. If",
"\n"],
["Strings is a (possibly empty) list","\n"],
["[\"Txt1\", ..., \"TxtN\"], the result","\n"],
["represents the source code text","\n"]],
["\n",
[[<<"\n %">>,
{inline,[<<"Txt1">>]},
<<"\n ...\n %">>,
{inline,[<<"TxtN">>]}],
"\n"]],
["\n",
[["Padding states the number of empty character positions",
"\n"],
["to the left of the comment separating it horizontally from",
"\n"],
["source code on the same line (if any). If Padding is",
"\n"],
["none, a default positive number is used. If","\n"],
["Padding is an integer less than 1, there should be no",
"\n"],
["separating space. Comments are in themselves regarded as source",
"\n"],
["program forms.","\n"]]]]],
[]},
{docsh_edoc_xmerl,fullDescription,4,
[{file,
"/Users/erszcz/work/erszcz/docsh/_build/test/lib/docsh/src/docsh_edoc_xmerl.erl"},
{line,125}]},
{xmerl,tagdef,5,[{file,"xmerl.erl"},{line,261}]},
{xmerl,export_content,2,[{file,"xmerl.erl"},{line,199}]},
{xmerl,export_content,2,[{file,"xmerl.erl"},{line,199}]},
{xmerl,export_element,2,[{file,"xmerl.erl"},{line,224}]},
{xmerl,export_content,2,[{file,"xmerl.erl"},{line,199}]},
{xmerl,export_content,2,[{file,"xmerl.erl"},{line,199}]}]}}]
----------------------------------------------------
2018-04-26 18:30:59.675
docsh_edoc fails for 14 out of 320 modules
%%% completeness_SUITE ==> docsh_works_for_each_file_with_edoc: SKIPPED
%%% completeness_SUITE ==> {tc_user_skip,"run just for the analysis printout"}
Skipped 1 (1, 0) tests. Passed 1 tests.
18:31:01 erszcz @ x4 : ~/work/erszcz/docsh (eep-48)
$