Size of resulting documentation with many container elements (e.g. class fields)

[bsutton]

I'm trying to work out f I'm doing something wrong. I'm generating docs using dartdoc and the size of the resulting doc directory is huge. I'm using dartdoc 5.0.1 from pub.dev.

I'm running: dartdoc --no-validate-links --output $pathToDocRoot --no-show-progress

I've seen dartdoc generate directories that are 23GB in size.

I'm hoping I'm doing something wrong.

As an example, the iconify_flutter package version 0.0.3 is 18MB tar.gz (I thought pub.dev limited uploads to 10MB?). After running dartdoc the resulting doc/api directory is: 11GB

It also takes an extreme amount of time: Documented 111 public libraries in 17569.5 seconds

This is on a AMD® Ryzen 7 3700x 8-core processor × 16 with 32GB ram and SSDs on linux.

[srawlins]

😮 That... sounds too large, haha. And also a ton of time. I'll take a look at iconify_flutter.

[parlough]

I haven't taken a look, but I think this goes back to the issue of a large amount of members each having their own file. A package with that that many icon sets is bound to have a lot.

See https://github.com/dart-lang/dartdoc/issues/1983

[srawlins]

I think we want to make a change where members, like static consts, which do not have a doc comment, don't get a page.

I think in the case of this package we see this scary growth: A library with a class with n members will generate O(n) files, each of which has a sidebar of length O(n), resulting in a total of O(n²) bytes.

[bsutton]

This package certainly has a lot of static consts and generates 109K files.

It has one directory with 11K files.

The fact that it takes 5 hours to generate is also rather problematic.

To be clear this isn't a one-off occurrence. We are seeing these sorts of numbers for many packages.

[parlough]

I think we want to make a change where members, like static consts, which do not have a doc comment, don't get a page.

I think in the case of this package we see this scary growth: A library with a class with n members will generate O(n) files, each of which has a sidebar of length O(n), resulting in a total of O(n2) bytes.

Sounds like a good idea to me. There's a few things to consider though, such as if it may break external expectations of where member links would be, as these would be exceptions.

[bsutton]

#1983 also discussed iframes for the TOC and such which I think would be a valuable idea.

[srawlins]

I wonder if something has been fixed here, in terms of the time. It just took me 2080 seconds at HEAD to document iconify_flutter.

[srawlins]

The size is not fixed though:

$ du -sh doc/
 82G    doc/
$ du -sh lib/
 80M    lib/

[bsutton]

I just did a run without the --no-validate-links which seem to run much faster but that doesn't make sense as I would have thought the no validate option would be faster.

[srawlins]

Yeah maybe you got them backwards? I "just" tried without --no-validate-links and took 19624.1 seconds. 💀 💀 💀

[bsutton]

Yes I think you are right.

On Sat, 26 Mar 2022, 5:45 am Sam Rawlins, @.***> wrote:

Yeah maybe you got them backwards? I "just" tried without --no-validate-links and took 19624.1 seconds. 💀 💀 💀

— Reply to this email directly, view it on GitHub https://github.com/dart-lang/dartdoc/issues/2995#issuecomment-1079308932, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAG32OEGTL74CBMNZ377F7DVBYCSVANCNFSM5RSN6BEQ . You are receiving this because you authored the thread.Message ID: @.***>

[srawlins]

I did some size benchmarking, and it does look either quadratic or even cubic. :/

In generating docs for 1 package with 1 library with 1 class with N methods with 1 parameter each:

method count	kbytes of output	ms to validate
1	164	329
10	240	433
100	1434	1067
1000	60416	24293
4000	930816	369820
8000	3670016	1460461
10000	5767168	2776674

Google Sheets gives a quadratic regression for the output size as: 1845 + -0.727x + 0.0576x^2 with R^2=1.

Google Sheets gives a quadratic regression for the validation time as: 22516 + -70.9x + 0.0338x^2 with R^2=0.994.