Improve performance on file lists / id lists

Open fsundermeyer opened this issue 8 years ago • 0 comments

Problem:

DAPS takes very long to create file-lists / ID lists. This makes it impossible to use file lists functions for macros inside editors or for auto-completion of the xml:ids. Root cause for this problem is profiling: In order to know which files belong to a set (XML and images) an XML representation of all files, images and IDs is generated in make/setfiles.mk. Creating this list takes approx. 2 seconds. This information is written to a tmp-file that is deleted once DAPS finishes. The list of XML files and images itself is directly written to variables for performance reasons. In order to be able to write the file lists to variables via VAR:=$(shell ...) we must be sure the XML is well-formed (otherwise the file list may only contain an error message, which will lead to unforseen behavior that is hard to debug), so the very first thing DAPS does is checking the XML sources for well-formedness. This takes another 2 seconds.

Suggestions for improvements (just a brain dump, needs checking and benchmarking):

permanently write the file containing the IDs and files to the file system, with dependencies on images and sources
- pro: grep'ing for IDs and filenames within that file will be quick and will not even need XSLT
- cons: dependencies can only be set for all XML and image files. The file list will be regenerated when any XML/image file in the source dirs has changes (but may better than always regenerating the list.
check the return status of the xsltproc call that generates the file containing the IDs and files rather than running a check for well-formedness. If the xsltproc call fails, the XML is not well-formed.