How to speed up meta-sca

[gomme600]

In my CI environment (gitlab-ci), I keep the sstate-cache directory and reuse it for multiple builds. The tmp directory is deleted for each build. This causes meta-sca to rerun everytime (and takes almost 4 hours!).

I can't keep the tmp directory because I am building different projects (with a common base). Is there anything similar to the sstate-cache for meta-sca which could be shared across projects ?

Also, is there some kind of whitelist for layers instead of a blacklist ? I am currently using SCA_SPARE_LAYER and I don't want to inherit sca on each layer individually if possible.

Thanks

[priv-kweihmann]

Hmmm, meta-sca should be sstate-safe, so you wouldn't require a tmp directory for the build. Regarding the allow-list - I think that would be an interesting enhancement, could you please open up a dedicated ticket for that?

In the past I used this layer in a way to always include everything unless explicitly excluded, but I get your point. BTW does your build without meta-sca fully support the sstate cache? meaning that there won't be a rerun of it when nothing got modified

[gomme600]

I'm pretty sure it does fully support sstate-cache, I will check again.

When rerunning a build without removing tmp it takes 2 minutes. When removing tmp it does everything meta-sca related again.

[gomme600]

I have opened a ticket for the whitelist here. Thanks for looking into it!

[priv-kweihmann]

@gomme600 it seems you were right, as a few host specific paths slipped into the task hash signatures - although they shouldn't be there - I fixed that with #9492. Could you please give it a try and report back if sstate caches are now working for you

[gomme600]

Is that included in the latest dunfell build now ?

Edit : Just checked and seen that it is, am trying now !

[gomme600]

For some reason I am getting this error in my CI environment but not on my local build : Parsing recipes...ERROR: ParseError at /app/gitlab-runner/builds/c89cebdd/1/BS_PAMELA/manifests/repo_project_dir/sources/meta-sca/dynamic-layers/meta-oe/php_%.bbappend:1: unparsed line: 'PACKAGECONFIG:class-native = "mbstring openssl zip"'

Edit : That's the error I was getting here , looks like I need to update my CI environnement.

[gomme600]

I just got this error :

ERROR: base-files-3.0.14-r89 do_sca_get_sources: Execution of '/pamela/pcm-imx6quad-10191/tmp/work/pcm_imx6quad-actia-linux-gnueabi/base-files/3.0.14-r89/temp/run.do_sca_get_sources.5406' failed with exit code 1 ERROR: Logfile of failure stored in: /pamela/pcm-imx6quad-10191/tmp/work/pcm_imx6quad-actia-linux-gnueabi/base-files/3.0.14-r89/temp/log.do_sca_get_sources.5406 Log data follows: | DEBUG: Executing shell function do_sca_get_sources | find: ‘/pamela/pcm-imx6quad-10191/tmp/work/pcm_imx6quad-actia-linux-gnueabi/base-files/3.0.14-r89/sstate-build-deploy_source_date_epoch’: No such file or directory | WARNING: /pamela/pcm-imx6quad-10191/tmp/work/pcm_imx6quad-actia-linux-gnueabi/base-files/3.0.14-r89/temp/run.do_sca_get_sources.5406:1 exit 1 from 'find /pamela/pcm-imx6quad-10191/tmp/work/pcm_imx6quad-actia-linux-gnueabi/base-files/3.0.14-r89 -type f -exec md5sum {} ; >> /pamela/pcm-imx6quad-10191/tmp/work/pcm_imx6quad-actia-linux-gnueabi/base-files/3.0.14-r89/temp/sca_seen_sources.txt' | ERROR: Execution of '/pamela/pcm-imx6quad-10191/tmp/work/pcm_imx6quad-actia-linux-gnueabi/base-files/3.0.14-r89/temp/run.do_sca_get_sources.5406' failed with exit code 1 NOTE: recipe base-files-3.0.14-r89: task do_sca_get_sources: Failed ERROR: Task (/pamela/sources/poky/meta/recipes-core/base-files/base-files_3.0.14.bb:do_sca_get_sources) failed with exit code '1'

[gomme600]

And I got this error on my local build :

ERROR: gobject-introspection-1.62.0-r0 do_sca_tracefiles: Error executing a python function in exec_func_python() autogenerated:

The stack trace of python calls that resulted in this exception/failure was: File: 'exec_func_python() autogenerated', lineno: 2, function: 0001: *** 0002:do_sca_tracefiles(d) 0003: File: '/home/slucas/meta-sca/repo3/sources/meta-sca/classes/sca-tracefiles.bbclass', lineno: 82, function: do_sca_tracefiles 0078: 0079:python do_sca_tracefiles() { 0080: import json 0081: with open(d.expand("${SCA_TRACEFILES_LIST}"), "w") as o: *** 0082: json.dump(sca_get_trace_files(d), o) 0083:} 0084: 0085:addtask do_sca_tracefiles after do_install before do_package 0086: File: '/home/slucas/meta-sca/repo3/sources/meta-sca/classes/sca-tracefiles.bbclass', lineno: 55, function: sca_get_trace_files 0051: _res = {} 0052: _pkg_files = sca_get_pkg_files(d) 0053: for k, v in _pkg_files.items(): 0054: try: *** 0055: cmd_out = subprocess.check_output(["tracefiles", d.getVar("D"), d.getVar("SCA_SOURCES_DIR")] + v) 0056: except subprocess.CalledProcessError as e: 0057: cmd_out = e.output or "" 0058: 0059: if not isinstance(cmd_out, str): File: '/usr/lib/python3.6/subprocess.py', lineno: 356, function: check_output 0352: # empty string. That is maintained here for backwards compatibility. 0353: kwargs['input'] = '' if kwargs.get('universal_newlines', False) else b'' 0354: 0355: return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, *** 0356: **kwargs).stdout 0357: 0358: 0359:class CompletedProcess(object): 0360: """A process that has finished running. File: '/usr/lib/python3.6/subprocess.py', lineno: 423, function: run 0419: if 'stdin' in kwargs: 0420: raise ValueError('stdin and input arguments may not both be used.') 0421: kwargs['stdin'] = PIPE 0422: *** 0423: with Popen(*popenargs, **kwargs) as process: 0424: try: 0425: stdout, stderr = process.communicate(input, timeout=timeout) 0426: except TimeoutExpired: 0427: process.kill() File: '/usr/lib/python3.6/subprocess.py', lineno: 729, function: init 0725: startupinfo, creationflags, shell, 0726: p2cread, p2cwrite, 0727: c2pread, c2pwrite, 0728: errread, errwrite, *** 0729: restore_signals, start_new_session) 0730: except: 0731: # Cleanup if the child failed starting. 0732: for f in filter(None, (self.stdin, self.stdout, self.stderr)): 0733: try: File: '/usr/lib/python3.6/subprocess.py', lineno: 1364, function: _execute_child 1360: if errno_num != 0: 1361: err_msg = os.strerror(errno_num) 1362: if errno_num == errno.ENOENT: 1363: err_msg += ': ' + repr(err_filename) *** 1364: raise child_exception_type(errno_num, err_msg, err_filename) 1365: raise child_exception_type(err_msg) 1366: 1367: 1368: def _handle_exitstatus(self, sts, _WIFSIGNALED=os.WIFSIGNALED, Exception: FileNotFoundError: [Errno 2] No such file or directory: 'tracefiles': 'tracefiles'

ERROR: Logfile of failure stored in: /home/slucas/meta-sca/repo3/build-imx_imx-distro_security_sota_test_xwayland/tmp/work/cortexa9t2hf-neon-actia-linux-gnueabi/gobject-introspection/1.62.0-r0/temp/log.do_sca_tracefiles.215 ERROR: Task (/home/slucas/meta-sca/repo3/sources/poky/meta/recipes-gnome/gobject-introspection/gobject-introspection_1.62.0.bb:do_sca_tracefiles) failed with exit code '1'