Failed to generate .clang-format

[mikhailramalho]

First of all, excellent project!

I'm trying to run on our project: esbmc but I get the following:

 $ python3 unformat/ --root /home/mramalho/esbmc-github/src/clang-c-frontend/ "/home/mramalho/esbmc-github/src/clang-c-frontend/**/*.h" "/home/mramalho/esbmc-github/src/clang-c-frontend/**/*.cpp"
Using the configuration file from the provided root, '/home/mramalho/esbmc-github/src/clang-c-frontend/'
0: .Traceback (most recent call last):
  File "/usr/lib64/python3.5/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib64/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "unformat/__main__.py", line 85, in <module>
    main(args, pool)
  File "unformat/__main__.py", line 52, in main
    (generation_fittest, population) = generate(population, source_filenames, args, pool)
  File "unformat/__main__.py", line 40, in generate
    return recombine(scored_population, args)
  File "unformat/recombine.py", line 91, in recombine
    range(args.population - 1)]
  File "unformat/recombine.py", line 90, in <listcomp>
    recombined_configs = [mutate(ranked[int(random() * random() * len(ranked))][1], args.mutation) for _ in
  File "unformat/recombine.py", line 79, in mutate
    return {key: visit_line(key, value, mutation_rate) for key, value in config.items()}
AttributeError: 'NoneType' object has no attribute 'items'

I'm actually setting the root one subdir of the project because that's the place with the best code style so far.

If I run on the actual root dir (esbmc/src), it goes out of memory but then keeps running, eventually failing with the same error.

[johnmcfarlane]

Thanks @mikhailramalho, what versions of Python and clang-format are you using? Also, how much RAM and what CPU do you have? To avoid the OOM issue, you might try reducing the number of parallel processes with -j N.

[mikhailramalho]

Hi,

$ clang-format -version
clang-format version 3.9.1 (tags/RELEASE_391/final)

$ python3 --version
Python 3.5.3

Processor: Intel® Xeon(R) CPU W3520 @ 2.67GHz × 8 Memory: 23.5 GiB Fedora 25

[johnmcfarlane]

Hmm, you have plenty of RAM. Nevertheless, -j 4 should reduce the chance of OOM. I'll look into how easy it is to catch this error and fail more gracefully.

[johnmcfarlane]

I've observed this problem and reducing jobs to one doesn't help. It seems to be a problem with files from my cmake build directory (which got pulled in when I specified the root of my project). There's likely a lot of extreme code in the average build folder. Could this be the case with you?

[Manu343726]

I think I've reproduced the same issue with 60 files approx:

.Traceback (most recent call last):
  File "/home/manuel-sanchez/Python-3.5.4/lib/python3.5/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/manuel-sanchez/Python-3.5.4/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/manuel-sanchez/Documentos/unformat/__main__.py", line 85, in <module>
    main(args, pool)
  File "/home/manuel-sanchez/Documentos/unformat/__main__.py", line 52, in main
    (generation_fittest, population) = generate(population, source_filenames, args, pool)
  File "/home/manuel-sanchez/Documentos/unformat/__main__.py", line 40, in generate
    return recombine(scored_population, args)
  File "/home/manuel-sanchez/Documentos/unformat/recombine.py", line 91, in recombine
    range(args.population - 1)]
  File "/home/manuel-sanchez/Documentos/unformat/recombine.py", line 90, in <listcomp>
    recombined_configs = [mutate(ranked[int(random() * random() * len(ranked))][1], args.mutation) for _ in
  File "/home/manuel-sanchez/Documentos/unformat/recombine.py", line 79, in mutate
    return {key: visit_line(key, value, mutation_rate) for key, value in config.items()}
AttributeError: 'NoneType' object has no attribute 'items'

after looking at the backtrace and the code I would say my problem is that make_initial_configs() is returning an empty set as population, since I called unformat this way:

python3.5 $HOME/Documentos/unformat --root . "<my-sourcedir>/**/*.h" "<my-sourcedir>/**/*.cpp" --command clang-format-3.9

with the root directory containing an empty .clang-format file. Maybe make_initial_configs() should be able to return a valid structured (but empty) set of args, or I should not be that lazy and feed the tool with some initial config.

Yes, I'm that guy, leaving all the work to write a .clang-format matching his code guidelines to a tool :)

[johnmcfarlane]

That's a good catch, thanks. I'm working on a change that adds uncrustify support or I'd address that case right now. That branch already has the fix. If it don't get it merged soon, I'll apply a patch to master.