Add support for f-strings

[mrdaybird]

Support f-strings in lpython, because f-strings are awesome. Previously, expression enclosed inside the f-strings was not parsed correctly. (Fixes #641 ) TODO:

• [x] Add new tokens to handle f-strings.
  • Divide f-strings into three parts start, middle, end to allow parsing of enclosed expression.
• [x] Add grammar for f-strings.
• [x] Add semantics for f-strings to create correct AST.
  • All the parts of f-string are merged using recursively apply add operation.
• [x] Implement visit_FormattedValue and visit_JoinedStr in python_ast_to_asr.cpp
  • This uses handle_intrinsic_str to cast expression to string
• ( It works! )
• [x] Add tests and clean up code.
• [x] Update previous tests.
• [x] Add support(+tests) for multi line f-strings Maybe for another PR: (this PR is already pretty large to review i think)
• Add escape for braces
• Add format specifiers

[mrdaybird]

@certik Can you provide me with some feedback on this PR? Are there any things that could be improved? Thanks beforehand.

[mrdaybird]

To support f-strings I have used re2c conditions. This was the best solution I could come up with to support lexing and parsing of expression enclosed within the string. So, this requires adding -c flag in re2c command. @certik Also, ~I think the release test failing is not due to this PR~. There are memory leaks while running tests on the main branch. Update: I think release test was failing due to my code. After I tried another approach without using condition mode of re2c there are no problems with the test.

[mrdaybird]

I think I have finally found an efficient/clean solution to handle lexing fstrings and parsing expressions within it. I noticed re2c conditions explode the code size of tokenizer.cpp from ~3000 to ~13000 lines of code. Using multiple blocks is way more efficient and all the tests are also passing.

[mrdaybird]

Now the following works:

a : i32 = 10
b : f64 = 15.5
c : str = "foobar"
print(f"Trying some calculations : {a*2+3}, {b*3.14}, some strings {c*2}")

This give the following output:

Trying some calculations : 23, 48.670000, some strings foobarfoobar

[mrdaybird]

@Shaikh-Ubaid @czgdp1807 Can you please review this PR?

[Thirumalai-Shaktivel]

Thank you very much, @mrdaybird, for this. I need some time to review this thoroughly. I will report back soon.

[certik]

@mrdaybird my apologies that nobody reviewed it yet. @Thirumalai-Shaktivel, @Shaikh-Ubaid would one of you have time to review it please?

[ubaidsk]

The changes seem considerate. I will give this a thorough look today/tomorrow.