Ensure standalone `format_args!()` invocations get lowered properly

[CohenArthur]

e.g. make sure that this code:

fn main() {
    format_args!("heyo");
}

actually goes through the right ASTLoweringExpr

[badumbatish]

can i try this? My plan is to step into the debugger for one of the currently already implemented macros and see what steps i need to take to reach ASTLoweringExpr.

[CohenArthur]

@badumbatish yes! that'd be great. please note that at the moment, we are not using HIR lowering for expanding AST::FormatArg nodes - this development will happen later on. still, it eventually needs to happen so that work won't be for nothing. sorry that part of the compiler is undergoing such churn at the moment lol

[badumbatish]

sounds good, i'll do a WIP for this and touch it up after #2874 is merged

[badumbatish]

I'm getting some trouble getting through any call of this Rust::HIR::ASTLoweringExpr::visit* nor Rust::HIR::ASTLoweringExpr::translate

via this example

// testing_file.rs
#![feature(rustc_attrs)]

#[rustc_builtin_macro]
macro_rules! concat {
    () => {{}};
}

fn main() {
    // concat!();
    concat!("message");
}

I thought since "message" is a constant expression, I'd hit a ASTLoweringExpr at some point.

My terminal call:

gccrs -frust-incomplete-and-experimental-compiler-do-not-use testing_file.rs -o dummy -wrapper cgdb,--args

and my break:

b Rust::HIR::ASTLoweringExpr::translate
rb Rust::HIR::ASTLoweringExpr::visit*

[CohenArthur]

Yeah I think this might require digging in the MacroExpander implementation but it's not the easiest part of the code :/ I would suggest looking at the SingleAstNode being emitted by that concat!() invocation and following it until expansion. We might miss it because we're expecting a statement and it's actually an expression or something

[badumbatish]

Are you referring expansion to these lines in rust-session-manager.cc?

  if (last_step == CompileOptions::CompileStep::Expansion)
    return;

  auto name_resolution_ctx = Resolver2_0::NameResolutionContext ();
  // expansion pipeline stage

  expansion (parsed_crate, name_resolution_ctx);
  rust_debug ("\033[0;31mSUCCESSFULLY FINISHED EXPANSION \033[0m");
  if (options.dump_option_enabled (CompileOptions::EXPANSION_DUMP))
    {
      // dump AST with expanded stuff
      rust_debug ("BEGIN POST-EXPANSION AST DUMP");
      dump_ast_pretty (parsed_crate, true);
      rust_debug ("END POST-EXPANSION AST DUMP");
    }

I also have some questions about Items, in the stage where we resolve from AST to HIR, we call HIR::ASTLowering::Resolve (parsed_crate); which calls std::unique_ptr<HIR::Crate> ASTLowering::go () which translate the crate's AST Item to HIR items but I'm not sure what are items and how by translating these items we can translate other type of AST like expr? Is there some nested translate calls?

[CohenArthur]

Are you referring expansion to these lines in rust-session-manager.cc?

if (last_step == CompileOptions::CompileStep::Expansion) return;

auto name_resolution_ctx = Resolver2_0::NameResolutionContext (); // expansion pipeline stage

expansion (parsed_crate, name_resolution_ctx); rust_debug ("\033[0;31mSUCCESSFULLY FINISHED EXPANSION \033[0m"); if (options.dump_option_enabled (CompileOptions::EXPANSION_DUMP)) { // dump AST with expanded stuff rust_debug ("BEGIN POST-EXPANSION AST DUMP"); dump_ast_pretty (parsed_crate, true); rust_debug ("END POST-EXPANSION AST DUMP"); }

Yes - expansion will go through the AST, find macro invocations and replace them with their expanded tokens. so we'll go from something like this:

macro_rules! add {
    ($a:expr, $b:expr) => { $a + $b }
}

fn main() {
    let a = add!(15, 14);
}

to something like this post expansion:

macro_rules! add {
    ($a:expr, $b:expr) => { $a + $b }
}

fn main() {
    let a = 15 + 14;
}

this is done by creating AST::Fragments, which are pieces of AST we'll insert instead of macro invocations, directly in the AST. because one macro invocation can expand to multiple "things", like multiple expressions or multiple types, or two types and a function, etc, well an AST fragment contains a list of nodes which can contain either an AST::Expr, or an AST::Stmt, or an AST::Item... etc. and based on the context we will try to fetch the node's expression, or the statement and so on.

For example:

let a = add!(15, 14);

here, the macro invocation is in an expression place - so we'll try and access the fragment's expression. But if we do something like:

fn main() {
    add!(15, 14);
}

then, the invocation is a statement within a function block. Does that make sense?

I also have some questions about Items, in the stage where we resolve from AST to HIR, we call HIR::ASTLowering::Resolve (parsed_crate); which calls std::unique_ptr<HIR::Crate> ASTLowering::go () which translate the crate's AST Item to HIR items but I'm not sure what are items and how by translating these items we can translate other type of AST like expr? Is there some nested translate calls?

The Rust reference has a good chapter on items: https://doc.rust-lang.org/reference/items.html

You're right that there are nested translate calls - for example, a function is an item. A function usually has a block, which will be translated/lowered as well. Recursively, we'll go through that block and translate all of its statements as well as translate its tail expression

[badumbatish]

Hi Arthur, sorry I haven't been too active on the issue

Thank you for your write up! I think my approach of walking through the debugger with concat is not really effective?

I did step through the codebase and found ASTLoweringExpr::visit (AST::FormatArgs &fmt) in rust-ast-lower-expr.cc around 825 and HIR::Expr * FormatArgsLowering::go (AST::FormatArgs &fmt) in rust-ast-lower-format-args.cc around 33.

I did step through the debugger with breakpoints and neither went through. The problem I'm having is with all these translate calls nesting with each other and with visit() also, I'm not sure where to focus on the point that can enable format_args! to be lowered on all cases. Is there an easy way to go about this such as #2911 ?