Infinite loop during auto return type deduction

[Weverything]

Bugzilla Link	34742
Version	unspecified
OS	Linux
CC	@DougGregor

Extended Description

The auto return type of a member function depends on its own return value. When instantiating the function, template deduction recursively tries to deduce the return type with no end.

$ cat foo.cc
struct S {
  template<class X = S>
  static auto foo() -> decltype(X::foo());
};

void run() {
  S::foo<>();
};

$ clang foo.cc -std=c++11
#0 0x0000000001e191e4 PrintStackTraceSignalHandler(void*) (/usr/local/bin/clang-3.5+0x1e191e4)
#1 0x0000000001e19546 SignalHandler(int) (/usr/local/bin/clang-3.5+0x1e19546)
#2 0x00007fcfb4081330 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x10330)
#3 0x00000000033736d6 clang::Sema::BuildOverloadedCallExpr(clang::Scope*, clang::Expr*, clang::UnresolvedLookupExpr*, clang::SourceLocation, llvm::MutableArrayRef<clang::Expr*>, clang::SourceLocation, clang::Expr*, bool, bool) (/usr/local/bin/clang-3.5+0x33736d6)
#4 0x000000000315196c clang::Sema::ActOnCallExpr(clang::Scope*, clang::Expr*, clang::SourceLocation, llvm::MutableArrayRef<clang::Expr*>, clang::SourceLocation, clang::Expr*, bool) (/usr/local/bin/clang-3.5+0x315196c)
#5 0x000000000349c0ff clang::TreeTransform<(anonymous namespace)::TemplateInstantiator>::TransformCallExpr(clang::CallExpr*) (/usr/local/bin/clang-3.5+0x349c0ff)
#6 0x0000000003488c9e clang::TreeTransform<(anonymous namespace)::TemplateInstantiator>::TransformType(clang::TypeLocBuilder&, clang::TypeLoc) (/usr/local/bin/clang-3.5+0x3488c9e)
#7 0x000000000348cb99 clang::Sema::SubstFunctionDeclType(clang::TypeSourceInfo*, clang::MultiLevelTemplateArgumentList const&, clang::SourceLocation, clang::DeclarationName, clang::CXXRecordDecl*, unsigned int) (/usr/local/bin/clang-3.5+0x348cb99)
#8 0x00000000034c5bc4 clang::TemplateDeclInstantiator::SubstFunctionType(clang::FunctionDecl*, llvm::SmallVectorImpl<clang::ParmVarDecl*>&) (/usr/local/bin/clang-3.5+0x34c5bc4)
#9 0x00000000034c312c clang::TemplateDeclInstantiator::VisitCXXMethodDecl(clang::CXXMethodDecl*, clang::TemplateParameterList*, bool) (/usr/local/bin/clang-3.5+0x34c312c)
#10 0x00000000034cab16 clang::Sema::SubstDecl(clang::Decl*, clang::DeclContext*, clang::MultiLevelTemplateArgumentList const&) (/usr/local/bin/clang-3.5+0x34cab16)
#11 0x000000000344b4c6 clang::Sema::FinishTemplateArgumentDeduction(clang::FunctionTemplateDecl*, llvm::SmallVectorImpl<clang::DeducedTemplateArgument>&, unsigned int, clang::FunctionDecl*&, clang::sema::TemplateDeductionInfo&, llvm::SmallVectorImpl<clang::Sema::OriginalCallArg> const*, bool, llvm::function_ref<bool ()>) (/usr/local/bin/clang-3.5+0x344b4c6)
#12 0x000000000344cec0 clang::Sema::DeduceTemplateArguments(clang::FunctionTemplateDecl*, clang::TemplateArgumentListInfo*, llvm::ArrayRef<clang::Expr*>, clang::FunctionDecl*&, clang::sema::TemplateDeductionInfo&, bool, llvm::function_ref<bool (llvm::ArrayRef<clang::QualType>)>) (/usr/local/bin/clang-3.5+0x344cec0)
#13 0x000000000335fc24 clang::Sema::AddTemplateOverloadCandidate(clang::FunctionTemplateDecl*, clang::DeclAccessPair, clang::TemplateArgumentListInfo*, llvm::ArrayRef<clang::Expr*>, clang::OverloadCandidateSet&, bool, bool) (/usr/local/bin/clang-3.5+0x335fc24)
#14 0x00000000033731bd AddOverloadedCallCandidate(clang::Sema&, clang::DeclAccessPair, clang::TemplateArgumentListInfo*, llvm::ArrayRef<clang::Expr*>, clang::OverloadCandidateSet&, bool, bool) (/usr/local/bin/clang-3.5+0x33731bd)
#15 0x0000000003372fc8 clang::Sema::AddOverloadedCallCandidates(clang::UnresolvedLookupExpr*, llvm::ArrayRef<clang::Expr*>, clang::OverloadCandidateSet&, bool) (/usr/local/bin/clang-3.5+0x3372fc8)
#16 0x00000000033733db clang::Sema::buildOverloadedCallSet(clang::Scope*, clang::Expr*, clang::UnresolvedLookupExpr*, llvm::MutableArrayRef<clang::Expr*>, clang::SourceLocation, clang::OverloadCandidateSet*, clang::ActionResult<clang::Expr*, true>*) (/usr/local/bin/clang-3.5+0x33733db)
#17 0x00000000033737ec clang::Sema::BuildOverloadedCallExpr(clang::Scope*, clang::Expr*, clang::UnresolvedLookupExpr*, clang::SourceLocation, llvm::MutableArrayRef<clang::Expr*>, clang::SourceLocation, clang::Expr*, bool, bool) (/usr/local/bin/clang-3.5+0x33737ec)

< Repeat from #4 Sema::ActOnCallExpr >

[jyknight]

Issue still present. Current Clang prints:

/tmp/foo.cc:3:33: warning: stack nearly exhausted; compilation time may suffer, and crashes due to stack overflow are likely [-Wstack-exhausted]
  static auto foo() -> decltype(X::foo());
                                ^
/tmp/foo.cc:3:33: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
/tmp/foo.cc:3:33: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
/tmp/foo.cc:3:33: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
/tmp/foo.cc:3:33: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
/tmp/foo.cc:3:33: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
/tmp/foo.cc:3:33: note: (skipping 565 contexts in backtrace; use -ftemplate-backtrace-limit=0 to see all)
/tmp/foo.cc:3:33: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
/tmp/foo.cc:3:33: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
/tmp/foo.cc:3:33: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
/tmp/foo.cc:3:33: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
/tmp/foo.cc:7:3: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
  S::foo<>();
  ^
/tmp/foo.cc:3:33: fatal error: recursive template instantiation exceeded maximum depth of 1024
  static auto foo() -> decltype(X::foo());
                                ^
/tmp/foo.cc:3:33: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
/tmp/foo.cc:3:33: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
/tmp/foo.cc:3:33: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
/tmp/foo.cc:3:33: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
/tmp/foo.cc:3:33: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
/tmp/foo.cc:3:33: note: (skipping 1015 contexts in backtrace; use -ftemplate-backtrace-limit=0 to see all)
/tmp/foo.cc:3:33: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
/tmp/foo.cc:3:33: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
/tmp/foo.cc:3:33: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
/tmp/foo.cc:3:33: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
/tmp/foo.cc:7:3: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
  S::foo<>();
  ^
1 warning and 1 error generated.

For comparison, GCC 12 prints:

/tmp/foo.cc: In substitution of ‘template<class X> static decltype (X::foo()) S::foo() [with X = S]’:
/tmp/foo.cc:3:39:   recursively required by substitution of ‘template<class X> static decltype (X::foo()) S::foo() [with X = S]’
/tmp/foo.cc:3:39:   required by substitution of ‘template<class X> static decltype (X::foo()) S::foo() [with X = S]’
/tmp/foo.cc:7:11:   required from here
/tmp/foo.cc:3:39: fatal error: template instantiation depth exceeds maximum of 900 (use ‘-ftemplate-depth=’ to increase the maximum)
    3 |   static auto foo() -> decltype(X::foo());
      |                                 ~~~~~~^~

[llvmbot]

@llvm/issue-subscribers-clang-frontend

[Endilll]

Nothing has changed as of post-17 trunk since last comment: https://godbolt.org/z/no65r1hza

<source>:3:33: warning: stack nearly exhausted; compilation time may suffer, and crashes due to stack overflow are likely [-Wstack-exhausted]
    3 |   static auto foo() -> decltype(X::foo());
      |                                 ^
<source>:3:33: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
<source>:3:33: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
<source>:3:33: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
<source>:3:33: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
<source>:3:33: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
<source>:3:33: note: (skipping 494 contexts in backtrace; use -ftemplate-backtrace-limit=0 to see all)
<source>:3:33: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
<source>:3:33: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
<source>:3:33: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
<source>:3:33: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
<source>:7:3: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
    7 |   S::foo<>();
      |   ^
<source>:3:33: fatal error: recursive template instantiation exceeded maximum depth of 1024
    3 |   static auto foo() -> decltype(X::foo());
      |                                 ^
<source>:3:33: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
<source>:3:33: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
<source>:3:33: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
<source>:3:33: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
<source>:3:33: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
<source>:3:33: note: (skipping 1015 contexts in backtrace; use -ftemplate-backtrace-limit=0 to see all)
<source>:3:33: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
<source>:3:33: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
<source>:3:33: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
<source>:3:33: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
<source>:7:3: note: while substituting deduced template arguments into function template 'foo' [with X = (no value)]
    7 |   S::foo<>();
      |   ^
1 warning and 1 error generated.

[zygoloid]

What action is this bug report requesting? The original report showed a crash, which is now fixed, and Clang diagnoses the infinite instantiation issue for the testcase, so are we done? Or is this a diagnostic-quality issue requesting some kind of special-case handling here beyond fixing the crash and producing a diagnostic explaining what's wrong?

[jyknight]

I think when I commented before, I was worried that it prints -Wstack-exhausted before failing, since the message emitted for that indicates that it may actually run out of stack soon and then crash. And, thus, it seemed like maybe we were just very lucky to avoid the hard limit in this particular example. But, I see now that Clang actually spawns a whole new thread to let it keep recursing without ever running out of stack space, so it's not actually going to crash.

So, yea, we can probably call it fixed. It may be nice to detect the recursion in the diagnostics printer, like GCC does, but I don't feel strongly about that.