Threads mistakenly attach to main thread's span

[Ten0]

trafficstars

The first time we call sentry-related functions in a thread, it attaches to whichever Span the main thread is currently attached to. This causes very wrong sub-spanning (which I just spend 6 hours investigating). Attaching to sentry client, DSN, parameters..., etc seems correct, but span definitely not.

I feel like maybe the Scope should be a property that's not inherited from whichever scope the main thread happens to be in at that moment, because if there is a sub-thread it's very possible that the main thread is doing unrelated work at that moment, and that unrelated work may very well involve its own spans and data, so it seems that it's not correct to attach to those.

Related issues: #507 Related writeup: https://swatinem.de/blog/log-contexts/

[Ten0]

Aha digging further into this it seems that it already attaches to the top scope of the main thread, so the issue may be that the sentry initialization leaves without actually pushing a scope level, leading to it accidentally polluting what others need to attach to...

[Ten0]

so the issue may be that the sentry initialization leaves without actually pushing a scope level

That actually wouldn't work: any Hub::new_from_top(Hub::current()) on the main thead itself wouldn't attach to the proper thing.

[Ten0]

Ok so I'm trying to understand how it's supposed to work but there's something I don't understand: Hub::push_scope() ends up calling this function, but: https://github.com/getsentry/sentry-rust/blob/8cc4848f115152af6f2ad66d7aeb17bd3bbe38fd/sentry-core/src/scope/real.rs#L86-L89 actually pushes a clone of StackLayer which contains an Arc<Scope>: https://github.com/getsentry/sentry-rust/blob/8cc4848f115152af6f2ad66d7aeb17bd3bbe38fd/sentry-core/src/scope/real.rs#L72-L76 without actually touching the top, so in effect we aren't changing the scope at all, since the top of the stack (and even the new layer) end up referencing the same scope as before! So, what's the point of this push_scope function, since it actually can't change the active scope?

EDIT: Aah the update actually happens here, and all of the data of the scope is cloned and they all have Arc wrappers: https://github.com/getsentry/sentry-rust/blob/8cc4848f115152af6f2ad66d7aeb17bd3bbe38fd/sentry-core/src/hub.rs#L210-L213

[Ten0]

Ok so my understanding is that there's no way to prevent the Hub to attach to the current scope of the main thread. It looks like the "main Hub" shouldn't really be attached to any thread, and even the main thread should have its own Hub. That would fix this issue.

[szokeasaurusrex]

Hi @Ten0, I am a bit confused by your most recent comment. Is this issue resolved, or does something still need to be fixed?

If something needs to be fixed, please explain in more detail exactly what is going wrong. A reproduction would be extremely helpful.

[Ten0]

What's going wrong is in the scenario where I do:

1. Init sentry on main thread
2. Spawn a sub-thread
3. Keep doing work on main thread
4. sub-thread calls any sentry-related function

Then the sub-thread attaches to whatever scope the main thread was running at that exact moment. You end up with wrong sub-spanning, that is, things showing as sub-spans of unrelated other things that the main thread is working on. That is irrelevant, wrong in the general case (above) and hard to debug.

The reason why this happens is that there's a "main hub" that gets attached to whichever thread is currently running when the first Sentry-related function initializes, and then stays linked to it and shared with other things that get attached to it.

The fix would be to have the main hub, whose state would be described by whatever we put in init, and modifiable explicitly, not be attached to any thread, so that the thread that first calls sentry-related stuff doesn't have a special and unreliable behavior of things implicitly attaching to it.

My current workaround for this behavior is this:

pub fn init_sentry(
	before_send: Option<BeforeCallback>,
) -> std::result::Result<sentry::ClientInitGuard, std::env::VarError> {
	// Initialize sentry on a separate thread to workaround:
	// https://github.com/getsentry/sentry-rust/issues/674#issuecomment-2250386287
	// (So that sub-threads don't bind to random scopes the main thread may be using at any given time)
	std::thread::spawn(|| {
		use std::env::var;
		let options = sentry::ClientOptions {
			dsn: Some(var("SentryDsn")?.parse().expect("Invalid value for sentry DSN")),
			release: Some(Cow::Owned(var("CommitShaShort")?)),
			environment: Some(Cow::Owned(
				var("SENTRY_OVERRIDE_ENVIRONMENT_NAME")
					.ok()
					.map_or_else(|| var("SCD_GROUP"), Ok)?,
			)),
			before_send,
			..sentry::ClientOptions::default()
		};
		std::env::set_var("RUST_BACKTRACE", "1");
		let guard = sentry::init(options);

		// Make sure that we are the main hub, so that any thread will bind to our hub and have the client,
		// otherwise Sentries wouldn't be sent
		if !std::sync::Arc::ptr_eq(&sentry::Hub::current(), &sentry::Hub::main()) {
			panic!("Sentry-related functions should never be called before initializing Sentry");
		}

		#[cfg(feature = "tracing")]
		tracing_integration::init();

		Ok(guard)
	})
	.join()
	.expect("Sentry initialization thread failed")
}