cef icon indicating copy to clipboard operation
cef copied to clipboard
verified publisher icon chromiumembedded

Dynamic loading of libcef.so on Linux ARM64 not possible anymore due to TLS size increase

Open Hethsron opened this issue 1 year ago • 16 comments

Describe the bug Dynamically loading libcef.so into a running process on Linux arm64 fails since CEF 129 with the error message "cannot allocate memory in static TLS block". Since a lot of CEF wrappers depend on the ability to dynamically extract and load libcef.so (known: CEF4Delphi, JCEF, probably more), this issue effectively makes CEF 129+ unusable for these wrappers on Linux arm64.

To Reproduce Either use CEF 129 with for example JCEF on Linux, or use the following synthetic console application to dlopen libcef.so:

#include <stdio.h>
#include <dlfcn.h>

int main (void)
{
  void* g_libcef_handle = dlopen("./libcef.so", RTLD_LAZY | RTLD_LOCAL);
  if (!g_libcef_handle) {
    fprintf(stderr, "dlerror %s\n", dlerror());
  }
  return 0;
}

Compile the source code :

g++ -o main.cpp -ldl

Run :

bash-4.4$ ./a.out 
dlerror ./libcef.so: cannot allocate memory in static TLS block
bash-4.4$

Expected behavior libcef.so can be dynamically loaded into a Linux process.

Screenshots If applicable, add screenshots to help explain your problem.

Versions (please complete the following information):

  • OS: Linux (all ARM popular distributions)
  • CEF Version: 129.x (6668)

Background/Analysis The TLS segment size of Chromium 129 increases by over 700 bytes, as can be seen here:

bash-4.4$ readelf -Wl libcef.so | grep -E 'PhysAddr|TLS'
  Type           Offset   VirtAddr           PhysAddr           FileSiz  MemSiz   Flg Align
  TLS            0xc7cc340 0x000000000c7ec340 0x000000000c7ec340 0x000090 0x000430 R   0x40
bash-4.4$

This issue is similar to #3616.

Hethsron avatar Oct 14 '24 14:10 Hethsron

@magreenblatt Hi Marshall, do you have a fix for that ?

Hethsron avatar Oct 14 '24 14:10 Hethsron

Does this also impact x64 Intel builds? You can use the tips from #3616 to debug the issue.

magreenblatt avatar Oct 14 '24 15:10 magreenblatt

Does this also impact x64 Intel builds? You can use the tips from #3616 to debug the issue.

Intel x64 versions are not impacted. I will try to debug the issue with these tips and let you as soon as possible a further analysis.

Hethsron avatar Oct 14 '24 15:10 Hethsron

@magreenblatt

After following tips, here is my analysis :

  • In case of libcef.so (M~129), the output shows that the TLS segment has a file size (FileSiz) of 0x000090 and a memory size (MemSiz) of 0x000430. Thus, the segment size in hex is 0x000430 and equal to 1072 bytes.

  • Earlier versions only had a TLS segment size of 0x0003d8 or 984 bytes. For example, here is the result of the command on libcef.so version M~125 :

 readelf -Wl libcef.so | grep -E 'PhysAddr|TLS'
  Type           Offset   VirtAddr           PhysAddr           FileSiz  MemSiz   Flg Align
  TLS            0xc214fc0 0x000000000c234fc0 0x000000000c234fc0 0x000090 0x0003d8 R   0x40

Here are the two TLS segment results compared side by side:

Property First Result Second Result Difference
Offset 0xc7cc340 0xc214fc0 -
VirtAddr 0x000000000c7ec340 0x000000000c234fc0 -
PhysAddr 0x000000000c7ec340 0x000000000c234fc0 -
FileSiz 0x000090 (144 bytes) 0x000090 (144 bytes) No difference
MemSiz 0x000430 (1072 bytes) 0x0003d8 (984 bytes) 88 bytes (1072 - 984)
Flags R R No difference
Align 0x40 0x40 No difference

Key Differences:

  • The memory size (MemSiz) is different:
    • First result: 1072 bytes (0x000430)
    • Second result: 984 bytes (0x0003d8) The difference in size is 88 bytes. Other fields such as Offset, VirtAddr, PhysAddr, FileSiz, and flags remain the same, except for different starting addresses.

Version used for test

cef_binary_129.0.12+gf09539f+chromium-129.0.6668.101_linuxarm64_client

Workaround Just for the record, this workaround works. But it's not acceptable in our case:

export LD_PRELOAD=<FULL-PATH-TO-libcef.so>

And the following patch is already present in config.h:

diff --git third_party/libxml/linux/config.h third_party/libxml/linux/config.h
index c064071ce1545..65110af9a78f5 100644
--- third_party/libxml/linux/config.h
+++ third_party/libxml/linux/config.h
@@ -171,7 +171,7 @@
/* #undef XML_SOCKLEN_T */
 
/* TLS specifier */
-#define XML_THREAD_LOCAL _Thread_local
+/* #undef XML_THREAD_LOCAL */
 
/* Define for Solaris 2.5.1 so the uint32_t typedef from <sys/synch.h>,
<pthread.h>, or <semaphore.h> is not used. If the typedef were allowed, the

Deep Investigation Further investigation found the following usage (.tdata and .tbss are the symbol sections of interest when looking for the actual users of all that TLS space - this article here (A dynamic linker murder mystery) was very helpful in learning interesting tidbits about TLS, and commands to investigate the situation) :

bash-4.4$ llvm-objdump -C -t libcef.so | grep -F '.tdata'
0000000000000000 l       .tdata	0000000000000000 $d
0000000000000000 l       .tdata	0000000000000010 .hidden absl::cord_internal::cordz_next_sample
0000000000000010 l       .tdata	0000000000000004 v8::internal::(anonymous namespace)::current_per_thread_assert_data
0000000000000010 l       .tdata	0000000000000000 $d
0000000000000014 l       .tdata	0000000000000004 base::internal::(anonymous namespace)::current_sequence_token
0000000000000018 l       .tdata	0000000000000004 base::internal::(anonymous namespace)::current_task_token
000000000000001c l       .tdata	0000000000000001 base::internal::(anonymous namespace)::current_task_is_thread_bound
0000000000000014 l       .tdata	0000000000000000 $d
0000000000000018 l       .tdata	0000000000000000 $d
000000000000001c l       .tdata	0000000000000000 $d
000000000000001d l       .tdata	0000000000000001 base::internal::(anonymous namespace)::task_priority_for_current_thread
000000000000001d l       .tdata	0000000000000000 $d
0000000000000020 l       .tdata	0000000000000004 base::(anonymous namespace)::current_thread_type
0000000000000020 l       .tdata	0000000000000000 $d
0000000000000028 l       .tdata	0000000000000008 base::(anonymous namespace)::thread_name
0000000000000028 l       .tdata	0000000000000000 $d
0000000000000030 l       .tdata	0000000000000004 base::(anonymous namespace)::g_thread_id
0000000000000034 l       .tdata	0000000000000001 base::(anonymous namespace)::g_is_main_thread
0000000000000030 l       .tdata	0000000000000000 $d
0000000000000034 l       .tdata	0000000000000000 $d
0000000000000038 l       .tdata	0000000000000004 partition_alloc::internal::base::(anonymous namespace)::g_thread_id
000000000000003c l       .tdata	0000000000000001 partition_alloc::internal::base::(anonymous namespace)::g_is_main_thread
0000000000000038 l       .tdata	0000000000000000 $d
000000000000003c l       .tdata	0000000000000000 $d
0000000000000040 l       .tdata	0000000000000000 $d
0000000000000040 l       .tdata	0000000000000040 .hidden google::protobuf::internal::ThreadSafeArena::thread_cache_
0000000000000080 l       .tdata	0000000000000004 simd_support
0000000000000088 l       .tdata	0000000000000004 simd_features
0000000000000084 l       .tdata	0000000000000004 simd_huffman
0000000000000080 l       .tdata	0000000000000000 $d
0000000000000084 l       .tdata	0000000000000000 $d
0000000000000088 l       .tdata	0000000000000000 $d
000000000000008c l       .tdata	0000000000000000 $d
000000000000008c l       .tdata	0000000000000004 .hidden blink::next_world_id
0000000000000090 l       .tdata	0000000000000020 std::sys::thread_local::destructors::list::DTORS::h9d4e40d7fcc5ca6c
0000000000000090 l       .tdata	0000000000000000 $d         

bash-4.4$ llvm-objdump -C -t libcef.so | grep -F '.tbss' 
00000000000000b0 l       .tbss	0000000000000000 $d
00000000000000b0 l       .tbss	0000000000000008 .hidden perfetto::DataSource<base::perfetto_track_event::TrackEvent, perfetto::internal::TrackEventDataSourceTraits>::tls_state_
00000000000000b8 l       .tbss	0000000000000008 (anonymous namespace)::g_isolate_manager
00000000000000b8 l       .tbss	0000000000000000 $d
00000000000000c4 l       .tbss	0000000000000001 guard variable for absl::base_internal::GetCachedTID()::thread_id
00000000000000c0 l       .tbss	0000000000000004 absl::base_internal::GetCachedTID()::thread_id
00000000000000c0 l       .tbss	0000000000000000 $d
00000000000000c4 l       .tbss	0000000000000000 $d
00000000000000c8 l       .tbss	0000000000000018 absl::cord_internal::cordz_should_profile_slow(absl::cord_internal::SamplingState&)::exponential_biased_generator
00000000000000c8 l       .tbss	0000000000000000 $d
00000000000000e0 l       .tbss	0000000000000001 absl::log_internal::(anonymous namespace)::ThreadIsLoggingStatus()::thread_is_logging
00000000000000e0 l       .tbss	0000000000000000 $d
00000000000000e1 l       .tbss	0000000000000001 ipcz::(anonymous namespace)::is_thread_within_api_call
00000000000000e1 l       .tbss	0000000000000000 $d
00000000000000e8 l       .tbss	0000000000000000 $d
00000000000000e8 l       .tbss	0000000000000008 .hidden perfetto::DataSource<webrtc::perfetto_track_event::TrackEvent, perfetto::internal::TrackEventDataSourceTraits>::tls_state_
00000000000000f8 l       .tbss	0000000000000001 guard variable for SkStrikeCache::GlobalStrikeCache()::cache
00000000000000f0 l       .tbss	0000000000000008 SkStrikeCache::GlobalStrikeCache()::cache
00000000000000f0 l       .tbss	0000000000000000 $d
00000000000000f8 l       .tbss	0000000000000000 $d
0000000000000100 l       .tbss	0000000000000008 dawn::native::(anonymous namespace)::tlDevice
0000000000000100 l       .tbss	0000000000000000 $d
0000000000000108 l       .tbss	0000000000000000 $d
0000000000000108 l       .tbss	0000000000000008 .hidden perfetto::DataSource<v8::perfetto_track_event::TrackEvent, perfetto::internal::TrackEventDataSourceTraits>::tls_state_
0000000000000110 l       .tbss	0000000000000001 v8::internal::(anonymous namespace)::tls_singleton_taken
0000000000000118 l       .tbss	0000000000000018 v8::internal::(anonymous namespace)::tls_singleton_storage
0000000000000110 l       .tbss	0000000000000000 $d
0000000000000118 l       .tbss	0000000000000000 $d
0000000000000130 l       .tbss	0000000000000000 $d
0000000000000138 l       .tbss	0000000000000000 $d
0000000000000138 l       .tbss	0000000000000008 .hidden v8::internal::g_current_isolate_
0000000000000130 l       .tbss	0000000000000008 .hidden v8::internal::g_current_per_isolate_thread_data_
0000000000000140 l       .tbss	0000000000000004 v8::internal::(anonymous namespace)::thread_id
0000000000000140 l       .tbss	0000000000000000 $d
0000000000000148 l       .tbss	0000000000000008 v8::internal::(anonymous namespace)::current_marking_barrier
0000000000000148 l       .tbss	0000000000000000 $d
0000000000000150 l       .tbss	0000000000000008 v8::internal::(anonymous namespace)::pending_layout_change_object_address
0000000000000150 l       .tbss	0000000000000000 $d
0000000000000158 l       .tbss	0000000000000008 v8::internal::(anonymous namespace)::current_local_heap
0000000000000158 l       .tbss	0000000000000000 $d
0000000000000160 l       .tbss	0000000000000000 $d
0000000000000160 l       .tbss	0000000000000008 .hidden perfetto::DataSource<v8::internal::CodeDataSource, v8::internal::CodeDataSourceTraits>::tls_state_
0000000000000168 l       .tbss	0000000000000000 $d
0000000000000168 l       .tbss	0000000000000004 .hidden v8::internal::trap_handler::g_thread_in_wasm_code
0000000000000170 l       .tbss	0000000000000008 v8::internal::wasm::(anonymous namespace)::current_code_refs_scope
0000000000000170 l       .tbss	0000000000000000 $d
0000000000000178 l       .tbss	0000000000000000 $d
0000000000000178 l       .tbss	0000000000000008 .hidden v8::base::ContextualVariable<v8::internal::compiler::turboshaft::Tracing, v8::internal::compiler::turboshaft::Tracing>::top_
0000000000000180 l       .tbss	0000000000000000 $d
0000000000000180 l       .tbss	0000000000000008 .hidden v8::base::ContextualVariable<v8::internal::compiler::turboshaft::TypeInferenceReducerArgs, v8::internal::compiler::turboshaft::TypeInferenceReducerArgs>::top_
0000000000000188 l       .tbss	0000000000000000 $d
0000000000000188 l       .tbss	0000000000000008 .hidden content::media_stream_manager
0000000000000190 l       .tbss	0000000000000000 $d
0000000000000198 l       .tbss	0000000000000000 $d
0000000000000190 l       .tbss	0000000000000008 .hidden openscreen::internal::ScopedTraceOperation::traces_
0000000000000198 l       .tbss	0000000000000008 .hidden openscreen::internal::ScopedTraceOperation::root_node_
00000000000001a0 l       .tbss	0000000000000008 content::(anonymous namespace)::utility_thread
00000000000001a0 l       .tbss	0000000000000000 $d
00000000000001a8 l       .tbss	0000000000000000 $d
00000000000001a8 l       .tbss	0000000000000008 .hidden blink::g_thread_specific_
00000000000001b0 l       .tbss	0000000000000008 blink::(anonymous namespace)::current_thread
00000000000001b0 l       .tbss	0000000000000000 $d
00000000000001b8 l       .tbss	0000000000000008 webrtc::(anonymous namespace)::jingle_thread_wrapper
00000000000001b8 l       .tbss	0000000000000000 $d
00000000000001c0 l       .tbss	0000000000000008 extensions::(anonymous namespace)::contexts
00000000000001c0 l       .tbss	0000000000000000 $d
00000000000001c8 l       .tbss	0000000000000008 extensions::(anonymous namespace)::service_worker_data
00000000000001c8 l       .tbss	0000000000000000 $d
00000000000001d0 l       .tbss	0000000000000010 __cxxabiv1::(anonymous namespace)::__globals()::eh_globals
00000000000001d0 l       .tbss	0000000000000000 $d
00000000000001e0 l       .tbss	0000000000000001 __cxxabiv1::(anonymous namespace)::dtors_alive
00000000000001e8 l       .tbss	0000000000000008 __cxxabiv1::(anonymous namespace)::dtors
00000000000001e0 l       .tbss	0000000000000000 $d
00000000000001e8 l       .tbss	0000000000000000 $d
00000000000001f0 l       .tbss	0000000000000000 $d
00000000000001f0 l       .tbss	0000000000000008 .hidden base::internal::current_notification
00000000000001f8 l       .tbss	0000000000000008 base::(anonymous namespace)::delegate
0000000000000200 l       .tbss	0000000000000008 base::(anonymous namespace)::run_loop_timeout
00000000000001f8 l       .tbss	0000000000000000 $d
0000000000000200 l       .tbss	0000000000000000 $d
0000000000000208 l       .tbss	0000000000000001 base::internal::(anonymous namespace)::current_task_is_running_synchronously
0000000000000208 l       .tbss	0000000000000000 $d
0000000000000210 l       .tbss	0000000000000008 base::(anonymous namespace)::scoped_defer_task_posting
0000000000000210 l       .tbss	0000000000000000 $d
0000000000000218 l       .tbss	0000000000000008 base::(anonymous namespace)::current_pending_task
0000000000000228 l       .tbss	0000000000000008 base::(anonymous namespace)::current_long_task_tracker
0000000000000220 l       .tbss	0000000000000008 base::(anonymous namespace)::current_scoped_ipc_hash
0000000000000218 l       .tbss	0000000000000000 $d
0000000000000220 l       .tbss	0000000000000000 $d
0000000000000228 l       .tbss	0000000000000000 $d
0000000000000230 l       .tbss	0000000000000008 base::sequence_manager::(anonymous namespace)::thread_local_sequence_manager
0000000000000230 l       .tbss	0000000000000000 $d
0000000000000238 l       .tbss	0000000000000008 base::(anonymous namespace)::current_default_handle
0000000000000238 l       .tbss	0000000000000000 $d
0000000000000240 l       .tbss	0000000000000008 base::(anonymous namespace)::current_default_handle
0000000000000240 l       .tbss	0000000000000000 $d
0000000000000248 l       .tbss	0000000000000004 base::internal::(anonymous namespace)::fizzle_block_shutdown_tasks_ref
0000000000000248 l       .tbss	0000000000000000 $d
0000000000000250 l       .tbss	0000000000000008 base::internal::(anonymous namespace)::current_thread_group
0000000000000250 l       .tbss	0000000000000000 $d
0000000000000258 l       .tbss	0000000000000008 base::(anonymous namespace)::hang_watch_state
0000000000000258 l       .tbss	0000000000000000 $d
0000000000000260 l       .tbss	0000000000000008 base::internal::(anonymous namespace)::blocking_observer
0000000000000268 l       .tbss	0000000000000008 base::internal::(anonymous namespace)::last_scoped_blocking_call
0000000000000260 l       .tbss	0000000000000000 $d
0000000000000268 l       .tbss	0000000000000000 $d
0000000000000270 l       .tbss	0000000000000008 base::internal::(anonymous namespace)::current_sequence_local_storage
0000000000000270 l       .tbss	0000000000000000 $d
0000000000000278 l       .tbss	0000000000000001 base::(anonymous namespace)::tls_blocking_disallowed
0000000000000279 l       .tbss	0000000000000001 base::(anonymous namespace)::tls_base_sync_primitives_disallowed
000000000000027a l       .tbss	0000000000000001 base::(anonymous namespace)::tls_cpu_intensive_work_disallowed
000000000000027b l       .tbss	0000000000000001 base::(anonymous namespace)::tls_singleton_disallowed
0000000000000278 l       .tbss	0000000000000000 $d
0000000000000279 l       .tbss	0000000000000000 $d
000000000000027a l       .tbss	0000000000000000 $d
000000000000027b l       .tbss	0000000000000000 $d
0000000000000280 l       .tbss	0000000000000008 base::(anonymous namespace)::fd_watcher
0000000000000280 l       .tbss	0000000000000000 $d
0000000000000288 l       .tbss	0000000000000008 base::trace_event::(anonymous namespace)::thread_local_event_buffer
0000000000000290 l       .tbss	0000000000000001 base::trace_event::(anonymous namespace)::thread_blocks_message_loop
0000000000000291 l       .tbss	0000000000000001 base::trace_event::(anonymous namespace)::thread_is_in_trace_event
0000000000000298 l       .tbss	0000000000000008 base::trace_event::TraceLog::ShouldAddAfterUpdatingState(char, unsigned char const*, char const*, unsigned long, int, base::TimeTicks, base::trace_event::TraceArguments*)::current_thread_name
0000000000000288 l       .tbss	0000000000000000 $d
0000000000000290 l       .tbss	0000000000000000 $d
0000000000000291 l       .tbss	0000000000000000 $d
0000000000000298 l       .tbss	0000000000000000 $d
00000000000002a0 l       .tbss	0000000000000001 base::tracing::GetThreadIsInTraceEvent()::thread_is_in_trace_event
00000000000002a0 l       .tbss	0000000000000000 $d
00000000000002a8 l       .tbss	0000000000000008 mojo::internal::(anonymous namespace)::g_thread_local_node
00000000000002a8 l       .tbss	0000000000000000 $d
00000000000002b0 l       .tbss	0000000000000008 mojo::(anonymous namespace)::g_end_to_end_metric
00000000000002b0 l       .tbss	0000000000000000 $d
00000000000002c8 l       .tbss	0000000000000001 __tls_guard
00000000000002b8 l       .tbss	0000000000000010 mojo::(anonymous namespace)::g_sub_sampler
00000000000002b8 l       .tbss	0000000000000000 $d
00000000000002c8 l       .tbss	0000000000000000 $d
00000000000002c9 l       .tbss	0000000000000001 mojo::(anonymous namespace)::is_in_urgent_message_scope
00000000000002c9 l       .tbss	0000000000000000 $d
00000000000002f0 l       .tbss	0000000000000001 guard variable for quiche::(anonymous namespace)::Xoshiro256PlusPlus()::rng_state
00000000000002d0 l       .tbss	0000000000000020 quiche::(anonymous namespace)::Xoshiro256PlusPlus()::rng_state
00000000000002d0 l       .tbss	0000000000000000 $d
00000000000002f0 l       .tbss	0000000000000000 $d
00000000000002f8 l       .tbss	0000000000000008 quic::(anonymous namespace)::current_context
00000000000002f8 l       .tbss	0000000000000000 $d
0000000000000300 l       .tbss	0000000000000001 IPC::(anonymous namespace)::off_sequence_binding_allowed
0000000000000300 l       .tbss	0000000000000000 $d
0000000000000308 l       .tbss	0000000000000008 IPC::(anonymous namespace)::received_queue
0000000000000308 l       .tbss	0000000000000000 $d
0000000000000310 l       .tbss	0000000000000000 $d
0000000000000310 l       .tbss	0000000000000008 .hidden perfetto::DataSource<tracing::MetadataDataSource, perfetto::DefaultDataSourceTraits>::tls_state_
0000000000000318 l       .tbss	0000000000000000 $d
0000000000000318 l       .tbss	0000000000000008 .hidden perfetto::DataSource<tracing::PerfettoTracedProcess::DataSourceProxy<tracing::TraceEventMetadataSource>, perfetto::DefaultDataSourceTraits>::tls_state_
0000000000000320 l       .tbss	0000000000000000 $d
0000000000000320 l       .tbss	0000000000000008 .hidden perfetto::DataSource<tracing::TriggersDataSource, perfetto::DefaultDataSourceTraits>::tls_state_
0000000000000328 l       .tbss	0000000000000001 variations::(anonymous namespace)::in_set_field_trial_group_from_browser
0000000000000328 l       .tbss	0000000000000000 $d
0000000000000330 l       .tbss	0000000000000008 webrtc::(anonymous namespace)::current
0000000000000330 l       .tbss	0000000000000000 $d
0000000000000338 l       .tbss	0000000000000008 SkSL::sMemPool
0000000000000338 l       .tbss	0000000000000000 $d
0000000000000340 l       .tbss	0000000000000008 skgpu::ganesh::gCache
0000000000000340 l       .tbss	0000000000000000 $d
0000000000000348 l       .tbss	0000000000000008 ui::(anonymous namespace)::event_source
0000000000000348 l       .tbss	0000000000000000 $d
0000000000000350 l       .tbss	0000000000000008 gl::(anonymous namespace)::current_context
0000000000000358 l       .tbss	0000000000000008 gl::(anonymous namespace)::current_real_context
0000000000000350 l       .tbss	0000000000000000 $d
0000000000000358 l       .tbss	0000000000000000 $d
0000000000000360 l       .tbss	0000000000000008 gl::ThreadLocalCurrentGL()::current_gl
0000000000000360 l       .tbss	0000000000000000 $d
0000000000000368 l       .tbss	0000000000000008 gl::(anonymous namespace)::current_surface
0000000000000368 l       .tbss	0000000000000000 $d
0000000000000370 l       .tbss	0000000000000000 $d
0000000000000370 l       .tbss	0000000000000008 .hidden re2::hooks::context
0000000000000378 l       .tbss	0000000000000008 gpu::(anonymous namespace)::current_task_runner
0000000000000378 l       .tbss	0000000000000000 $d
0000000000000384 l       .tbss	0000000000000001 guard variable for WTF::CurrentThread()::g_id
0000000000000380 l       .tbss	0000000000000004 WTF::CurrentThread()::g_id
0000000000000380 l       .tbss	0000000000000000 $d
0000000000000384 l       .tbss	0000000000000000 $d
0000000000000385 l       .tbss	0000000000000000 $d
0000000000000385 l       .tbss	0000000000000001 .hidden WTF::g_is_main_thread
0000000000000388 l       .tbss	0000000000000008 v8::base::(anonymous namespace)::thread_stack_start
0000000000000388 l       .tbss	0000000000000000 $d
0000000000000390 l       .tbss	0000000000000000 $d
0000000000000390 l       .tbss	0000000000000008 .hidden perfetto::DataSource<tracing::PerfettoTracedProcess::DataSourceProxy<memory_instrumentation::TracingObserver>, perfetto::DefaultDataSourceTraits>::tls_state_
0000000000000398 l       .tbss	0000000000000008 metrics::(anonymous namespace)::provider
0000000000000398 l       .tbss	0000000000000000 $d
00000000000003a0 l       .tbss	0000000000000008 gpu::webgpu::(anonymous namespace)::parent_decoder
00000000000003a0 l       .tbss	0000000000000000 $d
00000000000003a8 l       .tbss	0000000000000008 content::(anonymous namespace)::child_process
00000000000003a8 l       .tbss	0000000000000000 $d
00000000000003b0 l       .tbss	0000000000000008 content::(anonymous namespace)::child_thread_impl
00000000000003b0 l       .tbss	0000000000000000 $d
00000000000003b8 l       .tbss	0000000000000000 $d
00000000000003e8 l       .tbss	0000000000000000 $d
00000000000003e8 l       .tbss	0000000000000008 .hidden guard variable for blink::HeapSizeCache::ForCurrentThread()::heap_size_cache
00000000000003b8 l       .tbss	0000000000000030 .hidden blink::HeapSizeCache::ForCurrentThread()::heap_size_cache
00000000000003f0 l       .tbss	0000000000000000 $d
00000000000003f0 l       .tbss	0000000000000004 .hidden blink::script_forbidden_counter
00000000000003f8 l       .tbss	0000000000000008 content::(anonymous namespace)::render_thread
00000000000003f8 l       .tbss	0000000000000000 $d
0000000000000400 l       .tbss	0000000000000008 content::(anonymous namespace)::render_thread
0000000000000400 l       .tbss	0000000000000000 $d
0000000000000408 l       .tbss	0000000000000008 content::(anonymous namespace)::worker_data
0000000000000408 l       .tbss	0000000000000000 $d
0000000000000410 l       .tbss	0000000000000008 AffixMgr::compound_check(std::__Cr::basic_string<char, std::__Cr::char_traits<char>, std::__Cr::allocator<char>> const&, short, short, short, short, hentry**, hentry**, char, char, int*)::timelimit
0000000000000418 l       .tbss	0000000000000008 AffixMgr::compound_check_morph(char const*, int, short, short, short, short, hentry**, hentry**, char, std::__Cr::basic_string<char, std::__Cr::char_traits<char>, std::__Cr::allocator<char>>&, std::__Cr::basic_string<char, std::__Cr::char_traits<char>, std::__Cr::allocator<char>> const*)::timelimit
0000000000000410 l       .tbss	0000000000000000 $d
0000000000000418 l       .tbss	0000000000000000 $d
0000000000000428 l       .tbss	0000000000000001 guard variable for sentencepiece::random::GetRandomGenerator()::mt
0000000000000420 l       .tbss	0000000000000008 sentencepiece::random::GetRandomGenerator()::mt
0000000000000420 l       .tbss	0000000000000000 $d
0000000000000428 l       .tbss	0000000000000000 $d
0000000000000430 l       .tbss	0000000000000000 $d
0000000000000430 l       .tbss	0000000000000008 .hidden gwp_asan::internal::ThreadLocalState<gwp_asan::internal::ThreadLocalRandomBitGenerator>::state_
0000000000000438 l       .tbss	0000000000000000 $d
0000000000000438 l       .tbss	0000000000000008 .hidden gwp_asan::internal::ThreadLocalState<gwp_asan::internal::SamplingState<(gwp_asan::internal::ParentAllocator)2>>::state_
0000000000000440 l       .tbss	0000000000000000 $d
0000000000000440 l       .tbss	0000000000000008 .hidden gwp_asan::internal::ThreadLocalState<gwp_asan::internal::SamplingState<(gwp_asan::internal::ParentAllocator)0>>::state_
0000000000000448 l       .tbss	0000000000000000 $d
0000000000000448 l       .tbss	0000000000000008 .hidden gwp_asan::internal::ThreadLocalState<gwp_asan::internal::SamplingState<(gwp_asan::internal::ParentAllocator)1>>::state_
0000000000000450 l       .tbss	0000000000000000 $d
0000000000000450 l       .tbss	0000000000000018 std::hash::random::RandomState::new::KEYS::_$u7b$$u7b$constant$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::VAL::hd9d6164a1b6881c6

However, please let me know @magreenblatt if anyone knows a better solution :

Hethsron avatar Oct 15 '24 09:10 Hethsron

I confirm this issue is present in Ubuntu 24.10 for ARM64 running in a Raspberry Pi 5. The workaround works in this case.

However, the official "Raspberry Pi OS" distribution is not affected by this issue and it's possible to load libcef.so without problems.

All tests were made with CEF 129.0.12 and CEF4Delphi in Lazarus

salvadordf avatar Oct 15 '24 15:10 salvadordf

@Hethsron Thanks for the details. I'm not seeing any large new contributors to MemSiz in your output.

In case of libcef.so (M-129), the output shows [...] a memory size (MemSiz) of 0x000430 [...] libcef.so version M~125 [...] MemSiz 0x0003d8

Can you compare the objdump -t libcef.so | grep -F '.tbss' output more closely between the versions to see where the additional memory usage is coming from? If you can identify a specific new culprit then we can potentially address it. It would be best to compare with M128, or whichever version most recently did not exhibit the problem.

Additionally:

Thus, the segment size in hex is 0x000430 and equal to 1072 bytes.

Whereas, from issue #3616:

The effective surplus allocated by default in x64 Linux (by ld-linux, which is the lib responsible to do this allocation) is sufficient for about 1600 or so bytes.

This suggests that your ARM64 system has a substantially lower threshold than the x64 default. You might want to see if you can adjust that in your system or kernel configuration.

magreenblatt avatar Oct 18 '24 17:10 magreenblatt

@Hethsron Thanks for the details. I'm not seeing any large new contributors to MemSiz in your output.

In case of libcef.so (M-129), the output shows [...] a memory size (MemSiz) of 0x000430 [...] libcef.so version M~125 [...] MemSiz 0x0003d8

Can you compare the objdump -t libcef.so | grep -F '.tbss' output more closely between the versions to see where the additional memory usage is coming from? If you can identify a specific new culprit then we can potentially address it. It would be best to compare with M128, or whichever version most recently did not exhibit the problem.

@magreenblatt Here is a script I developed to answer this request :

#!/bin/bash

# Array of folder names
folders=(cef125 cef126 cef127 cef128 cef129 cef130 cef131)

# Create a directory to store temporary files for comparisons
temp_dir=$(mktemp -d)
trap "rm -rf $temp_dir" EXIT  # Clean up temp files on exit

# Collect the output of llvm-objdump for each folder
for folder in "${folders[@]}"; do
    if [[ -f "$folder/libcef.so" ]]; then
        echo "Processing $folder/libcef.so"
        # Run llvm-objdump and filter for .tbss, store in a temp file
        llvm-objdump -C -t "$folder/libcef.so" | grep -F '.tbss' > "$temp_dir/$folder.tbss"
    else
        echo "Warning: $folder/libcef.so not found, skipping."
    fi
done

# Compare each version's output line-by-line using diff
for ((i=0; i<${#folders[@]}-1; i++)); do
    for ((j=i+1; j<${#folders[@]}; j++)); do
        folder1=${folders[i]}
        folder2=${folders[j]}
        file1="$temp_dir/$folder1.tbss"
        file2="$temp_dir/$folder2.tbss"
        
        if [[ -f "$file1" && -f "$file2" ]]; then
            echo "Comparing $folder1 with $folder2:"
            diff -y --suppress-common-lines "$file1" "$file2" || echo "Differences found"
            echo "---------------------------------------"
        fi
    done
done

To get the output, you can execute the following command:

bash-4.4$ chmod +x compare_cef_tbss.sh
bash-4.4$ ./compare_cef_tbss.sh > tbss_comparison.txt

Hethsron avatar Oct 21 '24 07:10 Hethsron

@Hethsron Thanks, that's the first step. Now compare the output to identify what's changed.

magreenblatt avatar Oct 21 '24 19:10 magreenblatt

I'm testing on Ubuntu 24.10 Concept for Snapdragon X Elite arm64 laptops, I've tested CEF 118, 127, 128 and 130 and for me they all show the "cannot allocate memory in static TLS block" error when dynamically loading libcef.so

FlavionTrack avatar Oct 29 '24 09:10 FlavionTrack

I'm testing on Ubuntu 24.10 Concept for Snapdragon X Elite arm64 laptops, I've tested CEF 118, 127, 128 and 130 and for me they all show the "cannot allocate memory in static TLS block" error when dynamically loading libcef.so

@magreenblatt I will let you know as soon as possible if i perform to identify what's changed on those versions of CEF.

Hethsron avatar Nov 12 '24 13:11 Hethsron

@magreenblatt As promised, I'll get back to you with the results. I used the following CEF versions from spotify :

  1. cef_binary_125.0.22+gc410c95+chromium-125.0.6422.142_linuxarm64_client.tar.bz2

  2. cef_binary_126.2.18+g3647d39+chromium-126.0.6478.183_linuxarm64_client.tar.bz2

  3. cef_binary_127.3.5+g114ea2a+chromium-127.0.6533.120_linuxarm64_client.tar.bz2

  4. cef_binary_128.4.12+g1d7a1f9+chromium-128.0.6613.138_linuxarm64_client.tar.bz2

  5. cef_binary_129.0.12+gf09539f+chromium-129.0.6668.101_linuxarm64_client.tar.bz2

  6. cef_binary_130.1.16+g5a7e5ed+chromium-130.0.6723.117_linuxarm64_client.tar.bz2

  7. cef_binary_131.2.1+ge081753+chromium-131.0.6778.13_linuxarm64_client.tar.bz2

And when I run my script, I have the following output : tbss_comparison.txt

Can you see the changes for yourself? Thanks in advance for your feedback

Regards

Hethsron

Hethsron avatar Nov 15 '24 00:11 Hethsron

Disabling PA_CONFIG_THREAD_CACHE_FAST_TLS in the Chromium build here may help, similar to https://issues.chromium.org/issues/40642257#comment207 which was applied for ARM ChromeOS.

EDIT: It looks like Chromium stopped using thread_local on Linux in this commit (M120).

magreenblatt avatar Jan 09 '25 18:01 magreenblatt

Disabling PA_THREAD_CACHE_FAST_TLS in the Chromium build may help, similar to https://issues.chromium.org/issues/40642257#comment207 which was applied for ARM ChromeOS.

Thank you @magreenblatt for this solution. I'll test it and let you know as soon as possible if it fixes our issue.

Hethsron avatar Jan 09 '25 19:01 Hethsron

@Hethsron Sorry, it's not a solution for versions >= M120. See EDIT above.

magreenblatt avatar Jan 09 '25 19:01 magreenblatt

@Hethsron Sorry, it's not a solution for versions >= M120. See EDIT above.

Ok Thank you for update.

Hethsron avatar Jan 09 '25 22:01 Hethsron

Added the example code from the first comment and repro here: https://github.com/djrecipe/CefIssue3803-TLSAlloc

Tested using CEF 109 with same error. LD_PRELOAD works but not viable

djrecipe avatar Apr 16 '25 08:04 djrecipe

I think this isn't a big TLS increase, but rather death by one thousand cuts. I'm trying to do CEF builds in Fedora, and 136 worked with a size of 0x0004a4 while 137 failed with a size of 0x0004f0. The client app in this case is OBS, which loads other libraries which probably use TLS. So it was probably already close to the edge, and small additions brought it over.

hoshinolina avatar Jun 17 '25 12:06 hoshinolina

I think I'm getting somewhere. I think the issue is that if any TLS access in the entire libcef.so uses static TLS, then the entire TLS usage of libcef.so becomes a static allocation. So the right fix here is not to try to reduce TLS usage, but rather to identify static TLS usage and change it to dynamic.

One culprit I found so far is the linux_blink_thread_local patch. It is an incorrect fix, as it changes the TLS model from local-exec to initial-exec. What you actually want in a shared library is local-dynamic, which would be set by BLINK_HEAP_HIDE_THREAD_LOCAL_IN_LIBRARY, which is set by BLINK_HEAP_INSIDE_SHARED_LIBRARY, which comes from the blink_heap_inside_shared_library GN define. It's kind of funny, the comment says:

declare_args() {
  # Whether the blink heap code is compiled into a shared library.
  # Embedders like CEF use this flag to include blink in a shared library under
  # non-component build.
  blink_heap_inside_shared_library = is_component_build
}

... but CEF does not actually set this!

I'm trying a build now with that patched in, to see if there are any others left (I saw some more object files with static TLS relocs but I'm not sure if they contribute to the final libcef.so yet).

hoshinolina avatar Jun 17 '25 13:06 hoshinolina

Another one is components/gwp_asan/client/thread_local_state.h. This needs the extra define THREAD_LOCAL_STATE_USES_PARTITION_ALLOC_TLS. That probably needs a patch similar to v8_build.patch to set it.

And that's it! With those two changes, libcef.so no longer uses static TLS at all, and the issue is fixed.

For future reference, this shell snippet can be used to look through the CEF build tree and find object files with static TLS relocations (on aarch64):

find . -name '*.o' | while read a; do
    if llvm-objdump -r "$a" 2>/dev/null | grep TPREL; then
        echo "> $a"
    fi
done

hoshinolina avatar Jun 17 '25 13:06 hoshinolina

@asahilina can you share your fixes as a pull request? Thanks.

magreenblatt avatar Jun 20 '25 16:06 magreenblatt

Another one is components/gwp_asan/client/thread_local_state.h. This needs the extra define THREAD_LOCAL_STATE_USES_PARTITION_ALLOC_TLS. That probably needs a patch similar to v8_build.patch to set it.

And that's it! With those two changes, libcef.so no longer uses static TLS at all, and the issue is fixed.

For future reference, this shell snippet can be used to look through the CEF build tree and find object files with static TLS relocations (on aarch64):

find . -name '*.o' | while read a; do if llvm-objdump -r "$a" 2>/dev/null | grep TPREL; then echo "> $a" fi done

Thanks @asahilina for your efforts on the subject. Would it be possible for you to submit your solution in the form of a pull request so that it benefits the whole community?

Hethsron avatar Jun 27 '25 15:06 Hethsron

I'm pretty busy the next couple of weeks and I don't have a "stand alone" CEF tree (outside the Fedora RPM infra) that's ready to build and test, so it's a bit tricky for me to test a PR. Perhaps someone else can give it a go?

It's just adding that GN define and writing a simple BUILD.gn patch to add that C define to the gwp_asan build.

On June 28, 2025 12:20:40 AM GMT+09:00, "@" @.***> wrote:

Hethsron left a comment (chromiumembedded/cef#3803)

Another one is components/gwp_asan/client/thread_local_state.h. This needs the extra define THREAD_LOCAL_STATE_USES_PARTITION_ALLOC_TLS. That probably needs a patch similar to v8_build.patch to set it.

And that's it! With those two changes, libcef.so no longer uses static TLS at all, and the issue is fixed.

For future reference, this shell snippet can be used to look through the CEF build tree and find object files with static TLS relocations (on aarch64):

find . -name '*.o' | while read a; do if llvm-objdump -r "$a" 2>/dev/null | grep TPREL; then echo "> $a" fi done

Thanks @asahilina for your efforts on the subject. Would it be possible for you to submit your solution in the form of a pull request so that it benefits the whole community?

-- Reply to this email directly or view it on GitHub: https://github.com/chromiumembedded/cef/issues/3803#issuecomment-3013439473 You are receiving this because you were mentioned.

Message ID: @.***> ~~ Lina

hoshinolina avatar Jun 27 '25 15:06 hoshinolina

I just tested the latest CEF4Delphi version which uses CEF 138.0.15 in a fully updated Raspberry Pi OS running on a Raspberry Pi 5 and I got the "cannot allocate memory in static TLS block" error when trying to load libcef.so.

salvadordf avatar Jun 29 '25 15:06 salvadordf

For future reference, this shell snippet can be used to look through the CEF build tree and find object files with static TLS relocations (on aarch64):

find . -name '*.o' | while read a; do
    if llvm-objdump -r "$a" 2>/dev/null | grep TPREL; then
        echo "> $a"
    fi
done

Looks like the search strings are R_AARCH64_TLS_TPREL64 for ARM64 and R_X86_64_TPOFF64 for x86_64.

Testing with M138 before any changes, using a local x86_64 Release ASAN build, I'm not seeing any results with your sample script. However, testing the binary does show some results:

$ readelf -Wr "out/Release_GN_x64/libcef.so" | grep -E 'R_AARCH64_TLS_TPREL64|R_X86_64_TPOFF64'
0000000058893510  0000000000000012 R_X86_64_TPOFF64                          238
0000000058898028  0000000000000012 R_X86_64_TPOFF64                          2d28
0000000058898038  0000000000000012 R_X86_64_TPOFF64                          2d30

I will attempt to apply your changes and will post a PR if the build succeeds.

magreenblatt avatar Jul 03 '25 19:07 magreenblatt

The readelf output is empty after applying the changes, so it likely worked.

@asahilina can you review https://bitbucket.org/chromiumembedded/cef/pull-requests/919? Thanks.

magreenblatt avatar Jul 03 '25 21:07 magreenblatt

Reviewed!

hoshinolina avatar Jul 04 '25 09:07 hoshinolina

For anyone else who is stumbling upon this issue and looking for a workaround until the upstream fix is ready: GLIBC_TUNABLES=glibc.rtld.optional_static_tls=2048 will fix it on glibc systems and doesn't involve LD_PRELOAD.

Edit: This doesn't seem to always work, unfortunately. I'm not sure why...

hoshinolina avatar Jul 04 '25 10:07 hoshinolina

Please test in the upcoming M139 beta build. If it works there we can also merge to M138.

magreenblatt avatar Jul 10 '25 20:07 magreenblatt

I just tested a custom version of CEF4Delphi with CEF 139 and it works perfectly in the latest Raspberry Pi OS version and also in the latest Ubuntu version. I used a Raspberry Pi 5 and Lazarus 4.0.

Thank you very much! :)

salvadordf avatar Jul 11 '25 19:07 salvadordf

Please test in the upcoming M139 beta build. If it works there we can also merge to M138.

Hi

I've also run my test with the ARM64 version of CEF 139 and I no longer get any errors.

Run :

bash-4.4$ ./a.out 
bash-4.4$

The output is now empty.

Thank you for all

Hethsron avatar Jul 12 '25 08:07 Hethsron