[Epic] EuiDataGrid long-term plan and investigation

[tkajtoch]

✨ Summary

EuiDataGrid is one of the largest and most important components EUI provides. Since its initial implementation in 2019, it has received numerous fixes and improvements making it suit growing Kibana use cases better.

❗ Problem or Opportunity

We're at a point where the current architecture of EuiDataGrid limits us quite heavily. Features like sticky rows and columns aren't possible to implement without workarounds due to the underlying virtualization library limitations, and a major refactor is long overdue.

The goal of this work is to investigate the current state of the component and create a plan to improve it for long-term maintainability and extensibility.

Objectives

1. Understand and document the current EuiDataGrid technical implementation Before we can work on any improvements or refactors, we need to list all features (and quirks) that we currently provide, including graphs to visualize the data flow and various relationships. This is a prerequisite for any comparisons.
2. Document and analyze all internal usages of EuiDataGrid, including the most used and requested features UnifiedDataTable and other wrappers extending or otherwise controlling the functionality of EuiDataGrid. We should aim to provide a consistent data grid experience across the product and minimize solution-specific solutions. This point includes features we weren't able to implement in the past due to technical limitations, the lack of time, or prioritization.
3. Understand real Kibana needs for the component Due to the age of EuiDataGrid, we need to ensure we don't maintain unnecessary features and provide the most value possible. This should be documented alongside the list of currently supported features from point objective 1
4. Estimate the efforts of maintaining the component as is (1), refactoring it to improve maintainability (2), and refactoring its internals to integrate an off-the-shelf solution like TanStack Table to offload some of the base-level functionality (3) We're looking forward to switching to a solution like TanStack Table, but before we commit to it, we must understand the scale of changes it would require. Krzysztof and Eyo have recently experimented with that library in Kibana with excellent results.
5. Plan the future of EuiDataGrid, considering the points above and more Aspects like cross-team collaboration, implementing user research feedback, and providing long-term extensibility are key We should aim for proper separation of concerns - EUI should focus on providing great UI, while the logic should likely live in a separate package where any team can contribute.

🎯 Value

This investigation aims to address questions about the current state of EuiDataGrid and the technical strategy moving forward. We want to get a deep understanding of the current implementation and its limitations, and assess how big of an investment this component needs, including future product needs and requests that we have been unable to fulfill due to the current EuiDataGrid architecture.

🎯 Urgency

We're highly aware of the current limitations and complexities of EuiDataGrid. Each issue and feature request related to it comes with a risk and a significant time investment, which directly affects the EUI team. While the need for this effort may not be visible externally, I strongly believe it should be considered as urgent due to:

1. The EUI team currently lacks understanding of some EuiDataGrid internals, which increases the time we have to invest in debugging, thinking, and implementing changes.
2. EuiDataGrid comes with a set of challenges specific to the internal implementation, especially around cell rendering and virtualization. Used solutions work, but greatly increase complexity, which we want to reduce whenever possible.
3. Other open-source solutions like TanStack Table seem to provide a solid base layer for our usage and include features like virtualization and drag-and-drop out of the box. Outsourcing features like these to an external library allows us to invest more time in product-specific functionality, rather than working on problems that have already been solved.
4. The logic is tightly connected to UI elements, making it impossible to reuse pieces of EuiDataGrid in other places.
5. The API is cumbersome and limiting at times. Its inflexibility necessitates the use of wrapper components, such as UnifiedDataTable, which provide extended functionality. These may lead to fragmentation and uneven UX.

🧠 Feasibility & Direction

It's very feasible to execute this investigation. The goals are clear - see below.

❓ Open Questions to answer

Definition of Done

• EuiDataGrid internals are documented in an easy-to-follow format
• The investigation findings and results are documented
• The long-term, detailed plan for EuiDataGrid is documented in an RFC and shared internally
• All feature requests are circled back on and prioritized or declined; a finalized list of required and optional features is put together.