[DISCUSS]: Roadmap 2024

[chunshao90]

Here are some of the features I have planned for 2024. Everyone is welcome to discuss.

Distributed Solution

• Meta Procedure Module
  • Refine operations such as retries and rollbacks for different procedures.
• Automated Load Balancing Based on Real Load
  • Implement dynamic load information collection mechanisms, with horaedb uploading real load data.
  • Incorporate advanced scheduling algorithms to balance load distribution based on actual load conditions.
• Increase chaos testing to validate system robustness.

Query Optimization

• Optimize PQL by supporting operator pushdown to reduce data retrieval costs.
• Add SST-level sorted keys to improve data-fetching efficiency during queries.
• Pre-Aggregation and Pre-Downsample.

Storage Optimization

• Improve query performance for both high and low cardinality by enhancing storage formats
  • Introduce inverted indexes.
  • Organize data along timelines.
  • Implement a columnar storage Memtable.
• Compaction Offload
  • Offload compaction tasks to dedicated nodes or services to minimize the impact on real-time query nodes.
• Import opendal.

Other

• horaecontrol cluster command-line tool, supporting functions such as automated deployment, monitoring integration, and fault diagnosis.
• Optimize the code directory structure.

Need to discuss

~~- Evaluate the feasibility of reimplementing horaemeta with Rust to unify the development language and improve development efficiency.~~

[ShiKaiWi]

I guess Pre-Aggregation and Pre-Downsample is also an attractive feature to help improve the performance of the queries involving massive data points.

[tanruixiang]

• import opendal
• file structure simplify

[kcrazy]

Support Windows system.

[jiacai2050]

Support Windows system.

Hi, would you give us more background about your user case?

We haven't test horaedb on Windows since there is no one asking for this feature.

[kcrazy]

Support Windows system.

Hi, would you give us more background about your user case?

We haven't test horaedb on Windows since there is no one asking for this feature.

Although my production environment is Linux, my development and testing environments are on Windows. Sometimes I also perform tasks on Windows.

If there is a native support version for Windows, it would be more convenient to use, and this is also a factor I consider in whether to use horaedb.

[jiacai2050]

Thanks, that's sounds reasonable for me.

In theory, IMO horaedb don't use any platform specific features, so it may work on Windows, but I can't ensure that, I think we need someone to try build/run on windows, would you like to give it a try? I create a tracking issue here.

[kcrazy]

Thanks, that's sounds reasonable for me.

In theory, IMO horaedb don't use any platform specific features, so it may work on Windows, but I can't ensure that, I think we need someone to try build/run on windows, would you like to give it a try? I create a tracking issue here.

ok