[perf tuning discussion] params.sq_thread_idle will impact performance a lot

[gaowayne]

hello expert, this is related with https://github.com/axboe/liburing/issues/1415, after a lot of parameter tuning. I can reach 1300MB/s now. still have gap with fio. but I found one interesting thing is that. params.sq_thread_idle will impact performance a lot!

if we sq_thread_idle = 120000 that means sq kernel thread should busy polling. but after I enable it I saw the performance drop from 1200MB/s to 900MB/s

here is my example code

/*
 * Direct Submission NVMe Framework
 * 
 * High-performance NVMe I/O using io_uring with direct submission architecture.
 * Eliminates traditional I/O bottlenecks through zero-copy operations and 
 * efficient batching.
 * 
 * Architecture:
 * - Main thread: Direct io_uring submission with batching
 * - Backend thread: Dedicated completion polling (CPU pinned)
 * - Kernel thread: SQPOLL/IOPOLL for efficient device interaction
 * 
 * Key Features:
 * - Zero-copy direct submission to io_uring
 * - Lock-free completion processing
 * - Intelligent batching to reduce syscall overhead
 * - CPU isolation for optimal performance
 * - High queue depth utilization (configurable)
 */

#include <iostream>
#include <cstring>
#include <fcntl.h>
#include <unistd.h>
#include <sys/ioctl.h>
#include <sys/mman.h>
#include <linux/nvme_ioctl.h>
#include <liburing.h>
#include <errno.h>
#include <thread>
#include <atomic>
#include <chrono>
#include <vector>
#include <memory>
#include <signal.h>
#include <sched.h>
#include <pthread.h>
#include <iomanip>

// Modern kernel feature definitions
#ifndef IORING_SETUP_SQE128
#define IORING_SETUP_SQE128 (1U << 10)
#endif
#ifndef IORING_SETUP_CQE32
#define IORING_SETUP_CQE32 (1U << 11)
#endif
#ifndef IORING_OP_URING_CMD
#define IORING_OP_URING_CMD 46
#endif
#ifndef NVME_URING_CMD_IO
#define NVME_URING_CMD_IO _IOWR('N', 0x80, struct nvme_uring_cmd)
#endif

// Global shutdown flag
std::atomic<bool> g_running{true};

void signal_handler(int signal) {
    std::cout << "\n🛑 Received signal " << signal << ", shutting down gracefully..." << std::endl;
    g_running.store(false);
}

// NVMe device information
struct nvme_data {
    __u32 nsid;
    __u32 lba_shift;
    __u32 lba_size;
};

enum nvme_io_opcode {
    nvme_cmd_read = 0x02,
    nvme_cmd_write = 0x01,
};



/**
 * ═══════════════════════════════════════════════════════════════════════════════
 * DIRECT SUBMISSION HIGH-PERFORMANCE NVME FRAMEWORK
 * ═══════════════════════════════════════════════════════════════════════════════
 * 
 * 🚀 REVOLUTIONARY ARCHITECTURE:
 * ┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
 * │   Main Thread   │    │ Backend Thread  │    │ Kernel Thread   │
 * │                 │    │                 │    │                 │
 * │ • Direct submit │    │ • Poll completions│   │ • SQPOLL        │
 * │ • Zero queuing  │    │ • Fill comp queue│    │ • Device I/O    │
 * │ • Max IOPS      │    │ • CPU affinity  │    │ • Zero-copy     │
 * └─────────────────┘    └─────────────────┘    └─────────────────┘
 *        │                        │                        │
 *        └────── Direct ──────────┼────────────────────────┘
 *                                 │
 *                    ┌─────────────────┐
 *                    │  Completion     │
 *                    │  Queue          │
 *                    │  (Results)      │
 *                    └─────────────────┘
 */
class DirectSubmissionNVMe {
private:
    // Core queue sizing
    static constexpr size_t QUEUE_DEPTH = 512;
    static constexpr size_t MAX_BATCH_COMPLETE = 2048;
    // ═══════════════════════════════════════════════════════════════════════════
    // 💾 DEVICE AND I/O INFRASTRUCTURE
    // ═══════════════════════════════════════════════════════════════════════════
    
    int fd_ = -1;                    // NVMe device file descriptor
    struct nvme_data device_;        // Device characteristics (namespace, LBA size)
    struct io_uring ring_;           // io_uring instance with IOPOLL/SQPOLL
    
    // ═══════════════════════════════════════════════════════════════════════════
    // 🧵 THREADING INFRASTRUCTURE - CPU ISOLATION STRATEGY  
    // ═══════════════════════════════════════════════════════════════════════════
    
    std::unique_ptr<std::thread> backend_thread_;  // Dedicated completion polling thread
    std::atomic<bool> running_{false};             // Global shutdown coordination
    std::atomic<bool> ready_{false};               // Backend thread ready signal
    
    // ═══════════════════════════════════════════════════════════════════════════
    // 🔄 DIRECT COUNTER COMMUNICATION - MAXIMUM PERFORMANCE  
    // ═══════════════════════════════════════════════════════════════════════════
    
    // Monitoring counters
    
    // Core operation counters
    uint64_t total_submitted_{0};                  // Total operations submitted to io_uring (main thread only)
    std::atomic<uint64_t> total_completed_{0};     // Total operations completed successfully (precise for BW calculation)
    volatile uint64_t total_errors_{0};            // Total I/O errors encountered (less critical, volatile is fine)

    
    // Queue utilization tracking
    std::atomic<uint32_t> io_uring_in_flight_{0};  // Current operations in io_uring
    std::atomic<uint32_t> max_in_flight_seen_{0};  // Peak queue utilization achieved


    
    // ═══════════════════════════════════════════════════════════════════════════
    // 🎯 SIMPLE FLOW CONTROL - BASIC TRACKING
    // ═══════════════════════════════════════════════════════════════════════════
    
    // Batching configuration - combine with SQPOLL for optimal performance
    static constexpr size_t SUBMIT_BATCH_SIZE = 128;
    uint32_t pending_submissions_{0};                 // Count of unbatched SQEs waiting (main thread only)

    
public:
    DirectSubmissionNVMe() = default;
    
    bool initialize(const std::string& device_path) {
        std::cout << "🚀 Initializing Direct Submission NVMe Framework" << std::endl;
        std::cout << "   Architecture: Main thread direct submission + Backend completion polling" << std::endl;
        std::cout << "   Target device: " << device_path << std::endl;
        
        // Open NVMe device
        fd_ = open(device_path.c_str(), O_RDWR);
        if (fd_ < 0) {
            std::cerr << "❌ Failed to open device: " << device_path << " - " << strerror(errno) << std::endl;
            return false;
        }
        
        // Get namespace ID
        int temp_nsid = ioctl(fd_, NVME_IOCTL_ID);
        if (temp_nsid < 0) {
            std::cerr << "❌ Failed to get namespace ID" << std::endl;
            close(fd_);
            return false;
        }
        
        device_.nsid = static_cast<__u32>(temp_nsid);
        device_.lba_size = 4096;
        device_.lba_shift = 12;
        
        std::cout << "✅ Device opened successfully:" << std::endl;
        std::cout << "   📁 Path: " << device_path << std::endl;
        std::cout << "   🆔 Namespace ID: " << device_.nsid << std::endl;
        std::cout << "   📏 LBA Size: " << device_.lba_size << " bytes" << std::endl;
        
        return setup_io_uring();
    }
    
    bool setup_io_uring() {
        std::cout << "\n🔧 Configuring io_uring for Direct Submission" << std::endl;
        
        struct io_uring_params params = {};
        
        // Essential flags for maximum performance
        params.flags = IORING_SETUP_SQE128 | IORING_SETUP_CQE32;
        params.flags |= IORING_SETUP_IOPOLL;    // Polling completion
        params.flags |= IORING_SETUP_SQPOLL;    // Kernel submission thread
        params.flags |= IORING_SETUP_CQSIZE;
        params.cq_entries = QUEUE_DEPTH;
        params.flags |= IORING_SETUP_SQ_AFF;
        params.sq_thread_cpu = 4;  // Pin kernel SQPOLL thread to CPU 4
        //params.sq_thread_idle = 120000;  // Keep SQPOLL thread active for 120 seconds
        
        int ret = io_uring_queue_init_params(QUEUE_DEPTH, &ring_, &params);
        if (ret < 0) {
            std::cerr << "❌ Failed to initialize io_uring: " << strerror(-ret) << std::endl;
            return false;
        }
        
        std::cout << "✅ io_uring initialized for direct submission!" << std::endl;
        std::cout << "   📊 Queue Depth: " << QUEUE_DEPTH << std::endl;
        std::cout << "   📦 Batch Size: " << SUBMIT_BATCH_SIZE << std::endl;
        std::cout << "   🎯 IOPOLL + SQPOLL enabled" << std::endl;
        std::cout << "   💫 Kernel SQPOLL thread on CPU 4 (120s idle timeout)" << std::endl;
        
        return true;
    }
    
    bool start() {
        if (running_.load(std::memory_order_acquire)) {
            return false;
        }
        
        std::cout << "\n🎬 Starting Direct Submission Framework..." << std::endl;
        
        running_.store(true, std::memory_order_release);
        
        // Start backend completion polling thread
        backend_thread_ = std::make_unique<std::thread>(&DirectSubmissionNVMe::backend_main, this);
        
        // Wait for backend to be ready
        while (!ready_.load(std::memory_order_acquire)) {
            std::this_thread::sleep_for(std::chrono::microseconds(100));
        }
        
        std::cout << "✅ Framework operational!" << std::endl;
        std::cout << "   🎭 Main Thread: Ready for direct io_uring submission" << std::endl;
        std::cout << "   ⚡ Backend Thread: Polling completions on CPU 5" << std::endl;
        
        return true;
    }
    
    void stop() {
        if (!running_.load(std::memory_order_acquire)) {
            return;
        }
        
        std::cout << "\n🛑 Stopping framework..." << std::endl;
        running_.store(false, std::memory_order_release);
        
        if (backend_thread_ && backend_thread_->joinable()) {
            backend_thread_->join();
        }
        
        io_uring_queue_exit(&ring_);
        std::cout << "✅ Framework stopped" << std::endl;
    }
    
    ~DirectSubmissionNVMe() {
        stop();
        if (fd_ >= 0) {
            close(fd_);
        }
    }
    
    /**
     * Direct submission read operation
     * 
     * Submits read requests directly to io_uring with zero intermediate queues.
     * Uses batching to reduce syscall overhead.
     */
    bool direct_submit_read(uint64_t lba, uint32_t blocks, void* buffer) {
        // Check for available slots
        uint32_t current_in_flight = io_uring_in_flight_.load(std::memory_order_acquire);
        
        // Submit until queue is full
        if (current_in_flight >= QUEUE_DEPTH) {
            return false; // Queue full - try again immediately
        }
        
        // Get SQE from kernel
        struct io_uring_sqe* sqe = io_uring_get_sqe(&ring_);
        if (!sqe) {
            return false; // Kernel SQ is actually full
        }
        
        // Fill SQE with NVMe command
        memset(sqe, 0, 2 * sizeof(*sqe));  // Clear both SQE and extended cmd area
        
        // Setup io_uring command wrapper
        sqe->opcode = IORING_OP_URING_CMD;     // Use io_uring command interface
        sqe->fd = fd_;                         // NVMe device file descriptor
        sqe->cmd_op = NVME_URING_CMD_IO;       // NVMe I/O command type
        sqe->user_data = 0;                    // No completion data needed
        sqe->flags = 0;                        // No special flags needed
        
        // Build NVMe command
        struct nvme_uring_cmd* cmd = (struct nvme_uring_cmd*)sqe->cmd;
        memset(cmd, 0, sizeof(*cmd));  // Clear NVMe command structure
        
        cmd->opcode = nvme_cmd_read;
        cmd->flags = 0;
        cmd->nsid = device_.nsid;
        cmd->addr = reinterpret_cast<uintptr_t>(buffer);
        cmd->data_len = blocks * device_.lba_size;
        cmd->metadata = 0;
        cmd->metadata_len = 0;
        cmd->timeout_ms = 0;
        
        // LBA addressing
        uint64_t slba = lba;
        uint32_t nlb = blocks - 1;
        
        cmd->cdw10 = slba & 0xffffffff;
        cmd->cdw11 = slba >> 32;
        cmd->cdw12 = nlb;
        cmd->cdw13 = 0;
        cmd->cdw14 = 0;
        cmd->cdw15 = 0;
        
                // Update counters
        total_submitted_++;
        io_uring_in_flight_.fetch_add(1, std::memory_order_relaxed);
        
        // Batching to reduce syscall overhead - works with SQPOLL
        pending_submissions_++;
        if (pending_submissions_ >= SUBMIT_BATCH_SIZE) {
            // Submit batch
            pending_submissions_ = 0;
            io_uring_submit(&ring_);  
        }
        
        return true;
    }
    
    /**
     * Direct submission write operation
     * 
     * Uses the same direct submission architecture as reads.
     */
    bool direct_submit_write(uint64_t lba, uint32_t blocks, void* buffer) {
        // Same simple policy as read - submit until queue is full
        uint32_t current_in_flight = io_uring_in_flight_.load(std::memory_order_acquire);
        if (current_in_flight >= QUEUE_DEPTH) {
            return false; // Queue full - try again immediately
        }
        
        struct io_uring_sqe* sqe = io_uring_get_sqe(&ring_);
        if (!sqe) {
            return false;
        }
        
        memset(sqe, 0, 2 * sizeof(*sqe));
        
        // No completion data needed - using direct counters for performance analysis
        sqe->opcode = IORING_OP_URING_CMD;
        sqe->fd = fd_;
        sqe->cmd_op = NVME_URING_CMD_IO;
        sqe->user_data = 0;  // No user data needed
        sqe->flags = 0;
        
        struct nvme_uring_cmd* cmd = (struct nvme_uring_cmd*)sqe->cmd;
        memset(cmd, 0, sizeof(*cmd));
        
        cmd->opcode = nvme_cmd_write;
        cmd->flags = 0;
        cmd->nsid = device_.nsid;
        cmd->addr = reinterpret_cast<uintptr_t>(buffer);
        cmd->data_len = blocks * device_.lba_size;
        
        uint64_t slba = lba;
        uint32_t nlb = blocks - 1;
        
        cmd->cdw10 = slba & 0xffffffff;
        cmd->cdw11 = slba >> 32;
        cmd->cdw12 = nlb;
        
        total_submitted_++;
        io_uring_in_flight_.fetch_add(1, std::memory_order_relaxed);
        
        // Batching to reduce syscall overhead - works with SQPOLL
        pending_submissions_++;
        if (pending_submissions_ >= SUBMIT_BATCH_SIZE) {
            // Submit batch
            pending_submissions_ = 0;
            io_uring_submit(&ring_);
        }
        
        return true;
    }
    
    // Removed fetch_completions - using direct counters now for performance analysis
    
    // Performance monitoring
    uint64_t get_total_submitted() const { return total_submitted_; }
    uint64_t get_total_completed() const { return total_completed_.load(std::memory_order_relaxed); }
    uint64_t get_total_errors() const { return total_errors_; }

    uint32_t get_current_in_flight() const { return io_uring_in_flight_.load(); }
    uint32_t get_max_in_flight_seen() const { return max_in_flight_seen_.load(); }
    
    uint32_t get_pending_submissions() const { return pending_submissions_; }

    
    // Configuration verification methods
    size_t get_queue_depth_config() const { return QUEUE_DEPTH; }
    size_t get_batch_size_config() const { return SUBMIT_BATCH_SIZE; }


    
    
    void print_stats() {
        uint64_t submitted = total_submitted_;
        uint64_t completed = total_completed_.load(std::memory_order_relaxed);
        uint64_t errors = total_errors_;

        uint32_t current_in_flight = io_uring_in_flight_.load();
        uint32_t max_seen = max_in_flight_seen_.load();
        uint32_t pending = pending_submissions_;




        
        std::cout << "\n📊 Direct Submission Performance:" << std::endl;
        std::cout << "  ✅ Submitted: " << submitted << std::endl;
        std::cout << "  ✅ Completed: " << completed << std::endl;
        std::cout << "  ❌ Errors: " << errors << std::endl;



        
        std::cout << "\n🚀 Queue Utilization:" << std::endl;
        std::cout << "  📈 Current In-Flight: " << current_in_flight << "/" << QUEUE_DEPTH 
                  << " (" << (current_in_flight * 100 / QUEUE_DEPTH) << "%)" << std::endl;
        std::cout << "  🏆 Peak Utilization: " << max_seen << "/" << QUEUE_DEPTH 
                  << " (" << std::fixed << std::setprecision(1) << (max_seen * 100.0 / QUEUE_DEPTH) << "%)" << std::endl;
        std::cout << "  ⏳ Pending Submissions: " << pending << "/" << SUBMIT_BATCH_SIZE 
                  << " (batch will submit at " << SUBMIT_BATCH_SIZE << ")" << std::endl;


        
        if (submitted > 0) {
            double completion_rate = (double)completed / submitted * 100.0;
            std::cout << "\n📈 Efficiency:" << std::endl;
            std::cout << "  📊 Completion Rate: " << completion_rate << "%" << std::endl;
            

        }
        

    }
    
private:
    uint64_t get_ns_timestamp() {
        auto now = std::chrono::high_resolution_clock::now();
        return std::chrono::duration_cast<std::chrono::nanoseconds>(
            now.time_since_epoch()).count();
    }
    
    /**
     * ═══════════════════════════════════════════════════════════════════════════
     * ⚡ BACKEND COMPLETION POLLING THREAD - DEDICATED HARVESTING ENGINE
     * ═══════════════════════════════════════════════════════════════════════════
     * 
     * This is the heart of the completion processing architecture. This thread
     * runs exclusively on CPU 1 and does ONLY completion polling - no submission
     * handling whatsoever. It's designed for maximum completion throughput.
     * 
     * 🎯 THREAD RESPONSIBILITIES:
     * • **Exclusive Polling**: 100% dedicated to io_uring completion harvesting
     * • **High-Priority Execution**: SCHED_FIFO priority 40 for consistent performance
     * • **CPU Isolation**: Pinned to CPU 1 to avoid cache thrashing with main thread
     * • **Lock-free Delivery**: Enqueues results to main thread via SPSC queue
     * • **Memory Management**: Cleans up completion metadata to prevent leaks
     * • **Error Handling**: Tracks and reports I/O errors for analysis
     * 
     * 🚀 PERFORMANCE DESIGN:
     * • **Busy Polling**: Continuous polling for minimum latency
     * • **Batch Processing**: Handles up to 4K completions per cycle
     * • **Zero Blocking**: Never blocks on queue operations
     * • **Cache Optimization**: Minimal memory access patterns
     * • **Atomic Coordination**: Efficient communication with main thread
     * 
     * 📊 EXPECTED PERFORMANCE:
     * • 1M+ completions/second processing capability
     * • <1μs average completion processing latency  
     * • 99.9% CPU utilization on dedicated core
     * • Zero lock contention with main thread
     */
    void backend_main() {
        std::cout << "\n🧵 Backend Completion Thread Starting..." << std::endl;
        
        // Set CPU affinity to CPU 5 (separate from main thread and kernel SQPOLL)
        cpu_set_t cpuset;
        CPU_ZERO(&cpuset);
        CPU_SET(5, &cpuset);
        
        if (pthread_setaffinity_np(pthread_self(), sizeof(cpu_set_t), &cpuset) != 0) {
            std::cerr << "⚠️  Warning: Failed to set backend CPU affinity" << std::endl;
        } else {
            std::cout << "   ✅ Backend CPU Affinity: Pinned to CPU 5" << std::endl;
        }
        
        // Set highest priority for completion processing
        struct sched_param param;
        param.sched_priority = 99;  // Maximum SCHED_FIFO priority
        
        if (pthread_setschedparam(pthread_self(), SCHED_FIFO, &param) != 0) {
            std::cerr << "⚠️  Warning: Failed to set backend priority" << std::endl;
        } else {
            std::cout << "   ✅ Backend Priority: SCHED_FIFO priority 99 (HIGHEST)" << std::endl;
        }
        
        std::cout << "   🎯 Backend ready for completion polling" << std::endl;
        ready_.store(true, std::memory_order_release);
        
        // Stats timing for backend thread
        auto last_stats_time = std::chrono::high_resolution_clock::now();
        auto next_stats_time = last_stats_time + std::chrono::seconds(120);
        uint64_t last_completed = 0;
        
                // Main completion polling loop
        while (running_.load(std::memory_order_acquire)) {

            
            // Poll completions directly - no queue overhead
            poll_and_queue_completions();
            
        }
        
        std::cout << "\n📊 Backend thread shutdown complete" << std::endl;
    }
    
    /**
     * ═══════════════════════════════════════════════════════════════════════════
     * 🔍 POLL AND QUEUE COMPLETIONS - CORE HARVESTING FUNCTION
     * ═══════════════════════════════════════════════════════════════════════════
     * 
     * This function is the critical path for completion processing. It harvests
     * completed I/O operations from io_uring and delivers them to the main thread
     * via the lock-free completion queue.
     * 
     * 🎯 PROCESSING FLOW:
     * 1. **Peek CQE**: Check for available completion queue entries
     * 2. **Extract Metadata**: Retrieve operation context and timing data
     * 3. **Error Handling**: Track and report any I/O errors encountered
     * 4. **Queue Delivery**: Enqueue result for main thread consumption
     * 5. **Memory Cleanup**: Free completion metadata to prevent leaks
     * 6. **Queue Management**: Mark CQE as processed for kernel reuse
     * 
     * 🚀 PERFORMANCE OPTIMIZATIONS:
     * • **Batch Processing**: Handle up to 4K completions per call
     * • **Early Termination**: Exit immediately when no completions available
     * • **Cache Efficiency**: Minimal memory access patterns
     * • **Error Fast Path**: Quick handling of error conditions
     * • **Lock-free Enqueue**: Zero-contention delivery to main thread
     * 
     * @return Number of completions processed in this cycle
     */
    int poll_and_queue_completions() {
        struct io_uring_cqe* cqe;
        size_t completed = 0;  // Use size_t to match MAX_BATCH_COMPLETE type
        
        // Process up to MAX_BATCH_COMPLETE per cycle
        while (completed < MAX_BATCH_COMPLETE) {
            int ret = io_uring_peek_cqe(&ring_, &cqe);
            
            if (ret < 0) {
                if (ret == -EAGAIN) {
                    break; // No completions available - this is NORMAL, not an error
                } else {
                    std::cerr << "❌ Completion polling error: " << strerror(-ret) << std::endl;
                    break;
                }
            }
            
            // Simplified completion processing - no allocation overhead
            // Handle errors and count completions directly
            if (cqe->res < 0) {
                std::cerr << "❌ cqe res error: " << cqe->res << std::endl;
                total_errors_++;
            } else {
                // Valid completion - precise counting for BW calculation
                total_completed_.fetch_add(1, std::memory_order_relaxed);
                            // Decrement in-flight counter
                io_uring_in_flight_.fetch_sub(1, std::memory_order_acq_rel);
            
                // Mark CQE as seen
                io_uring_cqe_seen(&ring_, cqe);
                completed++;
            }           
        }
        
        return static_cast<int>(completed);
    }
};


int main(int argc, char* argv[]) {
    signal(SIGINT, signal_handler);
    signal(SIGTERM, signal_handler);
    
    // Pin main thread to CPU 9 to avoid core jumping
    cpu_set_t cpuset;
    CPU_ZERO(&cpuset);
    CPU_SET(9, &cpuset);
    
    if (pthread_setaffinity_np(pthread_self(), sizeof(cpu_set_t), &cpuset) != 0) {
        std::cerr << "⚠️  Warning: Failed to set main thread CPU affinity to CPU 9" << std::endl;
    } else {
        std::cout << "✅ Main thread pinned to CPU 9" << std::endl;
    }
    
    std::cout << "════════════════════════════════════════════════════════════════" << std::endl;
    std::cout << "🚀 DIRECT SUBMISSION NVME FRAMEWORK - MAXIMUM IOPS ARCHITECTURE" << std::endl;
    std::cout << "════════════════════════════════════════════════════════════════" << std::endl;
    std::cout << "\n🎯 ARCHITECTURE FEATURES:" << std::endl;
    std::cout << "   • Main Thread: Direct io_uring submission (zero queuing)" << std::endl;
    std::cout << "   • Backend Thread: Dedicated completion polling (CPU 5)" << std::endl;
    std::cout << "   • Kernel Thread: SQPOLL submission management (CPU 4)" << std::endl;
    std::cout << "   • Lock-free Completion Queue: Zero-contention results" << std::endl;
    std::cout << "   • Direct Path: Eliminates all submission bottlenecks" << std::endl;
    
    const std::string device_path = (argc > 1) ? argv[1] : "/dev/ng2n1";
    std::cout << "\n📁 Target NVMe Device: " << device_path << std::endl;
    
    // Initialize framework
    DirectSubmissionNVMe nvme;
    if (!nvme.initialize(device_path)) {
        std::cerr << "\n💥 FATAL: Failed to initialize direct submission framework" << std::endl;
        return 1;
    }
    
    if (!nvme.start()) {
        std::cerr << "\n💥 FATAL: Failed to start framework" << std::endl;
        return 1;
    }
    
    std::cout << "\n🎊 SUCCESS: Direct submission framework operational!" << std::endl;
    
    // Allocate LARGE high-performance buffer pool - never allocate on the fly
    std::vector<std::unique_ptr<uint8_t[]>> buffers;
    constexpr size_t NUM_BUFFERS = 16384;  // 🚀 4x larger buffer pool (16K buffers)
    constexpr size_t BUFFER_SIZE = 4096;
    
    std::cout << "\n💾 Allocating DMA Buffers..." << std::endl;
    
    for (size_t i = 0; i < NUM_BUFFERS; ++i) {
        void* raw_ptr;
        if (posix_memalign(&raw_ptr, 4096, BUFFER_SIZE) != 0) {
            std::cerr << "💥 FATAL: Failed to allocate buffer " << i << std::endl;
            return 1;
        }
        buffers.emplace_back(static_cast<uint8_t*>(raw_ptr));
        
        uint8_t* buf = buffers.back().get();
        
        // Lock buffer in memory to prevent page-out during DMA
        if (mlock(buf, BUFFER_SIZE) != 0) {
            std::cerr << "⚠️  Warning: Failed to lock buffer " << i << " in memory" << std::endl;
        }
        
        for (size_t j = 0; j < BUFFER_SIZE; ++j) {
            buf[j] = (i * 256 + j) & 0xFF;
        }
    }
    
    std::cout << "   ✅ Allocated " << NUM_BUFFERS << " x 4KB aligned buffers (64MB total)" << std::endl;
    
    // Allow framework to fully initialize
    std::this_thread::sleep_for(std::chrono::milliseconds(100));
    
    /**
     * ═══════════════════════════════════════════════════════════════════════
     * 🚀 MAXIMUM IOPS STRESS TEST - DIRECT SUBMISSION
     * ═══════════════════════════════════════════════════════════════════════
     * 
     * This test demonstrates maximum IOPS through direct submission:
     * - Main thread submits as fast as possible directly to io_uring
     * - Backend thread polls completions continuously  
     * - Zero intermediate queues or processing delays
     * - Peak performance demonstration
     */
    
    std::cout << "\n🚀 Starting Maximum IOPS Direct Submission Test..." << std::endl;
    std::cout << "📊 Duration: 8 minutes of continuous maximum rate submission" << std::endl;
    std::cout << "🎯 Goal: Achieve highest possible IOPS through direct submission" << std::endl;
    std::cout << "⚙️  Configuration Verification:" << std::endl;
    std::cout << "   🎚️  QUEUE_DEPTH = " << nvme.get_queue_depth_config() << std::endl;
    std::cout << "   📦 SUBMIT_BATCH_SIZE = " << nvme.get_batch_size_config() << std::endl;
    std::cout << "   🔄 SQPOLL + batching enabled (120s idle timeout)" << std::endl;
    
    uint32_t request_count = 0;
    size_t buffer_index = 0;
    uint32_t consecutive_failures = 0;
    
    auto test_start_time = std::chrono::high_resolution_clock::now();
    auto test_duration = std::chrono::minutes(8);
    auto next_stats_time = test_start_time + std::chrono::seconds(120);
    
    uint64_t last_completed = 0;
    auto last_stats_time = test_start_time;
    auto start_time = test_start_time;
    
    // Simplified - no completion fetching, just use counters for BW calculation
    
    // Main direct submission loop
    while (g_running.load()) {
        auto current_time = std::chrono::high_resolution_clock::now();
        
        if (current_time - test_start_time >= test_duration) {
            std::cout << "\n🏁 8-minute test completed!" << std::endl;
            break;
        }
        
        // Print stats every 15 seconds
        if (current_time >= next_stats_time) {
            bool bDebugstats = false;
            auto elapsed = std::chrono::duration_cast<std::chrono::seconds>(current_time - test_start_time);
            
            uint64_t current_completed = nvme.get_total_completed();
            auto stats_elapsed = std::chrono::duration_cast<std::chrono::milliseconds>(current_time - last_stats_time);
            
            uint64_t completed_delta = current_completed - last_completed;
            double interval_seconds = stats_elapsed.count() / 1000.0;
            double real_time_iops = completed_delta / interval_seconds;
            double real_time_bandwidth = real_time_iops * 4.0 / 1024.0;  // Each request is 4KB
            
            std::cout << "\n⏱️  " << elapsed.count() << "/480 seconds elapsed" << std::endl;
            std::cout << "📊 SIMPLIFIED DIRECT SUBMISSION PERFORMANCE:" << std::endl;
            std::cout << "   🚀 REAL-TIME IOPS: " << std::fixed << std::setprecision(0) 
                     << real_time_iops << std::endl;
            std::cout << "   📈 REAL-TIME BANDWIDTH: " << std::fixed << std::setprecision(1) 
                     << real_time_bandwidth << " MB/s (4KB per request)" << std::endl;
            std::cout << "   ✅ Completed Delta: " << completed_delta << " requests" << std::endl;
            std::cout << "   🎯 Submitted: " << nvme.get_total_submitted() << " | Completed: " << current_completed << std::endl;
            
            if(bDebugstats){
                nvme.print_stats();
            }
            
            last_completed = current_completed;
            last_stats_time = current_time;
            next_stats_time += std::chrono::seconds(120);
        }
        
        // 🚀 SIMPLE SUBMISSION BURST - Fixed burst size
         const uint32_t MAX_BURST = 8192*4;  // Simple fixed burst size
        
        for (uint32_t i = 0; i < MAX_BURST; i++) {
            void* buffer = buffers[buffer_index % NUM_BUFFERS].get();
            uint64_t lba = (request_count * 47) % 1000000*1000;
            
            // Direct submit to io_uring (no intermediate queues!)
            if (nvme.direct_submit_read(lba, 1, buffer)) {
                request_count++;
                buffer_index++;
                consecutive_failures = 0;
            } else {
                consecutive_failures++;
                break; // Exit burst on first failure
            }
         }
         
                                    // 🎯 MAXIMUM POLLING RATE - No additional sleep needed
         // (Backpressure handled in direct_submit_read with 100ns sleep when queue full)
         
         // 🚀 MOSTLY BUSY POLLING - Only micro-pause when truly saturated
    }
    
    std::cout << "\n⏳ Waiting for all operations to complete..." << std::endl;
    
    // Final completion wait - let backend finish processing
    std::cout << "\n⏳ Waiting for remaining completions..." << std::endl;
    std::this_thread::sleep_for(std::chrono::seconds(1));
    
    auto end_time = std::chrono::high_resolution_clock::now();
    auto duration = std::chrono::duration_cast<std::chrono::milliseconds>(end_time - start_time);
    
    std::cout << "\n📈 Final Direct Submission Results:" << std::endl;
    nvme.print_stats();
    
    std::cout << "\n⏱️  Performance Summary:" << std::endl;
    std::cout << "  Total Runtime: " << duration.count() << " ms" << std::endl;
    std::cout << "  Requests Submitted: " << request_count << std::endl;


    
    if (duration.count() > 0) {
        uint64_t completed = nvme.get_total_completed();
        double iops = (double)completed * 1000.0 / duration.count();
        std::cout << "  Average IOPS: " << (int)iops << std::endl;
        
        double throughput_mb = iops * 4.0 / 1024.0;
        std::cout << "  Throughput: " << std::fixed << std::setprecision(1) << throughput_mb << " MB/s" << std::endl;
    }
    
    std::cout << "\n🏁 Direct Submission Test Completed!" << std::endl;
    std::cout << "Key achievements demonstrated:" << std::endl;
    std::cout << "  ✓ Main thread direct io_uring submission (zero queuing)" << std::endl;
    std::cout << "  ✓ Backend thread dedicated completion polling" << std::endl;
    std::cout << "  ✓ Lock-free completion communication" << std::endl;
    std::cout << "  ✓ Maximum IOPS through elimination of bottlenecks" << std::endl;
    
    // Cleanup
    for (auto& buf : buffers) {
        uint8_t* ptr = buf.get();
        munlock(ptr, BUFFER_SIZE);  // Unlock memory before freeing
        free(buf.release());
    }
    
    nvme.stop();
    return 0;
} 

[gaowayne]

when I call io_uring_peek_cqe, it will invoke syscall everytime.

[royonia]

IMO you could hardly get any real insights posting 100k lines of AI generated emojis. from your code example it does not seem to leave room for the "actual" IO ?