ESP32-S3 sense 無法檢測 USB 音訊設備的問題 / Issue with ESP32-S3 Sense Unable to Detect USB Audio Device (AEGHB-819)

[LiTipo]

Answers checklist.

• [X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
• [X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
• [X] I have searched the issue tracker for a similar issue and not found a similar issue.

General issue report

問題描述

作業系統：Windows 11
idf版本：v5.2.2 release
目標晶片：ESP32-S3 (QFN56) (revision v0.2)
專案建置方式：使用idf建置專案
執行命令類型：CMD
使用套件：usb_stream

目前我使用Python撰寫socket server端，並且讓ESP32-S3 sense連到我的路由器網路，並且透過TCP/IP的方式傳送資訊到server端。WIFI功能與TCP/IP功能都是正常執行的，但我的UAC代碼目前無法正常執行，會停滯在usb_streaming_connect_wait()方法的地方。我認為是ESP32-S3 sense沒有掃描到我的音訊設備。

我的硬體串接方式為：

• ESP32-S3上的PIN20 (D+) 和 PIN19 (D-) 串接到USB音訊設備。
• 該USB音訊設備支持UAC1.0和2.0規範。
• 輸入電源是透過USB接口供電，但該USB線為僅供電的連接線，並沒有使用傳輸線。

我購買的ESP32-S3 sense是從Seeed Studio網站購買的。

問題

1. 我的ESP32-S3 sense無法在usb_streaming_connect_wait()方法中掃描到USB音訊設備，請問可能的原因是什麼？
2. ESP32-S3 sense與USB音訊設備之間的正確連接方式應該是什麼？
3. 是否有任何配置或設定需要調整以確保ESP32-S3 sense能夠成功掃描和連接到USB音訊設備？
4. 有什麼方法可以檢查ESP32-S3 sense是否已正確識別到USB音訊設備？
5. 是否有任何範例代碼或文檔可以提供給我，以幫助我解決此問題？

謝謝。

Issue Description

Operating System: Windows 11
IDF Version: v5.2.2 release
Target Chip: ESP32-S3 (QFN56) (revision v0.2)
Project Build Method: Using IDF to build the project
Command Type: CMD
Used Library: usb_stream

I am using Python to write a socket server and have the ESP32-S3 sense connect to my router network, transmitting information to the server via TCP/IP. The WiFi and TCP/IP functions are working correctly, but my UAC code is currently not functioning properly and stalls at the usb_streaming_connect_wait() method. I believe the ESP32-S3 sense is not detecting my audio device.

My hardware connection is as follows:

• ESP32-S3 PIN20 (D+) and PIN19 (D-) are connected to the USB audio device.
• The USB audio device supports UAC1.0 and 2.0 standards.
• The power input is via the USB interface, but the USB cable is a power-only cable and does not use data lines.

I purchased the ESP32-S3 sense from the Seeed Studio website.

Questions

1. My ESP32-S3 sense is unable to detect the USB audio device at the usb_streaming_connect_wait() method. What could be the possible reasons for this issue?
2. What is the correct way to connect the ESP32-S3 sense to the USB audio device?
3. Are there any configurations or settings that need to be adjusted to ensure the ESP32-S3 sense can successfully detect and connect to the USB audio device?
4. How can I check if the ESP32-S3 sense has correctly recognized the USB audio device?
5. Are there any sample codes or documentation that can help me resolve this issue?

Thank you.

Code

ESP32-s3 Code

// RTOS 實時操作系統
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"

// 基礎系統 API
#include "esp_system.h"
#include "esp_log.h"
#include "nvs_flash.h"

// WIFI API
#include "freertos/event_groups.h"
#include "esp_wifi.h"
#include "esp_event.h"

/* 透過項目配置設置 SSID 和密碼，或直接在此處設置 */
#define DEFAULT_SSID CONFIG_EXAMPLE_WIFI_SSID
#define DEFAULT_PWD CONFIG_EXAMPLE_WIFI_PASSWORD

// 選擇掃描和排序方法
#if CONFIG_EXAMPLE_WIFI_ALL_CHANNEL_SCAN
#define DEFAULT_SCAN_METHOD WIFI_ALL_CHANNEL_SCAN
#elif CONFIG_EXAMPLE_WIFI_FAST_SCAN
#define DEFAULT_SCAN_METHOD WIFI_FAST_SCAN
#else
#define DEFAULT_SCAN_METHOD WIFI_FAST_SCAN
#endif /*CONFIG_EXAMPLE_SCAN_METHOD*/

#if CONFIG_EXAMPLE_WIFI_CONNECT_AP_BY_SIGNAL
#define DEFAULT_SORT_METHOD WIFI_CONNECT_AP_BY_SIGNAL
#elif CONFIG_EXAMPLE_WIFI_CONNECT_AP_BY_SECURITY
#define DEFAULT_SORT_METHOD WIFI_CONNECT_AP_BY_SECURITY
#else
#define DEFAULT_SORT_METHOD WIFI_CONNECT_AP_BY_SIGNAL
#endif /*CONFIG_EXAMPLE_SORT_METHOD*/

// 設置掃描閾值和認證模式
#if CONFIG_EXAMPLE_FAST_SCAN_THRESHOLD
#define DEFAULT_RSSI CONFIG_EXAMPLE_FAST_SCAN_MINIMUM_SIGNAL
#if CONFIG_EXAMPLE_FAST_SCAN_WEAKEST_AUTHMODE_OPEN
#define DEFAULT_AUTHMODE WIFI_AUTH_OPEN
#elif CONFIG_EXAMPLE_FAST_SCAN_WEAKEST_AUTHMODE_WEP
#define DEFAULT_AUTHMODE WIFI_AUTH_WEP
#elif CONFIG_EXAMPLE_FAST_SCAN_WEAKEST_AUTHMODE_WPA
#define DEFAULT_AUTHMODE WIFI_AUTH_WPA_PSK
#elif CONFIG_EXAMPLE_FAST_SCAN_WEAKEST_AUTHMODE_WPA2
#define DEFAULT_AUTHMODE WIFI_AUTH_WPA2_PSK
#else
#define DEFAULT_AUTHMODE WIFI_AUTH_OPEN
#endif
#else
#define DEFAULT_RSSI -127
#define DEFAULT_AUTHMODE WIFI_AUTH_OPEN
#endif /*CONFIG_EXAMPLE_FAST_SCAN_THRESHOLD*/

// TCP 客戶端相關的頭文件和全域變數
#include <string.h>
#include <unistd.h>
#include <sys/socket.h>
#include <errno.h>
#include <netdb.h>
#include <arpa/inet.h>

#define PORT 9999  // 替換成你的伺服器端口號

static const char *TAG = "example";

char ip_address_str[IP4ADDR_STRLEN_MAX];  // 儲存從 IP_EVENT_STA_GOT_IP 獲取的 IP 地址字串
int sock = -1;  // 保存socket描述符

void tcp_client(void);

// USB Stream API
#include "usb_stream.h"

// 定義狀態消息
#define STATUS_CONNECTED "STATUS: Device connected\n"
#define STATUS_DISCONNECTED "STATUS: Device disconnected\n"
#define STATUS_UAC_STARTED "STATUS: UAC started\n"
#define STATUS_UAC_ERROR "STATUS: UAC error\n"

void send_status_message(const char *message) {
    if (sock >= 0) {
        int to_write = strlen(message);
        while (to_write > 0) {
            int written = send(sock, message, to_write, 0);
            if (written < 0) {
                ESP_LOGE(TAG, "Error occurred during sending status: errno %d", errno);
                break;
            }
            to_write -= written;
        }
    }
}

static void event_handler(void* arg, esp_event_base_t event_base,
                          int32_t event_id, void* event_data)
{
    if (event_base == WIFI_EVENT && event_id == WIFI_EVENT_STA_START) {
        esp_wifi_connect();
    } else if (event_base == WIFI_EVENT && event_id == WIFI_EVENT_STA_DISCONNECTED) {
        esp_wifi_connect();
    } else if (event_base == IP_EVENT && event_id == IP_EVENT_STA_GOT_IP) {
        ip_event_got_ip_t* event = (ip_event_got_ip_t*) event_data;
        esp_ip4_addr_t ip = event->ip_info.ip;  // 使用 ip4_addr_t 型別來表示 IP 地址

        // 將 IP 地址轉換為字串表示並保存到 ip_address_str
        esp_ip4addr_ntoa(&ip, ip_address_str, IP4ADDR_STRLEN_MAX);
        
        // 使用 ESP_LOGI 函數來印出轉換後的 IP 地址字串
        ESP_LOGI(TAG, "獲取到 IP 地址: %s", ip_address_str);

        // After obtaining IP address, start TCP client
        tcp_client();
    }
}

/* 初始化 Wi-Fi 為 STA 模式並設置掃描方法 */
static void fast_scan(void)
{
    ESP_ERROR_CHECK(esp_netif_init());
    ESP_ERROR_CHECK(esp_event_loop_create_default());

    wifi_init_config_t cfg = WIFI_INIT_CONFIG_DEFAULT();
    ESP_ERROR_CHECK(esp_wifi_init(&cfg));

    ESP_ERROR_CHECK(esp_event_handler_instance_register(WIFI_EVENT, ESP_EVENT_ANY_ID, &event_handler, NULL, NULL));
    ESP_ERROR_CHECK(esp_event_handler_instance_register(IP_EVENT, IP_EVENT_STA_GOT_IP, &event_handler, NULL, NULL));

    // 初始化默認的 STA 網路接口實例（esp-netif）
    esp_netif_t *sta_netif = esp_netif_create_default_wifi_sta();
    assert(sta_netif);

    // 初始化並啟動 Wi-Fi
    wifi_config_t wifi_config = {
        .sta = {
            .ssid = DEFAULT_SSID,
            .password = DEFAULT_PWD,
            .scan_method = DEFAULT_SCAN_METHOD,
            .sort_method = DEFAULT_SORT_METHOD,
            .threshold.rssi = DEFAULT_RSSI,
            .threshold.authmode = DEFAULT_AUTHMODE,
        },
    };
    ESP_ERROR_CHECK(esp_wifi_set_mode(WIFI_MODE_STA));
    ESP_ERROR_CHECK(esp_wifi_set_config(WIFI_IF_STA, &wifi_config));
    ESP_ERROR_CHECK(esp_wifi_start());
}

// 定義UAC相關變數和回調函數
#define ENABLE_UAC_MIC_FUNCTION 1

#if (ENABLE_UAC_MIC_FUNCTION)
static uint32_t s_mic_samples_frequence = 0;
static uint32_t s_mic_ch_num = 0;
static uint32_t s_mic_bit_resolution = 0;
static EventGroupHandle_t s_evt_handle;
static void mic_frame_cb(mic_frame_t *frame, void *ptr)
{
    ESP_LOGD(TAG, "mic callback! bit_resolution = %u, samples_frequence = %"PRIu32", data_bytes = %"PRIu32,
             frame->bit_resolution, frame->samples_frequence, frame->data_bytes);
    // Send audio data over TCP
    if (sock >= 0) {
        int to_write = frame->data_bytes;
        while (to_write > 0) {
            int written = send(sock, frame->data, to_write, 0);
            if (written < 0) {
                ESP_LOGE(TAG, "Error occurred during sending: errno %d", errno);
                break;
            }
            to_write -= written;
        }
    }
}
#endif //ENABLE_UAC_MIC_FUNCTION

static void stream_state_changed_cb(usb_stream_state_t event, void *arg)
{
    switch (event) {
    case STREAM_CONNECTED: {
        size_t frame_size = 0;
        size_t frame_index = 0;
#if (ENABLE_UAC_MIC_FUNCTION)
        uac_frame_size_list_get(STREAM_UAC_MIC, NULL, &frame_size, &frame_index);
        if (frame_size) {
            ESP_LOGI(TAG, "UAC MIC: get frame list size = %u, current = %u", frame_size, frame_index);
            uac_frame_size_t *mic_frame_list = (uac_frame_size_t *)malloc(frame_size * sizeof(uac_frame_size_t));
            uac_frame_size_list_get(STREAM_UAC_MIC, mic_frame_list, NULL, NULL);
            for (size_t i = 0; i < frame_size; i++) {
                ESP_LOGI(TAG, "\t [%u] ch_num = %u, bit_resolution = %u, samples_frequence = %"PRIu32 ", samples_frequence_min = %"PRIu32 ", samples_frequence_max = %"PRIu32,
                         i, mic_frame_list[i].ch_num, mic_frame_list[i].bit_resolution, mic_frame_list[i].samples_frequence,
                         mic_frame_list[i].samples_frequence_min, mic_frame_list[i].samples_frequence_max);
            }
            s_mic_samples_frequence = mic_frame_list[frame_index].samples_frequence;
            s_mic_ch_num = mic_frame_list[frame_index].ch_num;
            s_mic_bit_resolution = mic_frame_list[frame_index].bit_resolution;
            if (s_mic_ch_num != 1) {
                ESP_LOGW(TAG, "UAC MIC: only support 1 channel in this example");
            }
            ESP_LOGI(TAG, "UAC MIC: use frame[%u] ch_num = %"PRIu32", bit_resolution = %"PRIu32", samples_frequence = %"PRIu32,
                     frame_index, s_mic_ch_num, s_mic_bit_resolution, s_mic_samples_frequence);
            free(mic_frame_list);
        } else {
            ESP_LOGW(TAG, "UAC MIC: get frame list size = %u", frame_size);
        }
#endif
        ESP_LOGI(TAG, "Device connected");
        send_status_message(STATUS_CONNECTED);
        break;
    }
    case STREAM_DISCONNECTED:
        ESP_LOGI(TAG, "Device disconnected");
        send_status_message(STATUS_DISCONNECTED);
        break;
    default:
        ESP_LOGE(TAG, "Unknown event");
        send_status_message(STATUS_UAC_ERROR);
        break;
    }
}

void tcp_client(void)
{
    char host_ip[IP4ADDR_STRLEN_MAX] = "192.168.50.116";
    int addr_family = 0;
    int ip_protocol = 0;

    // Copy IP address string to host_ip
    // strcpy(host_ip, ip_address_str);

    // Determine address family and protocol based on configuration
    struct sockaddr_in dest_addr;
    inet_pton(AF_INET, host_ip, &dest_addr.sin_addr);
    dest_addr.sin_family = AF_INET;
    dest_addr.sin_port = htons(PORT);
    addr_family = AF_INET;
    ip_protocol = IPPROTO_IP;

    // Create socket
    sock =  socket(addr_family, SOCK_STREAM, ip_protocol);
    if (sock < 0) {
        ESP_LOGE(TAG, "Unable to create socket: errno %d", errno);
        return;
    }
    ESP_LOGI(TAG, "Socket created, connecting to %s:%d", host_ip, PORT);

    // Connect to server
    int err = connect(sock, (struct sockaddr *)&dest_addr, sizeof(dest_addr));
    if (err != 0) {
        ESP_LOGE(TAG, "Socket unable to connect: errno %dS", errno);
        return;
    }
    ESP_LOGI(TAG, "Successfully connected");

    // Send initial message
    send_status_message("ESP32s3 successfully connected\n");

    // Initialize UAC after TCP connection
    ESP_LOGI(TAG, "Initializing UAC");
    send_status_message("Initializing UAC\n");
    s_evt_handle = xEventGroupCreate();
    if (s_evt_handle == NULL) {
        ESP_LOGE(TAG, "Event group create failed");
        send_status_message("Event group create failed\n");
        return;
    }
    
    send_status_message("Event group create success\n");
    uac_config_t uac_config = {
        .mic_bit_resolution = UAC_BITS_ANY,
        .mic_samples_frequence = UAC_FREQUENCY_ANY,
        .mic_cb = &mic_frame_cb,
        .mic_cb_arg = NULL,
        .flags = 0,
    };
    esp_err_t ret = uac_streaming_config(&uac_config);
    if (ret != ESP_OK) {
        ESP_LOGE(TAG, "UAC streaming config failed");
        send_status_message(STATUS_UAC_ERROR);
        return;
    }

    send_status_message("TP1\n");
    ESP_ERROR_CHECK(usb_streaming_state_register(&stream_state_changed_cb, NULL));
    send_status_message("TP2\n");
    ESP_ERROR_CHECK(usb_streaming_start());
    send_status_message("TP3\n");
    ESP_ERROR_CHECK(usb_streaming_connect_wait(portMAX_DELAY));
    send_status_message("TP4\n");

    send_status_message(STATUS_UAC_STARTED);
}

void app_main(void)
{
    // 初始化 NVS
    esp_err_t ret = nvs_flash_init();
    if (ret == ESP_ERR_NVS_NO_FREE_PAGES || ret == ESP_ERR_NVS_NEW_VERSION_FOUND) {
        ESP_ERROR_CHECK(nvs_flash_erase());
        ret = nvs_flash_init();
    }
    ESP_ERROR_CHECK(ret);

    // 執行 Wi-Fi 連接和 TCP 客戶端
    fast_scan();
}

Python Server Code

import socket

def get_ip_address():
    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    try:
        s.connect(("8.8.8.8", 80))
        ip_address = s.getsockname()[0]
    except Exception:
        ip_address = "127.0.0.1"
    finally:
        s.close()
    return ip_address

def start_server(ip, port):
    server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    server_socket.bind((ip, port))
    server_socket.listen(5)
    print(f"Server started at {ip}:{port}")
    
    while True:
        client_socket, client_address = server_socket.accept()
        print(f"Connection from {client_address}")
        while True:
            data = client_socket.recv(1024).decode()
            if not data:
                break
            print(f"Received data: {data}")
        response = "Message received"
        client_socket.send(response.encode())
        client_socket.close()

if __name__ == "__main__":
    ip_address = get_ip_address()
    print("IP Address:", ip_address)
    start_server(ip_address, 9999)

Python Server Execution Result

IP Address: 192.168.50.116
Server started at 192.168.50.116:9999
Connection from ('192.168.50.96', 49470)
Received data: ESP32s3 successfully connected

Received data: Initializing UAC
Event group create success
TP1
TP2
TP3