Choosing The Right Mode For SIMCOM or Any Modem... (AT or DialUP) (IDFGH-12638)

[parthbhat13]

Answers checklist.

• [X] I have read the documentation for esp-protocols components and the issue is not addressed there.
• [ ] I have updated my esp-protocols branch (master or release) to the latest version and checked that the issue is present there.
• [X] I have searched the issue tracker for a similar issue and not found a similar issue.

General issue report

Hi all, and specifically @david-cermak

Just a bit of background

apologies in advance as i do not know, what i am going to produce below in terms of issue or suggestion would fall under this title or not. bare with me as i would be summarizing my issue and work of past 3 years together in some limited words.

###The Problem Statement Basically since past 3 years ive been working and developing some UPI (Unified Payments Interface) system based payment devices for the vending machine, there has been a lot of upgrades and different versions have been developed and now there another final version which i am supposed to be launching. so far ive completed around 1000pcs in the market and we are trying to deploy even more. the stack which i am using is as follows

• ESP32
• GSM(A7672S) dialUp connection or PPPos over uart bus. NOTE: I've not used the CTS and RTS lines. now in terms of the software I've got two different things working , MQTT is meant for sending and receiving payment transactions and state of the machine. plus we have used HTTP to make sure there is a heartbeat and also for verification of the device. during the runtime there is HTTP and MQTT both are technically running 24X7 and every 60Sec there is a HTTP request sent to server more like a patch request and MQTT is just ideal in KEEP-ALIVE of 120Sec. that is 2 mins.

summary of a problem statement i need a real time communication with a server over GSM.. simple.

The Problem I'm facing and reason for this post ?

This will definitely be very long so bare with me please. after deploying multiple products in the market we have started receiving multiple complains from the field. mainly where the network drops, or the ping request fails or some other minor things which had been hard for me to figure out since i did not use my enough brains to have the OTA update function and remote LOG monitoring function.

Taking matters in my own hands i started talking with some of the other people in the market, including my close mentors in india,

• Mr.Franz @diplfranzhoepfinger
• Mr. Mahesh AsiaPacific Technical Head of Simcom
• Mr. Rajesh AsiaPacific Director of Simcom
• Mr.Ravi Pujjar Whos a youtuber and have made many great gsm projects.

as well since it was hard to get in touch with mr.David Cenmark i was lucky enough to watch your youtube video where you mentioned few things which lead me to more confusions.

to summarize the decisions from all the mentors mentioned above, all i could say is the ones from india asked to go with AT commands and use the internal GSM Http and MQTT stack to talk with the server. where in Mr.Franz and Mr.David asked to go for PPPOS dial Up connection.

now with the updated and latest product, i need to not only have the payment transactions working, but also OTA update, real time logs and much more other things. where in i spent 2 months to write my own AT commands based library for the GSM which can be used by any modem (that was the plan), i came with some results .

• HTTP GET request to the server with custom AT library takes 3Seconds (Thats too much)
• The dialUp connection and esp32 internal http library takes 1.5Seconds.

Ofcourse the right answer is to go with the esp-modem library but again cant forget the issues which ive had from the field where they complained of network loss, connection drop, http failed to ping and on and on.

out of my curiosity, i even tested the Arduino and TinyGsm library, and it seems they are doing all AT commands and its all over the TCP stack of the gsm modem. to my surprise arduino with a vanilla HTTP request over tcp. takes 840mS for response lol. idk what have they used in the tinyGsm library for it to be so fast. all of my stack is developed on esp-idf V4.4.6 and C .

Solutions I Can Think Of..

eventually, i am genuinely stuck between what to do? and below are my options

1. add CTS and RTS lines, Use the PPPOS dialUp connection, and use only single protocol MQTT/HTTP not both
2. add custom code on top of esp-modem library to handle the GSM HTTP/MQTT Stack AT commands.
3. Try and find ways to implement the TinyGsm based communication DTE driver for the GSM for faster Result in reading and writing to and from uart.

with the above solutions, i can add the cts and rts to my existing payment device since i have spare pins, but my other products which involves Tea Coffee Vending Machine and Even Vending Machine controller, has no ESP32 pins left whatsoever. so idk how would that be of any help. yes i can shift to esp32S3 and use the USB. but now there no time for redeveloping the whole pcbs and sample it and test all the other loads of protocols which are running on the board..

so what shall be done and are there any inputs ?

My Custom AT library At parser code..

1. Function To Send AT Command

gsm_err_t DTECommand(gsmConfig_t *config, char *at)
{
    char sendBuf[strlen(at) + 3];

    vTaskDelay(1 / portTICK_PERIOD_MS);
    uart_flush(config->gsmUartConfig.uartNum);

    memset(sendBuf, 0, sizeof(sendBuf));
    memcpy(sendBuf, at, strlen(at));

    strncat(sendBuf, "\r\n", strlen(sendBuf) + 2);

    if (config->debugDTE)
        ESP_LOGI(TAG, "Sending Cmd: %s", sendBuf);

    uart_write_bytes(config->gsmUartConfig.uartNum, sendBuf, strlen(sendBuf));
    uart_wait_tx_done(config->gsmUartConfig.uartNum, 100 / portTICK_PERIOD_MS);

    return GSM_OK;
}

2. Function To Receive AT Command

gsm_err_t DTEResponse(gsmConfig_t *config, char buffer[], size_t bufferSize, unsigned long timeout)
{
    unsigned long t = esp_timer_get_time() / 1000;
    while (1)
    {
        uart_get_buffered_data_len(config->gsmUartConfig.uartNum, (size_t *)&config->gsmBuffer.bufferedReadLength);
       
        // we keep this just so that we can make the size of the buffer according to this in the other function which we use.
        config->gsmBuffer.lastBufferedReadLength = config->gsmBuffer.bufferedReadLength;

        if (config->gsmBuffer.bufferedReadLength > 0)
            break;
        
        if ((esp_timer_get_time() / 1000) - t > timeout)
        {
            if (config->debugDTE)
                ESP_LOGW(TAG, "No Response, Timeout");

            return GSM_TIMEOUT;
        }

        vTaskDelay(100 / portTICK_PERIOD_MS);
    }

    t = esp_timer_get_time() / 1000;
    unsigned int i = 0;
    
    while (1)
    {
        while ((config->gsmBuffer.bufferedReadLength) > 0)
        {
            uart_read_bytes(config->gsmUartConfig.uartNum, (char *)&buffer[i], 1, 1 / portTICK_PERIOD_MS);
            config->gsmBuffer.bufferedReadLength--;
            i++;
            t = esp_timer_get_time() / 1000;
        }

        if (i > 2)
        {
            if (buffer[i - 2] == '\r' && buffer[i - 1] == '\n')
            {
                buffer[i - 2] = '\0';
                if (strlen(buffer) > 0)
                    break;
                else
                    i = 0;
            }
        }

        if ((esp_timer_get_time() / 1000) - t > 50 || i >= (bufferSize - 1))
        {
            buffer[i] = '\0';
            break;
        }
    }

    // we can log the data here
    if (config->debugDTE)
    {
        ESP_LOGI(TAG, "Response: %s", buffer);
        ESP_LOG_BUFFER_HEX(TAG, buffer, strlen(buffer) + 1);
    }

    return GSM_OK;
}

3. Function to receive AT command but it checks if there is valid ASCII. ive made this just because after power reset of gsm it sends garbage and the code crashes at strstr. and i cannot use this everywhere as during OTA update we need nonAscii data.

gsm_err_t DTEResponseForAT(gsmConfig_t *config, char buffer[], size_t bufferSize, unsigned long timeout)
{
    unsigned long t = esp_timer_get_time() / 1000;
    uint8_t gsmRespBuff[1030] = {'\0'};
    gsm_err_t ret = GSM_ERROR;
    while (1)
    {
        uart_get_buffered_data_len(config->gsmUartConfig.uartNum, (size_t *)&config->gsmBuffer.bufferedReadLength);

        // we keep this just so that we can make the size of the buffer according to this in the other function which we use.
        config->gsmBuffer.lastBufferedReadLength = config->gsmBuffer.bufferedReadLength;

        if (config->gsmBuffer.bufferedReadLength > 0)
            break;

        if ((esp_timer_get_time() / 1000) - t > timeout)
        {
            if (config->debugDTE)
                ESP_LOGW(TAG, "No Response, Timeout");

            return GSM_TIMEOUT;
        }

        vTaskDelay(100 / portTICK_PERIOD_MS);
    }

    t = esp_timer_get_time() / 1000;
    unsigned int i = 0;
    uint8_t noAsciiFlag = 0;
    memset(gsmRespBuff, 0, sizeof(gsmRespBuff));

    while (1)
    {
        while ((config->gsmBuffer.bufferedReadLength) > 0)
        {
            uart_read_bytes(config->gsmUartConfig.uartNum, &gsmRespBuff[i], 1, 1 / portTICK_PERIOD_MS);

            // check if the data is falling into the ASCII
            // TODO: okay parth, you do know you are being an absolute idiot by adding same functions twice.. make sure you dont show this to anyone and fix it. 
            // TODO: maybe make two tasks? send and receive?
            if (isascii(gsmRespBuff[i]))
            {
                buffer[i] = (char)gsmRespBuff[i];
            }
            else
            {
                noAsciiFlag = 1;
            }
            config->gsmBuffer.bufferedReadLength--;
            i++;
            t = esp_timer_get_time() / 1000;
        }

        if ((i > 2) && (noAsciiFlag == 0))
        {
           
            if (buffer[i - 2] == '\r' && buffer[i - 1] == '\n')
            {
                buffer[i - 2] = '\0';
                if (strlen(buffer) > 0)
                {
                    ret = GSM_OK;
                    break;
                }
                    
                else
                    i = 0;
            }
        }
        

        if (((esp_timer_get_time() / 1000) - t > 50 || i >= (bufferSize - 1)) && noAsciiFlag == 0)
        {
           
            buffer[i] = '\0';
            ret = GSM_OK;
            break;
        }
        else if(noAsciiFlag == 1)
        {
           
            memset(buffer, '\0', bufferSize);
            ret = GSM_ERROR;
            break;
        }
    }

    if (config->debugDTE)
    {
        ESP_LOGI(TAG, "Response: %s", buffer);
        ESP_LOG_BUFFER_HEX(TAG, buffer, strlen(buffer) + 1);
    }

    return ret;
}

i am not at all a good programmer so im sorry if its too much of stupid things i must've done.

thank you very much if you have read all of the above. thanks a lot.