Fixes and Improvements

[savejeff]

Hi, as discussed in the other thread, here are the improvements I implemented for the lib

#55

I have attached multiple versions that are more and more modified from the original version. the earlier might be easier to integrate.

sqlite3lib_github_fixes.zip

i renamed esp32.cpp with jhal.cpp

V1: only minor changes but added debug prints for understanding what the lib is doing V2: most of the fixes implemented but still similar to the original code <- I recommend this for importing fixes V3: refactoring and cleanup V4: my current version with working truncate.

V2-V4 uses my own implementation of a filesystem. it has the same functions as the esp fs classes like sd, sd_mmc, spiffs. replacing with the esp functions should be very easy.

if you have a question, just contact me. I recommend using a diff tool like beyond compare to compare the changes

Here are some additional explanations:

################## The File Struct #######################

struct ESP32File {
	sqlite3_file base;              /* Base class. Must be first. */
	jFileSystemClass* jfs; 	// pointer to filesystem (replace this with esps fs instance like sd, mmc, spiffs, ...)
	jFile* fp;                   	// pointer to file (replace this with File from esps fs::File 
	int lock;				// this is the lock applied to this database file
	int id;				// unique identifier for the database file to make debug prints easy

	char *aBuffer;                  /* Pointer to malloc'd buffer */
	int nBuffer;                    /* Valid bytes of data in zBuffer */
	sqlite3_int64 iBufferOfst;      /* Offset in file of zBuffer[0] */
};

##################### Log functions #########################

Log and LogD are just two levels of logging where LogD indicates debug prints only to the console, Log could also log to a logfile. just replace all LogD with Log

#if JSQL_DEBUG==1 // <- set JSQL_DEBUG to 1 to enable debug printing
#define LogX Log
#define LogXD LogD // <- remove "D" to only use Log
#else
#define LogX(...)
#define LogXD(...)
#endif

#include <stdarg.h>

String s_printf(const char* format, ...)
{
	char loc_buf[64];
	char* temp = loc_buf;
	va_list arg;
	va_list copy;
	va_start(arg, format);
	va_copy(copy, arg);
	int len = vsnprintf(temp, sizeof(loc_buf), format, copy);
	va_end(copy);
	if (len < 0) {
		va_end(arg);
		return "";
	};
	if (len >= sizeof(loc_buf)) {
		temp = (char*)malloc(len + 1);
		if (temp == NULL) {
			va_end(arg);
			return "";
		}
		len = vsnprintf(temp, len + 1, format, arg);
	}
	va_end(arg);
	
	String s = String(temp);
	
	if (temp != loc_buf) {
		free(temp);
	}
	return s;
}



extern void Log(const char* format, ...) __attribute__((format(printf, 1, 2)));
void Log(const char* format, ...)
{
	char loc_buf[64];
	char* temp = loc_buf;
	va_list arg;
	va_list copy;
	va_start(arg, format);
	va_copy(copy, arg);
	int len = vsnprintf(temp, sizeof(loc_buf), format, copy);
	va_end(copy);
	if (len < 0) {
		va_end(arg);
		return;
	};
	if (len >= sizeof(loc_buf)) {
		temp = (char*)malloc(len + 1);
		if (temp == NULL) {
			va_end(arg);
			return;
		}
		len = vsnprintf(temp, len + 1, format, arg);
	}
	va_end(arg);


	SerialOut.print("[");
	SerialOut.print(millis());
	SerialOut.print("] ");
	SerialOut.println(temp);

	

	if (temp != loc_buf) {
		free(temp);
	}
}

void Log(const String& msg)
{
	Log(msg.c_str());
}

##################### truncate #############################

here the implementation of truncate in my file system class:

here the file flags:

// Read and Write with Position at end of file. appends starting from end of file
#define _FILE_FLAG_RW_APPEND "a+"

// Read and Write with Position at start file. overrides starting from position 0
// Can be replaced by APPEND and move to first position (seek(0))
#define _FILE_FLAG_RW "r+"
#define _FILE_FLAG_RW_OVERRIDE "w+"

// For read file only
// in General this might work better than w+ when reading a whole file from start to end
#define _FILE_FLAG_R "r"

bool jFileSystemClass::truncate(const String& path, uint32_t size) 
{
	if(!exists(path))
		return false;

	String ftmp = get_tmp_filename();

	LogXD("jfs.%s.truncate: ftmp=%s", get_tag(), ftmp.c_str());

	rename(path, ftmp);

	jFile* f_old = open(ftmp, jFILE_MODE::FILE_FLAG_R);
	jFile* f_new = open(path, jFILE_MODE::FILE_FLAG_RW_OVERRIDE);

	//if(!f_old->isOpen() || !f_new->isOpen())
	//	return false;

	uint32_t size_new = 0;
	uint8_t buff[128];
	while (size_new < size && f_old->available())
	{
		int len = MIN(128, size - size_new);
		int rlen = f_old->read(buff, len);

		f_new->write(buff, rlen);
		size_new += rlen;
	}

	LogXD("jfs.%s.truncate: size=%d, size_new=%d", get_tag(), size, size_new);

	f_old->close();
	f_new->close();

	delete(f_old);
	delete(f_new);

	remove(ftmp);

	return true;
}

String jFileSystemClass::get_tmp_filename() 
{
	//TODO deadlock in case of all temp files are present -> not likely
	while(true) {
		int x = get_random_int(0, 999);
		String fname = s_printf("TMP%02d.BIN", x);
		if(!exists(fname))
			return fname;
	}
}

[siara-cc]

Hi @savejeff , Thanks for sharing this and sorry about the late response. I will work on it and update the library soon!!

[savejeff]

no problem. just wanted to give something back to the open source community.

I think making SQlite available to the Arduino community is a big thing. With the Filesystem abstraction, I'm also able to use the SdFat library and target other processors like RP2040 and SAMD etc. Teensy still has some problems though