PR for building wheels and other improvements

[laggykiller]

I have created a fork: https://github.com/laggykiller/sqlcipher3

What is improved:

• Using conan to install openssl (dependency of sqlcipher)
• Building with sqlcipher version 4
• Building without scripts in build-scripts, reducing code to maintain
• Compatibility with openssl3
• Including amalgamation files (sqlite3.c and sqlite3.h) in sdist that is uploaded to pypi
• Workflow
  • Building wheels for Windows, MacOS and Linux with cibuildwheels. All architectures supported by cibuildwheel should be working (x86_64, i686, aarch64, ppc64le, s390x)
  • Test wheels and source distribution
  • Auto upload wheels to pypi (PYPI_API_TOKEN need to be added in repository secrets)
• Using the more modren pyproject.toml
  • Specifies build time requirements in pyproject.toml so that they don't have to be explicitly installed using pip before building
  • Note that name, version, package_dir and packages need not to be specified in setup.py if they are specified in pyproject.toml but they are kept or else python <= 3.6 wheel building would fail
• Rename python test scripts to test_*.py for running python -m unittest easily

Note that I referenced with this fork that improved on pysqlite3 (https://github.com/Dobatymo/pysqlite3). See if you also want to get a PR from there for pysqlite3.

Here is the diff: https://github.com/coleifer/sqlcipher3/compare/master...laggykiller:sqlcipher3:master

I have uploaded the wheels to pypi: https://pypi.org/project/sqlcipher3-wheels

If you are interested in PR, please tell me what improvements you want and don't want, then I will open a PR.

If you are not interested in PR, I can just maintain sqlcipher3-wheels

[LesHol]

Hi I have been trying for a long time to get sqlcipher to work on android with python and buildozer and your wheels but failed. I´m a newbie with compilation so that may be part of the answer. When I try to install the wheel for aarch on my pc it says it is the wrong architecture of course but I don´t understand how I can get it to work with buildozer! I really would appreciate a description how to if it is possible.

TIA Leslie

[laggykiller]

@LesHol You should had asked in my repo, not here.

Anyway, you should build with x86_64 wheels if you are testing on PC (Assuming you are using x86_64 PC and you are running x86_64 Android emulator, not running aarch64 Android emulator), build with aarch64 if you are testing on Android phone (Assuming you have aarch64 phone). You should set "android.arch" in buildozer.

However if you are building iOS applications wuth buildozer, I think you might have to build sqlcipher3 by yourself...

[LesHol]

Hi Sorry about asking in the wrong place and thank you for your immediate answer. Just to be clear: clone sqlcipher clone sqlcipher3 pip sqlcipher wheels (your source) In buildozer requirements: sqlcipher3 Put android.arch in buildozer.spec exactly where?

Is this right or have I misunderstood something?

TIA Leslie

Den tors 9 jan. 2025 kl 00:54 skrev laggykiller @.***>:

@LesHol https://github.com/LesHol You should had asked in my repo, not here.

Anyway, you should build with x86_64 wheels if you are testing on PC (Assuming you are using x86_64 PC and you are running x86_64 Android emulator, not running aarch64 Android emulator), build with aarch64 if you are testing on Android phone (Assuming you have aarch64 phone). You should set "android.arch" in buildozer.

However if you are building iOS applications wuth buildozer, I think you might have to build sqlcipher3 by yourself...

— Reply to this email directly, view it on GitHub https://github.com/coleifer/sqlcipher3/issues/24#issuecomment-2578901164, or unsubscribe https://github.com/notifications/unsubscribe-auth/AG3IZRG3ECQDIOUGWN3HDYL2JW3D3AVCNFSM6AAAAABU2IXJL6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNZYHEYDCMJWGQ . You are receiving this because you were mentioned.Message ID: @.***>

[laggykiller]

@LesHol I already said this is the wrong place to ask, and you keep asking in the wrong place. Let's move your problem to https://github.com/laggykiller/sqlcipher3/issues/2

[mxmlnkn]

It would be nice if this were merged to avoid yet another fork! But I am thankful that the packages already exist on PyPI anyway.

And thank you for adding a short ReadMe explaining the usage! I also recommend merging that into this repo because this repo seems to have no explanation as to how to use the Python module. And even after the assumption that it should behave like the sqlite3 Python core module, my first try would have been import sqlcipher3; sqlcipher3.connect( ... ) instead of from sqlcipher3 import dbapi2 as sqlite.

[richbodo]

It might be a pain to get this working on macs, but it seems super valuable as the older drivers for encrypted db use are all worse off than this one, so there really aren't great options.

I had an AI write a little bug report about the mac issue that the mac people are seeing:

sqlcipher3 Compilation Bug Report

Issue Summary

sqlcipher3 fails to compile on macOS ARM64 (Apple Silicon) due to header file path issues. The package is hardcoded to look for sqlcipher/sqlite3.h but cannot find the headers even when SQLCipher is properly installed via Homebrew.

Adding symlinks to these libraries fixes the issue, but it's a brittle solution:

ln -s /opt/homebrew/lib/libsqlcipher.a /usr/local/lib/libsqlcipher.a
ln -s /opt/homebrew/include/sqlcipher /usr/local/include/sqlcipher

Environment Details

System Information

• OS: macOS 15.6 (Sequoia)
• Architecture: ARM64 (Apple Silicon)
• Python Version: 3.13.1
• Platform: macOS-15.6-arm64-arm-64bit-Mach-O

Package Versions

• sqlcipher3: 0.5.4 (latest)
• SQLCipher: 4.6.1 (installed via Homebrew)
• pip: 24.3.1
• setuptools: Latest

Installation Method

# SQLCipher installation
brew install sqlcipher

# sqlcipher3 installation attempt
pip install sqlcipher3

Error Details

Compilation Error

clang -Qunused-arguments -DMODULE_NAME=\"sqlcipher3.dbapi2\" -DSQLITE_HAS_CODEC=1 -I/usr/include -I/Users/richardbodo/src/prt/prt_env/include -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.13/include/python3.13 -c src/blob.c -o build/temp.macosx-14.0-arm64-cpython-313/src/blob.o

In file included from src/blob.c:1:
src/blob.h:4:10: fatal error: 'sqlcipher/sqlite3.h' file not found
    4 | #include "sqlcipher/sqlite3.h"
      |          ^~~~~~~~~~~~~~~~~~~~~
1 error generated.
error: command '/usr/bin/clang' failed with exit code 1

SQLCipher Installation Verification

# SQLCipher is properly installed
$ brew list | grep sqlcipher
sqlcipher

# Headers exist in the expected location
$ ls -la /opt/homebrew/opt/sqlcipher/include/sqlcipher/
total 1344
drwxr-xr-x@ 4 richardbodo  admin     128 Aug 14  2024 .
drwxr-xr-x@ 3 richardbodo  admin      96 Aug 14  2024 ..
-rw-r--r--@ 1 richardbodo  admin  646009 Aug 14  2024 sqlite3.h
-rw-r--r--@ 1 richardbodo  admin   38149 Aug 14  2024 sqlite3ext.h

Attempted Solutions

1. Environment Variables

export CFLAGS="-I/opt/homebrew/opt/sqlcipher/include"
export LDFLAGS="-L/opt/homebrew/opt/sqlcipher/lib"
pip install sqlcipher3

Result: Same error - headers not found

2. Direct Include Path

export CFLAGS="-I/opt/homebrew/opt/sqlcipher/include/sqlcipher"
export LDFLAGS="-L/opt/homebrew/opt/sqlcipher/lib"
pip install sqlcipher3

Result: Same error - headers not found

3. pkg-config Approach

export CFLAGS="$(pkg-config --cflags sqlcipher)"
export LDFLAGS="$(pkg-config --libs sqlcipher)"
pip install sqlcipher3

Result: pkg-config not available for sqlcipher on macOS

Root Cause Analysis

The issue appears to be that:

1. sqlcipher3 expects headers at: sqlcipher/sqlite3.h
2. SQLCipher installs headers at: /opt/homebrew/opt/sqlcipher/include/sqlcipher/sqlite3.h
3. Compiler cannot find headers even with correct include paths

This suggests the sqlcipher3 build system is not properly handling the include paths on macOS ARM64.

Impact

This bug prevents sqlcipher3 from being used on macOS ARM64 systems, which is a significant limitation given the growing adoption of Apple Silicon Macs.

Requested Fix

1. Update build system to properly detect SQLCipher installation on macOS ARM64
2. Add proper include path handling for Homebrew installations
3. Consider adding macOS ARM64 wheels to avoid compilation issues
4. Update documentation with macOS ARM64-specific installation instructions

Additional Context

• sqlcipher3-binary package is not available for macOS (Linux only)
• pysqlcipher3 has the same compilation issues
• This affects the broader ecosystem of encrypted SQLite applications on macOS ARM64

Reproduction Steps

1. Install macOS on Apple Silicon Mac
2. Install Homebrew
3. Install SQLCipher: brew install sqlcipher
4. Create Python virtual environment
5. Attempt to install sqlcipher3: pip install sqlcipher3
6. Observe compilation failure
7. Add the symlinks to the homebrew-installed libraries
8. Works.

[coleifer]

I'm surprised specifying the environment variables does not work - is this due to pip ignoring them?

[laggykiller]

I'm surprised specifying the environment variables does not work

@coleifer Specifying environment variables would had worked. Don't get fooled by AI. It is not necessary to do linking as suggested by the AI if you patch the culprit.

The actual culprit is in: https://github.com/coleifer/sqlcipher3/blob/e2176211155d0cc0a6580a9d9788b84fd77ef254/setup.py#L29-L31

Actually sqlcipher3 would build fine either using CFLAGS="-I/opt/homebrew/include" LDFLAGS="-L/opt/homebrew/lib" python setup.py build or CFLAGS="-I/opt/homebrew/opt/sqlcipher/include" LDFLAGS="-L/opt/homebrew/opt/sqlcipher/lib" python setup.py build if the culprit code does not exist. However, the culprit code overwrites CFLAGS supplied, causing compiler unable to find sqlcipher header.

I suggest:

if sys.platform == "darwin":
    os.environ['CFLAGS'] = os.environ.get("CFLAGS", "") + " -Qunused-arguments -I/opt/homebrew/include"
    os.environ['LDFLAGS'] = os.environ.get("LDFLAGS", "") + " -L/opt/homebrew/lib"
    log.info("CFLAGS: " + os.environ['CFLAGS'])
    log.info("LDFLAGS: " + os.environ['LDFLAGS'])

PS: Mostly a pain to build on Windows, not on *nix

[laggykiller]

@coleifer You may close this issue freely if you are not interested in my proposed PR, I can just continue to maintain my branch and the sqlcipher3-wheels PyPI package. I don't want this issue to get hijacked.

[coleifer]

I'm not sure whether or not I want to go ahead w/the merge. The Sqlcipher project itself is moving in a weird direction, so I've kept it open for now.

[richbodo]

Thanks @laggykiller. Do not want to hijack the issue. Appreciate the patch.