Support for Windows Phone 8.1

[Aurora12]

Hello,

I want to use libsodium-net in a Windows Store Universal App. This type of projects was introduced this spring and features sharing code between the mobile and desktop parts of the project. This works starting with Windows 8.1 and uses a restricted subset of .NET apis (somewhat even more restricted than Windows Phone 8 and earlier versions had). NuGet package for this kind of projects should target 'WindowsPhoneApp,Version=v8.1'.

There are two problems here:

1. .NET apis:
   • AppDomain class is unavailable and there is no loading dlls at runtime as far as I know. So, you can not use Sodium.DynamicInvoke.
   • System.Security.Cryptography is now Windows.Security.Cryptography
   • Array.Copy() last parameter is no longer long, it's int.
   • ProtectedMemory and MemoryProtectionScope are to be somehow replaced with Windows.Security.Cryptography.DataProtection.DataProtectionProvider. Here's the full list of replacements: http://msdn.microsoft.com/en-us/library/windows/apps/hh464945.aspx
2. I need to statically link sodium library to Windows Phone part of the project. Current libsodium-1.0.1 compiles for a variety of platforms, but not for Windows Phone ARM (let alone 8.1, for that matter). On MSDN forums they suggest that you should create a C++ Static Library or Windows Runtime component and compile the code with Visual Studio (so you can use the result in a Universal Store App).

While I could possibly cope with the first part, I have no clues as to what to do with the second one. Any ideas?

[Aurora12]

This is what I get when I try to compile libsodium for Windows ARM in Visual Studio 2013:

error C3861: '__cpuid': identifier not found -- sodium\runtime.c
error C2079: 'tv' uses undefined struct 'sodium_hrtime::timeval' -- randombytes\salsa20\randombytes_salsa20_random.c    
+ some more errors related to timeval
error C3861: 'VirtualLock': identifier not found -- sodium\utils.c
error C3861: 'VirtualUnlock': identifier not found -- sodium\utils.c
error C3861: 'GetSystemInfo': identifier not found -- sodium\utils.c
error C3861: 'VirtualProtect': identifier not found -- sodium\utils.c
error C3861: 'VirtualProtect': identifier not found -- sodium\utils.c
error C3861: 'VirtualProtect': identifier not found -- sodium\utils.c
error C3861: 'VirtualAlloc': identifier not found -- sodium\utils.c
error C3861: 'VirtualFree': identifier not found -- sodium\utils.c

This is what I did:

• added WIndows Phone 8.1 project to libsodium solution,
• referenced sources and headers from original project,
• set up compilation options appropriate for ARM
• added SDK folders to compile path: /I"c:\Program Files (x86)\Windows Kits\8.1\Include\um" /I"c:\Program Files (x86)\Windows Kits\8.1\Include\shared"
• used nosse version of pwhash_scryptsalsa208sha256

[Aurora12]

There are memoryapi classes (VirtualLock etc) in Windows Phone Kits\8.1\Include\minwin\memoryapi.h, but, alas, they are inside #if WINAPI_FAMILY_PARTITION(WINAPI_PARTITION_DESKTOP) block.

[jedisct1]

That can be worked around. What is the proper way to detect Windows 8.1 Mobile?

[jedisct1]

Maybe something like https://github.com/jedisct1/libsodium/commit/820c84ab894667f2ad0cef4e9cf849bee0a1fe71 can do the trick?

Is timeval actually not being defined anywhere on this target? It's safe to remove the randombytes_salsa20 folder from the project if you don't use it, but timeval being unavailable is quite surprising.

[alexbakker]

I'm sorry for jumping in on the discussion like this but I figured this wasn't worth creating a separate issue. I'm trying to compile libsodium as a runtime component too but I'm getting some errors. jedisct1/libsodium@820c84a fixed some of the errors Aurora12 was having but I'm still stuck with these:

error C2719: '_A': formal parameter with __declspec(align('16')) won't be aligned (src\libsodium\crypto_pwhash\scryptsalsa208sha256\sse\pwhash_scryptsalsa208sha256_sse.c)  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\xmmintrin.h  186 1   libsodium
error C2719: '_B': formal parameter with __declspec(align('16')) won't be aligned (src\libsodium\crypto_pwhash\scryptsalsa208sha256\sse\pwhash_scryptsalsa208sha256_sse.c)  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\xmmintrin.h  186 1   libsodium
error C2719: '_A': formal parameter with __declspec(align('16')) won't be aligned (src\libsodium\crypto_pwhash\scryptsalsa208sha256\sse\pwhash_scryptsalsa208sha256_sse.c)  C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\xmmintrin.h  187 1   libsodium
error C1083: Cannot open include file: 'wincrypt.h': No such file or directory

(about 100+ of C2719)

[jedisct1]

To work around the first issue, remove || defined(_MSC_VER) in pwhash_scryptsalsa208sha256_sse.c and crypto_scrypt-common.c.

The absence of wincrypt.h is more concerning. Try to remove the #include <wincrypt.h> statement in utils.c and see how it goes.

[alexbakker]

Forget about wincrypt.h not being found. It exists, I just wasn't including the right directory, sorry about that. After that I got an error about the fact that advapi32.lib was missing, I fixed that by adding C:\Program Files (x86)\Windows Kits\8.1\Lib\winv6.3\um\arm to the list of library directories

Now this error pops up:

error LNK2019: unresolved external symbol __cpuid referenced in function _cpuid runtime.obj

Thanks for your help so far.

[jedisct1]

Replace #ifdef _MSC_VER with #if 0 in runtime.c.

[alexbakker]

That's it, it compiled. Thank you for your help!

[jedisct1]

Woo-ooh!

[Aurora12]

@jedisct1 Thanks for looking into the issue!

I excluded randombytes_salsa20_random.c from the project since I don't use it, and this eliminated timeval errors. Windows Store apps do not support Winsock APIs (unlike Windows Phone 8 apps), which is the reason for timeval not being there, as far as I know.

The project doesn't compile still, looks like the fix is not effective. This is what VS adds to definitions in WP8.1 project: PSAPI_VERSION=2;WINAPI_FAMILY=WINAPI_FAMILY_PHONE_APP;_UITHREADCTXT_SUPPORT=0;%(PreprocessorDefinitions) I guess this is the way to detect WP platform. Could you please modify your fix?

[Aurora12]

@jedisct1 I added a couple of lines in utils.c and runtime.c just above your previous additions. Did so according to http://blogs.msdn.com/b/chuckw/archive/2012/09/18/dual-use-coding-techniques-for-games-part-3.aspx

#ifdef _WIN32
# define WINAPI_DESKTOP
# if defined(WINAPI_FAMILY) && (WINAPI_FAMILY == WINAPI_FAMILY_PHONE_APP)
#  undef WINAPI_DESKTOP
# elif defined(WINAPI_FAMILY_ONE_PARTITION) && defined(WINAPI_FAMILY_DESKTOP) && !WINAPI_FAMILY_ONE_PARTITION(WINAPI_FAMILY_DESKTOP)
#  undef WINAPI_DESKTOP
# endif
#endif

Now the project compiles! Currently I'm testing it to see if all works fine on devices.

[Aurora12]

Oh, well, shit just got real. While the project does compile for windows phone 8.1, it still does not compile as a part of a Windows Store project (universal project for WP and desktop), which is what I need in the end. I made some corrections to code, but got stuck with several things.

Corrections I've made:

• __inline instead of inline
• disabled /ZW and /Yu for all C files
• #include < windows.security.cryptography.h > instead of # include < wincrypt.h >

I also excluded files I don't need. I only need crypto_box, crypto_sign and crypto_secretbox with their dependencies.

Errors I have now:

• C2099: Initializer is not a constant in randombytes.c on line:

static const randombytes_implementation *implementation =
    &randombytes_sysrandom_implementation;

• C4146: unary minus operator applied to unsigned type, result still unsigned in fe_cswap_curve25519_ref10.c, fe_cmov.c, randombytes_sysrandom.c in expressions like b = -b; with b being unisigned int.

I'd appreciate any hints as to what to do with this stuff (why does it compile for other platforms anyway?)

[alexbakker]

@Aurora12 The former can be solved by defining SODIUM_STATIC. The latter can be solved by disabling SDL checks.

[jedisct1]

Can you check if the current libsodium code in git properly works for you in a universal app (or at least on Windows Phone)?

[Aurora12]

@jedisct1 Will give it another try this week! For several months I've been using js implementation of nacl through web view, which is of course ridiculous (not to say hilarious), but I had more important business logic problems to solve. Now I need to get it really working both in direct and in stream mode. Will update this thread on the results.

[Aurora12]

@jedisct1 So, I gave it another try and was partly successfull.

I was able to compile libsodium for ARM and x86/x64 in a Windows Runtime Component project for Universal Apps (targeting Windows 8.1 and Windows Phone 8.1). I had to do quite a lot of stuff to get it compiling (I can list it if you like), and had to write a wrapper class that exposes methods to C# using C++/CX types. I then created a test project of a Universal Store App in C# and tried to use the library.

Works fine in Windows 8.1 part of the project (I tried boxing and signing).

Doesn't work for Windows Phone 8.1 part. It looks like sodium fails to even initialize: the first reference to C++ wrapper class yields exception "The specified module could not be found" with no details as to which module failed. I tried launching x86 build in phone emulator and ARM build on device and the behaviour is the same.

What I do now is I'm adding code files to libsodium sources one by one to see when it stops loading in winphone. I started with just core.c, utils.c and runtime.c. It's rather hard since I'm not all that comfortable with C or C++ projects.

[Aurora12]

@jedisct1 @adamcaudill Looks like windows phone 8.1 stops loading library assemblies once I include randombytes_sysrandom.c into project. The library compiles. I commented out a couple of lines in core.c and utils.c, so the code is actually not triggered. Once randombytes_sysrandom.c is there, library fails to load on windows phone 8.1. Any ideas?

1> randombytes.c 1> randombytes_sysrandom.c 1> core.c 1> runtime.c 1> utils.c 1> version.c

These are the only files from libsodium I compile now.

[Aurora12]

@jedisct1 @adamcaudill Looks like RtlGenRandom is not available on wp 8.1, so this doesn't work:

// randombytes_sysrandom.c

#ifdef _WIN32
# include <windows.h>
# define RtlGenRandom SystemFunction036
# if defined(__cplusplus)
extern "C"
# endif
BOOLEAN NTAPI RtlGenRandom(PVOID RandomBuffer, ULONG RandomBufferLength);
# pragma comment(lib, "advapi32.lib")
#endif

As I mentioned in previous comment, libsodium fails to load if I compile randombytes_sysrandom.c. But when I comment out the following lines, the library loads (but is not working, obviously):

if (! RtlGenRandom((PVOID) buf, (ULONG) size)) {
    abort(); /* LCOV_EXCL_LINE */
}

https://msdn.microsoft.com/en-us/library/windows/desktop/aa387694%28v=vs.85%29.aspx This page suggests that we should use CryptGenRandom instead, but the latter resides in Wincrypt.h, which is not available on wp8.1. We have GenerateRandom function in windows.security.cryptography.h, may be that's the way...

[jedisct1]

Thanks for pinpointing the culprit. Is it the only remaining thing that doesn't work on WP 8.1?

CryptGenRandom() not being available is a bummer.

Windows::Security::Cryptography::CryptographicBuffer::GenerateRandom() looks like a way to do it, but is it available as a C function as well? Mind grepping for "generaterandom` in the available header files?

[Aurora12]

@jedisct1 There is wincrypt.h file in Windows Kit 8.1, and it does contain CryptGenRandom(). But almost all of the code there, including CryptGenRandom(), is inside #if WINAPI_FAMILY_PARTITION(WINAPI_PARTITION_DESKTOP), so it's desktop only. And Windows Phone Kit 8.1 does not contain wincrypt.h at all.

The only other file to contain generaterandom something is schannel.h which is in Windows Kit and contains Public Definitions for SCHANNEL Security Provider.

// Pre NT4.0 SP2 calls.  Call CAPI1 or CAPI2
// to get the same functionality instead.
VOID
WINAPI
SslGenerateRandomBits(
    PUCHAR      pRandomData,
    LONG        cRandomData
    );

[Aurora12]

@jedisct1 I wonder if I could use another approach. What if I generate random bytes in my wrapper class using CryptographicBuffer::GenerateRandom() and pass them to libsodium methods thus avoiding using randombytes_buf() altogether? This would be awkward, yes, and would require writing wrapper methods inside libsodium, but I need it working at long last. :)

[Aurora12]

@jedisct1 Just discovered one another problem: SecureZeroMemory doesn't seem to be accessible on WP in utils.c sodium_memzero().

[Aurora12]

@jedisct1 There's also some good news. I wrote the following and compiled it with /ZW.

// h:
extern "C" int GenerateRandomBytes(unsigned char *buffer, int length);

// cpp:
#include "pch.h"
#include "Extern.h"

using namespace Platform;
using namespace Platform::Collections;
using namespace Windows::Foundation::Collections;
using namespace Windows::Security::Cryptography;
using namespace Windows::Storage::Streams;

extern "C" int GenerateRandomBytes(unsigned char *bytes, unsigned int length)
{
    IBuffer^ buffer = CryptographicBuffer::GenerateRandom(length);
    DataReader^ reader = DataReader::FromBuffer(buffer);
    reader->ReadBytes(ArrayReference<unsigned char>(bytes, buffer->Length));
    return 1;
}

I then replaced lines with RtlGenRandom in randombytes_sysrandom.c with the following:

if (!GenerateRandomBytes((unsigned char*)buf, (unsigned int)size)) {
    abort();
}

It seems to work fine on Windows 8.1 part of the project. The library still fails to load on Windows Phone 8.1, but, apparently, the problem with random generator is fixed.

[jedisct1]

As a cleaner alternative to patching randombytes_sysrandom.c, you can register a custom implementation: http://doc.libsodium.org/advanced/custom_rng.html

Or, maybe even cleaner, declare a SystemFunction036() polyfill.

Still, that's fairly annoying. There has to be a way to do that in C.

[Aurora12]

@jedisct1 I totally agree. I left it for now as I'm still investigating the reasons the library fails to run. The problem with SecureZeroMemory I mentioned above may be a false alarm. It's quite difficult to debug this when running a winrt project.

[jedisct1]

SecureZeroMemory() not being available is not a big deal; the portable implementation is totally fine with all versions of Visual Studio so far.

But not having a PRNG is a showstopper.

[Aurora12]

@jedisct1 I finally got libsodium working in my universal store project. Here's the list of modifications to libsodium sources I had to make:

1. Replace static inline with static __inline in all files
2. Replace calls to RtlGenRandom with calls to a function (a couple of comments above) that internally uses winrt apis and is compiled with /ZW. Alternatively, define SystemFunction036 as you suggested above. This affects randombytes_sysrandom.c and randombytes_salsa20_random.c.
3. Undefine WINAPI_DESKTOP symbol for windows phone (a patch to runtime.c from the beginning of this thread).

I did not compile the following (I don't use the functionality): crypto_aead crypto_auth crypto_pwhash crypto_shorthash

[GregorSpagnolo]

@Aurora12 how did you linked C -> C++ code? In C project I created a cpp file and tried to compile as ZW. When hitting compile I receive the following error: LNK2038 mismatch detected for 'vccorlib_lib_should_be_specified_before_msvcrt_lib_to_linker': value '1' doesn't match value '0' in MSVCRTD.lib(app_appinit.obj)

[Aurora12]

@GregorSpagnolo Here's what I did:

1. Created a project from template Visual C++ : Store Apps : Universal Apps : Windows Runtime Component (Universal Apps) (VS2013).
2. Added libsodium/src to Shared project
3. Removed Consume Winrt Extensions (/ZW) and Use Precompiled Headers (/Yu) from all *.c files.
4. Copied properties from VS solution included in libsodium distribution (preprocessor symbols, disabled warnings)
5. Added reference to Microsoft Visual C++ 2013 Runtime Package for Windows Phone to Windows Phone project.
6. Made fixes to C code listed in previous comment.

I believe, you should be able to compile and initialize library with this setup.