Possible clashing symbols from multiple support libraries

[artwyman]

I'm logging this issue to capture a known potential issue so it doesn't get forgotten. It's possible to use Djinni to build a standalone library which could then be included into an app which also uses Djinni, or which uses other Djinni-using libraries. In such a case, depending on how the linker is set up, it's possible to end up with duplicate symbols for pieces of the support library. That could cause problems if the different components were built using different versions of the support library, leading to one of them using an incompatible version.

This is distinct from the possibility of duplicate symbols in generated code, which can at least be addressed by changing names and prefixes. Since the support lib names are fixed, they would need to be addressed in some other way. If the two components are built by different companies, it might be hard for a developer to address this easily, though an annoying workaround would be to hack up their local copy of Djinni to use customized names.

I haven't heard of any real instances of this, so it's not a priority to fix immediately. However I've heard of use of Djinni in projects like PSPDFKit which are libraries which could run into it. I think it would only be a problem if the distinct components are linked together statically, or if the support lib is in a separate dynamic library from the code which uses it, but I haven't tested to confirm anything. I'm mostly putting this issue here as a placeholder for tracking any discussion or any real failures which come up.

[mknejp]

What about inline namespaces? I know libc++ uses them. All their symbols are actually in namespace std::__1. We obviously can't use a double underscores though.

I don't know about PSPDFKit, but if they compile the support library into a separate libDjinni at least it's clear to consumers of such libraries that there might be conflicts. This could be further encouraged with a CMake project specifically for the support lib itself.

[artwyman]

Inline namespaces might be a good approach, though it would require a bit of scaffolding around making it easy to change the version, and remembering to do so when the support lib changes, which so far has been a much more informal process. This would be easier if we started doing numbered releases of Djinni rather than just an ongoing stable master branch.

[artwyman]

I thought about a shortcut for this, which would be to use the inline namespace approach, but make the inline namespace name switchable with preprocessor macro. I don't love relying on the pre-processor, but letting someone pass "-DDJINNI_SUPPORT_LIB_NAMESPACE" would be a quick way to let them ensure unique symbols for their product, without requiring a careful versioning scheme.

[mknejp]

If someone is using Djinni and a library that also relies on Djinni chances are high they want to exchange these types, like putting a library type in a record or interface. That's a recipe for disaster if those two Djinni implementations are different. The cache is definitely going to misbehave as passing the same interface to an app or library method will not preserve object identity.

It's just like with every other library, you don't want two versions of the same library running side-by-side. The one integrating the support library statically into their framework is the one to blame, that's just bad practice. People should treat the Djinni support library like any other dependency. And having a proper versioning scheme would certainly help with that. AFAIK the use of inline namespaces in libc++ is reserved only for ABI breaking changes.

[artwyman]

I think that's a valid use case, and tricky to support given no standard ABI for C++. However I'd argue it's probably not the most common case. I think it's more likely that two independent libraries, or the app plus a library make use of Djinni for their own purposes, but don't need to interact in a way which involves Djinni at all. That would be the case if PSPDFKit starts using Djinni, but then is used inside an app (like Dropbox) which uses Djinni. Djinni is far from standard enough for independently-developed components to be likely to use it in their external API immediately.

All that being said, a solution which covers all use cases would be best. However if this becomes a blocking issue for someone (it hasn't yet as far as I know), I'd settle for a more limited solution which works for the necessary cases.

[pellet]

I'm currently investigating the possibility of this issue arising within a static library which I develop for ios. Correct me if I'm wrong but won't the name mangling done by the C++ compiler ensure that this won't happen? I guess C++ mangles symbols by default making them unique unlike obj-c and c.

[artwyman]

Name mangling doesn't guarantee uniqueness, or you'd never successfully find a symbol. What it does is make sure you get a unique symbol for every function/variable, including things like namespaces and overloaded argument types. A function with the same name, namespace, and argument types will always generate the same mangled name, so the situation for duplicate symbols is unchanged from what it would be in C.

In particular, the Djinni support library contains some helper classes/functions with fixed names. If you use it in more than one sub-part of your app, you'll have duplicates. That'll likely be fine, unless they're built from different incompatible versions of the support library code, in which case some callers might not get the behavior they expected.

[pellet]

Ok, thanks for the clarification. I think this can be fixed for the android jni stuff by this parameter, if it's working correctly for djinni/src/run

   --jni-out $jni_out \
   --ident-jni-class NativeFooBar \
   --ident-jni-file NativeFooBar \

As for the namespace 'djinni' issue in the support library/generated files, I guess some changes will need to be made. The djinni namespace could be placed under the "--cpp-namespace", so it will not clash. The djinni support files could be copied and namespace modified and placed into a path specified by a new parameter called "--support-out".

[artwyman]

Yeah, the existing cmd-line arguments can ensure no conflicts between generated code by using unique prefixes. It's the support library which is at issue, since that code doesn't go through the generator, and thus is always the same. We could consider making the support-library source into a template with replaceable names which the generator can replace while copying to an output dir. Or it might be sufficient to hack a solution with the pre-processor alone. E.g. put everything in a namespace named with a macro like DJINNI_SUPPORT_NAMESPACE, defined to "djinni" by default via #ifndef, but something which users can pass on their compiler command-line to override.

[pellet]

The pre-processor idea sounds good too. I guess pre-processors just scare me a little bit when things go wrong :) For example, if there was a crash in the support library, the stack trace would be pointing to a namespace not actually existent in the source code. It's much of a muchness however, there could be reasons for using the preprocessor instead of the generator, like in projects which are referencing the djinni support code directly.

[pellet]

I came across this https://gcc.gnu.org/wiki/Visibility The fix could be as simple as using the -fvisibility=hidden flag.

[artwyman]

That will probably work for some cases. I think it only helps if the Djinni support lib and the code which uses it are in the same library. Also it looks like that feature is primarily aimed at shared libraries, so I'm not sure how it might apply to static libraries, if you're putting together several of those in your app.

[pellet]

It works the same with static libraries, if the symbols are made invisible they wont be linkable, it also works with obj-c not just c++.

[moderateepheezy]

I am currently encountering this issue on Xcode ld: 50 duplicate symbols for architecture arm64, it is as you stated above, but unfortunately, I don't understand the whole conversation and how to fix it.

This issue occurs when I pod install Firebase or Appsee, I would so love if someone can help me with a fix for this.

[artwyman]

Hmm, do Firebase and Appsee use Djinni? That's not something I'd heard of, and a bit surprising. Can you share more details of exactly which symbols are duplicated, and which libraries they're coming from?

[moderateepheezy]

[artwyman]

Those don't all look like Djinni symbols to me. E.g. OAKLibUtility is an ObjC class, which could possibly be Djinni-generated, but I'm doubtful that both you and a third-party would create a class of exactly that name. Is that one of your classes, or something from a 3rd-party library? The screenshot shows that class defined in both liboaktree_objc.a and libapplication_objc.a. That sounds to me like it could be a case where the same source was built in two different targets, or where the source was built into some static-lib, which was then linked into both liboaktree and libapplication, which then causes this problem later down the line.

[moderateepheezy]

OAKLibUtility is one of the classes that was created in c++ for Djinni, I didn't build the c++ class for Djinni, the guy who built it is not available for now.

The screenshot shows that class defined in both liboaktree_objc.a and libapplication_objc.a. That sounds to me like it could be a case where the same source was built in two different targets, or where the source was built into some static-lib, which was then linked into both liboaktree and libapplication, which then causes this problem later down the line.

How can I approach the problem above to a point where I get the actual cause, and I don't really understand why it only happens to Firebase and Appsee, every other pod libraries work fine.

[artwyman]

Dunno, but it sounds like the problem is not related to this issue, or to Djinni in particular. Assuming that the OAKLibUtility class isn't defined by any 3rd-party code, the problem is somewhere within your build system. In your position, I'd focus on trying to figure where the source for those symbols is compiled, then where the resulting libraries get linked, to make sure they're not linked in more than one place.

[moderateepheezy]

Thanks very much, I will try that.

[moderateepheezy]

Sorry, one more thing please, I am trying to do a trace of the issues, and all the 50 duplicates came from files generated by Djinni, OAKLibUtility and the likes. If the issue is not from Djnni, do I have to look into the C++ files?

[artwyman]

Depends on what symbols are duplicated, and how you build your code, so I don't think I can give specific guidance. All the symbols I see in your screenshots look like they're either ObjC (e.g. OBJC_CLASS$_OAKLibUtility) or ObjC++ (e.g. _Zn16djinni_generated4Test10fromCppOpt...). That means at least from that list you should be looking at the .m and .mm files not the .cpp files.

[moderateepheezy]

Yea, I have more of the .mm. below are two of the classes causing the duplicate, i tried changing the cppRef and _cppRefHandle of both classes since they are the only two common symbol, but still the same error.

OAKLIBTest

// AUTOGENERATED FILE - DO NOT MODIFY!
// This file generated by Djinni from application.djinni

#import "OAKLIBTest+Private.h"
#import "OAKLIBTest.h"
#import "DJICppWrapperCache+Private.h"
#import "DJIError.h"
#import "DJIMarshal+Private.h"
#include <exception>
#include <stdexcept>
#include <utility>

static_assert(__has_feature(objc_arc), "Djinni requires ARC to be enabled for this file");

@interface OAKLIBTest ()

- (id)initWithCpp:(const std::shared_ptr<::oaktree_gen::Test>&)cppRef;

@end

@implementation OAKLIBTest {
    ::djinni::CppProxyCache::Handle<std::shared_ptr<::oaktree_gen::Test>> _cppRefHandle;
}

- (id)initWithCpp:(const std::shared_ptr<::oaktree_gen::Test>&)cppRef
{
    if (self = [super init]) {
        _cppRefHandle.assign(cppRef);
    }
    return self;
}

- (nonnull NSString *)getName {
    try {
        auto objcpp_result_ = _cppRefHandle.get()->getName();
        return ::djinni::String::fromCpp(objcpp_result_);
    } DJINNI_TRANSLATE_EXCEPTIONS()
}

+ (nullable OAKLIBTest *)create {
    try {
        auto objcpp_result_ = ::oaktree_gen::Test::create();
        return ::djinni_generated::Test::fromCpp(objcpp_result_);
    } DJINNI_TRANSLATE_EXCEPTIONS()
}

namespace djinni_generated {

auto Test::toCpp(ObjcType objc) -> CppType
{
    if (!objc) {
        return nullptr;
    }
    return objc->_cppRefHandle.get();
}

auto Test::fromCppOpt(const CppOptType& cpp) -> ObjcType
{
    if (!cpp) {
        return nil;
    }
    return ::djinni::get_cpp_proxy<OAKLIBTest>(cpp);
}

}  // namespace djinni_generated

@end

OAKLIBUtility

// AUTOGENERATED FILE - DO NOT MODIFY!
// This file generated by Djinni from application.djinni

#import "OAKLIBUtility+Private.h"
#import "OAKLIBUtility.h"
#import "DJICppWrapperCache+Private.h"
#import "DJIError.h"
#import "DJIMarshal+Private.h"
#include <exception>
#include <stdexcept>
#include <utility>

static_assert(__has_feature(objc_arc), "Djinni requires ARC to be enabled for this file");

@interface OAKLIBUtility ()

- (id)initWithCpp:(const std::shared_ptr<::oaktree_gen::Utility>&)cppRef;

@end

@implementation OAKLIBUtility {
    ::djinni::CppProxyCache::Handle<std::shared_ptr<::oaktree_gen::Utility>> _cppRefHandle;
}

- (id)initWithCpp:(const std::shared_ptr<::oaktree_gen::Utility>&)cppRef
{
    if (self = [super init]) {
        _cppRefHandle.assign(cppRef);
    }
    return self;
}

+ (nonnull NSString *)toJsonString:(nonnull NSDictionary<NSString *, NSString *> *)param {
    try {
        auto objcpp_result_ = ::oaktree_gen::Utility::to_json_string(::djinni::Map<::djinni::String, ::djinni::String>::toCpp(param));
        return ::djinni::String::fromCpp(objcpp_result_);
    } DJINNI_TRANSLATE_EXCEPTIONS()
}

+ (nonnull NSString *)toQueryString:(nonnull NSDictionary<NSString *, NSString *> *)param {
    try {
        auto objcpp_result_ = ::oaktree_gen::Utility::to_query_string(::djinni::Map<::djinni::String, ::djinni::String>::toCpp(param));
        return ::djinni::String::fromCpp(objcpp_result_);
    } DJINNI_TRANSLATE_EXCEPTIONS()
}

namespace djinni_generated {

auto Utility::toCpp(ObjcType objc) -> CppType
{
    if (!objc) {
        return nullptr;
    }
    return objc->_cppRefHandle.get();
}

auto Utility::fromCppOpt(const CppOptType& cpp) -> ObjcType
{
    if (!cpp) {
        return nil;
    }
    return ::djinni::get_cpp_proxy<OAKLIBUtility>(cpp);
}

}  // namespace djinni_generated

@end

[artwyman]

OAKLibTest's _cppRefHandle and OAKLibUtility's _cppRefHandle aren't the same symbol. The duplication of OBJC_CLASS$_OAKLibUtility suggests to me that OAKLibUtility.m is being compiled more than once, or that it's getting compiled into a library which is linked more than once.