Support for multiple data sources

[reddogaw]

Is your feature request related to a problem? Please describe. I would love to see the SDK provide a way to support multiple update processor factories.

The use case for this is that we'd like to have our local FileDataSource (that's used by developers on local machines) extend values from the default web based update processor.

We followed the guide at: https://docs.launchdarkly.com/docs/reading-flags-from-a-file to set up the file source however, we want to overlay that with the actual values from the shared Launch Darkly environment so that developers can feature toggle the specific features that they're working on at the specific point in time. Using the approach in that guide required that all feature flags be saved into the local file source.

Describe the solution you'd like We'd like to be able to support multiple update processor factories, with basic known rules for what happens when flags exist in multiple sources.

Suggested API might use an extension method like AndWithUpdateProcessorFactory to append the file based factory:

                var config = LaunchDarklyConfiguration.Default(launcDarklySettings.Key);
                if (launcDarklySettings.UseLocalConfig)
                {
                    if (!File.Exists(launcDarklySettings.LocalConfigFilePath))
                    {
                        throw new FileNotFoundException(
                            $"Local Launch Darkly config file was not found at {launcDarklySettings.LocalConfigFilePath}");
                    }

                    var fileSource = FileComponents.FileDataSource()
                        .WithFilePaths(launcDarklySettings.LocalConfigFilePath)
                        .WithAutoUpdate(true);

                    config = config
                        .WithUpdateProcessorFactory(Components.DefaultUpdateProcessor)
                        .AndWithUpdateProcessorFactory(fileSource);
                }

                return new LdClient(config);

[eli-darkly]

Well... I think I understand your desired use case, but the SDK architecture makes your proposed solution less straightforward than you may think.

The problem is that the update processor is not a component that LdClient requests flags from as needed; if it were, then it would be easy to implement a configuration like in your example so that if the first update processor did not have a value for the flag, we would ask the second one, and so on. Instead, it is a component that pushes flag values to the feature store. So, if both a FileDataSource and the DefaultUpdateProcessor are active, the latter could receive a flag from LaunchDarkly and put it into the feature store, overwriting a flag with the same key that had been previously put there by FileDataSource.

I think the only way to ensure that flags from source A would never be overwritten by flags from source B would be to create some kind of intermediate feature store component that wraps the real store—delegating updates to the real one, but with some way for source A to designate certain flags as write-protected—and point source B at that one. Maybe there's a different approach that I'm missing, but if not, that seems a bit elaborate.

We are planning to change the feature store architecture in several backward-incompatible ways in the next major version, so it might be more feasible to try to address this use case as part of that process, rather than trying to shoehorn it into the current architecture which is not a good fit.

[reddogaw]

I'll add my code below that uses an adapter pattern to support adding multiple processor factories like shown above...

However, one of the main stumbling blocks I found is that the Feature Stores were Init'd for each processor and the default InMemoryFeatureStore cleared all it's items when Init was called for the second time... So I had to hack around with merging multiple feature stores together.

using LaunchDarkly.Client;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using LaunchDarklyConfiguration = LaunchDarkly.Client.Configuration;

namespace Infrastructure.LaunchDarkly
{
    public static partial class ConfigurationExtensions
    {

        public static LaunchDarklyConfiguration AndWithUpdateProcessorFactory(this LaunchDarklyConfiguration configuration, IUpdateProcessorFactory factory)
        {
            var existing = configuration.UpdateProcessorFactory ?? Components.DefaultUpdateProcessor;
            var merged = new MergedUpdateProcessorFactory(existing).AndUpdateProcessorFactory(factory);
            return configuration
                .WithUpdateProcessorFactory(merged)
                .WithFeatureStoreFactory(new MergedFeatureStoreFactory())
                ;
        }

        /// <summary>
        /// Allows creation of multiple update processors
        /// </summary>
        private class MergedUpdateProcessorFactory : IUpdateProcessorFactory
        {
            private readonly List<IUpdateProcessorFactory> _factories;

            public MergedUpdateProcessorFactory(params IUpdateProcessorFactory[] factories)
            {
                _factories = new List<IUpdateProcessorFactory>(factories);
            }

            /// <summary>
            /// Creates a single update processor that adapts calls to each of the children
            /// </summary>
            /// <param name="config"></param>
            /// <param name="featureStore"></param>
            /// <returns></returns>
            public IUpdateProcessor CreateUpdateProcessor(LaunchDarklyConfiguration config, IFeatureStore featureStore)
            {
                if (featureStore.GetType().Name.Equals("FeatureStoreClientWrapper"))
                {
                    // NOTE: No way around this... They've wrapped it against my will and hidden the underlying feature store from us :-(
                    var storeFieldAccessor = featureStore.GetType().GetField("_store", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance);
                    if (storeFieldAccessor != null)
                    {
                        featureStore = ((IFeatureStore)storeFieldAccessor.GetValue(featureStore)) ?? featureStore;
                    }
                }

                // Use the multiple factories to generate each of the processors
                // However, initialize each of them using their own feature store (if configured this way)
                var processors = new List<IUpdateProcessor>(_factories.Count);
                foreach (var factory in _factories)
                {
                    var featureStoreToUse = featureStore;
                    if (featureStore is IMergedFeatureStore mergedFeatureStore)
                    {
                        featureStoreToUse = mergedFeatureStore.NewChildFeatureStore();
                    }

                    var processor = factory.CreateUpdateProcessor(config, featureStoreToUse);
                    processors.Add(processor);
                }
                return new MergedUpdateProcessorAdapter(processors.ToArray());
            }

            /// <summary>
            /// Append the update processor factory to our list of adapted factories
            /// </summary>
            /// <remarks>
            /// Note that in terms of features, the last added factory's processor will provide the values for duplicated items
            /// </remarks>
            /// <param name="factory"></param>
            /// <returns></returns>
            public MergedUpdateProcessorFactory AndUpdateProcessorFactory(IUpdateProcessorFactory factory)
            {
                _factories.Add(factory);
                return this;
            }

            /// <summary>
            /// Acts as an adapter for echo-ing calls to multiple child processors
            /// </summary>
            private class MergedUpdateProcessorAdapter : IUpdateProcessor
            {
                private readonly List<IUpdateProcessor> _processors;

                public MergedUpdateProcessorAdapter(params IUpdateProcessor[] processors)
                {
                    _processors = new List<IUpdateProcessor>(processors);
                }

                public void Dispose()
                {
                    _processors.ForEach(p => p.Dispose());
                    _processors.Clear();
                }

                public bool Initialized()
                {
                    return _processors.All(p => p.Initialized());
                }

                public async Task<bool> Start()
                {
                    var results = await Task.WhenAll(_processors.Select(p => p.Start()).ToArray());
                    return results.All(s => s);
                }
            }
        }

        /// <summary>
        /// Indicates that this feature store coordinates state for multiple child feature stores
        /// </summary>
        private interface IMergedFeatureStore : IFeatureStore
        {
            IFeatureStore[] Children { get; }

            IFeatureStore NewChildFeatureStore();
        }

        /// <summary>
        /// Generates merged feature store items
        /// </summary>
        private class MergedFeatureStoreFactory : IFeatureStoreFactory
        {
            public IFeatureStore CreateFeatureStore()
            {
                return new MergedFeatureStoreAdapter();
            }

            /// <summary>
            /// Adapts multiple feature stores to provide logic for combining common state
            /// </summary>
            /// <remarks>
            /// The main logic below means that for duplicate keys, features from the **last** store win
            /// </remarks>
            private class MergedFeatureStoreAdapter : IMergedFeatureStore, IFeatureStore
            {
                private readonly List<IFeatureStore> _stores;

                public MergedFeatureStoreAdapter()
                {
                    _stores = new List<IFeatureStore>();
                }

                public IFeatureStore NewChildFeatureStore()
                {
                    var nextStore = new InMemoryFeatureStore();
                    _stores.Add(nextStore);
                    return nextStore;
                }

                public IFeatureStore[] Children
                {
                    get => _stores.ToArray();
                }

                public void Delete<T>(VersionedDataKind<T> kind, string key, int version) where T : IVersionedData
                {
                    _stores.ForEach(s => s.Delete<T>(kind, key, version));
                }

                public void Dispose()
                {
                    _stores.ForEach(s => s.Dispose());
                    _stores.Clear();
                }

                public void Init(IDictionary<IVersionedDataKind, IDictionary<string, IVersionedData>> allData)
                {
                    _stores
                        .Where(s => !s.Initialized())
                        .ToList()
                        .ForEach(s => s.Init(allData));
                }

                public bool Initialized()
                {
                    return _stores.Any() && _stores.TrueForAll(s => s.Initialized());
                }

                public void Upsert<T>(VersionedDataKind<T> kind, T item) where T : IVersionedData
                {
                    _stores.ForEach(s => s.Upsert<T>(kind, item));
                }

                IDictionary<string, T> IFeatureStore.All<T>(VersionedDataKind<T> kind)
                {
                    var results = _stores.Select(s => s.All<T>(kind)).ToArray();

                    // Merge values, but for dupes the last feature value wins
                    return results
                        .SelectMany(dict => dict)
                        .ToLookup(pair => pair.Key, pair => pair.Value)
                        .ToDictionary(group => group.Key, group => group.Last());
                }

                T IFeatureStore.Get<T>(VersionedDataKind<T> kind, string key)
                {
                    return _stores.Select(s => s.Get<T>(kind, key)).LastOrDefault();
                }
            }
        }
    }
}

[reddogaw]

⬆️ A few nasty hacks (like the field reflection for the FeatureStoreClientWrapper) but hopefully it kind of makes sense or at least illustrates a workaround path for the SDK architecture issues that you've mentioned @eli-darkly

[eli-darkly]

Well, as I said, that's a bit elaborate. Your code is more or less what I was imagining (although I'm not sure why you have implemented Init, Upsert, and Delete for the merged store - no one is going to be calling those methods as far as I can tell - and I don't see what the Children getter is used for), but I'm not sure this is a direction we want to go in, until we have an architecture that's more suited to it. It's clever, but as you say, it's hacky; building something like that into the SDK means that we have to maintain it, adding further constraints to what we can change elsewhere in the SDK... all to support a very specific test configuration that we haven't had any other requests for.

If it works for you, great; I don't see any reason why you couldn't use it just like this, without it having to be built into the SDK.

[reddogaw]

Totally agree that the code above is unsuitable for your library which is why I couldn't make a PR for it. In our deploy we'll only be utilizing it on debug builds.

I guess the original feature request of being able support values fed from multiple sources would still be ideal (maybe in a similar manner to how .NET Core Configuration providers work) if you'd be willing to keep it open.

I would imagine such a feature would provide value to LaunchDarkly by enabling large teams of developers to override feature flags for only their own actively developing/exploring code changes.

[eli-darkly]

I'll definitely add it to a list of feature requests for the upcoming rewrite.