Support for multiple firestore clients
This pull request intends to resolve issue #86.
I have followed the discussion there and created what I believe to be a nice solution based on Option 3.
Primary things to note about this solution:
- The current way of configuring a client/db via the
configure()method remains in tact. - Support for multiple clients builds on top of that and relies on storing of a ConfigItem in the CONFIGURATIONS dict.
- Upon creation of a config, both a sync and async client will be set for that config
- Code examples introduced below are included in a
multiple_clients_examplesdirectory, and can be removed before merge. - I have added ~20 more unit tests, but there may be some gaps. Happy to add more with your suggestions.
Other small things:
- readme was updated with some more small details, and I recommend using pytest instead for development. It may be that I don't know poetry well enough to fix this, but I found that upon running
poetry run invoke testmy current changes were rolled back to the current firedantic version. I suppose this is because poetry is locked toversion = "0.11.0"in the pyproject.toml and so I used pytest instead. - Aside from testing support of multiple clients, some test cases were added to check more types of operands, i.e.
op.EQand"==". All work as expected. - Apologies for the added spacing everywhere. I believe one of the formatters from the poetry setup did that. If it bothers anyone i can go back and remove it.
Now for the examples...
Example of client creation:
## OLD WAY:
def configure_client():
# Firestore emulator must be running if using locally.
if environ.get("FIRESTORE_EMULATOR_HOST"):
client = Client(
project="firedantic-test",
credentials=Mock(spec=google.auth.credentials.Credentials),
)
else:
client = Client()
configure(client, prefix="firedantic-test-")
print(client)
def configure_async_client():
# Firestore emulator must be running if using locally.
if environ.get("FIRESTORE_EMULATOR_HOST"):
client = AsyncClient(
project="firedantic-test",
credentials=Mock(spec=google.auth.credentials.Credentials),
)
else:
client = AsyncClient()
configure(client, prefix="firedantic-test-")
print(client)
## NEW WAY:
def configure_multiple_clients():
config = Configuration()
# name = (default)
config.create(
prefix="firedantic-test-",
project="firedantic-test",
credentials=Mock(spec=google.auth.credentials.Credentials),
)
# name = billing
config.create(
name="billing",
prefix="test-billing-",
project="test-billing",
credentials=Mock(spec=google.auth.credentials.Credentials),
)
config.get_client() ## will pull the default client
config.get_client("billing") ## will pull the billing client
config.get_async_client("billing") ## will pull the billing async client
Under the hood of config.create():
- (see
configuration.pyfor more details) Created suggestedConfigurationclass for added support of multiple configurations/clients. - The general idea is to save each configuration to the CONFGURATIONS Dict. When using the singular
configuremethod, it will be saved there as(default). - New single clients can also be configured with the new multi client method. Multi is not required to use multi.
class Configuration:
def __init__(self):
self.configurations: Dict[str, ConfigItem] = {}
def create(
self,
name: str = "(default)",
prefix: str = "",
project: str = "",
credentials: Credentials = None,
) -> None:
self.configurations[name] = ConfigItem(
prefix=prefix,
project=project,
credentials=credentials,
client=Client(
project=project,
credentials=credentials,
),
async_client=AsyncClient(
project=project,
credentials=credentials,
),
)
# add to global CONFIGURATIONS
global CONFIGURATIONS
CONFIGURATIONS[name] = self.configurations[name]
For saving/finding entries:
## With single client
def old_way():
# Firestore emulator must be running if using locally.
configure_client()
# Now you can use the model to save it to Firestore
owner = Owner(first_name="John", last_name="Doe")
company = Company(company_id="1234567-8", owner=owner)
company.save()
# Prints out the firestore ID of the Company model
print(f"\nFirestore ID: {company.id}")
# Reloads model data from the database
company.reload()
## Now with multiple clients/dbs:
def new_way():
config = Configuration()
# 1. Create first config with config_name = "(default)"
config.create(
prefix="firedantic-test-",
project="firedantic-test",
credentials=Mock(spec=google.auth.credentials.Credentials),
)
# Now you can use the model to save it to Firestore
owner = Owner(first_name="Alice", last_name="Begone")
company = Company(company_id="1234567-9", owner=owner)
company.save() # will use 'default' as config name
# Reloads model data from the database
company.reload() # with no name supplied, config refers to "(default)"
# 2. Create the second config with config_name = "billing"
config_name = "billing"
config.create(
name=config_name,
prefix="test-billing-",
project="test-billing",
credentials=Mock(spec=google.auth.credentials.Credentials),
)
# Now you can use the model to save it to Firestore
account = BillingAccount(name="ABC Billing", billing_id="801048", owner="MFisher")
bc = BillingCompany(company_id="1234567-8", billing_account=account)
bc.save(config_name)
# Reloads model data from the database
bc.reload(config_name) # with config name supplied, config refers to "billing"
# 3. Finding data
# Can retrieve info from either database/client
# When config is not specified, it will default to '(default)' config
# The models do not know which config you intended to use them for, and they
# could be used for a multitude of configurations at once.
print(Company.find({"owner.first_name": "Alice"}))
print(BillingCompany.find({"company_id": "1234567-8"}, config=config_name))
print(
BillingCompany.find(
{"billing_account.billing_id": "801048"}, config=config_name
)
)
A simplified example for the find() method: Note here, the multi client method is using the (default) config, as no config was specified.
print("\n---- Running OLD way ----")
configure_client()
companies1 = Company.find({"owner.first_name": "John"})
companies2 = Company.find({"owner.first_name": {op.EQ: "John"}})
companies3 = Company.find({"owner.first_name": {"==": "John"}})
assert companies1 != []
assert companies1 == companies2 == companies3
print("\n---- Running NEW way ----")
configure_multiple_clients()
companies1 = Company.find({"owner.first_name": "Alice"})
companies2 = Company.find({"owner.first_name": {op.EQ: "Alice"}})
companies3 = Company.find({"owner.first_name": {"==": "Alice"}})
assert companies1 != []
assert companies1 == companies2 == companies3
A big thanks for the PR!
I'm really sorry to say we're pretty busy for the next month, so going to frankly say we're most likely not able to have a deeper look into the PR within the next month.
Hi @joakimnordling , Thanks so much for the detailed review and suggestions. I've been MIA on this since your message but I should have time next week to give it another go. Will keep you posted. M