With claude code, our team can add async support to this lovely mongoengine.

Our team is testing in our application(fastAPI).

---

Comprehensive Async Support for MongoEngine

This PR adds full asynchronous support to MongoEngine, enabling high-performance async/await operations while maintaining 100% backward compatibility with existing synchronous code.

🎯 Overview

MongoEngine now supports comprehensive async operations using PyMongo's AsyncMongoClient. This implementation allows developers to build scalable applications with async/await syntax without requiring any changes to existing document models.

🚀 Key Features

Core Async Operations

• Document Operations: async_save(), async_delete(), async_reload(), async_ensure_indexes()
• QuerySet Operations: async_get(), async_first(), async_count(), async_create(), async_to_list()
• Bulk Operations: async_update(), async_update_one(), async_delete()
• Async Iteration: Full support for async for with QuerySets

Advanced Features

• Reference Fields: Async dereferencing with AsyncReferenceProxy and async_fetch()
• GridFS: Complete async file operations (async_put(), async_get(), async_read(), async_delete())
• Transactions: async_run_in_transaction() context manager
• Context Managers: async_switch_db(), async_switch_collection(), async_no_dereference()
• Aggregation: async_aggregate(), async_distinct()
• Cascade Operations: Full support for all deletion rules (CASCADE, NULLIFY, etc.)

🔧 Technical Implementation

Connection Management

• New connect_async() function with proper AsyncMongoClient handling
• Automatic connection type detection and enforcement
• Clear error messages for wrong connection usage
• PyMongo 4.13+ dependency management with graceful fallback

Backward Compatibility

• 100% Compatible: All existing sync code works unchanged
• No Model Changes: Document classes require no modifications
• Mixed Usage: Sync and async connections can coexist in the same application
• Clear Separation: Async methods use async_ prefix pattern

Error Handling

• Runtime errors prevent mixing sync/async methods with wrong connections
• Helpful error messages guide users to correct usage
• Proper ImportError handling for PyMongo version compatibility

📊 Test Coverage

• 79+ async-specific tests covering all features
• Complete test suite passes (1000+ existing tests remain functional)
• Integration tests for complex async workflows
• Error handling tests for connection type validation

📚 Documentation

Updated Documentation

• README.rst: Comprehensive async examples and installation guide
• docs/guide/async-support.rst: 500+ line detailed usage guide
• Migration examples: Step-by-step sync-to-async conversion
• Performance tips and best practices

Installation Support

# Install with async support
pip install mongoengine[async]

🔄 Intentionally Deferred Features

The following features were strategically deferred to maintain focus and ensure core stability:

• async_values(), async_values_list(): Low usage frequency, can use aggregation workaround
• async_explain(): Debugging feature, PyMongo direct access available
• Hybrid Signal System: Complex implementation, consider as separate enhancement
• ListField ReferenceField Auto-conversion: Requires deep structural changes

💻 Usage Examples

Basic Operations

import asyncio
from mongoengine import Document, StringField, connect_async

class User(Document):
    name = StringField(required=True)
    email = StringField(required=True)

async def main():
    # Connect asynchronously
    await connect_async('mydatabase')
    
    # Document operations
    user = User(name="John", email="[email protected]")
    await user.async_save()
    
    # QuerySet operations  
    users = await User.objects.filter(name="John").async_to_list()
    count = await User.objects.async_count()
    
    # Async iteration
    async for user in User.objects.filter(active=True):
        print(f"User: {user.name}")

asyncio.run(main())

Advanced Features

# Transactions
async with async_run_in_transaction():
    await user1.async_save()
    await user2.async_save()

# Context managers
async with async_switch_db(User, 'analytics_db'):
    await analytics_user.async_save()

# GridFS
await MyDoc.file.async_put(file_data, instance=doc)
content = await MyDoc.file.async_read(doc)

🏗️ Development Process

This implementation was developed through a systematic 4-phase approach:

1. Phase 1: Async connection foundation and basic document operations
2. Phase 2: QuerySet async methods and iteration support
3. Phase 3: Reference fields, GridFS, and complex field types
4. Phase 4: Advanced features (transactions, context managers, aggregation)

Each phase included comprehensive testing and documentation updates.

✅ Quality Assurance

• All pre-commit hooks pass: black, flake8, isort, pyupgrade
• CONTRIBUTING.rst compliance: Follows all project guidelines
• Dependency management: Proper PyMongo version handling
• Python 3.9-3.13 support: Aligned with upstream version policy

🔮 Future Compatibility

This implementation is designed for long-term stability:

• Follows PyMongo async patterns and best practices
• Maintains MongoEngine's existing architecture and conventions
• Extensible design allows for future enhancements
• Clean separation enables independent sync/async development

I will leave our live test results later. We are also not sure if it is safe to use and if it is worth to use.

We found some issues and are working on.

We fully migrated and deployed our application project. There were some issues and we fixed.

I only skimmed the PR, but I am not sure how I feel about AI writing an async version. I made a couple comments about what I would expect might need to be applied across the whole change-set (this is my personal opinion and is not exhaustive)

@terencehonles

First of all, it’s truly an honor to receive a reply from the maintainer of what I consider to be one of the best libraries out there. I know how great this project is and genuinely hope it continues to thrive, which is why I completely understand your concerns about parts of the PR that were written with the help of AI.

That said, our team has been using these async changes in production with FastAPI and has consistently seen performance improvements of over 30–40% on average. Based on that experience, we also felt that this feature deserves to be redesigned in a more elegant and robust way — not just to work, but to feel native to the spirit of the library.

I’d love to hear more of your thoughts. In the meantime, here are some of the directions we’ve been considering based on our usage:

• Separate AsyncBaseQuerySet: Rather than duplicating methods with an async_ prefix, we think it would be cleaner to define an AsyncBaseQuerySet class, using the same method names where possible. This separation helps avoid confusion and aligns with async conventions.
• LazyReference and Reference behavior: This part gets tricky. Since the original behavior was sync, accessing references as properties worked fine. But in async usage, if they haven’t been prefetched (via something like select_related or dereference), requiring await on property access feels awkward and unintuitive.
• Dual connection support: We’re also exploring whether it would make sense to support both sync and async connections side-by-side and allow developers to choose or even mix usage depending on context.

Let me know your thoughts!

Thanks for your reply, I to be fair I am a contributor and not the maintainer, so that's why I suggested these are my personal opinions. I understand that the API was definitely not written in a way that expects async usage, and there are certain places this might be problematic or harder to migrate.

For your first suggestion I think it depends what route you go with the third suggestion. If you want the same names then you'll need two separate connections, but if you want to mix things you need to have a prefix in order to call the right implementation. For references, they could be dereferenced eagerly on load, or lazily via an async dereference method / or an async property. This design decision would need to be made with a wide enough audience (or at least sufficient maintainer agreement)