Implement ecosystem CI for backward compatibility testing

[dahlia]

Description

Implement an ecosystem CI system similar to Vite's ecosystem-ci to test backward compatibility of new Fedify versions against real-world projects in the Fedify ecosystem. This will help catch breaking changes before they affect production applications and ensure the stability of the Fedify ecosystem.

Background

As Fedify grows and more projects adopt it in production, it becomes increasingly important to ensure that new releases don't break existing applications. An ecosystem CI would automatically test new Fedify versions against known projects using Fedify, providing early warning about potential compatibility issues.

Implementation Approach

Phase 1: Internal Examples Testing

Extend the existing CI to test examples against different Fedify versions:

• Current examples: blog, express, hono-sample, next14-app-router, next15-app-router, cloudflare-workers
• Test matrix: Test examples against current, pre-release, and previous Fedify versions
• Failure reporting: Alert on breaking changes with detailed logs

Phase 2: Production Projects Integration

Include real-world Fedify projects identified in the Success Stories:

High Priority Targets

• Ghost ActivityPub - https://github.com/TryGhost/ActivityPub
  • Major publishing platform used by thousands
  • Active contributor to Fedify ecosystem
• Hollo - https://github.com/fedify-dev/hollo
  • Created by Fedify's original developer
  • Serves as showcase and test case
• Encyclia - https://encyclia.pub/
  • Academic research bridge to fediverse
  • Specialized use case validation
• Hackers' Pub - https://hackers.pub/
  • Community-driven project

Medium Priority Targets

• Fedified Microblog - https://github.com/fedify-dev/microblog
• Fedi Badge - https://github.com/dahlia/fedi-badge
• Other community projects as they emerge

Phase 3: Automated Ecosystem Management

• Project discovery: Automatically discover new projects using Fedify
• Opt-in system: Allow projects to register for ecosystem testing
• Community integration: Integrate with fedify-dev organization projects

Technical Implementation

Infrastructure Components

1. Test Runner

   • Scheduled CI jobs (daily/weekly)
   • On-demand testing for release candidates
   • Matrix testing across Node.js/Deno/Bun environments

2. Project Management

   • Configuration files defining test projects
   • Version compatibility matrix
   • Test commands and success criteria for each project

3. Reporting System

   • Dashboard showing ecosystem health
   • Automated issue creation for failures
   • Integration with release process

4. Integration Points

   • Pre-release testing pipeline
   • Release candidate validation
   • Community notification system

Example Configuration Structure

# ecosystem-projects.yml
projects:
  - name: "ghost-activitypub"
    repo: "TryGhost/ActivityPub"
    test_command: "npm test"
    build_command: "npm run build"
    environments: ["node-18", "node-20"]
    
  - name: "hollo"
    repo: "fedify-dev/hollo"
    test_command: "deno task test"
    build_command: "deno task build"
    environments: ["deno"]

Benefits

1. Early Detection: Catch breaking changes before they reach production
2. Ecosystem Stability: Ensure the entire Fedify ecosystem remains healthy
3. Release Confidence: Provide confidence in new releases through real-world testing
4. Community Trust: Demonstrate commitment to backward compatibility
5. Regression Prevention: Prevent regressions that could affect critical projects

Success Criteria

• [ ] Set up basic ecosystem CI infrastructure
• [ ] Include all internal examples in ecosystem testing
• [ ] Integrate at least 2-3 production projects (Ghost, Hollo)
• [ ] Automated testing on release candidates
• [ ] Dashboard or reporting system for ecosystem health
• [ ] Documentation for projects wanting to join ecosystem CI
• [ ] Integration with existing release process

Implementation Considerations

1. Performance: Ecosystem CI should not significantly slow down development
2. Maintenance: Automated project discovery and management where possible
3. Privacy: Respect project privacy and opt-in requirements
4. Resources: Efficient use of CI resources and parallelization
5. Communication: Clear communication channels for ecosystem issues

References

• Vite Ecosystem CI - Similar system for inspiration
• Fedify Success Stories - Projects to include
• Current CI Configuration - Existing testing infrastructure

Future Enhancements

• Performance benchmarking across ecosystem projects
• Security scanning for ecosystem projects
• Documentation generation from ecosystem testing
• Community metrics and ecosystem health reporting