feat: Comprehensive storage provider system (S3-compatible, FTP/SFTP)

[gabelul]

Summary

Extends the storage system to support any S3-compatible provider and traditional file transfer protocols, providing users with maximum flexibility in choosing their storage backend.

Features Added

✅ Generic S3-compatible storage provider

• Works with AWS S3, MinIO, DigitalOcean Spaces, Backblaze B2, Wasabi, and any S3-compatible service
• Configurable public URLs with support for CDNs and custom domains
• Signed URL generation for private buckets
• Path-style and virtual-hosted-style URL support
• Enhanced Cloudflare R2 compatibility

✅ FTP storage provider

• Support for both FTP and FTPS (FTP over SSL/TLS)
• Configurable passive/active modes
• Connection pooling and timeout handling
• Date-based directory organization

✅ SFTP storage provider

• SSH key and password authentication support
• Connection keepalive and timeout management
• Secure file transfer over SSH

✅ Enhanced upload factory

• Comprehensive environment variable validation
• Clear error messages for missing configuration
• Support for all providers: local, cloudflare, s3-compatible, ftp, sftp

✅ Comprehensive documentation

• Detailed .env.example with examples for all providers
• Clear separation between upload credentials and public URLs
• Provider-specific configuration options

Benefits over existing solutions

Compared to PR #873:

• Broader compatibility: Supports ANY S3-compatible service, not just S3/MinIO
• Flexible URL configuration: Configurable public URLs essential for CDNs
• Additional protocols: FTP/SFTP support for legacy systems
• Production-ready: Better error handling, connection management, and retries
• Better architecture: Single s3-compatible provider handles all S3-compatible services

Real-world use cases:

• Self-hosters: Can use MinIO, local FTP servers, or any cloud provider
• Enterprise: SFTP integration with existing file servers
• Cost optimization: Use cheaper S3-compatible providers like Wasabi or Backblaze
• Geographic distribution: Use regional providers like DigitalOcean Spaces
• CDN integration: Proper public URL configuration for CloudFront, CloudFlare, etc.

Configuration Examples

AWS S3

STORAGE_PROVIDER="s3-compatible"
S3_COMPATIBLE_ACCESS_KEY="AKIA..."
S3_COMPATIBLE_SECRET_KEY="..."
S3_COMPATIBLE_REGION="us-east-1"
S3_COMPATIBLE_BUCKET="my-bucket"

MinIO

STORAGE_PROVIDER="s3-compatible"
S3_COMPATIBLE_ENDPOINT="https://minio.example.com"
S3_COMPATIBLE_PATH_STYLE=true
S3_COMPATIBLE_PUBLIC_URL="https://cdn.example.com"

DigitalOcean Spaces

STORAGE_PROVIDER="s3-compatible"
S3_COMPATIBLE_ENDPOINT="https://nyc3.digitaloceanspaces.com"
S3_COMPATIBLE_PUBLIC_URL="https://my-space.nyc3.cdn.digitaloceanspaces.com"

FTP/SFTP

STORAGE_PROVIDER="ftp"  # or "sftp"
FTP_HOST="ftp.example.com"
FTP_USER="username"
FTP_PASSWORD="password"
FTP_REMOTE_PATH="/public_html/uploads"
FTP_PUBLIC_URL="https://example.com/uploads"

Closes

• Fixes #322
• Supersedes #873

Testing

• All storage providers implement the same IUploadProvider interface
• Comprehensive error handling and validation
• Environment variable validation with clear error messages
• Backward compatibility maintained with existing local and cloudflare providers

This implementation provides a production-ready, flexible storage system that addresses the community's need for diverse storage options while maintaining code quality and usability.

[vercel[bot]]

Someone is attempting to deploy a commit to the Listinai Team on Vercel.

A member of the Team first needs to authorize it.

[egelhaus]

@nevo-david This looks a lot like AI

[gabelul]

@nevo-david This looks a lot like AI

how so?

[r1me75]

We need this merged. Idc if it is AI or not if it works?

Any updates on this @nevo-david