gofr-dev
File System Implementation Refactoring
File System Implementation Refactoring
Background
Current file system implementations (S3, GCS, SFTP) contain significant code duplication and inconsistent metrics.
Required Changes
-
Create a unified metrics system
- Move metrics definitions to the main file package
- Add storage_type label to differentiate providers (AWS/GCP/FTP)
-
Separate common code from provider-specific code
- Common file operations → main file package
- Storage/retrieval specifics → provider packages
-
Standardize shared components
- Create a single FileLog implementation in the main package
- Move common constants (SUCCESS, ERROR) to the main package
- Define standard histogram buckets in one place
- Implement a CloudStorageClient interface for S3/GCS
Benefits
- Reduced code duplication
- Consistent metrics across all storage providers
- Simplified implementation of new providers
- Better visibility for application developers
File System Refactoring Update
I have made some progress regrading this in PR #2435 and currently updated only GCS implementation to follow the same. Once merged we can focus on other providers to first match the GCS pattern for common logger and metrics functionality and then we can go ahead with larger refactorings.
####Completed
- Extracted common observability components to main file package.
- Standardized logging and metrics across providers.
- Unified constants (StatusSuccess, StatusError) and histogram buckets
- Updated GCS implementation to use these components
Future Work (Post-Merge)
- Create CloudStorageClient interface for all providers
- Separate common operations from provider-specific storage logic
- Implement version key/locking for consistent remote writes
These changes have significantly reduced code duplication while maintaining compatibility with existing implementations. The remaining work will be addressed in a dedicated follow-up PR to complete the architecture improvements.
Hi @Umang01-hash, I’d love to contribute to this issue if it’s available or if you’d like help with any part of it. I have experience in Go and refactoring file system–related components. Could you please assign me or guide me on how I can help?
