Implement Unified Cache Layer

[dask-58]

Goal

Implement a unified caching layer under pkg/cache that is:

• Simple to use
• Production-ready
• Configurable and extensible
• Integrated with observability tooling

---

Deliverables

• Core Cache Interface
• In-Memory Backend
• Redis Backend
• Observability Wrapper
  • Logging: Log each operation and failure with context.
  • Tracing: Wrap backend calls in OpenTelemetry spans.
  • Metrics: Emit Prometheus metrics for operations, latencies, and errors.
• Configuration Support Support runtime configuration and environment-based backend switching.

[dask-58]

@coolwednesday Can you assign this to me? I’m already working on it.

[coolwednesday]

Sure! Do update the PR description with the architecture you wish to follow.