Tracking issue: Refactor services to use the `backend.rs` and `core.rs` convention

[erickguan]

Problem Statement

OpenDAL implements services with the following convention:

• backend.rs
  • Builder: Builds a service's config. For example, see S3's builder. Refer to the Configurator too.
  • Backend: References to the core module and implements the fundamental Access trait.
• core.rs: Contains code that directly interacts with cloud services, databases, and platforms.
• other code modules

Currently, some services only implement backend.rs (a lack of core.rs), primarily for historical reasons. The recommended approach is to refactor these services so that each has both backend.rs and core.rs. This new convention makes the code base easier to maintain, read, and contribute to, thereby benefiting OpenDAL’s overall development.

Tasks

• [x] FTP #5949
• [x] GridFS #5966
• [ ] HDFS @uruemu
• [ ] HDFS native
• [x] HTTP https://github.com/apache/opendal/pull/5879
• [x] IPFS #5950
• [x] IPFS Mutable File System (IPMFS) #5980
• [x] OneDrive https://github.com/apache/opendal/pull/5733
• [x] SFTP https://github.com/apache/opendal/pull/5902
• [x] VercelArtifacts https://github.com/apache/opendal/pull/5873
• [x] Web HDFS https://github.com/apache/opendal/pull/5893

If you’re interested in tackling one of these services, feel free to comment on this issue and we’ll help get you started!

[uruemu]

Interested in picking up one of these. FTP?

Would appreciate the help to get setup.

[erickguan]

@uruemu Great to hear—and thanks for your interest! This task should be fairly straightforward, requiring only a bit of familiarity with OpenDAL and some basic Rust concepts.

If you’d like to start exploring the code early, here’s what the work involves:

• Create a new module core/src/services/ftp/core.rs with a struct FtpCore. Here's a rough example:

  // It’s up to you to decide what belongs in `FtpCore`.
  struct FtpCore {
      endpoint: String,
      user: String,
      password: String,
      enable_secure: bool,
      pool: OnceCell<bb8::Pool<Manager>>,
  }
  

• Keep FtpBuilder and FtpConfig in backend.rs. Inside Builder::build, you’ll construct the FtpCore.

• Use an Arc<FtpCore> to share state across structs like FtpBackend and FtpReader.

• OpenDAL requires the FTP service to implement the Access trait. The logic for this should go into core.rs, as it's considered internal implementation detail.

• For example, functions like ftp_connect and ftp_stat can be moved to core.rs.

• As you go, you’ll iterate on what should live in core.rs versus what stays in backend.rs.

• Use the helper method parse_error in ftp/err.rs to handle error propagation.

• Refer to other services (e.g., sftp, fs) to see how helper functions and logic are structured.

• To test FTP behavior with OpenDAL, run:

  OPENDAL_TEST=ftp cargo test behavior::test \
    --features tests,services-ftp \
    -- \
    --show-output
  

  You’ll need to set the following environment variables:

  • OPENDAL_FTP_ENDPOINT=ftp://<endpoint>
  • OPENDAL_FTP_ROOT=/path/to/dir
  • OPENDAL_FTP_USER=<user>
  • OPENDAL_FTP_PASSWORD=<password>

  You can run an FTP server locally, for example, using a Docker container.

[uruemu]

I can take up the GridFS and HDFS next