java: Add spring-boot-starter-opendal

[Xuanwo]

Spring is a crucial framework in the Java ecosystem. It would be beneficial for OpenDAL to develop a spring-boot-starter-opendal, enabling Java users to seamlessly integrate OpenDAL immediately.

Notes

• We can place spring-boot-starter-opendal under integrations/spring-boot.

[hezhangjian]

I can deep in that next week.

[Xuanwo]

I can deep in that next week.

Thanks a lot!

[hezhangjian]

@Xuanwo Sorry for my late, recently I was kind of busy. But I still want to work on this :). I think in the short term, we will not be putting the spring starter opendal into the Spring repository. Therefore, we are facing the issue of maintaining OpenDAL with multiple versions of Spring in the OpenDAL repository. I suggest the following strategy:

opendal-spring-xx will adapt to the corresponding Spring Boot version. For example, opendal-33 will adapt to Spring Boot 3.3:

<dependency>
  <groupId>org.apache.opendal</groupId>
  <artifactId>opendal-spring-33</artifactId>
  <version>${latest.version}</version>
</dependency>

<dependency>
  <groupId>org.apache.opendal</groupId>
  <artifactId>opendal-spring-27</artifactId>
  <version>${latest.version}</version>
</dependency>

opendal-spring-boot-starter will by default depend on the adapted latest version of Spring Boot. If you need to run it in a lower version of a Spring Boot project, you can replace it in the following way:

<dependency>
  <groupId>org.apache.opendal</groupId>
  <artifactId>opendal-spring-boot-starter</artifactId>
  <version>${latest.version}</version>
  <exclusions>
      <exclusion>
          <groupId>org.apache.opendal</groupId>
          <artifactId>opendal-spring-33</artifactId>
      </exclusion>
  </exclusions>
</dependency>
<dependency>
  <groupId>org.apache.opendal</groupId>
  <artifactId>opendal-spring-27</artifactId>
  <version>${latest.version}</version>
</dependency>

[Xuanwo]

Hi @shoothzj, thanks for your efforts. However, I feel targeting multiple Spring versions might be too complex for a starter proejct. I'm concerned about its future maintainability. Perhaps we could focus on the latest version and provide guidance for older versions?

[hezhangjian]

@Xuanwo I agree, we can focus on 3.3.x version.

[hezhangjian]

@Xuanwo, SpringBoot now supports two types of starters: reactive for asynchronous operations and non-reactive for synchronous operations. Let's look at a Redis example:

• spring-data-redis contains the core logic.
• spring-boot-starter-redis includes auto-configuration for Spring WebMVC (synchronous).
• spring-boot-starter-redis-reactive includes auto-configuration for Spring WebFlux (asynchronous).

If a project is popular, the spring-boot-autoconfigure repository will include auto-configuration in their main repo, and spring-boot-starter-redis and spring-boot-starter-redis-reactive can be empty or non-existent.

I think we should first maintain these three sub-projects with the latest SpringBoot version in our repo.

Regarding the artifact ID, I prefer using "opendal" as the suffix, which is common in the Spring ecosystem. What do you think?

Opendal as Prefix

• opendal-spring
• opendal-spring-starter
• opendal-spring-starter-reactor

Opendal as Suffix

• spring-opendal
• spring-starter-opendal
• spring-starter-opendal-reactor

[Xuanwo]

I'm happy to continue using the common behavior in the ecosystem. spring-opendal looks fine to me.

[hezhangjian]

@Xuanwo I put some sample codes in https://gitcode.com/shoothzj/opendal-demo/overview. And here is some docs, would you please take a look?

Usage

Configuration

First, configure the application.yml file with the necessary OpenDAL settings:

spring:
  opendal:
    schema: "fs"
    conf:
      root: "/tmp"

Code Integration

Features

• OpenDAL/OpenDALReactive Bean: Easily operate on data using OpenDAL/OpenDALReactive.
• Auto Serialize/Deserialize Data: Leverage Jackson for automatic serialization and deserialization.

I plan to implement Auto Serialize/Deserialize Data in the next.

Implementation

Spring WebMVC

To use OpenDAL with Spring WebMVC:

1. Configure the application.yml file as shown above.
2. Autowire the OpenDAL bean in your controller.

import org.apache.opendal.Entry;
import org.apache.opendal.Metadata;
import org.apache.opendal.PresignedRequest;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.*;

import java.time.Duration;
import java.util.List;

@RestController
@RequestMapping("/api/v1/opendal")
public class OpenDALController {

    @Autowired
    private OpenDAL openDAL;

    @PostMapping("/write")
    public ResponseEntity<Void> write(@RequestParam String path, @RequestBody String content) {
        openDAL.write(path, content);
        return ResponseEntity.ok().build();
    }

    @PostMapping("/writeBytes")
    public ResponseEntity<Void> writeBytes(@RequestParam String path, @RequestBody byte[] content) {
        openDAL.write(path, content);
        return ResponseEntity.ok().build();
    }

    @PostMapping("/append")
    public ResponseEntity<Void> append(@RequestParam String path, @RequestBody String content) {
        openDAL.append(path, content);
        return ResponseEntity.ok().build();
    }

    @PostMapping("/appendBytes")
    public ResponseEntity<Void> appendBytes(@RequestParam String path, @RequestBody byte[] content) {
        openDAL.append(path, content);
        return ResponseEntity.ok().build();
    }

    @GetMapping("/stat")
    public ResponseEntity<Metadata> stat(@RequestParam String path) {
        Metadata metadata = openDAL.stat(path);
        return ResponseEntity.ok(metadata);
    }

    @GetMapping("/read")
    public ResponseEntity<byte[]> read(@RequestParam String path) {
        byte[] content = openDAL.read(path);
        return ResponseEntity.ok(content);
    }

    @GetMapping("/presignRead")
    public ResponseEntity<PresignedRequest> presignRead(@RequestParam String path, @RequestParam long duration) {
        PresignedRequest request = openDAL.presignRead(path, Duration.ofSeconds(duration));
        return ResponseEntity.ok(request);
    }

    @GetMapping("/presignWrite")
    public ResponseEntity<PresignedRequest> presignWrite(@RequestParam String path, @RequestParam long duration) {
        PresignedRequest request = openDAL.presignWrite(path, Duration.ofSeconds(duration));
        return ResponseEntity.ok(request);
    }

    @GetMapping("/presignStat")
    public ResponseEntity<PresignedRequest> presignStat(@RequestParam String path, @RequestParam long duration) {
        PresignedRequest request = openDAL.presignStat(path, Duration.ofSeconds(duration));
        return ResponseEntity.ok(request);
    }

    @DeleteMapping("/delete")
    public ResponseEntity<Void> delete(@RequestParam String path) {
        openDAL.delete(path);
        return ResponseEntity.ok().build();
    }

    @PostMapping("/createDir")
    public ResponseEntity<Void> createDir(@RequestParam String path) {
        openDAL.createDir(path);
        return ResponseEntity.ok().build();
    }

    @PostMapping("/copy")
    public ResponseEntity<Void> copy(@RequestParam String sourcePath, @RequestParam String targetPath) {
        openDAL.copy(sourcePath, targetPath);
        return ResponseEntity.ok().build();
    }

    @PostMapping("/rename")
    public ResponseEntity<Void> rename(@RequestParam String sourcePath, @RequestParam String targetPath) {
        openDAL.rename(sourcePath, targetPath);
        return ResponseEntity.ok().build();
    }

    @DeleteMapping("/removeAll")
    public ResponseEntity<Void> removeAll(@RequestParam String path) {
        openDAL.removeAll(path);
        return ResponseEntity.ok().build();
    }

    @GetMapping("/list")
    public ResponseEntity<List<Entry>> list(@RequestParam String path) {
        List<Entry> entries = openDAL.list(path);
        return ResponseEntity.ok(entries);
    }
}

Spring WebFlux

To use OpenDALReactive with Spring WebFlux:

1. Configure the application.yml file as shown above.
2. Autowire the OpenDALReactive bean in your controller.

import com.shoothzj.opendal.spring.OpenDALReactive;
import org.apache.opendal.Entry;
import org.apache.opendal.Metadata;
import org.apache.opendal.PresignedRequest;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.*;
import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;

import java.time.Duration;

@RestController
@RequestMapping("/api/v1/opendal/reactive")
public class OpenDALReactiveController {

    @Autowired
    private OpenDALReactive<byte[]> openDALReactive;

    @PostMapping("/write")
    public Mono<ResponseEntity<Void>> write(@RequestParam("path") String path, @RequestBody String content) {
        return openDALReactive.write(path, content).then(Mono.just(ResponseEntity.ok().<Void>build()));
    }

    @PostMapping("/writeBytes")
    public Mono<ResponseEntity<Void>> writeBytes(@RequestParam("path") String path, @RequestBody byte[] content) {
        return openDALReactive.write(path, content).then(Mono.just(ResponseEntity.ok().<Void>build()));
    }

    @PostMapping("/append")
    public Mono<ResponseEntity<Void>> append(@RequestParam("path") String path, @RequestBody String content) {
        return openDALReactive.append(path, content).then(Mono.just(ResponseEntity.ok().<Void>build()));
    }

    @PostMapping("/appendBytes")
    public Mono<ResponseEntity<Void>> appendBytes(@RequestParam("path") String path, @RequestBody byte[] content) {
        return openDALReactive.append(path, content).then(Mono.just(ResponseEntity.ok().<Void>build()));
    }

    @GetMapping("/stat")
    public Mono<ResponseEntity<Metadata>> stat(@RequestParam("path") String path) {
        return openDALReactive.stat(path).map(ResponseEntity::ok);
    }

    @GetMapping("/read")
    public Mono<ResponseEntity<byte[]>> read(@RequestParam("path") String path) {
        return openDALReactive.read(path).map(ResponseEntity::ok);
    }

    @GetMapping("/presignRead")
    public Mono<ResponseEntity<PresignedRequest>> presignRead(@RequestParam("path") String path, @RequestParam("duration") long duration) {
        return openDALReactive.presignRead(path, Duration.ofSeconds(duration)).map(ResponseEntity::ok);
    }

    @GetMapping("/presignWrite")
    public Mono<ResponseEntity<PresignedRequest>> presignWrite(@RequestParam("path") String path, @RequestParam("duration") long duration) {
        return openDALReactive.presignWrite(path, Duration.ofSeconds(duration)).map(ResponseEntity::ok);
    }

    @GetMapping("/presignStat")
    public Mono<ResponseEntity<PresignedRequest>> presignStat(@RequestParam("path") String path, @RequestParam("duration") long duration) {
        return openDALReactive.presignStat(path, Duration.ofSeconds(duration)).map(ResponseEntity::ok);
    }

    @DeleteMapping("/delete")
    public Mono<ResponseEntity<Void>> delete(@RequestParam("path") String path) {
        return openDALReactive.delete(path).then(Mono.just(ResponseEntity.ok().<Void>build()));
    }

    @PostMapping("/createDir")
    public Mono<ResponseEntity<Void>> createDir(@RequestParam("path") String path) {
        return openDALReactive.createDir(path).then(Mono.just(ResponseEntity.ok().<Void>build()));
    }

    @PostMapping("/copy")
    public Mono<ResponseEntity<Void>> copy(@RequestParam("sourcePath") String sourcePath, @RequestParam("targetPath") String targetPath) {
        return openDALReactive.copy(sourcePath, targetPath).then(Mono.just(ResponseEntity.ok().<Void>build()));
    }

    @PostMapping("/rename")
    public Mono<ResponseEntity<Void>> rename(@RequestParam("sourcePath") String sourcePath, @RequestParam("targetPath") String targetPath) {
        return openDALReactive.rename(sourcePath, targetPath).then(Mono.just(ResponseEntity.ok().<Void>build()));
    }

    @DeleteMapping("/removeAll")
    public Mono<ResponseEntity<Void>> removeAll(@RequestParam("path") String path) {
        return openDALReactive.removeAll(path).then(Mono.just(ResponseEntity.ok().<Void>build()));
    }

    @GetMapping("/list")
    public Flux<Entry> list(@RequestParam("path") String path) {
        return openDALReactive.list(path);
    }
}

Testing with curl

Use the following curl commands to test the endpoints of your Spring WebMVC and WebFlux controllers:

1. Write String Content

   curl -X POST "http://localhost:8080/api/v1/opendal/write?path=/example/path" -H "Content-Type: application/json" -d "This is a test content"
   

2. Write Byte Content

   curl -X POST "http://localhost:8080/api/v1/opendal/writeBytes?path=/example/path
   
   

3. Append String Content

   curl -X POST "http://localhost:8080/api/v1/opendal/append?path=/example/path" -H "Content-Type: application/json" -d "This is additional content"
   

4. Append Byte Content

   curl -X POST "http://localhost:8080/api/v1/opendal/appendBytes?path=/example/path" -H "Content-Type: application/octet-stream" --data-binary @/path/to/your/file
   

5. Stat

   curl -X GET "http://localhost:8080/api/v1/opendal/stat?path=/example/path"
   

6. Read

   curl -X GET "http://localhost:8080/api/v1/opendal/read?path=/example/path"
   

7. Presign Read

   curl -X GET "http://localhost:8080/api/v1/opendal/presignRead?path=/example/path&duration=3600"
   

8. Presign Write

   curl -X GET "http://localhost:8080/api/v1/opendal/presignWrite?path=/example/path&duration=3600"
   

9. Presign Stat

   curl -X GET "http://localhost:8080/api/v1/opendal/presignStat?path=/example/path&duration=3600"
   

10. Delete

    curl -X DELETE "http://localhost:8080/api/v1/opendal/delete?path=/example/path"
    

11. Create Directory

    curl -X POST "http://localhost:8080/api/v1/opendal/createDir?path=/example/path"
    

12. Copy

    curl -X POST "http://localhost:8080/api/v1/opendal/copy?sourcePath=/example/source&targetPath=/example/target"
    

13. Rename

    curl -X POST "http://localhost:8080/api/v1/opendal/rename?sourcePath=/example/source&targetPath=/example/target"
    

14. Remove All

    curl -X DELETE "http://localhost:8080/api/v1/opendal/removeAll?path=/example/path"
    

15. List

    curl -X GET "http://localhost:8080/api/v1/opendal/list?path=/example/path"
    

[Xuanwo]

Hi, @tisonkun, would you like to take a look? I'm not a Java expert like you are.

[Xuanwo]

Some ideas in my mind for now:

• We don't need to seperate writeString and writeBytes, we only need to have write which accept bytes.
• We should remove append, it's an option for write
• I don't need we need to play with presign at current stage.

[tisonkun]

Hi, @tisonkun, would you like to take a look? I'm not a Java expert like you are.

Looks like a good start. Perhaps create a repo for this starter and we can collaborate as normal issues + PRs.

I don't think we need to include these code into the main repo since it would cause extra release burden. Let @shoothzj be the maintainer of the starters and modifying anything necessary in binding-java should be fine.

[hezhangjian]

• We don't need to seperate writeString and writeBytes, we only need to have write which accept bytes.

We can only maintain writeBytes now, Java has powerful overload and generic, we can support write(Object obj) leverage jackson's serial.

• We should remove append, it's an option for write

I need take a deep look, but why the java binding expose this method?

• I don't need we need to play with presign at current stage. Ok.

[hezhangjian]

Hi, @tisonkun, would you like to take a look? I'm not a Java expert like you are.

Looks like a good start. Perhaps create a repo for this starter and we can collaborate as normal issues + PRs.

I don't think we need to include these code into the main repo since it would cause extra release burden. Let @shoothzj be the maintainer of the starters and modifying anything necessary in binding-java should be fine.

I would be happy to help maintain the starters. :)

[Xuanwo]

Hi, @shoothzj, the subproject for spring-opendal has been started. Would you like to create a tracking issue for it?

[hezhangjian]

@Xuanwo https://github.com/apache/opendal/issues/5015

[Xuanwo]

Thanks!