storage.create function deprecated - storage.createFrom() hangs

[oliverjstack]

Overview

Using the com.google.cloud:spring-cloud-gcp-storage:5.0.4 library the storage.create() function has now been listed as deprecated to upload files to GCS bucket.

They now have a createFrom() function that achieves the same outcome however seems when it interacts with the GCS server (fake-gcs-server) it fails.

Notes

• The post request sent to the GCS Server using the createFrom() returns a 200

"POST /upload/storage/v1/b/<bucketName>/o?ifGenerationMatch=0&name=<fileName>&uploadType=resumable HTTP/1.1\" 200 772\n"

• The post request sent to the GCS Server using the create() function returns 200

"POST /upload/storage/v1/b/<bucketName>/o?ifGenerationMatch=0&projection=full&uploadType=multipart HTTP/1.1\" 200 1007\n"

• The create() function successfully uploads the file to the container and is verified with buy running a java script which lists all files within the bucket using the Storage client. The createFrom() does not have the file listed.
• The -log-level "debug" does not show any debug logs within the container.
• The -log-level "trace, panic" return a "not a valid logging level" message in the container on startup.

Summary

I appreciate that you guys are seemingly the only way to create a fake gcs server allowing integration testing to proceed. I think its crazy that Google has no simple image that can emulate their own server functionality.

Let me know if I need to add anything or what you guys require from me. For now, Ill just be using the GCP SDK local helper client to emulate the storage but would love to use your solution if this problem is resolved.

[fsouza]

Can you share a stack trace of the failure and how to reproduce it?

[marcuslinke]

@fsouza There is no stacktrace. It just hangs indefinitely.

[fsouza]

@marcuslinke can you share a complete reproducer? I suspect the new method is expecting something different but I'm not sure what.

[sigpwned]

I ran across this issue from the Java GCS client. I have no idea what languages other folks are using.

Here is a (reasonably) simple Java application that shows off the issue:

package io.socialytiqs.bsky.backfill.rehost;

import static java.time.Duration.ofSeconds;
import static org.assertj.core.api.Assertions.assertThat;
import static org.junit.jupiter.api.Assertions.assertTimeoutPreemptively;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.testcontainers.containers.GenericContainer;
import com.google.cloud.NoCredentials;
import com.google.cloud.storage.Blob;
import com.google.cloud.storage.BlobInfo;
import com.google.cloud.storage.Storage;
import com.google.cloud.storage.StorageOptions;

public class WriteableByteChannelHangTest {

  private static final String PROJECT_ID = "test-project";
  private static final String BUCKET_NAME = "test-bucket";

  private static GenericContainer<?> gcsContainer;
  private static Storage storage;

  @BeforeAll
  public static void setup() {
    // Start the fake GCS server
    gcsContainer = new GenericContainer<>("fsouza/fake-gcs-server:latest").withExposedPorts(4443)
        .withCommand("-scheme", "http");
    gcsContainer.start();

    // Configure the GCS client to use the fake GCS server
    String gcsEndpoint = "http://localhost:" + gcsContainer.getMappedPort(4443);
    storage = StorageOptions.newBuilder().setHost(gcsEndpoint).setProjectId(PROJECT_ID)
        .setCredentials(NoCredentials.getInstance()).build().getService();

    // Create a bucket in the fake GCS server
    storage.create(com.google.cloud.storage.BucketInfo.newBuilder(BUCKET_NAME).build());
  }

  @AfterAll
  public static void cleanup() {
    if (gcsContainer != null) {
      gcsContainer.stop();
    }
  }

  @Test
  public void testWritableByteChannelHangs() throws Exception {
    final String payload = "Hello, World!";

    // This works
    Blob blob1a = storage.create(BlobInfo.newBuilder(BUCKET_NAME, "blob1").build(),
        new ByteArrayInputStream(payload.getBytes(StandardCharsets.UTF_8)));

    // I know it works because I can read it back
    Blob blob1b = storage.get(blob1a.getBlobId());
    String blob1data = new String(blob1b.getContent(), StandardCharsets.UTF_8);
    assertThat(blob1data).isEqualTo(payload);

    // However, this times out...
    assertTimeoutPreemptively(ofSeconds(30), () -> {
      // Create a BlobInfo object for the object to be written
      BlobInfo info2a = BlobInfo.newBuilder(BUCKET_NAME, "blob2").build();

      // Open a WritableByteChannel to write to the object
      try {
        storage.createFrom(info2a,
            new ByteArrayInputStream(payload.getBytes(StandardCharsets.UTF_8)));
      } catch (IOException e) {
        // Handle exceptions (e.g., if the server becomes unresponsive)
        e.printStackTrace();
      }
    });
  }
}

The first assertThat() against blob1 passes. The assertTimeoutPreemptively() never returns.

EDIT: Actually, if I set the timeout high enough (30 seconds does it, but fewer might, too), I get this error:

com.google.cloud.storage.StorageException: Unknown Error
	at com.google.cloud.storage.ResumableSessionFailureScenario.toStorageException(ResumableSessionFailureScenario.java:120)
	at com.google.cloud.storage.JsonResumableSessionQueryTask.call(JsonResumableSessionQueryTask.java:138)
	at com.google.cloud.storage.JsonResumableSession.query(JsonResumableSession.java:58)
	at com.google.cloud.storage.JsonResumableSession.lambda$put$0(JsonResumableSession.java:74)
	at com.google.cloud.storage.Retrying.lambda$run$0(Retrying.java:102)
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:102)
	at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
	at com.google.cloud.storage.Retrying.run(Retrying.java:99)
	at com.google.cloud.storage.JsonResumableSession.put(JsonResumableSession.java:69)
	at com.google.cloud.storage.ApiaryUnbufferedWritableByteChannel.internalWrite(ApiaryUnbufferedWritableByteChannel.java:115)
	at com.google.cloud.storage.ApiaryUnbufferedWritableByteChannel.writeAndClose(ApiaryUnbufferedWritableByteChannel.java:65)
	at com.google.cloud.storage.UnbufferedWritableByteChannelSession$UnbufferedWritableByteChannel.writeAndClose(UnbufferedWritableByteChannelSession.java:40)
	at com.google.cloud.storage.DefaultBufferedWritableByteChannel.close(DefaultBufferedWritableByteChannel.java:167)
	at com.google.cloud.storage.StorageByteChannels$SynchronizedBufferedWritableByteChannel.close(StorageByteChannels.java:151)
	at com.google.cloud.storage.StorageException.wrapIOException(StorageException.java:223)
	at com.google.cloud.storage.BaseStorageWriteChannel.close(BaseStorageWriteChannel.java:98)
	at com.google.cloud.storage.StorageImpl.createFrom(StorageImpl.java:304)
	at com.google.cloud.storage.StorageImpl.createFrom(StorageImpl.java:292)
	at io.socialytiqs.bsky.backfill.rehost.WriteableByteChannelHangTest.lambda$0(WriteableByteChannelHangTest.java:70)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.Net.pollConnect(Native Method)
	at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
	at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:549)
	at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:592)
	at java.base/java.net.Socket.connect(Socket.java:751)
	at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:178)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:531)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:636)
	at java.base/sun.net.www.http.HttpClient.<init>(HttpClient.java:280)
	at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:386)
	at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:408)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1304)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1237)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1123)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1052)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1446)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1417)
	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:113)
	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84)
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012)
	at com.google.cloud.storage.JsonResumableSessionQueryTask.call(JsonResumableSessionQueryTask.java:62)
	... 22 more

I know from previous experience that anything using a WriteableByteChannel fails, too. This example uses a WriteableByteChannel under the covers in the client. That doesn't mean WriteableByteChannel is the root cause, but it's an interesting note.

The ConnectException is odd. We know that the storage client can talk to the emulator, since it does earlier in the test. But now it can't see the emulator? I wrapped the timeout code in a try/finally, put a breakpoint in the finally, and then was able to curl the emulator while the program was paused, so I know the emulator didn't just crash.

This makes me lean towards a client error. But this is the only mention I've found of the bug, so I figured I'd put my learnings here in the hopes that it helps others in the future!

[mihohren]

Can confirm issue is still present in latest as of today

[lberrymage]

I don't think this is a bug with fake-gcs-server, but a configuration issue when running fake-gcs-server in a container as described in #637. createFrom() initiates a resumable upload with a POST to fake-gcs-server and then makes subsequent requests to the endpoint returned in the Location header returned in the first request. If you run fake-gcs-server in a Docker container with a port mapping different than the default (which testcontainers in particular encourages), then only the first request will succeed — subsequent requests will hang because fake-gcs-server's default value for the Location header uses the listening port, not the port that is exposed to the Docker host, causing those requests to be made to an unreachable endpoint. My guess is that's where the ConnectionException is coming from.

I ran into what appears to be this same issue today and confirmed the problem by statically mapping port 4443 to the same port on my Docker host for my testcontainers container, which successfully worked around the issue.

I believe passing a proper value for -external-url would work as well, although obviously that's more difficult since you don't know the dynamically mapped port of the container until it's started.