Upload without storing whole file in memory

[biji]

Is it possible to Upload() big files like videos and documents without reading whole file to variable? Probably that will save some memory

Thanks

[serenity-77]

Upload once and cache the upload result

[tulir]

Currently not possible, but files are also limited to 100mb so there's no problem just reading the whole file to memory.

[Duckduckgot]

Upload once and cache the upload result

But it is limited in time, then you will need to upload again.

[mamur-rezeki]

Client.Upload encrypt whole []byte in code before uploaded.

https://github.com/tulir/whatsmeow/blob/be0edabb0bf3d3f6401d9e6d9922373ed58a3487/upload.go#L63-L134

Seems impossible to implement io Reader directly from path to avoid high memory usage.

[tulir]

It's of course possible to implement, just requires a bunch of refactoring

[mamur-rezeki]

Alright, we will wait for it to be implement.

[Duckduckgot]

Someone please help to realize this useful feature.

[Duckduckgot]

Something's wrong. Download failure at the recipient

func EncryptStream(key, iv []byte, reader io.Reader, writer io.Writer) error {
	block, err := aes.NewCipher(key)
	if err != nil {
		return err
	}

	var stream cipher.Stream
	if iv != nil {
		stream = cipher.NewCFBEncrypter(block, iv)
	} else {
		iv = make([]byte, aes.BlockSize)
		if _, err := io.ReadFull(rand.Reader, iv); err != nil {
			return err
		}

		stream = cipher.NewCFBEncrypter(block, iv)
		writer.Write(iv)
	}

	writer = &cipher.StreamWriter{S: stream, W: writer}

	if _, err := io.Copy(writer, reader); err != nil {
		return err
	}

	return nil
}

[Duckduckgot]

func encryptStream(key, iv, macKey []byte, reader io.Reader, writer io.Writer) ([]byte, []byte, error) {
	block, err := aes.NewCipher(key)
	if err != nil {
		return nil, nil, err
	}
	h1 := sha256.New()
	if _, err := io.Copy(h1, reader); err != nil {
		return nil, nil, err
	}
	FileSHA256 := h1.Sum(nil)
	var buf bytes.Buffer
	stream := cipher.NewCFBEncrypter(block, iv)
	encWriter := &cipher.StreamWriter{S: stream, W: &buf}

	h := hmac.New(sha256.New, macKey)
	if _, err := io.Copy(h, reader); err != nil {
		return nil, nil, err
	}
	mac := h.Sum(nil)
	mac10 := mac[:10]

	h2 := sha256.New()
	if _, err := io.Copy(encWriter, reader); err != nil {
		return nil, nil, err
	}
	if _, err := writer.Write(mac10); err != nil {
		return nil, nil, err
	}
	if _, err := io.Copy(h2, &buf); err != nil {
		return nil, nil, err
	}
	FileEncSHA256 := h2.Sum(nil)

	return FileSHA256, FileEncSHA256, nil
}

[ahmedRSA]

func encryptStream(key, iv, macKey []byte, reader io.Reader, writer io.Writer) ([]byte, []byte, error) {
	block, err := aes.NewCipher(key)
	if err != nil {
		return nil, nil, err
	}
	h1 := sha256.New()
	if _, err := io.Copy(h1, reader); err != nil {
		return nil, nil, err
	}
	FileSHA256 := h1.Sum(nil)
	var buf bytes.Buffer
	stream := cipher.NewCFBEncrypter(block, iv)
	encWriter := &cipher.StreamWriter{S: stream, W: &buf}

	h := hmac.New(sha256.New, macKey)
	if _, err := io.Copy(h, reader); err != nil {
		return nil, nil, err
	}
	mac := h.Sum(nil)
	mac10 := mac[:10]

	h2 := sha256.New()
	if _, err := io.Copy(encWriter, reader); err != nil {
		return nil, nil, err
	}
	if _, err := writer.Write(mac10); err != nil {
		return nil, nil, err
	}
	if _, err := io.Copy(h2, &buf); err != nil {
		return nil, nil, err
	}
	FileEncSHA256 := h2.Sum(nil)

	return FileSHA256, FileEncSHA256, nil
}

you make it work?

[arugaz]

i can make it but it requires temporary file to place the enc data, if thats not a problem i'll do pull request

edit: ill find out to avoid that

[Duckduckgot]

i can make it but it requires temporary file to place the enc data, if thats not a problem i'll do pull request

edit: ill find out to avoid that

Is there any progress?

[gamersindo1223]

@Duckduckgot any progress?

[mamur-rezeki]

According to @arugaz comment.

Temporarily storing plaintext and dataToUpload to disk for read-write may reduce RAM memory usage rather than use buffer variable like bytes.Reader or []byte to pass to the another func.

[gamersindo1223]

yeah