Showing file sizes in the file list

[nh2]

Hi, here's a feature request and associated question.

It would be great if android-file-transfer could show the size of the file next to the name.

From https://medium.com/@bandan/media-transfer-protocol-and-4gb-file-transfers-effe74663a63 I know that normal file size attributes are 32-bit uints and thus limited to 4 GiB, and the size field for files larger than that is to be set to 0xFFFFFFFF according to the spec, and that for transfer of files larger than that the way to do it is described in the MTP spec (section "Retrieving a >4GB Object with a GetObject Operation").

android-file-transfer gets the correct size during transfer, e.g. in console output like this:

downloading  1  file(s),  11131933122  bytes

and the code related to getting the file size is probably this:

https://github.com/whoozle/android-file-transfer-linux/blob/774e27a7a5a34f2df5646211de808a631991e64b/fuse/fuse_ll.cpp#L116

Knowing nothing about MTP, now my question:

Is it possible to get the file size efficiently for each file? Or do you have to initiate + abort a transfer for that? Or does GetObjectInfo() perhaps already get called for getting the file names, and so you already have the real sizes but they aren't shown in the GUI? If it's possible, could android-file-transfer show the sizes in the list?

Then, I also do not understand the MTP spec. One section says:

5.3.1.4 Object Compressed Size The size of the data component of the object in bytes. If the object is larger than 2^32 bytes in size (4GB), this field shall contain a value of 0xFFFFFFFF.

So this says ObjectCompressedSize is the field that's 32-bits.

But in Appendix H they say:

Retrieving a >4GB Object with a GetObject Operation When retrieving an object, the size of the object to be retrieved shall be identified using the ObjectCompressedSize Object Property, which is not limited to 32 bits.

This now says that ObjectCompressedSize is not limited to 32 bits, directly contradicting the earlier section.

The code I quoted from android-file-transfer seems to implement the earlier section. So I believe Appendix H of the MTP spec (Revision 1.1) is wrong. Do you agree?

---

What other implementations do:

• Apparently all current GNOME tools not only display the size of files > 4 GiB wrong (claiming it is 4294967295), but also silently corrupt the data during copy,creating files that are 4 GiB, throwing away the rest. I've lost important videos because of this, in my case in Thunar over gphoto2://.
  • How can I find out which protocol (PTP or MTP) libgphoto2 uses? Does this question even make sense?
• Same for the gphoto2 CLI app.
• libptp2's ptpcam -L shows the file sizes as 4294967295 and also currently crashes after 3 minutes transferring (over USB 2 -> USB C on my 11 GB file):

  % ptpcam --get-file=0x0000043f
  Camera: RICOH THETA Z1
  Saving file: "R0011054.MP4" error!
  PTP: Protocol error: response expected
  ERROR: Could not close session!
  1,78s user 24,20s system 12% cpu 3:23,72 total
  

  It creates a 4 GiB file on the destination immediately (probably ftruncate'd or so).
• mtp-detect from libmtp version: 1.1.13 doesn't even find the camera (Ricoh Theta Z1).
• Windows 10 shows the file sizes correctly already on the device, and also copies fine.

Click here to expand older side note that turned out to be false

The below turned out to be wrong, I measured across different USB ports, and apparently 1 USB port only has this problem with the long cable but works with the short one. Probably it's underpowered for the long cable. On another port/laptop it works as well with android-file-transfer as it does with Linux.

Finally, I'd like to let you know (and write for myself here, as it did initially confuse me when investigating these >4-GiB-issues) that with a 3 meter USB 2 -> USB C flat band cable, `android-file-transfer` often stalls the transport speed and then crashes with:

device found, opening session...
device info "Ricoh Company, Ltd."   "RICOH THETA Z1"
mtpz-data path:  "/home/niklas/.mtpz-data"
switching to storage id  131073
session opened, starting
downloading to  "/home/niklas/tmp/ricoh"
downloading  1  file(s),  11131933122  bytes
downloading  1087 to "/home/niklas/tmp/ricoh/R0011054.MP4"
10010 ms since the last poll call
downloading file  "/home/niklas/tmp/ricoh/R0011054.MP4"  failed:  "timeout reaping usb urb"
finishing queue
got unknown urb: 0x7f9bebdf2490 of size 30000
30013 ms since the last poll call
[1]    23752 segmentation fault (core dumped)

This doesn't happen with a short cable (where it also has consistent transfer speed), so it's probably cable related, but Windows's MTP does not have this problem with the same cable and camera, so maybe it can be improved.

Thanks!

[nh2]

I believe libgphoto2 is supposed to handle large files here:

case PTP_OPC_ObjectSize:
	if (prop->datatype == PTP_DTC_UINT64) {
		ob->oi.ObjectCompressedSize = prop->propval.u64;
	} else if (prop->datatype == PTP_DTC_UINT32) {
		ob->oi.ObjectCompressedSize = prop->propval.u32;
	}

But that doesn't seem to work, or this code is not called.

[nh2]

While comparing how libgphoto2 deals with large files on our Samsung Galaxy S7, I found that there's a recent regression that breaks it for that phone while it was working before:

https://github.com/gphoto/libgphoto2/issues/581

Interestingly there, even in the unregressed case where --list-files shows the correct size, --get-file corrupted the file by truncating to 2^32 Bytes.

android-file-transfer transfers that file correctly.

[whoozle]

Hello! Thanks for the thorough investigation

From https://medium.com/@bandan/media-transfer-protocol-and-4gb-file-transfers-effe74663a63 I know that normal file size attributes are 32-bit uints and thus limited to 4 GiB, and the size field for files larger than that is to be set to 0xFFFFFFFF according to the spec, and that for transfer of files larger than that the way to do it is described in the MTP spec (section "Retrieving a >4GB Object with a GetObject Operation").

That's correct, event though fuse_ll is fuse wrapper for user space mount, code you found is the code to get size. Code to get the size in gui is in MtpObjectsModel::ObjectInfo MtpObjectsModel::getInfoById(mtp::ObjectId objectId) const

	qint64 size = oi.ObjectCompressedSize;
	if (size == mtp::MaxObjectSize)
		size = _session->GetObjectIntegerProperty(objectId, mtp::ObjectProperty::ObjectSize);

This now says that ObjectCompressedSize is not limited to 32 bits, directly contradicting the earlier section.

Not really. First paragraph was about ObjectCompressedSize field of ObjectInfo structure. It has fixed size of 32 bits. Second paragraph is about object property, and they are typed and it's implementation defined if they 8/16/32 or 64 bits.

I'm getting size with size = _session->GetObjectIntegerProperty(objectId, mtp::ObjectProperty::ObjectSize); and it depends how much data we get from device. Take a look into InputStream::ReadSingleInteger

		switch(data.size())
		{
		case 8: return s.Read64();
		case 4: return s.Read32();
		case 2: return s.Read16();
		case 1: return s.Read8();
		default:
			throw std::runtime_error("unexpected length for numeric property");
		}

Libptp code below seems fine to me, as long as device properly report property data type :wink:

case PTP_OPC_ObjectSize:
	if (prop->datatype == PTP_DTC_UINT64) {
		ob->oi.ObjectCompressedSize = prop->propval.u64;
	} else if (prop->datatype == PTP_DTC_UINT32) {
		ob->oi.ObjectCompressedSize = prop->propval.u32;
	}

maybe it's not called at all, because again, it's code for parsing individual property, not MTP ObjectInfo, and you have explicitly call it in your client (e.g. GUI)

Regarding time-out, this is long standing issue to me, maybe even Linux bug. I found a few reproduction scenarios, but never been able to find the root cause of the issue. It looks like physically URB is in linux usbcore, but it never got signalled neither to ioctl no poll/select.

Regarding efficient way of getting sizes, there's a way defined in MTP specification, command is GetObjectPropertyList. It retrieves one or all properties of object set given.

https://github.com/whoozle/android-file-transfer-linux/blob/master/cli/Session.cpp#L520

Hope this helps.

Regarding original request, seems to be nice idea as long as device supports GetObjectPropertyList, or it won't be efficient at all. GetObjectInfo is really slow and I'm trying not to call it while listing files neither from UI nor from cli or fuse.

[nh2]

Hey, thanks for your reply.

First paragraph was about ObjectCompressedSize field of ObjectInfo structure. It has fixed size of 32 bits. Second paragraph is about object property

Ah I see. So that distinction is from these two parts of the spec, right?

5.3.1 ObjectInfo Dataset Description
5.3.1.4 Object Compressed Size

5.3.2 Object Properties
5.3.2.6 Required Object Properties
        Devices which implement Object Properties must implement as Object Properties the
        properties which map to the required fields of the ObjectInfo dataset, [..]
        Required properties for all object formats include:
        * ObjectCompressedSize

I have another related question:

In https://en.wikipedia.org/wiki/Picture_Transfer_Protocol#Version_1.1 it says

Support for objects larger than the 4GiB size limit set by PTP v1.0, by requiring 64 bits (8 bytes) for object size

Is this PTP 1.1 actually existent? I couldn't find a PDF for it, and I couldn't figure out whether anything implements that part, or whether any devices support it.

[whoozle]

So that distinction is from these two parts of the spec, right?

Yes, ObjectInfo is specific message, pretty much obsoleted in favour of properties. There's only a few fields still required there.

Is this PTP 1.1 actually existent?

I'm not sure about PTP 1.1, it looks like it's been superseded by MTP 1.1. You can legally download MTP 1.1 spec from here.

So just to sum it up, 4Gb+ support is essentially this:

• Every 32 bit field bigger than 2**32 - 1 becomes 0xFFFFFFFF. This also applies to MTP messages/containers sizes.
• USB Bulk transfer does not require any sizes in advance, though could theoretically be even more that 2**64 and end is indicated by partial packet (anything less than bulk transfer size, which is normally 512 bytes), so no changes here.
• ObjectSize property may be 64 bit for devices which support big files.