feat(server): Google Pixel motion photos

[fourkbomb]

Description

Add support for motion photos taken on Pixel phones. They have the exif property 'MotionPhoto' set to 1, and an embedded mp4 file appended to the JPEG file.

The implementation works like this:

• on metadata extraction, if a live photo is detected, examine the metadata to determine where in the file the embedded MP4 is.
• extract this MP4 and write it next to the JPEG.
• link it using the existing mechanism for live photos.

There is a "MotionPhotoPresentationTimestampUs" exif property, which we don't do anything with - I imagine that it refers to the timepoint in the video that the photo was taken at, but it probably warrants more investigation.

How Has This Been Tested?

• Tested against motion photos from a Pixel 6, Pixel 4a, and Pixel 2.

Checklist:

• [x] I have performed a self-review of my own code
• [x] I have made corresponding changes to the documentation if applicable

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
immich	⬜️ Ignored (Inspect)			Jul 22, 2023 2:54am

[alex-phillips]

@fourkbomb this is great! I was just thinking about adding this in myself when I saw this.

One note: Google Pixel's actually changed how their videos are embedded inside the motion photos at some time between the Pixel 2 and the Pixel 4 (maybe Pixel 3?). I'd love to see the older version get added in as well. If you would like to integrate this into this MR, I'll provide the metadata info below. Otherwise, I'll wait until this gets merged in an open an MR with the old version support.

Previous to the Pixel 4, Pixel live photos were embedded with different metadata to denote where in the file the videos were stored. Currently, there is a flag of MotionPhoto with a Length , Padding, etc in the Directory entry as you've shown in your MR. The old versions instead provide the following EXIF metadata (this is from a sample image I have):

  MicroVideo: 1
  MicroVideoVersion: 1
  MicroVideoOffset: 3114365
  MicroVideoPresentationTimestampUs: 1099537

I believe most of the logic you already have can be used for these types of files as well with 3 main differences:

1. The flag to determine the existence of a live photo is MicroVideo instead of MotionPhoto
2. The data is not stored in the Directory entry of the metadata, but instead in the above metadata fields
3. Instead of a length of the video, it provides an offset inside the file for the video.

If you wanted to tackle this, I believe you could abstract the directory reading to a more generic function that can support both the offset as well as the length and then pass this normalized information into the function extractPixelLivePhoto.

Let me know what you think and if you want to integrate this. Otherwise, I'll look into adding it afterwards. Thanks again!

[alex-phillips]

@fourkbomb Here's a sample motion photo from my old Pixel 2 you can use for testing if you'd like to take this on. Ultimate for all live photos, we just need the location of the file where the video lives and it's length. I think it would be best to have a centralized function that can check for various cases and build this data regardless of what tags it uses and then it could pass this normalized data into a generic extractLivePhoto function to perform the rest of the logic.

MVIMG_20200727_093505.jpg.zip

[fourkbomb]

I think it would be best to have a centralized function that can check for various cases and build this data regardless of what tags it uses and then it could pass this normalized data into a generic extractLivePhoto function to perform the rest of the logic.

I had a go at this, although I didn't move the logic to look for the EXIF tags into its own function - particularly for the old Pixel case, it's very simple and not worth splitting out IMO. Let me know what you think!

[alex-phillips]

I had a go at this, although I didn't move the logic to look for the EXIF tags into its own function - particularly for the old Pixel case, it's very simple and not worth splitting out IMO. Let me know what you think!

The older versions with MicroVideo seem to be working great, but now my "newer" photos that used the MotionPhoto method don't seem to be working. It looks like Immich is detecting that they are supposed to be live photos (I get the icon and all the UI components), but when I attempt to play, it shows an error. I also don't see an mp4 file was written to the filesystem for the MotionPhoto files but there is one written for the MicroVideo files.

[fourkbomb]

The older versions with MicroVideo seem to be working great, but now my "newer" photos that used the MotionPhoto method don't seem to be working. It looks like Immich is detecting that they are supposed to be live photos (I get the icon and all the UI components), but when I attempt to play, it shows an error. I also don't see an mp4 file was written to the filesystem for the MotionPhoto files but there is one written for the MicroVideo files.

I just tried, it seems to work for me. Are you able to provide a broken image file? Or can you see anything in the logs?

[alex-phillips]

I just tried, it seems to work for me. Are you able to provide a broken image file? Or can you see anything in the logs?

I must have some problematic photos as newer ones seem to be working fine. I might investigate more on my end, but it looks like this is working. False alarm!

[alex-phillips]

@fourkbomb Ok, follow-up with more information. This is interesting and maybe you can shed some light on what's happening. I think it may be potential differences between even later Pixel motion photos, in this case a Pixel 4a and a Pixel 7 Pro. It appears the images from my older 4a are working fine, but my newer Pixel 7 Pro images are presenting these issues.

Test image name I'm using and describing the issues below is from my Pixel 7 Pro, and have provided in the attachment: PXL_20221220_233509498.MP.jpg

The photo gets uploaded and interpreted as "live photos" because in the UI they have the play icon, but when you hover over them, it turns into a red !. Now this is where it gets interesting. If I download the image, it downloads 2 identical copies of the image: PXL_20221220_233509498.jpg PXL_20221220_233509498(1).jpg

Now if I select multiple files and download the image in that batch, it generates a zip file and the files related to the test image in this are as follows: PXL_20221220_233509498.jpg PXL_20221220_233509498.MP.mp4

So two things here:

1. During the processing, it appears the .MP. portion of the original filename is getting removed. I think we need to try and preserve this as it is part of the filename, not the extension.
2. The single download fails to get the video portion but the bulk download works. This may not be related to your PR but is worth mentioning in case we need to open a separate bug.

Now what is interesting is that the code is clearly detecting the live photo properly, but it is not displaying or playing it properly in the UI. And as you can see from the bulk download, it is creating the mp4 file and even providing it in the download. The mp4 file plays back fine on my local machine as well.

In the attachment provided, I've included a photo from my Pixel 4a (which is working properly) and a photo from my Pixel 7 Pro, each in 2 formats: a downloaded directory which is how the files are received when downloaded (in bulk) from the UI, and an original folder which is the original photos prior to uploading into Immich.

Clearly there's a difference between the Pixel 4a (working) and the Pixel 7 Pro (not working), but at a glance I can't seem to tell what that is.

immich-test-files.zip

[alex-phillips]

@fourkbomb sorry for all the pings. You can ignore the last post. It looks like the issue here is that the newer pixel phones are encoded with HEVC and due to a bug with how Google handles them, Immich is currently not transcoding them. Not sure if there's a fix for this, and any work can be followed against the other bug. I think we're good to go here.

https://github.com/immich-app/immich/issues/3012

[alextran1502]

Hello, what is the status of this PR?

[alex-phillips]

Hello, what is the status of this PR?

This looks good to me and seems to be working as expected. I think we just need to get the tests working.