react-native-skia
react-native-skia copied to clipboard
Shopify
Buggy .makeImageSnapshot() on IOS
Description
When I want to snapshot my Skia canvas (which is not that small), it gets buggy. These bugs don't occur always. If you try it multiple times, you will see that sometimes it works fine.
Screenrecording of the bug: https://www.youtube.com/shorts/mH7VuiT7CuU
What bugs?:
- For a short period of time, the elements of my canvas get transformed in a weird way
- Outputted image doesn't contain all items of canvas sometimes
Does RNSkia somehow rerender the Canvas, when the method makeImageSnapshot gets executed? If yes, I think, the snapshot is taken before the rerender process is completely finished.
It makes no sense to me, since the only code that gets triggered on the button click is this one:
const handleShare = async () => {
const image = canvasRef.current?.makeImageSnapshot();
if (image) {
const base64Image = image.encodeToBase64();
const options = {
mimeType: 'image/jpeg',
dialogTitle: 'Share the image',
UTI: 'public.jpeg',
};
const fileUri = FileSystem.documentDirectory + 'test.jpeg'; // todo ts name name
await FileSystem.writeAsStringAsync(fileUri, base64Image, {
encoding: FileSystem.EncodingType.Base64,
});
Sharing.shareAsync(fileUri, options)
.then((data: any) => {
console.log('SHARED');
})
.catch((err: any) => {
console.log(JSON.stringify(err));
});
}
};
Version
0.1.197 in repro but also appears in 0.1.202
Steps to reproduce
How to reproduce:
npm install
On Mac simulator
npx expo run:ios
On Iphone with EAS
eas build --profile development --platform ios
npx expo start --dev-client
In the App
Snack, code example, screenshot, or link to a repository
I couldn't really reproduce this in any test projects sadly. This only happened in my big project, i guess its because it has more heavy operations.
https://github.com/sravis02/flashes-app Branch: issue-react-native-skia I invited you both (chrfalch & wcandillion) to my repository
The Canvas is in PreviewCanvas.tsx The Snapshotfunction in preview.tsx
I'm struggling to understand the issue based on the video, could you walk me thought it? We know have a test suite for this feature any tests you would like to me add there I can add.
Same issue but I believe it only happens in dev builds. Does the issue happen in a production build for you @sravis02?
What happens for me: I have a canvas with various elements. When I call makeImageSnapshot in the local dev builds, the resulting image sometimes doesn't contain all my elements from the canvas. It quickly blinks (like in your screen recording) and it seems as the elements are scaled down and on the top left corner falsely positioned.
In production I cannot reproduce this so far...
I'm struggling to understand the issue based on the video, could you walk me thought it? We know have a test suite for this feature any tests you would like to me add there I can add.
thats exactly the issue i am having. it occurs on expo development builds, expo prebuild builds and expo internal preview builds for me.
I'm struggling to understand the issue based on the video, could you walk me thought it? We know have a test suite for this feature any tests you would like to me add there I can add.
i will take a look into it again and try to provide some minimal reproduceable tests
I've encountered the same issue. When I call the makeImageSnapshot sometimes there's a small blink in the canvas (image of the frame attached).
https://github.com/Shopify/react-native-skia/assets/10179494/434ea7e5-1c2a-4f18-8d65-ad125a38fa20
This is the code I'm using:
export default function ImagePreviewScreen() {
const navigation = useNavigation()
const route = useRoute<any>()
const { t } = useTranslation()
const { uri, serviceId, isFinal } = route.params ?? {}
console.log('ImagePreviewScreen')
const path = useSharedValue<string>('')
const [isTransitioning, setIsTransitioning] = useState(true)
const [initialCanvasSize, setInitialCanvasSize] = useState(null as Dimensions)
const [currentCanvasSize, setCurrentCanvasSize] = useState(null as Dimensions)
const [hasPath, setHasPath] = useState(false)
const [isSaving, setIsSaving] = useState(false)
useAnimatedReaction(
() => Boolean(path.value),
(currentValue, previousValue) => {
if (currentValue !== previousValue) {
runOnJS(setHasPath)(currentValue)
}
},
)
const canvasRef = useCanvasRef()
const image = useImage(uri)
const handleUndo = useCallback(() => {
// Remove last line remove until last M
path.value = path.value.replace(/ ?M[^M]*$/, '')
}, [path])
const handleLayout = useCallback(
(event: LayoutChangeEvent) => {
const { width, height } = event.nativeEvent.layout
if (!initialCanvasSize) {
setInitialCanvasSize({ width, height })
}
setCurrentCanvasSize({ width, height })
},
[initialCanvasSize],
)
const srcRect = useImageRect({ canvas: initialCanvasSize, uri })
const dstRct = useImageRect({ canvas: currentCanvasSize, uri })
const handleSave = useCallback(async () => {
console.log('handleSave')
setIsSaving(true)
// Simply save image if there are no annotations
if (!hasPath) {
addServiceAttachment(uri, serviceId)
} else {
try {
console.log('Make image snapshot')
const canvas = canvasRef.current
if (!canvas) return
console.log(canvas.state)
// TODO: use original image size
// eslint-disable-next-line @typescript-eslint/no-non-null-assertion
const snapshot = canvas.makeImageSnapshot(dstRct!)
const base64Image = snapshot.encodeToBase64(ImageFormat.PNG)
console.log('Image snapshot done')
if (!base64Image) return
const fileName =
`${serviceId}-${Date.now()}.${SIGNATURE_EXTENSION}` as const
const imageUri = documentDirectory + fileName
console.log('Write image to file')
await writeAsStringAsync(
imageUri,
base64Image.replace('data:image/png;base64,', ''),
{ encoding: 'base64' },
)
snapshot.dispose()
console.log('Add attachment to service')
addServiceAttachment(imageUri, serviceId)
} catch {
// TODO: show error
} finally {
setIsSaving(false)
}
}
navigation.goBack()
}, [canvasRef, dstRct, hasPath, navigation, serviceId, uri])
useLayoutEffect(() => {
navigation.addListener('transitionEnd' as any, () => {
setIsTransitioning(false)
})
navigation.setOptions({
headerTintColor: 'white',
headerStyle: { backgroundColor: 'black', color: 'white' },
headerLeft() {
return (
<Button
$variant="secondary"
onPress={() => navigation.goBack()}
$small
>
{t('cancel')}
</Button>
)
},
headerRight() {
return isFinal ? null : (
<Button
disabled={isSaving}
$variant="primary"
onPress={handleSave}
$small
>
{t('screens.image-preview.save')}
</Button>
)
},
})
}, [handleSave, handleUndo, isFinal, isSaving, navigation, path.value, t])
const panGesture = Gesture.Pan()
.onStart((event) => {
if (!srcRect || !dstRct) return
let { x, y } = event
if (
JSON.stringify(initialCanvasSize) !== JSON.stringify(currentCanvasSize)
) {
const [{ translateX }, { translateY }, { scaleX }, { scaleY }] =
rect2rect(srcRect, dstRct)
x = (x - translateX) / scaleX
y = (y - translateY) / scaleY
}
path.value = `${path.value} M ${x},${y}`
})
.onUpdate((event) => {
if (!srcRect || !dstRct) return
let { x, y } = event
if (
JSON.stringify(initialCanvasSize) !== JSON.stringify(currentCanvasSize)
) {
const [{ translateX }, { translateY }, { scaleX }, { scaleY }] =
rect2rect(srcRect, dstRct)
x = (x - translateX) / scaleX
y = (y - translateY) / scaleY
}
path.value = `${path.value} L ${x},${y}`
})
const tapGesture = Gesture.Tap()
.runOnJS(true)
.onStart(() => {
Keyboard.dismiss()
})
const composed = Gesture.Race(tapGesture, panGesture)
return (
<SafeAreaView edges={SAFE_AREA_EDGES} style={styles.container}>
<EXStatusBar style="light" />
{!isFinal && (
<View style={styles.actionsContainer}>
<Button
disabled={!hasPath}
$variant="secondary"
onPress={handleUndo}
$circle
>
<Button.Icon as={Ionicons} name="return-up-back-outline" />
</Button>
<View />
</View>
)}
<KeyboardAvoidingView
style={styles.container}
behavior="padding"
keyboardVerticalOffset={125}
>
<GestureDetector gesture={composed}>
{isTransitioning ? (
<View style={styles.empty} />
) : (
<Canvas
ref={canvasRef}
onLayout={handleLayout}
mode="continuous"
style={styles.canvas}
>
{dstRct ? (
<Group clip={dstRct}>
<Image
image={image}
x={dstRct.x}
y={dstRct.y}
width={dstRct.width}
height={dstRct.height}
/>
{srcRect && dstRct ? (
<FitBox src={srcRect} fit="fill" dst={dstRct}>
<Path
strokeCap="round"
path={path}
strokeWidth={5}
style="stroke"
color="red"
/>
</FitBox>
) : null}
</Group>
) : null}
</Canvas>
)}
</GestureDetector>
<View style={styles.textInputContainer}>
<TextInput
placeholderTextColor="gray"
style={{ color: 'white' }}
$color="white"
placeholder={
isFinal ? '' : t('screens.image-preview.description.placeholder')
}
readOnly={isFinal}
hideError
hideBorder
multiline
/>
</View>
</KeyboardAvoidingView>
</SafeAreaView>
)
}
Probably related to https://github.com/Shopify/react-native-skia/issues/2133
Following up on this, I think I found a workaround for the issue @cesargdm.
Indeed I have the same issue and I first thought it would only happen in my development builds but it also happened in production, only for less powerful devices. It was probably visible in the development builds because they are a bit less performant, and something obviously tries to render the moment the snapshot is taken. I assume in most attempts it's fast enough to take the image after this drawing process.
I figured what if I just delay the makeImageScreenshot by one render cycle using setTimeout. And it works. I have no idea why 😅, but with over 100 attempts, I now can no longer reproduce the issue, so I believe it fixed it. Let me know if it works for you too.
Here's my code:
const onSavePress = useCallback(async () => {
//... some other stuff
setTimeout(() => {
const image = ref.current?.makeImageSnapshot();
if (!image) {
// handle error
return;
}
setSkiaImage(image);
}, 0);
}, [ref]);
Update: Unfortunately the hack doesn't work fully reliable after all.. It causes the issue to disappear when I have just a handful of elements in the canvas. But when I add say 20 images to the canvas, I still get the glitching.
Feels to me like some sort of race condition where the snapshot happens just before the drawing on the canvas is finished?
this bug looks interesting and I will investigate it further but in the meantime, I recommend to draw the content offscreen and take a snapshot of that: https://shopify.github.io/react-native-skia/docs/animations/textures#usetexture
Thanks @wcandillon, indeed that's what I planned to do next... I'll keep you posted
Gang, I think that this PR may fix the issue: https://github.com/Shopify/react-native-skia/pull/2254, any chance you could try this patch on your app?
nice! Probably dumb question but do I need to build skia locally to do that? If I install from your branch it get's added as react-native-skia-dev-tools.
that's a fair question, I was asking just in case, we will publish this bug fix soon so you will be able to test on the publish version. But now I am realizing that these a not related most likely. I will try to investigate this particular issue still
On Mon, Feb 26, 2024 at 10:18 PM ludwighen @.***> wrote:
nice! Probably dumb question but do I need to build skia locally to do that? If I install from your branch it get's added as react-native-skia-dev-tools.
— Reply to this email directly, view it on GitHub or unsubscribe. You are receiving this email because you commented on the thread.
Triage notifications on the go with GitHub Mobile for iOS or Android.
So I tested your fix with 0.1.241 beta but the issue is still there.
Next I tried the useTexture approach but I cannot get the basic example working. It throws an error: Value is undefined, expected an Object at root.render(element); in Offscreen.tsx.
const texture = useTexture(
<Fill color="cyan" />,
{ width, height }
);
Any idea why?
While looking into all of this, I was wondering if my current approach is even the "right" way, I think I should switch to the imperative API. I know it's not the right thread for it, but given that it would likely mean a significant refactor for me, maybe you could point in me the right direction since the documentation is a bit limited for the imperative API 🥲.
Basically, I have a canvas that can contain lots of different elements from the user (images, svgs, paths, etc.), think something like the Instagram stories editor / stickers example. I use the declarative API like this:
<Canvas>
{elements.map((element, index) => (
<CanvasElement
gestureInfo={gestureInfo[element.uid]}
key={element.uid}
element={element}
canvasWidth={canvasWidth}
canvasHeight={canvasHeight}
clock={clock}
color={element.color}
/>
))}
</Canvas>
And then essentially the CanvasElement conditionally renders Skia images, svgs, paths, etc. and applies transforms from GestureHandlers.
The useTexture hook would not allow me to pass the same logic in, would it? Should I instead create all of the conditional drawing with the imperative API and just display the results on screen? Also performance wise, is there a difference between declarative vs imperative? I can have up to 100 elements/images drawn on the canvas in my case. Any pointers would be welcome! I also see quite a few people out there that use Skia for a creative "editor" with a preview and an export stage, it could be cool to have resources on how to handle offscreen vs onscreen.
Anyways, thanks a ton @wcandillon as always 🙏
Update: I finally was able to solve it (this time reliably @sravis02 @cesargdm).
Not using the canvas ref's makeImageSnapshot() but instead drawing offscreen indeed fixes the problem and it also revealed why the items glitched small in the top left corner. It has something to do with the scale/pixel density.
What I did is just putting the Canvas children 1 to 1 but it has the caveat that the sizing is not identical to the onscreen canvas:
import { drawAsImage } from '@shopify/react-native-skia';
const { scale } = useWindowDimensions();
const scaledHeight = canvasHeight * scale;
const scaledWidth = canvasWidth * scale;
const image = drawAsImage(
<>
<Rect
x={0}
y={0}
height={scaledHeight}
width={scaledWidth}
color={color}
}
/>
{elements.map((element, index) => (
<CanvasElement
gestureInfo={gestureInfo[element.uid]}
//... I also needed to adjust the scale of width/height of my elements here
/>
))}
</>,
{
width: scaledWidth,
height: scaledHeight,
},
);
const base64 = image.encodeToBase64();
So basically, when I use the normal canvasWidth/canvasHeight (via height/width from useWindowDimensions), the offscreen canvas is very small because it uses them as absolute pixel values whereas with the declarative API the pixels are scaled according to device (i.e. 3x for iPhones).
So the little glitch we see when using the ref's makeImageSnapshot method is basically the non-upscaled dimensions. Does this somehow help you in debugging @wcandillon ?
Hope that helps
Thank you for this. I filed a seperate bug report for the offscreen scaling: https://github.com/Shopify/react-native-skia/issues/2336 This is gonna sound silly but I knew that of course these need to be scaled to pixel density but couldn't build an example where I would notice the difference: https://github.com/Shopify/react-native-skia/blob/main/package/src/renderer/Offscreen.tsx#L35
But I will definitely revisit.
For makeImageSnapshot, we know offer makeImageSnapshotAsync that runs on the UI thread so it shares the same Skia context than your onscreen canvas: https://shopify.github.io/react-native-skia/docs/canvas/overview/#getting-a-canvas-snapshot