Combining tag_facets() with plot_annotation()

[RS-eco]

Is there an easy way to combine tag_facets() with plot_annotation() when combining a facet plot with another plot?

library(ggplot2); library(tagger); library(patchwork)

# Create facet_plot with tags
(p1 <- ggplot(mpg, aes(displ, cty)) + geom_point() + 
  facet_grid(rows = vars(drv)) + tag_facets())

# Create additional plot
p2 <- ggplot(mtcars) + geom_boxplot(aes(gear, disp, group = gear))

# Combine plots and add tags to 2nd plot
p1 + p2 + plot_annotation(tag_levels = 'a')

Right now, this would give me two different tags for the 1st plot and the wrong label for the 2nd plot (should be letter d). Is there a way to specify the starting letter, similar to tag_pool in tag_facets(), in plot_annotation? And is there a way to specify, which of the patchwork plots should be annotated and which not?