spacyr
spacyr copied to clipboard
nounphrase_consolidate() gives unexpected results
spacyr
has been great to learn! Thanks for making this available. I might be doing something wrong but sometimes nounphrase_consolidate()
gives me unexpected results. It seems to be related to how much data spacyr
has to process. In the results below, sometimes processing more rows gives the expected results (orange) and sometimes processing fewer rows gives the expected results (green). I hope you can shed some light on this for me.
library(tidyverse)
library(spacyr)
odd_words <- "right|most|paw|clear"
df <-
tibble(
doc_id = seq_along(sentences),
text = tolower(sentences)
)
all_rows <-
df |>
spacy_parse(nounphrase = TRUE) |>
nounphrase_consolidate()
filtered_rows <-
df |>
filter(str_detect(text, odd_words)) |> # <--- this line is different
spacy_parse(nounphrase = TRUE) |>
nounphrase_consolidate()
filtered_rows |>
select(doc_id:token) |>
left_join(
select(all_rows,doc_id, sentence_id, token_id, token2 = token)
) |>
filter(token != token2)