Error spans incorrect when using `flatten`

[wez]

Here's a link to my example/repro:

https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=5651e21bcb73e6fabf7e1d0ef67660d2

and inline for convenience:

use serde::Deserialize;
use std::collections::HashMap;

#[derive(Deserialize, Debug)]
struct Thing {
   #[serde(flatten)]
   entries: HashMap<String, Entry>,
}

#[derive(Deserialize, Debug)]
struct Entry {
   foo: u32,
}

fn main() {
    let data = "# comment line\n[\"default\"]\nfooo = 42";
    match toml::from_str::<Thing>(&data) {
        Ok(data) => println!("{data:#?}"),
        Err(err) => println!("{err:#}"),
    }
}

produces:

TOML parse error at line 1, column 1
  |
1 | # comment line
  | ^
missing field `foo`

I would expect the line number to be 3 rather than 1, and for the context in the error message to refer to the appropriate part of the file.

[epage]

Without debugging this, my best guess is that something in the way our code or serde is structured is preventing us from associating the foo error with the default table, making us associate it with the whole document which puts the error at the beginning of the file.

[soywod]

I have the same issue with a #[serde(flatten)] on a HashMap as well, which is a real problem for my tool because any wrong option points to the whole document:

#[derive(Clone, Debug, Default, Eq, PartialEq, Deserialize, Serialize)]
#[serde(rename_all = "kebab-case")]
pub struct TomlConfig {
    #[serde(alias = "name")]
    pub display_name: Option<String>,
    pub signature: Option<String>,
    pub signature_delim: Option<String>,
    pub downloads_dir: Option<PathBuf>,

    #[serde(flatten)]
    pub accounts: HashMap<String, TomlAccountConfig>,
}

[example]
backend = "imapp"

Error: cannot parse config file at "~/code/himalaya/config.sample.toml"

Caused by:
    TOML parse error at line 1, column 1
      |
    1 | [example]
      | ^
    unknown variant `imapp`, expected one of `maildir`, `imap`, `smtp`, `sendmail`

[baszalmstra]

I ran into this as well. I think it has something to do with flatten. See https://www.rustexplorer.com/b/uspq0k for a minimal reproducible case.

[epage]

If I'm reading the serde_derive expanded code, it looks like this is a problem with how serde_derive deserializes things.

serde_derive seems to flatten content into something like serde_value (which can't error) and then deserializes into the flattened type (can error). This means the error is coming out of the deserializer for the outer table, and not on the individual field, and that is all toml/toml_edit can rely on for associating it with the code.

[soywod]

Indeed, looks like a know issue at serde_derive, see https://github.com/serde-rs/serde/issues/2581 and https://github.com/serde-rs/serde/issues/1183. Should we close this issue or wait for serde_derive to fix it before closing it?

[epage]

I think keeping it open but blocked on serde will better help people discover it.

[epage]

I've generalized the title as they are all the same root cause. There isn't a point in fracturing the conversation between multiple issues.