datasets icon indicating copy to clipboard operation
datasets copied to clipboard

Labor seems to have structural issues

Open acbart opened this issue 9 years ago • 2 comments

Reevaluate it, make sure everything has the right type and that there are no redundant key paths?

The big things here:

  • [ ] "unit" should be stored as a comment
  • [ ] I believe that there might be some inconsistent dictionaries (I don't entirely recall what that means, my report generator needs better documentation; but I think it means that the dictionaries are repeated but with different internal structure, which wrecks havoc on the Java version): "White": "result.[0].data.Civilian noninstitutional population.White.Women", "Black or African American": "result.[0].data.Civilian noninstitutional population.Black or African American.Women"
  • [ ] The following keys are dictionaries in some cases and in other cases numbers: "result.[0].data.Civilian labor force participation rate.Asian.All" "result.[0].data.Employment-population ratio.Asian.All" "result.[0].data.Not in labor force.Asian.All" "result.[0].data.Unemployed.Asian.All" "result.[0].data.Civilian labor force.Asian.All" "result.[0].data.Employed.Asian.All" "result.[0].data.Not in labor force.White.All" "result.[0].data.Civilian noninstitutional population.Asian.All" "result.[0].data.Unemployment rate.Asian.All"
  • [ ] I believe the above is accounted for the fact that Asians don't have the same gender-level data available for all the years. If that's the case, we must either A) impute the missing demographic data with -1 or estimated values, B) trim the data from those years completely, C) Only report total for Asians.

acbart avatar Sep 26 '16 01:09 acbart

When there was no data present for certain demographic groups, this usually occurred with Asian populations, for part of the time range the data was incorrectly input. Instead of always having a dictionary of units and value only the value of 0 would be present. This is what was causing the inconsistent dictionaries.

Let me know if this is what you were looking for as a final structure

RyanWhitcomb-VT avatar Jan 09 '17 21:01 RyanWhitcomb-VT

Definitely don't want an inconsistent structure. I'd say we should probably put in 0 for the leaf fields. Do we have a sense of what percentage of data is missing here?

acbart avatar Jan 15 '17 20:01 acbart