dig-etl-engine

dig-etl-engine copied to clipboard

New project creation

[]

while creating new project, enable "Hit Enter" to save

saggu

Add support in myDIG for different XSD dates

[{"_id":"63615053cecf4e081a26986a","body":"Assigning to @GreatYYX to create new entries in myDIG for these types and to @saggu to handle the different formats in ETK.","issue_id":1662033440797,"origin_id":345425084,"user_origin_id":1074599,"create_time":1510990678,"update_time":1510990678,"id":1667321939771,"updated_at":"2022-11-01T16:58:59.770000Z","created_at":"2022-11-01T16:58:59.770000Z"},{"_id":"63615053cecf4e081a26986b","body":"@szeke what is this about ?","issue_id":1662033440797,"origin_id":404336613,"user_origin_id":6811931,"create_time":1531350001,"update_time":1531350001,"id":1667321939773,"updated_at":"2022-11-01T16:58:59.773000Z","created_at":"2022-11-01T16:58:59.773000Z"},{"_id":"63615053cecf4e081a26986c","body":"This is about the ability to store dates of different resolution. It is supported in etk2, but not in myDIG or DIG UI, this is important.","issue_id":1662033440797,"origin_id":404349824,"user_origin_id":1074599,"create_time":1531354658,"update_time":1531354658,"id":1667321939776,"updated_at":"2022-11-01T16:58:59.775000Z","created_at":"2022-11-01T16:58:59.775000Z"}]

Need separate myDIG types for year, year/month, year/month/day, year/month/day/time, so that we can properly parse and display dates to users

szeke

Implement a new action `delete` in etk filters.

[{"_id":"63614fd48041c95dfb1c11a1","body":"@szeke still relevant ?","issue_id":1662033440803,"origin_id":404337318,"user_origin_id":6811931,"create_time":1531350208,"update_time":1531350208,"id":1667321812194,"updated_at":"2022-11-01T16:56:52.194000Z","created_at":"2022-11-01T16:56:52.194000Z"},{"_id":"63614fd48041c95dfb1c11a2","body":"I forgot how the filters work, did we implement them?","issue_id":1662033440803,"origin_id":404349617,"user_origin_id":1074599,"create_time":1531354572,"update_time":1531354572,"id":1667321812197,"updated_at":"2022-11-01T16:56:52.197000Z","created_at":"2022-11-01T16:56:52.197000Z"}]

If mydig sees a doc, which it got from ES, with action `delete` set, it should delete it from the ES if mydig sees a doc with action `discard` it...

saggu

Define base class for custom extractors

[{"_id":"63615560ea01ec786e81d822","body":"Obsolete in ETK2","issue_id":1662033440805,"origin_id":404340038,"user_origin_id":6811931,"create_time":1531351145,"update_time":1531351145,"id":1667323232412,"updated_at":"2022-11-01T17:20:32.411000Z","created_at":"2022-11-01T17:20:32.411000Z"}]

At a minimum, custom extractors need a name that will appear in the myDIG menu of extractors.

szeke

Add ability to combine columns using templates

[{"_id":"63615d983056137e26644b2e","body":"Added `template` feature to combine fields.\r\n\r\nNot closing because we may need something smarter for the cases when some fields are missing, eg, sometimes we are supposed to have year, month and day, but day is missing so a simple template creates illegal dates.","issue_id":1662033440808,"origin_id":357878685,"user_origin_id":1074599,"create_time":1516088879,"update_time":1516088879,"id":1667325336813,"updated_at":"2022-11-01T17:55:36.813000Z","created_at":"2022-11-01T17:55:36.813000Z"},{"_id":"63615d983056137e26644b2f","body":"WIll keep this open, it should be fixed in etk2","issue_id":1662033440808,"origin_id":404333501,"user_origin_id":6811931,"create_time":1531349023,"update_time":1531349023,"id":1667325336817,"updated_at":"2022-11-01T17:55:36.817000Z","created_at":"2022-11-01T17:55:36.817000Z"}]

There are CSV files where the values are split over multiple columns and we need them as one column. For example, there can be `year`, `month` and `day` columns and...

szeke

Ensure that DIG ETK pipeline remains active

[{"_id":"63615e0fd297b621323af088","body":"This is working for sage news, but not for other sage projects.","issue_id":1662033440810,"origin_id":404332947,"user_origin_id":6811931,"create_time":1531348860,"update_time":1531348860,"id":1667325455771,"updated_at":"2022-11-01T17:57:35.770000Z","created_at":"2022-11-01T17:57:35.770000Z"}]

By default, ETK pipeline turns off after one hour. Make sure that for SAGE it is set to never exit.

szeke

Ensure that dates/times are parsed in DIG

[{"_id":"636161a38041c95dfb1c1acc","body":"@adityasundaram status?","issue_id":1662033440812,"origin_id":377379872,"user_origin_id":6811931,"create_time":1522359354,"update_time":1522359354,"id":1667326371751,"updated_at":"2022-11-01T18:12:51.751000Z","created_at":"2022-11-01T18:12:51.751000Z"},{"_id":"636161a38041c95dfb1c1acd","body":"@szeke is this still relevant ?","issue_id":1662033440812,"origin_id":404332739,"user_origin_id":6811931,"create_time":1531348805,"update_time":1531348805,"id":1667326371754,"updated_at":"2022-11-01T18:12:51.754000Z","created_at":"2022-11-01T18:12:51.754000Z"}]

Looks like DIG does not accept zulu times (trailing z) Verify that ES is correctly mapping the dates so that date queries work. Check the mapping file in ES to...

szeke

Create dialog to select extractors for a segment

[{"_id":"6361530ad297b621323aea20","body":"\r\n","issue_id":1662033440814,"origin_id":334305043,"user_origin_id":1074599,"create_time":1507155261,"update_time":1507155261,"id":1667322634476,"updated_at":"2022-11-01T17:10:34.475000Z","created_at":"2022-11-01T17:10:34.475000Z"},{"_id":"6361530ad297b621323aea21","body":"@szeke still to do ?","issue_id":1662033440814,"origin_id":377378416,"user_origin_id":6811931,"create_time":1522359011,"update_time":1522359011,"id":1667322634479,"updated_at":"2022-11-01T17:10:34.479000Z","created_at":"2022-11-01T17:10:34.479000Z"},{"_id":"6361530ad297b621323aea22","body":"Keep open, with ETK2 this will be possible to do, although we need to rethink the GUI","issue_id":1662033440814,"origin_id":377539333,"user_origin_id":1074599,"create_time":1522421134,"update_time":1522421134,"id":1667322634482,"updated_at":"2022-11-01T17:10:34.482000Z","created_at":"2022-11-01T17:10:34.482000Z"}]

Shows a dialog that has a checkbox for each extractor.

szeke

Add field attribute called "Ranking Multiplier"

[{"_id":"636159548041c95dfb1c1708","body":"There should be two UI elements for each field. One that enables the ranking multiplier field (like a check box) and then a text box for the value that defaults to 1.0 which is for the user to specify the attribute value. I've called the attribute `scoring_coefficient` instead of `ranking_multiplier`.","issue_id":1662033440816,"origin_id":376742176,"user_origin_id":5325227,"create_time":1522205070,"update_time":1522205070,"id":1667324244077,"updated_at":"2022-11-01T17:37:24.077000Z","created_at":"2022-11-01T17:37:24.077000Z"}]

Each field should have an attribute called `Ranking Multiplier` and the value can be a float number that defaults to `1.0`. Internally, the attribute can be called `ranking_multiplier` or whatever...

szeke

Cursor Positions and Form submit on Enter

[]

Cursor to be positioned on beginning of each dialog on open. Enter submits the form.

shrev