baker icon indicating copy to clipboard operation
baker copied to clipboard

Add lua filter

Open arl opened this issue 3 years ago • 2 comments

:question: What

Add a LUA filter that allows to add filter logic (implement the Process function) in lua. Needs further testing/benchmarking/polishing but first results shows a relatively good performance with respect to the same filter coded in Go (5% slower for a filter with a single lua operation).

Every Lua baker filter compiles a given script and will run a user-specific function (which should have the following prototype):

-- rec is a record object
-- next is function next(record)
function myLuaFilterFunction(rec, next)

lua script knows the record type which is the lua-equivalent of the Record Go interface. record has the following methods:

  • get(int) string
  • set(int, string)
  • clear()
  • copy() record

In the lua context (called lua state), some utility functions are already defined and can be used:

  • validateRecord(record) bool, int
  • createRecord record
  • fieldByName(string) int|nil different from Go which returns (baker.FieldIndex, bool)
  • fieldNames which is a lua table, same as fieldNames Go slice

:hammer: How to test

  1. List all steps necessary;
  2. To test this pull request.

:white_check_mark: Checklists

This section contains a list of checklists for common uses, please delete the checklists that are useless for your current use case (or add another checklist if your use case isn't covered yet).

  • [x] Is there unit/integration test coverage for all new and/or changed functionality added in this PR?
  • [x] Have the changes in this PR been functionally tested?
  • [x] Have new components been added to the related all.go files?
  • [x] Have new components been added to the documentation website?
  • [x] Has make gofmt-write been run on the code?
  • [x] Has make govet been run on the code? Has the code been fixed accordingly to the output?
  • [x] Have the changes been added to the CHANGELOG.md file?
  • [x] Have the steps in CONTRIBUTING.md been followed to update a Go module?

arl avatar Dec 09 '20 11:12 arl

Codecov Report

Merging #110 (e5da03b) into main (3ff24dd) will increase coverage by 0.79%. The diff coverage is 82.29%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #110      +/-   ##
==========================================
+ Coverage   49.43%   50.22%   +0.79%     
==========================================
  Files          56       57       +1     
  Lines        3880     3976      +96     
==========================================
+ Hits         1918     1997      +79     
- Misses       1752     1764      +12     
- Partials      210      215       +5     
Impacted Files Coverage Δ
filter/lua.go 82.29% <82.29%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 3ff24dd...e5da03b. Read the comment docs.

codecov-io avatar Dec 12 '20 19:12 codecov-io

Codecov Report

Merging #110 (68a1060) into main (40a57bc) will decrease coverage by 3.75%. The diff coverage is 81.25%.

:exclamation: Current head 68a1060 differs from pull request most recent head 4fe5623. Consider uploading reports for the commit 4fe5623 to get more accurate results Impacted file tree graph

@@            Coverage Diff             @@
##             main     #110      +/-   ##
==========================================
- Coverage   53.95%   50.20%   -3.76%     
==========================================
  Files          60       57       -3     
  Lines        4003     3976      -27     
==========================================
- Hits         2160     1996     -164     
- Misses       1646     1764     +118     
- Partials      197      216      +19     
Impacted Files Coverage Δ
filter/lua.go 81.25% <81.25%> (ø)
filter/set_string_from_url.go 37.50% <0.00%> (-19.03%) :arrow_down:
topology.go 60.89% <0.00%> (-13.41%) :arrow_down:
config.go 65.00% <0.00%> (-13.00%) :arrow_down:
filter/metadata_lastmodified.go 76.19% <0.00%> (-12.70%) :arrow_down:
input/list.go 60.78% <0.00%> (-10.51%) :arrow_down:
filter/notnull.go 70.58% <0.00%> (-9.42%) :arrow_down:
filter/clear_fields.go 56.25% <0.00%> (-8.04%) :arrow_down:
filter/partial_clone.go 65.00% <0.00%> (-7.23%) :arrow_down:
filter/string_match.go 69.56% <0.00%> (-6.63%) :arrow_down:
... and 26 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update a209176...4fe5623. Read the comment docs.

codecov-commenter avatar Nov 15 '21 18:11 codecov-commenter