Archery icon indicating copy to clipboard operation
Archery copied to clipboard

脱敏查询虽然支持union/union all ,但不支持count

Open liujiashun0520 opened this issue 3 years ago • 1 comments

重现步骤

select count() as phone_cnt from t1 like '7%' UNION ALL select count() as phone_cnt from t2 like '7%' UNION ALL select count() as phone_cnt from t3 where phone_no like '7%' UNION ALL select count() as phone_cnt from t4 where phone_no like '7%';

预期外的结果

Error

数据脱敏异常:'field'

日志文本

No response

版本

1.8.4

部署方式

手工部署

是否还有其他可以辅助定位问题的信息?比如数据库版本等

No response

liujiashun0520 avatar Jul 13 '22 07:07 liujiashun0520

您好 这个问题我复现了,这个报错是

[2022-07-20 11:28:00,171][Thread-1:139749714147072][task_id:default][data_masking.py:51][WARNING]- 数据脱敏异常,错误信息:Traceback (most recent call last):
  File "/u01/bianque/sql/utils/data_masking.py", line 31, in data_masking
    select_list = del_repeat(select_list, keywords_count) if keywords_count else select_list
  File "/u01/bianque/sql/utils/data_masking.py", line 71, in del_repeat
    result_index = df.groupby(['field']).filter(lambda g: len(g) > 1).to_dict('records')
  File "/u01/venv4archery/lib/python3.9/site-packages/pandas/core/frame.py", line 6515, in groupby
    return DataFrameGroupBy(
  File "/u01/venv4archery/lib/python3.9/site-packages/pandas/core/groupby/groupby.py", line 525, in __init__
    grouper, exclusions, obj = get_grouper(
  File "/u01/venv4archery/lib/python3.9/site-packages/pandas/core/groupby/grouper.py", line 786, in get_grouper
    raise KeyError(gpr)
KeyError: 'field'

[2022-07-20 11:28:00,171][Thread-1:139749714147072][task_id:default][query.py:125][WARNING]- 数据脱敏异常,按照配置放行,查询语句:select count(*) as phone_cnt  from users  t1 where phone like '%1%'
UNION ALL
select count(*) as phone_cnt   from users t2 where phone like '%2%'
UNION ALL
select count(*) as  phone_cnt   from users  t3  where phone like '%3%'
UNION ALL
select count(*) as phone_cnt   from users t4 where phone like '%4%' limit 100;,错误信息:'field'

这个问题是因为goInception返回结果是null

(venv4archery) [root@mysql01 goInception-v1.3.0-1]# python  test1.py 
((1, 'select count(*) from  users', 0, '[]', None),)
+----+-----------------------------+----------+------------+--------+
| id |          statement          | errlevel | query_tree | errmsg |
+----+-----------------------------+----------+------------+--------+
| 1  | select count(*) from  users |    0     |     []     |  None  |

相关issue https://github.com/hanchuanchuan/goInception/issues/457

并且前端展示是正常的 image 所以报错问题不大,

当查询字段包含脱敏列,前端正常脱敏并且后端无报错 image

下一步: 1 优化 goInception count的返回值
2 优化程序对于count返回值处理 2

unknowissue avatar Jul 20 '22 06:07 unknowissue