spark-acid icon indicating copy to clipboard operation
spark-acid copied to clipboard

Published 20 hours ago • verified publisher icon qubole

Revisit read Txn and Lock table for read operations

Open citrusraj opened this issue 5 years ago • 1 comments

Currently read does not acquire locks due to life time of RDD is unspecified hence would lead to blocking of the InsertOverwrites. So protecting it against the DDL / Truncate / Compaction is operational issue as it stands.

Currently whenever DF materialises first time the transactional snapshot of read is acquired at that time.

Above two need rethinking ..

citrusraj avatar Dec 19 '19 04:12 citrusraj

https://github.com/qubole/spark-acid/commit/2e66f76b3a3c974a73f03d9e80d9c66a0475a552

This commit adds frame work to perform operation under transactions.

citrusraj avatar Jan 16 '20 12:01 citrusraj