Adding a new mapper type isn't possible.

[dhirendrafkl]

While we want to add new Mapper that would do mapping based on some conditional logic we found out that due to MapperType being an Enum, it isn't possible to customize it to add new mapper.

[aryaKetan]

Can you elaborate on what conditional logic you want to implement. We can have a discussion around that.

[dhirendrafkl]

Example

Table1 EntityId,Key,Value

Table2 EntityId,...

In table1 there are some keys which in merged table would become columns.

MergedTable EntityId,KeyX,KeyY,...

We wanted this type of customization which wasn't possible due to aforementioned problem.

[aryaKetan]

So, let me see if I got the requirement correct.

You have table 1 as

ACC1, EMAIL, [email protected] ACC1, PHONE, 900 ACC2, EMAIL, [email protected]

and you want to transform it into a table MergedTable as follows

ACC1 ,[email protected],900 ACC2,[email protected],NULL

Is my understanding correct?

[dhirendrafkl]

Yes. Absolutely right. On Aug 13, 2015 8:53 AM, "Arya Ketan" [email protected] wrote:

So, let me see if I got the requirement correct.

You have table 1 as

ACC1, EMAIL, [email protected] ACC1, PHONE, 900 ACC2, EMAIL, [email protected]

and you want to transform it into a table MergedTable as follows

ACC1 ,[email protected],900 ACC2,[email protected],NULL

Is my understanding correct?

— Reply to this email directly or view it on GitHub https://github.com/Flipkart/aesop/issues/28#issuecomment-130516845.

---

This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Although Flipkart has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments

[aryaKetan]

Ok. So the problem with your ask is that data is to transform your data from EAV model to row model. This may work only if updates/deletes to both rows of ACC1 happen in the same transaction. Even then, it would be complex to merge the rows into 1 row. But more or less , updates within the same txn will never happen and there-fore you would have to maintain state for ACC1 for you to merge.

But actually, to do what you want to do, you dont need a mapper type interface. This login can be implemented in the destination client processor. Say for e.g. you want to insert into mysql the merged table.

just keep firing queries such as update MergedTable set KeyX='Val' where entityId = 'ACC1'; when you get KeyY, your query would be like update MergedTable set KeyY='Val2' where entityId = 'ACC1';

and your Merged Table is ready.

Say if you want to insert the merged table into HBase, then keep adding columns in the column family for the rowKey ACC1. .

This can be done for any of the databases.

[anshul10s]

This is interesting approach Arya, works good for consumer stores which can accept data in columnar fashion and support ACID transaction.

Was wondering what will be the approach if I want to index these 2 columns in ES for a given entity. I can model it as 2 different child documents of entity, so that updates can be independent or read the document and update it.

Reading and updating will have consistency issues in corner cases, as ES is does not support ACID transaction. Another approach can be to treat these as a change singal and query the source of truth for all rows related to ACC1 and create the complete document for indexing.

[aryaKetan]

So, if we want to model our destination store in ES under the same conditions. So the elasic search doc is like { "keyX" : "Val1", "keyV" : "Val2" } cant we can do this by running groovy scripts for update or using the UpdateRequest ( in java client) ? We dont need to read doc and update. Now, even if you are reading it and updating, there is version support in ES update and if version does not match, it can return error

IMO ,this will not be doable only in K-V stores where-in in-place updates are not possible. ( couchbase - membase bucket)

[dhirendrafkl]

We've already overriden and solved it. Only concern is if someone wants to add a mapper for some specific use case this won't be possible.

On Thu, Aug 13, 2015 at 11:51 AM, Arya Ketan [email protected] wrote:

So, if we want to model our destination store in ES under the same conditions. So the elasic search doc is like { "keyX" : "Val1", "keyV" : "Val2" } cant we can do this by running groovy scripts for update or using the UpdateRequest ( in java client) ? We dont need to read doc and update. Now, even if you are reading it and updating, there is version support in ES update and if version does not match, it can return error

IMO ,this will not be doable only in K-V stores where-in in-place updates are not possible. ( couchbase - membase bucket)

— Reply to this email directly or view it on GitHub https://github.com/Flipkart/aesop/issues/28#issuecomment-130551054.

---

This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Although Flipkart has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments

[regunathb]

@dhirendrafkl do you have a fix that will address this issue? Happy to accept a pull request if the solution is generic enough.

[dhirendrafkl]

@regunathb Solution was implemented seperately without forking from aesop. This can be found at https://github.com/Flipkart/erp-change-propagation/tree/master/fklogistics-migration/fklogistics-hbase-consumer