WIP: Adapter goes NodeJS

[georg-schwarz]

Open points

• [ ] Replace knex with our own postgres implementation
• [ ] Code styling and clean-up (Linter)

DONT MERGE YET!

[f3l1x98]

For this one I am not sure if there was something changed (concerning my TODO at the top) https://github.com/jvalue/ods/blob/4fe866fdb2228525e80e6414c946785166e7ff4a/adapter/src/adapter/api/rest/adapterEndpoint.ts#L46-L84

[f3l1x98]

Again as the TODO says, I am not sure why the previous implementation queried based on both the dataImport.id AND datasource.id. From my understanding the dataImport.id is unique in the whole DB not for a single datasource. https://github.com/jvalue/ods/blob/eb36b17fcf5a8910eceee5c14ff4895efc9864ca/adapter/src/datasource/repository/postgresDataImportRepository.ts#L97-L103

[f3l1x98]

This TODO I have kept, because I too am unsure whether that is necessary. (But I can't remember that we have done so in the other parts of ODS) https://github.com/jvalue/ods/blob/4fe866fdb2228525e80e6414c946785166e7ff4a/adapter/src/datasource/api/rest/dataImportEndpoint.ts#L43-L68

[f3l1x98]

This is the part that was rewritten from the old impl and I have kept it. Not sure if we want to rewrite it again to match old Java impl. https://github.com/jvalue/ods/blob/4fe866fdb2228525e80e6414c946785166e7ff4a/adapter/src/datasource/services/dataImportTriggerService.ts#L84-L88 https://github.com/jvalue/ods/blob/4fe866fdb2228525e80e6414c946785166e7ff4a/adapter/src/adapter/importer/HttpImporter.ts#L86-L100

[f3l1x98]

https://github.com/jvalue/ods/blob/4fe866fdb2228525e80e6414c946785166e7ff4a/adapter/src/adapter/importer/HttpImporter.ts#L102-L118 For this one I wanted to ask, whether somebody knows a better way, because I also do not know one. The old Java impl got the response as a byte[] and then created a new String from it using the passed encoding: https://github.com/jvalue/ods/blob/843943f87762db286600891e7cbe79eca9745441/adapter/src/main/java/org/jvalue/ods/adapterservice/adapter/importer/HttpImporter.java#L59-L61

[georg-schwarz]

https://github.com/jvalue/ods/blob/4fe866fdb2228525e80e6414c946785166e7ff4a/adapter/src/adapter/importer/HttpImporter.ts#L102-L118

For this one I wanted to ask, whether somebody knows a better way, because I also do not know one. The old Java impl got the response as a byte[] and then created a new String from it using the passed encoding: https://github.com/jvalue/ods/blob/843943f87762db286600891e7cbe79eca9745441/adapter/src/main/java/org/jvalue/ods/adapterservice/adapter/importer/HttpImporter.java#L59-L61

I think you would need to use fetch API instead of Axios... There you can get the byte stream and interpret based on encoding.

[georg-schwarz]

This is the part that was rewritten from the old impl and I have kept it. Not sure if we want to rewrite it again to match old Java impl.

https://github.com/jvalue/ods/blob/4fe866fdb2228525e80e6414c946785166e7ff4a/adapter/src/datasource/services/dataImportTriggerService.ts#L84-L88

https://github.com/jvalue/ods/blob/4fe866fdb2228525e80e6414c946785166e7ff4a/adapter/src/adapter/importer/HttpImporter.ts#L86-L100

I think the old solution was better. The importer should not know that there are differences in parameters but just blindly execute what it gets as input.

[georg-schwarz]

@f3l1x98 I refactored everything except the two open points I replied to. Would be nice if you could take care of them. CI is currently not working somehow for Adapter integration tests.. doesn't get the exit code somehow.

[f3l1x98]

https://github.com/jvalue/ods/blob/4fe866fdb2228525e80e6414c946785166e7ff4a/adapter/src/adapter/importer/HttpImporter.ts#L102-L118

For this one I wanted to ask, whether somebody knows a better way, because I also do not know one. The old Java impl got the response as a byte[] and then created a new String from it using the passed encoding: https://github.com/jvalue/ods/blob/843943f87762db286600891e7cbe79eca9745441/adapter/src/main/java/org/jvalue/ods/adapterservice/adapter/importer/HttpImporter.java#L59-L61

I think you would need to use fetch API instead of Axios... There you can get the byte stream and interpret based on encoding.

This is not quite possible, because the native fetch api is not available in Node.JS before version 18.0.0. But I have found a way in which axios returns its data as an ArrayBuffer so I have choosen to use this together with the TextDecoder for converting to string: https://github.com/jvalue/ods/blob/d6c6b2e2c00763038e5d72bbcd0aa1067ffacb26/adapter/src/adapter/importer/HttpImporter.ts#L69-L82

[f3l1x98]

@georg-schwarz I have changed it, but the first issue I have solved differently than proposed, so please have a look at the comment above: https://github.com/jvalue/ods/pull/408#issuecomment-1223929981 The other issue has been solved: https://github.com/jvalue/ods/blob/d6c6b2e2c00763038e5d72bbcd0aa1067ffacb26/adapter/src/datasource/DataImportTriggerService.ts#L86-L87

[georg-schwarz]

LGTM!

Do you want to tackle the CI issue as well? I can also do it but will get to it end of week soonest.