Store the resources in S3 buckets (data file and revisions offset files)

[JohannesLichtenberger]

As an alternative backend or a combination of a local store and an S3 bucket store we could add a new storage type using JClouds blob store for instance (as in Treetank, the former project from which this project was forked).

[sudip-unb]

Hi @JohannesLichtenberger I would like to work on this issue.

[JohannesLichtenberger]

@sudip-unb did you make any advances or do you need help?

[sudip-unb]

Hi @JohannesLichtenberger please give me a little bit time. I am currently busy for my exam. I already explored the link you have provided me. I can start development from this weekend.

[JohannesLichtenberger]

Sure, just trying to ping assigned people because usually it's likely that they do not have time at all in my experience ;-) but glad that you're starting to work on it afterward. Take your time -- as said, I just wanted to make sure that you're still up for the task :-)

[JohannesLichtenberger]

@sudip-unb any news? :)

[Yashendr]

Hi can I help out with this issue?

[JohannesLichtenberger]

@Yashendr you can have a look at Treetank (https://github.com/sebastiangraf/treetank) where Sebastian already implemented such a backend (also a combined storage)...

[JohannesLichtenberger]

https://github.com/sebastiangraf/treetank/tree/master/coremodules/core/src/main/java/org/treetank/io/jclouds

[sband]

Hi @JohannesLichtenberger ,

I went through the org.sirix.io package to understand the storage types that SirixDB has currently. From what I have understood, FileStorage is already supported, however the above issue speaks about the combination of local file storage and S3 bucket store. Could you please elaborate more on this ask ? Or is it just that a support for S3 storage is needed as per the issue ? Once I understand this, I'd be happy to contribute my bit to this project. However, for starters here are my approaches of how I could implement this:

1. Using the FileWriter write data and revision files on the filesystem and copy them to S3. (I may be completely wrong about this given that I have not looked at the logic of how the page references are written.
2. Extend the current logic to write on to S3 object store directly a. Externalize the configuration to be able to provide AWS keys and S3 bucket details b. Create a connection to S3 bucket using the above configuration (in point "a") c. Using AWS SDK APIs, use the current logic to write the page references on to S3 object directly.

In approach 2 I could refer/use the jclouds package you have mentioned

Let me know what you think about these approaches.

[JohannesLichtenberger]

I'd probably see it as a kind of automatic backup. A local file based store and an async store via JClouds. IIRC the pure S3 storage was way too slow. So, in short I prefer your second option. To combine the storage approaches we can implement something as simple as this combined storage: https://github.com/sebastiangraf/treetank/tree/master/coremodules/core/src/main/java/org/treetank/io/combined

BTW: If you dig a bit deeper into the storage mechanism (simply store word aligned page fragments instead of same sized full pages), it would also be interesting to find out, why the iouring based backend currently is slower than the simple file channel based solution and the memory mapped backend (I think somehow because of the event loop)...

[sband]

Ok looking at the above for combined, here is my understanding of the requirement: We should write a Storage type that would be a combination of two storages -

1. Local - Could be any type that is currently supported in SirixDB - for example file channel, file, iouring OR memory mapped
2. remote storage - this asynchronously writes to remote storage - For now S3, in future this could be Azure Blob store OR GCP blob store So here is plain english implementation detail:
3. Write a class to facilitate remote storage
4. This class reads the cloud platform type
5. Based on the cloud platform, reads appropriate properties from a config file to create a connection
6. Contains methods to read and write from remote storage. A CombinedStorage class (similar to the treetank url above) will encapsulate the SirixDB storage type for local storage and the above remote storage class with methods to write asynchronously to the remote storage using Executors framework and leveraging the read/write methods of the respective local storage used in this class. It could literally be this class with some tweaks that are suitable for SirixDB

[sband]

BTW: If you dig a bit deeper into the storage mechanism (simply store word aligned page fragments instead of same sized full pages), it would also be interesting to find out, why the iouring based backend currently is slower than the simple file channel based solution and the memory mapped backend (I think somehow because of the event loop)...

I Could take this up as a different task may be ?

[JohannesLichtenberger]

Thanks for working on this :-) and probably the upcoming task

[Yashendr]

Ok looking at the above for combined, here is my understanding of the requirement: We should write a Storage type that would be a combination of two storages -

1. Local - Could be any type that is currently supported in SirixDB - for example file channel, file, iouring OR memory mapped

2. remote storage - this asynchronously writes to remote storage - For now S3, in future this could be Azure Blob store OR GCP blob store
   So here is plain english implementation detail:

3. Write a class to facilitate remote storage

4. This class reads the cloud platform type

5. Based on the cloud platform, reads appropriate properties from a config file to create a connection

6. Contains methods to read and write from remote storage.
   A CombinedStorage class (similar to the treetank url above) will encapsulate the SirixDB storage type for local storage and the above remote storage class with methods to write asynchronously to the remote storage using Executors framework and leveraging the read/write methods of the respective local storage used in this class. It could literally be [this ](https://github.com/sebastiangraf/treetank/blob/master/coremodules/core/src/main/java/org/treetank/io/combined/CombinedStorage.java) class  with some tweaks that are suitable for SirixDB

@sband Hey do you need help with this. Do you have anything in particular you would like me to do?

[sband]

@Yashendr sure I will let you know if I need any help around this.

[sband]

Hi @JohannesLichtenberger ,

This is in progress. I am hoping to complete this by coming friday...

[sband]

hi @JohannesLichtenberger

quick question:

1. For coding the reader that is used in as part of the Cloud storage, for instance AWS, my approach is to get the object from S3(remote storage in this case). I would get the data in bytes. Should I write this byte data into a local file by using the File Storage and then use the FileReader ? OR should I just leverage the same reader as that would be used by the user in the CombinedStorage class that I would be writing ? - I would prefer the later to avoid any confusion, but please let me know.
2. Same question above for writer.
3. For now I am writing code to support AWS only, is that ok ?

[JohannesLichtenberger]

Hi @sband, I'd simply read-write the page(-fragments) into S3 buckets. If we want a local cache and/or use S3 as a backup more or less I'd use the CombinedStorage.

AWS is okay :-) in the future, we could also support for instance writing to/reading from Kafka or Pulsar/BookKeeper...

However, what I'm even more interested in is making the local storage first of all as fast as possible before even using horizontal scaling/sharding... so I'd be rather interested why the IO-uring storage is currently on my Notebook at least slower in comparison to the FileChannel based approach.

[JohannesLichtenberger]

Furthermore, it's kind of sad that Intel Optane Non-Volatile Memory isn't produced anymore, as the page(-fragments) are not aligned to a predefined size, and thus, sometimes if only a few nodes are changed due to the sliding snapshot algorithm only mainly these nodes are written to a new location instead of the full page (thus generating a page-fragment). However, for iouring, I guess it would be great to have classes of page sizes and to use predefined buffers (as in Umbra from Thomas Neumann...).

[JohannesLichtenberger]

Thanks for working on SirixDB, BTW :-) really looking forward to your PR (and maybe future contributions?)

[sband]

hi @JohannesLichtenberger ,

I have created a DRAFT pull request for this https://github.com/sirixdb/sirix/pull/611 Could not complete the implementation as promised on this. However, please feel free to advice if you see that I am going in the wrong direction in terms of implementing this fix for the required use-case.

[ighosh98]

Hi, I see no activity on this for some time. If it's not being, actively worked on can I work on this issue?

[JohannesLichtenberger]

@ighosh98 do you intend to work on this?

[ighosh98]

Hi @JohannesLichtenberger , yes I will be working on this issue.

[JohannesLichtenberger]

@ighosh98 ping

[ighosh98]

Hi. I've been tied up with some work. It would take me some time to raise the PR. If someone else can develop it faster, they can take over.