Travers

I'm not familiar with databricks, but if it's just a mount point I'm surprised. Was there an error message or was it silent? Is it reproducible with smaller objects?

Thanks, I'd like to understand more. What's the easiest way to set up a databricks system?

I signed up for the community edition, but I'm not able to repro the issue (see below). Any ideas? Since you're working with very large data, it could be an...

I'm not sure what the issue is, going to need more info. Did you test this on the community edition, what's the difference between your set up and the community...

Thank you for digging that up. I didn't even realize it's possible for a file system to not support random writes lol. I'll try to investigate a bit more later...

I think you're right. The community edition worked on all versions (maybe not a real FUSE drive), and I didn't get around to spinning up an Azure system. The new...

I loaded up Azure and did some testing finally. ``` devtools::install_github("traversc/stringfish") # dependancy, necessary to use github version for other reasons devtools::install_github("traversc/qs") library(qs) qsave(1, file="/dbfs/temp.qs") ``` Should give you an...

Interesting project! Thanks for letting me know about it!

I would be opposed to building in metadata directly to the qs file format, because I see it as more of a convenience feature (since you could simply store metadata...

It is a good idea, but CRAN doesn't allow using R-connections directly within C code. Glad you found a workaround!