nsq_to_file: invalid cross-device link when moving from work-dir to output-dir

[dselivanov]

I'm trying to use nsq_to_file to sync messages to S3. So I use --work-dir option: --work-dir=/nsq_logs/nsq_logs_workdir to store intermediate messages to local disk and and --output-dir=/nsq_logs/nsq_logs_store where /nsq_logs/nsq_logs_store is S3 an bucket mounted as locally using s3fs.

Logs however can't be moved to /nsq_logs/nsq_logs_store and I'm getting following FATAL exception invalid cross-device link. I believe this is due how exclusiveRename works.

Would it makes sense to catch such exceptions and use file copy instead of os.Link?

EDIT: Something like here

[mreiferson]

We use os.Link to ensure atomicity for filesystems that support it. My recommendation here would be to have something else responsible for uploading to S3 and cleaning up the --output-dir.

[dselivanov]

Thanks Matt, I've switched to external script for now. Do you think it worth to handle cross-device file copy/move within nsq_to_file?

[mreiferson]

I don't think so — we would have to rethink the behavior of other parts of the code given the lack of atomicity. However, we could consider at least documenting the fact that this expects --work-dir and --output-dir to be on the same device.

[dselivanov]

After trying several solutions I discovered https://www.benthos.dev/ which plays really well with nsq and share somewhat similar philosophy with nsq. It can buffer, batch, encode messages and route to different sinks. Highly recommend for those who looks to stream messages to s3.