"Out of memory" with large files

[philipdexter]

Is there a maximum supported size for files? I get the following error when trying to ls a file with size of 1.5gb:

fs9p [DEBUG] S ((tag (1))
                               (payload
                                (Err
                                 ((ename "Out of memory") (errno ())))))

With a slightly smaller file (700mb) I get the following error:

fs9p [DEBUG] S ((tag (1))
                                     (payload
                                      (Err
                                       ((ename
                                         "Unix.Unix_error(Unix.EACCES, \"open\", \"/data/.git/objects/pack/pack-5b5629393a4b074d789894c51cc237f0cc1232f2.idx\")")
                                        (errno ())))))

Smaller files I can successfully ls and cat

[talex5]

ls in 9p needs to get the size of each file, which it currently does by reading the file. I've now added support in ocaml-git for reading the size directly: https://github.com/mirage/ocaml-git/pull/144 This needs to be added to Irmin so DataKit can use it.

We also need a streaming interface in Irmin and Git to handle large files. Any plans for this @samoht?

[samoht]

We also need a streaming interface in Irmin and Git to handle large files. Any plans for this @samoht?

That's a very good idea: https://github.com/mirage/ocaml-git/issues/33