brimdata
Low consumption of RAM and CPU during line import
Low consumption of RAM and CPU during line import
Of course, this may not be a problem, but during data import, approximately 30-40% of system power is used linearly. But while if the same file is imported as CSV, 100% of the system power is used and less time is used for import.
Of course, this problem may have existed in my system and is not a global problem
@SoftTools59654: I've tried to reproduce the performance problem you've described, but if anything, I see the opposite: When importing the same data, I saw the CSV import took much longer than Line. To test I created a 1.6 GB CSV file and tried to import it first as CSV and then as Line on a 4-core Windows VM with 16 GB of RAM. As the videos below show, as CSV it took about 115 seconds to import but as Line it took only about 30 seconds. A count() query after each import shows that the expected number of input records was the same in both cases.
CSV
https://github.com/brimdata/zui/assets/5934157/c9a8d15b-83a2-4e3e-bb2a-a77677e862ce
Line
https://github.com/brimdata/zui/assets/5934157/01692582-1a63-4d73-b68b-4f2c5e37a462
If you can find something different about your data or environment that reliably reproduces a different effect for you and share the details, I can see if we can explain why your results are different and if there's anything that can be done to improve.
In general, we're always working over time to improve performance, but we don't guarantee that operations will always be able to leverage 100% of the available system resources to deliver absolute best case performance, since some operations don't lend themselves to multi-core processing, some workloads trigger bottlenecks that are out of our control, etc. Particularly when you can find examples where the Zed tools lag far behind the performance of what you see in other systems that perform similar functions we'd be interested in hearing comparative numbers and how it affects your key use cases, as this may help guide where we invest research in performance improvement (https://github.com/brimdata/zed/issues/3959).
I'll hold this issue open for a while to see if you can provide any more detail that would explain your differing experience.
