synthetic

synthetic copied to clipboard

Additional Streamers to Support Random Access Reads and Appends

2

One of the biggest selling points for me of [fst](https://github.com/fstpackage/fst) is the ability to randomly access (and append in [v0.9.2](https://github.com/fstpackage/fst/milestone/23)!) disk stored data. That is, load specific data into an...

phillc73

Numeric simulation takes levels into account

1

A numeric vector can have a limited amount of _levels_ that are replicated: ```r # 10 'levels' vec_levels

MarcusKlik

Method table definition takes correlation into account

1

Advanced feature to generate dataset samples from a source dataset with the correlations between column vectors retained: ``` r dt

MarcusKlik

Method table_definition() can create a model of a source dataset

2

And generate new sample data from that. Correlations between columns can be retained: ``` r dt

MarcusKlik

Using appropriate generators and the `synthetic` infrastructure

MarcusKlik

For example: ``` r synthetic_bench() %>% bench_tables(generator, column_mode = "single column") %>% bench_streamers(rds_streamer, fst_streamer, parguet_streamer, feather_streamer) %>% bench_rows(1e7, 5e7) %>% bench_compression(50, 80) %>% compute() ``` Parameter _column\_mode_ could specify the...

MarcusKlik

Allow for selection of number of threads

1

Although the arrow package doesn't directly support selection of the number of threads (I think)

MarcusKlik

See [here](https://arrow.apache.org/blog/)

MarcusKlik

Explore package bench

1

As it also tracks memory allocations. It would be nice to benchmark memory usage across packages as well.

MarcusKlik

For long benchmarks, the user should not use data when an error is encountered (or the system goed down). Instead, we can use a file to save temp results and...

MarcusKlik