Sampling an int with a large bound value raises a MemoryError

[GuillaumeBroggi]

The following code with a higher bound of 1e12 raises a MemoryError (the exact admissible value will depend on the machine).

The error could be explained if the full int sequence is evaluated at some point (I did not inspect the code).

from f3dasm.design import Domain
from f3dasm import ExperimentData

domain = Domain()
domain.add_int(name="x", low=0, high=int(1e12))
random_input_data = ExperimentData.from_sampling(
    domain=domain, n_samples=1, sampler="random", seed=2024
)

[mpvanderschelling]

Thanks! This MemoryError comes from numpy.random_generator.Generator.choice. We might think about an upper limit that is displayed in the documentation or find a work-around

[GuillaumeBroggi]

If I am not mistaken, the incriminated lines are: https://github.com/bessagroup/f3dasm/blob/6b0da8b70196515f0153b05b9c7102bc23df3bda/src/f3dasm/_src/experimentdata/samplers.py#L180-L184

Indeed, the range generator is evaluated for the choice.

The issue with displaying an upper limit in the documentation is that it's a user-specific limit, depending on the hardware.

A workaround should avoid this evaluation. For an int (inspired from https://stackoverflow.com/a/54082200, no extensive testing done):

import numpy as np


def current_implementation(n_samples, lower_bound, upper_bound, step):
    rng = np.random.default_rng(seed=42)
    return rng.choice(
        range(lower_bound, upper_bound + 1, step),
        size=n_samples,
    )


def proposed_workaround(n_samples, lower_bound, upper_bound, step):
    rng = np.random.default_rng(seed=42)
    return (
        lower_bound
        + rng.integers(
            low=0, high=(upper_bound - lower_bound) / step + 1, size=n_samples
        )
        * step  
    )

current_implementation(n_samples=10, lower_bound=32, upper_bound=42, step=2)

array([32, 40, 38, 36, 36, 42, 32, 40, 34, 32])

proposed_workaround(n_samples=10, lower_bound=32, upper_bound=42, step=2)

array([32, 40, 38, 36, 36, 42, 32, 40, 34, 32])

current_implementation(n_samples=10, lower_bound=0, upper_bound=int(1e12), step=1)

MemoryError ...

proposed_workaround(n_samples=10, lower_bound=0, upper_bound=int(1e12), step=1)

array([773956048556, 438878439752, 858597919912, 697368029060, 94177347887, 975622351637, 761139701991, 786064305277, 128113632675, 450385937896])