how to remove logged data at specific step?

[kimjanq]

using run['logs/data'].pop(), I can remove the field stored under the path completely and all data associated with it.

However, I cannot find a function that deletes(or modifies) data in a specific step. (for example, there are data logged among step1~step100, I want to delete data after step80 and log again from step80).

Are there any related functions?

Thank you

[Herudaio]

@kimjanq there are none now, but we will be adding it shortly - it's in the development queue and I will update this issue when it's available.

[Blaizzy]

Hey @kimjanq,

Prince here,

You can modify data from a specific step onward using the latest neptune release in the following way.

Code example

import neptune

run = neptune.init_run(
    with_id='ABC-533', # Replace with the ID of the run you want to resume
)

# Fetch metadata from Neptune run
params = run["params"].fetch()
acc = run['training/acc'].fetch_values() # pandas DF w/ columns [value, step, timestamp]

step = 100 # cutoff step

# Filter rows up to `step` X from pandas DataFrame
acc.query(f'step <= {step}')

# Download model checkpoint
run[f"model/checkpoints/cpkt_{step}"].download()

# Log all points till `step` X
del run['training/acc']
run['training/acc'].extend(value=acc["value"], step=acc["step"], timestamp=acc["timestamp"])

# Continuing training from step X onwards
for step, value in enumerate(range(step,params["epochs"]), start=step):
    ... # model training
    run['training/acc'].append(value=value, step=step)

Docs:

• https://docs.neptune.ai/logging/to_existing_object/#example
• https://docs.neptune.ai/api/field_types/#append_1
• https://docs.neptune.ai/api/field_types/#extend_1
• https://docs.neptune.ai/api/field_types/#fetch_values

[SiddhantSadangi]

@kimjanq - in addition to @Blaizzy's workaround on how you can do it now, we also have native support for this feature under consideration in our roadmap: https://portal.neptune.ai/c/85-partially-clear-a-series-field

It would be great if you could vote on how important this is for you so that we can prioritize this effectively.