David Slater

The basic concept would be to hook `__setattr__` (or similar) and filter so that when a value gets set on an object, we can tap into that value.

Add method hooking to probe functionality

1

This would make it much easier to grab values from various functions, such as in https://github.com/twosixlabs/armory/issues/1462, where we could simply hook this method: https://github.com/twosixlabs/armory/blob/388edde7d85f96dac6a96c13854b955f1bb5c3c3/armory/art_experimental/attacks/carla_obj_det_patch.py#L262 Something like our eval patch (but...

When calculating the adversarial accuracy, it would be nice if stochastic models predicted the output multiple times that way we’d get less variance in the metric which would make it...

Explicit Task and Model Definitions

1

We should be significantly more explicit about what the model needs to do (the task), what the model is actually doing, and potentially what the attack threat model is. -...

Partial results and "continue" functionality for interrupted scenarios

1

Currently, if a scenario is interrupted due to any factor (e.g., OOM or KeyboardInterrupt), no results are written to disk. This is particularly painful in long-running scenarios that fail several...

Refactor datasets

8

Edited on 11 March 2022 GOAL: Currently armory uses TFDS 3.2.0 which is outdated and makes some tasks (e.g. including local datasets) difficult. This ticket aims to update TFDS to...

Pin end-to-end tests and computational requirements to armory outputs?

5

(@shenshaw26, this may be related to your thoughts/discussion on resource requirements, but is broader.) My thought is that for end-to-end testing, we run the scenarios (with some subset of samples)...

Request from Cory (Intel)

added new typing and changed error response

5

Fixes #194 Details on approach in linked issue.