LBANN Core Interface

[bvanessen]

Create a new LBANN core interface and front end driver Note that the initial version will replicate the lbann_inf driver

• [x] Create external c++ "driver" file
• [x] External driver will initialize MPI and pass communicator to LBANN core
• [x] LBANN initialize from a checkpoint without a prototext
• [x] core receives a sample (or set of samples) and returns responses
• [ ] Improve data set state to allow for discrete set of samples
• [ ] Allow LBANN to take tensors that are wired into certain input layers
• [x] No requirement for data readers
• [ ] Ability to have functional data readers read from complex in-memory objects
• [ ] Pass in argument as to max mini-batch size into trainer to properly size internal buffers

MPI_Comm my_mpi_comm = ...;
unsigned long max_minibatch_size = /*user input*/;
lbann_comm* c = lbann_initialize(my_mpi_comm);
model* m = get_model(c, max_minibatch_size, ...);
while (!done)
{
  auto samples = get_samples();
  /* IMO, "infer()" should block "samples" into chunks of `max_minibatch_size`
   * internally, rather than requiring the driver to do explicit blocking.
   * In Elemental-speak, this just means "View"-ing subsets of columns.
   */
  append(outputs, infer(m, samples.key, samples.values));
  cleanup(samples);/*idk if these are malloc'd or what*/
}
print(outputs);
free(m);
free(c);

output_type infer(model*, char const* input_field_names[], samples_type samples[]);

enum sample_data_type { FLOAT, DOUBLE, INT, INT16, INT32, UCHAR, ... }
struct sample_type {
  void** values,
  unsigned long* values_sizes,
  sample_data_type* samples_data_types,
  char const** keys
};

std::unique_ptr<BaseDistMatrix> make_viewing_matrix(void* values, unsigned long values_extent, sample_data_type type);