monitor and graph loss

[dtmoodie]

It would be a really nice feature if I could define a regex for parsing out the current loss and then graph it in the app.

[richardartoul]

Hey dtmoodie,

This is actually a feature we're currently designing. We haven't quite settled on how to implement it yet, but we're hoping to have a design thats easier / less brittle than just using regex's.

What we're exploring:

1. Automatic integrations with keras/tensorflow/pytorch etc etc such that if you're using those particular libraries, you just get the loss and such for free.
2. A function that you can import that allows you to provide structured data. Example:

from hyperdash import log_data

for i, epoch in enumerate(epochs):
  loss = train()
  log_data({"epoch": i, "loss": loss})

Or something roughly like that. What are your thoughts on that?

[dtmoodie]

Hello,

Currently I use frameworks that need to be launched from the command line. Thus neither of these options would work terribly well for me, which is why a regex of the log might be my best option.

[andrewschreiber]

@dtmoodie interesting. Could you link us to those frameworks?

[dtmoodie]

Sure,

Currently we are using darknet (https://github.com/pjreddie/darknet) and caffe with modifications. The fork of caffe that we are using does not have multi GPU training through the python interface so to get training to scale on our training server we have to launch it through the command line.

[richardartoul]

@dtmoodie Sorry about the delayed response. We agree with you that parsing logs would be super useful, we're currently trying to think of the best interface to support this and will get back to you soon.

[dtmoodie]

Awesome thank you so much!

[richardartoul]

@dtmoodie Hey just wanted to update you. We've now reached a place where we're able to store the timeseries data on our backend and the SDK can send the timeseries data up, but we don't have a solution for visualizing the data yet. We're working on a re-write of the mobile app to support real-time graphs, but its gonna take us a few weeks to get there!

[dtmoodie]

Awesome news, looking forward to the update.

[richardartoul]

Hey @dtmoodie,

We haven't quite launched yet, but we have a web dashboard you can play around with now to visualize data: https://hyperdash.io/dashboard

Right now the SDK has to be manually instrumented in the code (I.e you need to call hd_client.metric("acccuracy, 0.95) etc etc but I know you're looking for a log parsing solution because you don't have as much control over the code.

With that said we should come up with an interface for the log parsing. Right now I'm imagining a file that is stored in the directory with the program that you are running and inside that directory you define a list of regex's which will be applied to every line and parse out the relevant scalar value.

Does that sound reasonable to you?

[richardartoul]

Basically almost like a CSV where the values look like:

<some_regex_to_capture_loss>, "loss" <some_regex_to_capture_epoch>, "epoch"

[dtmoodie]

That sounds awesome, I can definitely work with this.