Export of non-GPS based workout (e.g. indoor cycling)

[Rubbert]

Wonderful work setting up this repo! I am interesting in extracting both heart-rate and location information.

I was able to export all GPS based workouts to GPX. However, for the non-GPS based workouts like indoor cycling I get a more or less empty file, just containing a start date/time and a workout of type "None". This makes sense, as there is no position information.

However, if I try to export as CSV (or XSLX) instead, the script breaks the moment it encounters the first non-GPS workout, with the error at the bottom of this issue.

I have two questions: (1) Is it possible to still export the heart-rate measurements for non-GPS workouts to e.g. CSV or XLSX? (2) If this is not feasible, is there a way to have the script skip any non-GPS workouts when exporting as CSV/XLSX?

I hope my request is clear, but let me know if you have any questions.

PS. I make YouTube videos on www.youtube.com/thequantifiedscientist , and I'll make sure to link this repo in the description if I end up using it. Great work!

The error: INFO:src.scraper:Downloaded /Volumes/DataAnalysisSSD/Projects/2017-07-15_trackingMyself/data/amazfit/amazfitGTS4/pythonExport/Workout--2022-11-10--09-50-24.xlsx Traceback (most recent call last): File "/Users/robterhorst/Mi-Fit-and-Zepp-workout-exporter/main.py", line 67, in <module> scraper.run() File "/Users/robterhorst/Mi-Fit-and-Zepp-workout-exporter/src/scraper.py", line 40, in run self.exporter.export(output_file_path, summary, points) File "/Users/robterhorst/Mi-Fit-and-Zepp-workout-exporter/src/exporters/geopandas_exporter.py", line 60, in export gdf = gdf[ File "/Users/robterhorst/opt/anaconda3/envs/miZeppExport/lib/python3.10/site-packages/geopandas/geodataframe.py", line 1412, in __getitem__ result = super().__getitem__(key) File "/Users/robterhorst/opt/anaconda3/envs/miZeppExport/lib/python3.10/site-packages/pandas/core/frame.py", line 3810, in __getitem__ indexer = self.columns._get_indexer_strict(key, "columns")[1] File "/Users/robterhorst/opt/anaconda3/envs/miZeppExport/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 6111, in _get_indexer_strict self._raise_if_missing(keyarr, indexer, axis_name) File "/Users/robterhorst/opt/anaconda3/envs/miZeppExport/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 6174, in _raise_if_missing raise KeyError(f"{not_found} not in index") KeyError: "['track_date', 'timestamp', 'latitude', 'longitude', 'altitude'] not in index"

[rolandsz]

At the moment, all extra data (including the heart rate) are tied to the GPS points, exporting only the heart rate most likely won't work out of the box. Unfortunately, I no longer have access to a Xiaomi device, therefore I can only provide you with some guidance on how it could be done.

First of all, the export code should handle the scenario when there are no points in the input - this is a bug in the code. In order to fix this, you can put a conditional return statement somewhere around https://github.com/rolandsz/Mi-Fit-and-Zepp-workout-exporter/blob/master/src/exporters/geopandas_exporter.py#L51, maybe adding a log message is also a good idea.

After fixing the KeyError, you can remove the condition that filters out the raw data should there be no latitudes at https://github.com/rolandsz/Mi-Fit-and-Zepp-workout-exporter/blob/master/src/exporters/base_exporter.py#L240

You might need to adjust the exporter code as well, I'd expect more errors due to missing GPS points. I can investigate it for you, If you can provide me an example JSON coming from the Xiaomi API (print detail.json() at https://github.com/rolandsz/Mi-Fit-and-Zepp-workout-exporter/blob/master/src/scraper.py#L30, then remove any personal information it might have and send it to me).

[Rubbert]

Thanks for all the suggestions! I am not a hard-core python programmer, but I was able to hack together something to get GPX files with just heart-rate in the cases where there is no GPS info.

I changed the following parts (there are probably much more elegant solutions):

def parse_points(summary: WorkoutSummary, detail: WorkoutDetailData) -> List[ExportablePoint]:
    track_data = parse_track_data(summary, detail)

    #if not track_data.hr:
    #    return []

    if not track_data.lat:
        points = [
            ExportablePoint(
                time=datetime.utcfromtimestamp(
                    point.time + track_data.start_time),
                latitude=42.42424242,
                longitude=21.21212121,
                altitude=-20000.0,
                heart_rate=point.hr,
                cadence=1,
            )
            for point in track_points(interpolate_data(track_data))
        ]
    else:
        points = [
            ExportablePoint(
                time=datetime.utcfromtimestamp(
                    point.time + track_data.start_time),
                latitude=point.position.lat,
                longitude=point.position.lon,
                altitude=point.position.alt,
                heart_rate=point.hr,
                cadence=point.cadence,
            )
            for point in track_points(interpolate_data(track_data))
        ]
    return points


def get_type(summary: WorkoutSummary):
    if summary.type == 1:
        return "run"
    elif summary.type == 6:
        return "hike"
    elif summary.type == 9:
        return "ride"
    elif summary.type == 10:
        return "indoorride"
    elif summary.type == 52:
        return "weights"
    elif summary.type == 24:
        return "weights"
    else:
        LOGGER.error(
            f"Unhandled type for workout {summary.trackid}: {summary.type}")


def run(self) -> None:
    workout_history = self.api.get_workout_history()
    logging.info(f"There are {len(workout_history.data.summary)} workouts")

    for summary in workout_history.data.summary:
        detail = self.api.get_workout_detail(summary)
        #print(detail.data)

        track_id = int(summary.trackid)
        file_name = datetime.fromtimestamp(track_id).strftime(
            "Workout--%Y-%m-%d--%H-%M-%S"
        )

        output_file_path = self.get_output_file_path(file_name)
        output_file_path.parent.mkdir(exist_ok=True)

        # self.exporter.export(output_file_path, summary, points)
        # LOGGER.info(f"Downloaded {output_file_path}")

        try:
            points = parse_points(summary, detail.data)

            self.exporter.export(output_file_path, summary, points)
            LOGGER.info(f"Downloaded {output_file_path}")
        except:
            print(f"skipped: {output_file_path}")


def parse_track_data(summary: WorkoutSummary, detail: WorkoutDetailData):
    return RawTrackData(
        start_time=int(summary.trackid),
        end_time=int(summary.end_time),
        cost_time=-1,
        distance=float(summary.dis),
        times=array.array(
            "q",
            [int(val) for val in list(filter(None, detail.time.split(";")))]
            if detail.time and detail.longitude_latitude
            else [],
        ),
        lat=array.array(
            "q",
            [
                int(val.split(",")[0])
                for val in list(filter(None, detail.longitude_latitude.split(";")))
            ]
            if detail.longitude_latitude
            else [],
        ),
        lon=array.array(
            "q",
            [
                int(val.split(",")[1])
                for val in list(filter(None, detail.longitude_latitude.split(";")))
            ]
            if detail.longitude_latitude
            else [],
        ),
        alt=array.array(
            "q",
            [int(val) for val in list(filter(None, detail.altitude.split(";")))]
            if detail.altitude
            else [],
        ),
        hrtimes=array.array(
            "q",
            [
                int(val.split(",")[0] or 1)
                for val in list(filter(None, detail.heart_rate.split(";")))
            ]
            if detail.heart_rate
            else [],
        ),
        hr=array.array(
            "q",
            [
                int(val.split(",")[1])
                for val in list(filter(None, detail.heart_rate.split(";")))
            ]
            if detail.heart_rate
            else [],
        ),
        steptimes=array.array(
            "q",
            [
                int(val.split(",")[0])
                for val in list(filter(None, detail.gait.split(";")))
            ]
            if detail.gait and detail.longitude_latitude
            else [],
        ),
        stride=array.array(
            "q",
            [
                int(val.split(",")[2])
                for val in list(filter(None, detail.gait.split(";")))
            ]
            if detail.gait and detail.longitude_latitude
            else [],
        ),
        cadence=array.array(
            "q",
            [
                int(val.split(",")[3])
                for val in list(filter(None, detail.gait.split(";")))
            ]
            if detail.gait and detail.longitude_latitude
            else [],
        ),
    )

[Rubbert]

Thanks again for all the help! I had another question (happy to open another issue if that would make sense for search-ability). My question is: do you know of some kind of documentation for what data can be extracted from the API? I wanted to also extract the sleep stage data (detailed sleep stages over time), and was wondering if you came across this. Thanks!

[rolandsz]

There is no public documentation available for the API, you'll need to look at the network requests and responses. I'd recommend https://frida.re/ and https://httptoolkit.com/ if you have a rooted phone.