Exception when displaying graphs

[shenki]

 fpos show home
/usr/lib/python3/dist-packages/numpy/lib/polynomial.py:587: RuntimeWarning: invalid value encountered in true_divide
  lhs /= scale
/usr/lib/python3/dist-packages/numpy/lib/polynomial.py:589: RuntimeWarning: invalid value encountered in true_divide
  c = (c.T/scale).T  # broadcast scale coefficients
/usr/lib/python3/dist-packages/numpy/lib/polynomial.py:594: RankWarning: Polyfit may be poorly conditioned
  warnings.warn(msg, RankWarning)
/usr/lib/python3/dist-packages/numpy/lib/polynomial.py:594: RankWarning: Polyfit may be poorly conditioned
  warnings.warn(msg, RankWarning)
/usr/lib/python3/dist-packages/numpy/lib/polynomial.py:594: RankWarning: Polyfit may be poorly conditioned
  warnings.warn(msg, RankWarning)
/usr/lib/python3/dist-packages/numpy/lib/polynomial.py:594: RankWarning: Polyfit may be poorly conditioned
  warnings.warn(msg, RankWarning)
/usr/lib/python3/dist-packages/numpy/lib/polynomial.py:594: RankWarning: Polyfit may be poorly conditioned
  warnings.warn(msg, RankWarning)
/usr/lib/python3/dist-packages/numpy/lib/polynomial.py:594: RankWarning: Polyfit may be poorly conditioned
  warnings.warn(msg, RankWarning)
/usr/lib/python3/dist-packages/numpy/lib/polynomial.py:594: RankWarning: Polyfit may be poorly conditioned
  warnings.warn(msg, RankWarning)
/usr/lib/python3/dist-packages/numpy/lib/polynomial.py:594: RankWarning: Polyfit may be poorly conditioned
  warnings.warn(msg, RankWarning)
/usr/lib/python3/dist-packages/numpy/lib/polynomial.py:594: RankWarning: Polyfit may be poorly conditioned
  warnings.warn(msg, RankWarning)
/usr/lib/python3/dist-packages/numpy/lib/polynomial.py:594: RankWarning: Polyfit may be poorly conditioned
  warnings.warn(msg, RankWarning)
/usr/lib/python3/dist-packages/numpy/core/_methods.py:82: RuntimeWarning: Degrees of freedom <= 0 for slice
  warnings.warn("Degrees of freedom <= 0 for slice", RuntimeWarning)
/usr/lib/python3/dist-packages/numpy/core/_methods.py:116: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)
Traceback (most recent call last):
  File "/home/shenki/bin/fpos", line 48, in <module>
    main()
  File "/home/shenki/bin/fpos", line 45, in main
    args.subcmd(args)
  File "/home/shenki/.local/lib/python3.5/site-packages/fpos/db.py", line 117, in main
    args.db_func(args)
  File "/home/shenki/.local/lib/python3.5/site-packages/fpos/db.py", line 98, in db_show
    visualise(list(window(start, end, csv.reader(db))), save=args.save)
  File "/home/shenki/.local/lib/python3.5/site-packages/fpos/visualise.py", line 587, in visualise
    graph_xy_progressive_mean(months, d_grouped, m_income, description_groups, span)
  File "/home/shenki/.local/lib/python3.5/site-packages/fpos/visualise.py", line 504, in graph_xy_progressive_mean
    forecast_plt, = plt.plot(xs[dates[1].day - 1:mr[1]], d_forecast[dates[1].day - 1:], ls="-", marker="o", color="orange")
  File "/usr/lib/python3/dist-packages/matplotlib/pyplot.py", line 3154, in plot
    ret = ax.plot(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/matplotlib/__init__.py", line 1814, in inner
    return func(ax, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/matplotlib/axes/_axes.py", line 1424, in plot
    for line in self._get_lines(*args, **kwargs):
  File "/usr/lib/python3/dist-packages/matplotlib/axes/_base.py", line 386, in _grab_next_args
    for seg in self._plot_args(remaining, kwargs):
  File "/usr/lib/python3/dist-packages/matplotlib/axes/_base.py", line 364, in _plot_args
    x, y = self._xy_from_xy(x, y)
  File "/usr/lib/python3/dist-packages/matplotlib/axes/_base.py", line 223, in _xy_from_xy
    raise ValueError("x and y must have same first dimension")
ValueError: x and y must have same first dimension

Running 0b9c8b2871a84821d5064e7545220615b4516e2c (with my Woolworths Money patch, but that shouldn't affect this).

The values that it's trying to plot on line 504 are:

[20, 21, 22, 23, 24, 25, 26, 27, 28, 29]
197.5585

[amboar]

Thanks for the report, I'll look into it. Probably several things should come out of this:

• We should protect against exceptions in the various plotting functions killing the application
• We should have the ability to configure or select certain plots to be displayed, or not.

[shenki]

• we should add vertical whitespace to the functions On 26 Mar 2016 14:10, "Andrew Jeffery" [email protected] wrote:

Thanks for the report, I'll look into it. Probably several things should come out of this:

• We should protect against exceptions in the various plotting functions killing the application
• We should have the ability to configure or select certain plots to be displayed, or not.

— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub https://github.com/amboar/fpos/issues/14#issuecomment-201700388

[amboar]

• we should add vertical whitespace to the functions

Python has so much horizontal whitespace I had to compromise somewhere

[amboar]

Okay, a reduced case of your error seems to be:

>>> import matplotlib.pyplot as plt
>>> plt.plot(list(range(1)), list(range(2)))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.4/site-packages/matplotlib/pyplot.py", line 3154, in plot
    ret = ax.plot(*args, **kwargs)
  File "/usr/lib64/python3.4/site-packages/matplotlib/__init__.py", line 1812, in inner
    return func(ax, *args, **kwargs)
  File "/usr/lib64/python3.4/site-packages/matplotlib/axes/_axes.py", line 1424, in plot
    for line in self._get_lines(*args, **kwargs):
  File "/usr/lib64/python3.4/site-packages/matplotlib/axes/_base.py", line 386, in _grab_next_args
    for seg in self._plot_args(remaining, kwargs):
  File "/usr/lib64/python3.4/site-packages/matplotlib/axes/_base.py", line 364, in _plot_args
    x, y = self._xy_from_xy(x, y)
  File "/usr/lib64/python3.4/site-packages/matplotlib/axes/_base.py", line 223, in _xy_from_xy
    raise ValueError("x and y must have same first dimension")
ValueError: x and y must have same first dimension

The line generating the error is;

forecast_plt, = plt.plot(xs[dates[1].day - 1:mr[1]], d_forecast[dates[1].day - 1:], ...)

so we know that the two slices have different lengths, however the same backtrace as above is generated for >>> plt.plot(list(range(2)), list(range(1))), so we can't deduce which one is shorter.

mr is assigned as following: mr = calendar.monthrange(pm.prev.year, pm.prev.month), where pm.prev should be the day of the last transaction in the database. The docstring for calendar.monthrange says:

monthrange(year, month)
    Return weekday (0-6 ~ Mon-Sun) and number of days (28-31) for
    year, month.

Maybe there's an off-by-one issue with the slice endpoint on the xs variable, but requesting a slice end index beyond the end of the sliced list doesn't cause problems:

>>> l1 = ['a']
>>> l1[:2]
['a']

The length of d_forecast is governed by the date of the most recent transaction, and the number of days remaining in the month of the most recent transaction: remaining = (mr[1] - dates[1].day). datetime.day is 1-indexed, so 26 == datetime.datetime(2016, 3, 26).day holds. remaining seems to be assigned a value consistent with its intent. d_forecast is generated as follows:

    for i, df in enumerate(forecast(groups, dates, remaining)[0]):
        pm.update(dates[1] + timedelta(1 + i), df)
    d_forecast = pm.head()

Considering March, we have 31 days. If the most recent transaction is on the 26th, then substituting into remaining = (mr[1] - dates[1].day) gives remaining = 31 - 26 (== 5). The timedelta(1 + i) looks sus, but this is the picket fence problem, and we're interested in the pickets. So it doesn't look like there's a problem there.

I've checked the forecast function as well, where islice is passed the function's length parameter, which equates to remaining. A quick test there shows it gives what we want:

>>> from itertools import islice
>>> list(islice(range(10), 5))
>>> len(list(islice(range(10), 5)))
5

So we ask for and receive 5 elements. The outcome is that d_forecast = pm.head() should always give the month of the most recent transaction, and never spill into the following month. It will also always produce a fully-projected month (i.e., up to and including the last day). So it doesn't look like the bug is in these calculations.

There's another possibility though, which is that the span argument passed to graph_xy_progressive_mean in lib/fpos/visualise.py is generated with current_date set to True. In this case it's possible that the current date falls in a month with a different number of days to the month of the date of the last transaction, in which case the values held in mr are generated from the wrong month. But it's off by default and it doesn't look like you're invoking fpos in a way that would enable it.

Overall, quite curious. Can you provide more information around the dates involved? What was the date of the most recent transaction? Can you also provide the lengths of the arrays passed to plt.plot()?

Am I right deducing that the last transaction was in Feb?