Error in CoxPH piecewise baseline hazard prediction

hi @CamDavidsonPilon

Noticed the following issue predicting hazard rate, seems unexpected and probably a bug. Repro below

from lifelines import CoxPHFitter
from lifelines.datasets import load_rossi

rossi = load_rossi()
cph = CoxPHFitter(baseline_estimation_method="piecewise", breakpoints=[25])
cph = cph.fit(rossi[["week", "arrest"]], "week", "arrest")

covariates = rossi[["week", "arrest"]].head()


#This works
z1 = cph.predict_cumulative_hazard(covariates)
print(z1.shape)
#This doesn't
z2 = cph.predict_hazard(covariates)

Had a look at fixing, but i seemed to break calibration in the process. Happy to help fix it but may need some guidance.

Full stack trace for the error below

Traceback (most recent call last):
  File "/Users/richardgriffiths/anaconda3/envs/p2pml39/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3437, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-52ce976ffed3>", line 1, in <module>
    runfile('/Users/richardgriffiths/Library/Application Support/JetBrains/PyCharm2021.1/scratches/scratch_61.py', wdir='/Users/richardgriffiths/Library/Application Support/JetBrains/PyCharm2021.1/scratches')
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/Users/richardgriffiths/Library/Application Support/JetBrains/PyCharm2021.1/scratches/scratch_61.py", line 15, in <module>
    z2 = cph.predict_hazard(covariates)
  File "/Users/richardgriffiths/projects/lifelines/lifelines/lifelines/fitters/coxph_fitter.py", line 2917, in predict_hazard
    return super(ParametricCoxModelFitter, self).predict_hazard(df, times=times, conditional_after=conditional_after)
  File "/Users/richardgriffiths/projects/lifelines/lifelines/lifelines/fitters/__init__.py", line 2421, in predict_hazard
    return pd.DataFrame(self._hazard(params_dict, np.tile(times, (n, 1)).T, Xs), index=times, columns=df.index)
  File "/Users/richardgriffiths/projects/lifelines/lifelines/lifelines/fitters/__init__.py", line 1369, in _hazard
    return egrad(self._cumulative_hazard, argnum=1)(params, T, Xs)  # pylint: disable=unexpected-keyword-arg
  File "/Users/richardgriffiths/anaconda3/envs/p2pml39/lib/python3.9/site-packages/autograd/wrap_util.py", line 20, in nary_f
    return unary_operator(unary_f, x, *nary_op_args, **nary_op_kwargs)
  File "/Users/richardgriffiths/anaconda3/envs/p2pml39/lib/python3.9/site-packages/autograd/differential_operators.py", line 38, in elementwise_grad
    vjp, ans = _make_vjp(fun, x)
  File "/Users/richardgriffiths/anaconda3/envs/p2pml39/lib/python3.9/site-packages/autograd/core.py", line 10, in make_vjp
    end_value, end_node =  trace(start_node, fun, x)
  File "/Users/richardgriffiths/anaconda3/envs/p2pml39/lib/python3.9/site-packages/autograd/tracer.py", line 10, in trace
    end_box = fun(start_box)
  File "/Users/richardgriffiths/anaconda3/envs/p2pml39/lib/python3.9/site-packages/autograd/wrap_util.py", line 15, in unary_f
    return fun(*subargs, **kwargs)
  File "/Users/richardgriffiths/projects/lifelines/lifelines/lifelines/fitters/coxph_fitter.py", line 2753, in _cumulative_hazard
    return self._cumulative_hazard_sans_strata(params, T, Xs)
  File "/Users/richardgriffiths/projects/lifelines/lifelines/lifelines/fitters/coxph_fitter.py", line 3146, in _cumulative_hazard_sans_strata
    T = T.reshape((n, 1))
  File "/Users/richardgriffiths/anaconda3/envs/p2pml39/lib/python3.9/site-packages/autograd/numpy/numpy_vjps.py", line 552, in wrapped_reshape
    return anp.reshape(x, *args, **kwargs)
  File "/Users/richardgriffiths/anaconda3/envs/p2pml39/lib/python3.9/site-packages/autograd/tracer.py", line 44, in f_wrapped
    ans = f_wrapped(*argvals, **kwargs)
  File "/Users/richardgriffiths/anaconda3/envs/p2pml39/lib/python3.9/site-packages/autograd/tracer.py", line 48, in f_wrapped
    return f_raw(*args, **kwargs)
  File "<__array_function__ internals>", line 5, in reshape
  File "/Users/richardgriffiths/anaconda3/envs/p2pml39/lib/python3.9/site-packages/numpy/core/fromnumeric.py", line 299, in reshape
    return _wrapfunc(a, 'reshape', newshape, order=order)
  File "/Users/richardgriffiths/anaconda3/envs/p2pml39/lib/python3.9/site-packages/numpy/core/fromnumeric.py", line 58, in _wrapfunc
    return bound(*args, **kwds)
ValueError: cannot reshape array of size 245 into shape (49,1)

thanks Richard

Originally posted by @griffiri in https://github.com/CamDavidsonPilon/lifelines/discussions/1266

@CamDavidsonPilon what do you think of these fixes?

If you think ok, i can add some unit tests and raise a PR

Looks reasonable! But the proof is in the unit test :)

I'd appreciate a PR - we can work together on this from there.

ok - let me have a look at adding some unit tests thanks

hi @CamDavidsonPilon - i made some progress, but still at least one issue remains unresolved. I would like to show you the code changes, as you may know immediately how to resolve. However i don't seem to be able to create a branch, are permissions restricted for that? thanks

Hi @griffiri - are you trying to create a PR from your own fork? Usually the workflow is:

1. Fork this repo to under your own account
2. Git clone the new repo locally
3. Make changes and push back to Github.
4. From there, you'll be able to send a pull request to original repo

Ah i see - first time i have done a PR on github, thanks for the instructions will try it

@CamDavidsonPilon ok think have raised PR in a sane manner