prophet
prophet copied to clipboard
facebook
Program stalls when it runs to an end.
Hi, everyone, I am new here. Recently, I adopted prophet to do some forecast in stock market. It goes smoothly most of the time, but I encounter some problems from time to time because my program, specifically, one of the processes (I should mention that I use multi-process to accelerate the speed) seemed fail to exit after running to the end. It seemed to relate with the workload because I hardly encounter such problem when the workload is low (ie, fewer iterations on my parameter space), but it appears more often when the workload is relatively large. In addition, I initially ran my program in Jupyter Notebook, so later I ran that in the Linux terminal and I found that such problems reduce by some degree but it still occurred some time. I am quite confused and bothered about this, can anyone help me out? Many thanks! My backend is pystan, as is suggested by the author, maybe related to some faults in pystan?
Just like this, the occupancy of CPU is 100% or so all the time, it fails to exit. It should be close to 0 if the process exits successfully. I have read similar issues posted by others, it is true that if I change the parameter sets it will work, however I think it is inefficient and how can I promise that such scenario will not appear in the future? And the algorithm I use is L-BFGS all the time, and I think it is irrelevant with the algorithm since I ever change that to Newton method, consequently it does not work.
We upgraded to the latest prophet version 1.1.1 from 14 September 2022 and had exactly the same issue: the CPU usage is 100% of the pod requested value. Most of the time happened for training and after the last log message like this:
WARNING | prophet.models | fit | 101 | Optimization terminated abnormally. Falling back to Newton.
Not sure it's related to this issue https://github.com/facebook/prophet/issues/1854 or it's something new in the latest version
