Zipline My First Algorithm not working -- all interesting fields are zero in the pickle file

[trevor-richardson]

Dear Zipline Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

• Operating System: Ubuntu 18.04
• Python Version: I've tried 3.4 and 2.7
• Python Bitness: both are 64 bit
• How did you install Zipline: tried pip and conda
• Python packages:

Name Version Build Channel

_libgcc_mutex 0.1 main
alembic 0.7.7 py35_0 Quantopian asn1crypto 0.24.0 py35_0
backcall 0.1.0 py_0 conda-forge bcolz 0.12.1 np114py35_0 Quantopian beautifulsoup4 4.9.1 pypi_0 pypi blas 1.0 mkl
bleach 3.1.5 pyh9f0ad1d_0 conda-forge blosc 1.16.3 hd408876_0
bottleneck 1.2.1 py35h035aef0_1
bzip2 1.0.8 h7b6447c_0
ca-certificates 2020.4.5.1 hecc5488_0 conda-forge certifi 2018.8.24 py35_1001 conda-forge cffi 1.11.5 py35he75722e_1
chardet 3.0.4 py35_1
click 7.1.2 py_0
contextlib2 0.6.0.post1 py_0
cryptography 2.3.1 py35hc365091_0
cycler 0.10.0 py35hc4d5149_0
cyordereddict 0.2.2 py35_0 Quantopian cython 0.28.5 py35hf484d3e_0
dbus 1.13.14 hb2f20db_0
decorator 4.4.2 py_0
defusedxml 0.6.0 py_0 conda-forge empyrical 0.5.0 py35_0 Quantopian entrypoints 0.2.3 py35_2 conda-forge expat 2.2.6 he6710b0_0
fontconfig 2.13.0 h9420a91_0
freetype 2.9.1 h8a8886c_1
glib 2.63.1 h5a9c865_0
gst-plugins-base 1.14.0 hbbd80ab_1
gstreamer 1.14.0 hb453b48_1
hdf5 1.10.2 hba1933b_1
icu 58.2 he6710b0_3
idna 2.7 py35_0
inflection 0.4.0 pypi_0 pypi intel-openmp 2019.4 243
intervaltree 2.1.0 py35_0 Quantopian ipykernel 5.1.0 pyh24bf2e0_0 conda-forge ipython 7.0.1 py35h24bf2e0_0 conda-forge ipython_genutils 0.2.0 py_1 conda-forge jedi 0.12.1 py35_0 conda-forge jinja2 2.11.2 pyh9f0ad1d_0 conda-forge jpeg 9b h024ee3a_2
json5 0.9.0 py_0 conda-forge jsonschema 2.6.0 py35_2 conda-forge jupyter_client 5.3.3 py_0 conda-forge jupyter_core 4.5.0 py_0 conda-forge jupyterlab 2.1.0 py_0 conda-forge jupyterlab_server 1.0.0 py_0
kiwisolver 1.0.1 py35hf484d3e_0
libedit 3.1.20181209 hc058e9b_0
libffi 3.2.1 hd88cf55_4
libgcc-ng 9.1.0 hdf63c60_0
libgfortran-ng 7.3.0 hdf63c60_0
libpng 1.6.37 hbc83047_0
libsodium 1.0.18 h516909a_0 conda-forge libstdcxx-ng 9.1.0 hdf63c60_0
libuuid 1.0.3 h1bed415_2
libxcb 1.13 h1bed415_1
libxml2 2.9.9 hea5a465_1
libxslt 1.1.33 h7d1a2b0_0
logbook 0.12.5 py35_0 Quantopian lru-dict 1.1.4 py35_0 Quantopian lxml 4.2.5 py35hefd8a0e_0
lz4-c 1.8.1.2 h14c3975_0
lzo 2.10 h7b6447c_2
mako 1.1.2 py_0
markupsafe 1.0 py35h14c3975_1
matplotlib 3.0.0 py35h5429711_0
mistune 0.8.3 py35h470a237_2 conda-forge mkl 2018.0.3 1
mkl_fft 1.0.6 py35h7dd41cf_0
mkl_random 1.0.1 py35h4414c95_1
more-itertools 8.3.0 pypi_0 pypi multipledispatch 0.6.0 py35_0
nbconvert 5.6.0 py_0 conda-forge nbformat 5.0.6 py_0 conda-forge ncurses 6.2 he6710b0_1
networkx 1.11 py35_1
notebook 5.7.0 py35_0 conda-forge numexpr 2.6.1 np114py35_0 Quantopian numpy 1.14.6 py35h3b04361_4
numpy-base 1.14.6 py35h81de0dd_4
openssl 1.0.2u h516909a_0 conda-forge packaging 20.4 pyh9f0ad1d_0 conda-forge pandas 0.22.0 py35hf484d3e_0
pandas-datareader 0.8.1 py_0
pandoc 2.9.2.1 0 conda-forge pandocfilters 1.4.2 py_1 conda-forge parso 0.7.0 pyh9f0ad1d_0 conda-forge patsy 0.5.0 py35_0
pcre 8.43 he6710b0_0
pexpect 4.6.0 py35_0 conda-forge pickleshare 0.7.5 py35_0 conda-forge pip 10.0.1 py35_0
prometheus_client 0.8.0 pyh9f0ad1d_0 conda-forge prompt_toolkit 2.0.10 py_0 conda-forge ptyprocess 0.6.0 py_1001 conda-forge pycparser 2.20 py_0
pygments 2.6.1 py_0 conda-forge pyopenssl 18.0.0 py35_0
pyparsing 2.4.7 py_0
pyqt 5.9.2 py35h05f1152_2
pysocks 1.6.8 py35_0
pytables 3.4.4 py35ha205bf6_0
python 3.5.6 hc3d631a_0
python-dateutil 2.8.1 py_0
pytz 2020.1 py_0
pyzmq 17.1.2 py35hae99301_0 conda-forge qt 5.9.6 h8703b6f_2
quandl 3.5.0 pypi_0 pypi readline 7.0 h7b6447c_5
requests 2.20.1 py35_0 Quantopian requests-file 1.4.3 py35_0
scipy 1.1.0 py35hd20e5f9_0
send2trash 1.5.0 py_0 conda-forge setuptools 40.2.0 py35_0
simplegeneric 0.8.1 py_1 conda-forge sip 4.19.8 py35hf484d3e_0
six 1.11.0 py35_1
snappy 1.1.7 hbae5bb6_3
sortedcontainers 2.1.0 py35_0 Quantopian soupsieve 2.0.1 pypi_0 pypi sqlalchemy 1.2.11 py35h7b6447c_0
sqlite 3.31.1 h62c20be_1
statsmodels 0.9.0 py35h3010b51_0
tbb 2020.0 hfd86e86_0
tbb4py 2018.0.5 py35h6bb024c_0
terminado 0.8.1 py35_1 conda-forge testpath 0.4.4 py_0 conda-forge tk 8.6.8 hbc83047_0
toolz 0.10.0 py_0
tornado 5.1.1 py35h7b6447c_0
trading-calendars 1.11.2 py35_0 Quantopian traitlets 4.3.2 py35_0 conda-forge urllib3 1.23 py35_0
watermark 2.0.2 pypi_0 pypi wcwidth 0.2.3 pyh9f0ad1d_0 conda-forge webencodings 0.5.1 py_1 conda-forge wheel 0.31.1 py35_0
xz 5.2.5 h7b6447c_0
yahoofinancials 1.5 pypi_0 pypi zeromq 4.2.5 hfc679d8_4 conda-forge zipline 1.3.0 np114py35_0 Quantopian zlib 1.2.11 h7b6447c_3
zstd 1.3.7 h0b5b093_0

Now that you know a little about me, let me tell you about the issue I am having:

Description of Issue

The intro tutorial run on a custom bundle doesn't work. I had to make a custom bundle because quandl won't accept my emails (it says my work email is not a work email). I followed this tutorial to set up my custom bundle -- https://towardsdatascience.com/introduction-to-backtesting-trading-strategies-7afae611a35e

Here are the first couple of lines from csv file cat to terminal. /AlgorithmicTrading$ cat aapl_daily/AAPL.csv date,open,high,low,close,volume,dividend,split 2017-01-03,115.80000305175781,116.33000183105469,114.76000213623047,116.1500015258789,28781900,0,0 2017-01-04,115.8499984741211,116.51000213623047,115.75,116.0199966430664,21118100,0,0

I know have the following bundles: apple-prices-2017-2019 2020-06-05 12:49:38.522678 apple-prices-2017-2019 2020-06-05 11:47:48.113717 apple-prices-2017-2019 2020-06-05 11:37:22.842459 apple-prices-2017-2019 2020-06-05 11:08:31.692111 ayx-prices-2019-2020 2020-06-04 13:17:20.079053 ayx-prices-2019-2020 2020-06-04 13:16:18.612500 ayx-prices-2019-2020 2020-06-04 12:45:41.304859 csvdir quandl quantopian-quandl

I run the following command on your buyapple example

zipline run -f /anaconda3/pkgs/zipline-1.3.0-np114py35_0/lib/python3.5/site-packages/zipline/examples/buyapple.py --bundle apple-prices-2017-2019 --start 2017-1-1 --end 2018-1-1 -o backtest.pickle

Almost all of the values in the resulting pickle are zero. And the profit is a flat line

Here is how you can reproduce this issue on your machine:

Reproduction Steps

Made a new conda environment. Installed zipline registered new bundle run zipline run command visualize that nothing happened except for tracking stock price (attached image shows this) ...

What steps have you taken to resolve this already?

I've tried python 3.5 and 2.7. I've tried jupyter notebook. I've tried following this peron's exact blog https://towardsdatascience.com/introduction-to-backtesting-trading-strategies-7afae611a35e

I cannot seem to get the backtest to work using any intro strategies.

Thank you!

...

Sincerely, Trevor Richardson

[cemal95]

You can use the csvdir.py and extensions.py files to register your bundle if you have personal data and your algo.

[trevor-richardson]

@cemal95 my bundle is already registered. I used the extensions.py file to register it. Is there anyway to check if it is registered correctly besides what I show above which is the output from zipline bundles?

[cemal95]

currently I dont know how to check. Did you make sure that your data is in the right file, sometimes they forget to provide the right path? It says in the blog to use this command to check if it is regestered: !zipline bundles

Maybe show your code?

[trevor-richardson]

@cemal95 I didn't write any code. I am using the code inside /zipline/examples/buyapple.py (which is provided by zipline). The repo is inside my anaconda environment folder. I ran the following command which should execute that strategy on my bundle. Here is the command:

zipline run -f /anaconda3/pkgs/zipline-1.3.0-np114py35_0/lib/python3.5/site-packages/zipline/examples/buyapple.py --bundle apple-prices-2017-2019 --start 2017-1-1 --end 2018-1-1 -o ./backtest.pickle

[cemal95]

Hey sorry for the late reply, I ran the example. I used the bundle='quandl'. And added the run_algorithm() function and it worked. Did you check the data maybe, did you download it?

Also, the example only has specific dates on it, the algo runs in the year 2014. If you look at the last method. So maybe thats why, you provided the data maybe, but the time is wrong?

[trevor-richardson]

The first three lines of the csv file I'm using as a custom bundle is printed out above. That custom bundle has data from 2017-2019. This has been confirmed. I really appreciate your help but I don't think I'm conveying the problem to you properly....we can just close this (unanswered). This isn't progressing for me. I can write the python script myself to backtest (simple loop over the data accounting for dates and such). Writing backtest myself (super simple) will be faster than debugging I think. Thanks!

[trevor-richardson]

@cemal95 I really like the code you pasted! Thanks for helping out on this journey!

[cemal95]

My pleasure man, also on my THESIS repo, I have a short intro on how to use it on jupyter notebook. Also gonna upload my thesis algos.

[trevor-richardson]

I still haven't figured out how to use custom bundles but I thought you should know your sample code helped me get the initial algorithm running with quandl data! Call this effectively closed. Two weird things. I had to edit benchmarks.py in the /zipline/data/ filepath so that json errors weren't occuring. they talk about this issue on other issues pages sorry I can't remember which one exactly and I had to add the quandl api key to export in .bashrc. It has been a journey to say the least. Not like downloading pytorch or matplotlib which just work.... I will definitely check out your thesis repo. Thanks again!

[cemal95]

Alrighty, cool man. Wish you all the best bro. Please let me know if something can be done to improve on something :)

[belortavirin]

Hello, guys Im doing my first code, unfortunately Im having a lot problems with my code. I already bundle my data in custom-bundle

#%%
import pandas as pd
import zipline
import pytz
import numpy as np
from analysis import create_benchmark, analyze
from zipline.api import order_percent, symbols, order, record, order_target_percent, symbol, schedule_function, date_rules, time_rules
from zipline.finance import commission
from datetime import datetime
from matplotlib import pyplot as plt, ticker, rc
import trading_calendars as tc
#%%
def initialize(context):

    context.set_commission(commission.PerShare(cost=0.0, min_trade_cost=0))
    context.assets= symbols('HGRE11','KNRI11','BPFF11','XPCM11','MXRF11')
    context.n_assets=len(context.assets)
    context.has_position= False

def handle_data(context, data):
 
  if not context.has_position:
      for asset in context.assets:
          if data.can_trade(asset):
              order_percent(asset, 1/context.n_assets)
      context.has_position= True

result = zipline.run_algorithm(
    start=start, # Set start
    end=end,  # Set end
    initialize=initialize, # Define startup function
    capital_base=100000, # Set initial capital
    data_frequency = 'daily',  # Set data frequency
    bundle='custom-bundle', # Select bundle
    trading_calendar= tc.get_calendar('BVMF'),
    benchmark_returns=bench_series) 

 fig, ax =plt.subplots(1, 1, sharex=True, figsize=[16,9])
 #Portafolio value
 result.portfolio_value.plot(ax=ax[0])
 ax[0].set_ylabel('Portafolio value in$')
 plt.legend()
 plt.show()

However, I got zero values in the result file. Im using jupyter to run Zipline. Any clue about this issue?