Inadequate documentation

[charlieyou]

If more detailed documentation it does exist, please point me to it and make it more clear in the readme.

Otherwise, currently there is only a basic example with one client. No documentation of more advanced usage or error messages. I cannot even seem to find a list of valid time frames without looking into the code itself.

[domgolonka]

Agreed, you have to read the source code for Go client.

[goodboy]

@charlieyou so why don't you write some.

[chipmaker]

The more people trade with APIs on Alpaca, Alpaca makes more money. I am not able to trade because I am going through a learning curve setting up databases properly. Therefore, please kindly help provide detailed documentation for marketstore and pymarketstore.

With novice/intermediate knowledge of Python, I wrote a signal generator and I'm afraid to use it for live-trading because I need uninterrupted latest data as fast as possible. Instead of me reinventing and wasting time, I would like to use a marketstore like this. I could use Arctic/mangoDB but then if someone made a simpler database, I am willing to use it. How to install marketsore on a windows machine btw? I can 'google' and try something. However, its best if we have detailed documentation so that we concentrate on building algos and not worrrying about databases.

[goodboy]

@chipmaker I think part of the problem is there isn't someone maintaining the primary client along with the this core.

This is something my crew will be looking to do as part of alpacahq/pymarketstore#41.

I'm not sure sure the alpaca team has time to maintain docs on a fast moving project it seems atm.

[chipmaker]

goodboy, When I was a consulting micro-chip verification engineer, many designers/colleages/myself would first complete the tasks and then only go to documentation because management always pushes for results. If alpaca has very large investors each with a legion of python-coders, then what you wrote here is fine because large investors dump the apis on their legion to hook that api into their own automated python systems of trading. That's B2B business style(B2B=Business to Business). (For example TDAmeritrade appears to cater to only large investor base and neglect small investors regarding their api, though only thinkorswim is ok for small investors)

However, you see, Alpaca, advertises as B2C business(Business to Consumers) attracting small crowd like us also. Therefore, please kindly consider our request for detailed documentation. Also, still you did not provide guidance on how I could install pymarketstore on a windows machine with marketstore. Please kindly let us know what to do.

By the way, if you tell your management to expand into India, you'll have 10 hands at your service which will help Alpaca to grow rapidly. Largest Microsoft, Google, Adobe, Apple, Cisco, Suzuki, Samsung, several linux variants, etc. software bases have been in India for few decades now. You see Interactive Brokers, they are in many countries now. I am liking Alpaca due to their cost-structure and robust api. It didn't break down that often in my trial runs of my bad code compared to TdAmeritrade's api...Being an Indian living in USA, I am sure Indians will love Alpaca for Indian exchanges. Thus you improve softwares and bring those back to US exchanges...Binance does something similar. Binance USA is a seperate entity from Binance china or Japan or other countries. However, you could switch apis effortlessly in Python programs available.

[goodboy]

@chipmaker sorry I missed this.

I think at some level it's going to be up to use as users to document things from the client side and then report and fix bugs here as they're found. I personally don't expect a FOSS project like this to have immediate support or docs for everything especially if the api is changing. Getting the code for this db for free I don't think documenting it in detail has much to do with Alpaca's business plans - they just FOSSed this code base because they're good peeps. The bulk of their users are using their public facing api service.

I think if you want docs, try using the project, read through the issues and start contributing what you learn. This is exactly what I have done and I've added documentation that the maintainers were just fine to accept.

[conall88]

Hi everyone,

I've been trying to get to grips with understanding how marketstore is configured, as it could be a great foundation for a team project.

The things that stand out for me:

• There are no suggested minimum system requirements, I was somewhat surprised when the docker image used 25GB RAM out of the box with the default GDAX connector config.
• I have no idea where confs are relating to memory management, so can't really address the above well.
• While golang can be executed on windows, I have yet to succeed in building from source on this platform using the ported version of make, would be nice to see an example added to the readme. I can open an issue and provide output from an example if that is worthwhile.

[briancappello]

@conall88

The configuration is all in mkts.yaml. Some options are documented in READMEs. However, some of the "less critical" options are not, and you will need to dive into the source code to understand how those work (or that they even exist). Each plugin tends to have its own README, located in the contrib/<plugin> folders. Furthermore, some stuff you will just need to experiment with. For example, if I recall correctly, the ondiskagg plugin only works when marketstore is configured with timezone: America/New_York. Also related to timezones, I haven't fully wrapped my head around all the quirks, but let's just say that there is interplay between the server's local timezone, marketstore's configured timezone, and the querying client's local timezone.

As far as I can tell - I have no affiliation with Alpaca - there aren't any configuration options directly related to memory management. That said, 25GB sounds extremely high to me. The server uses less than 1GB on my machine (running natively under Linux). You can experiment with the marketstore estimate command to get an idea for disk space requirements. Otherwise minimum system requirements are - like any other server code - basically a function of how many users/clients the server will be supporting. What I can tell you is that for personal/research usage, as long as you have a reasonably fast SSD, it will run very well on any consumer grade machine from the last 5-10 years.

As for Windows. MarketStore uses go plugins, which aren't supported on Windows by golang itself: Currently plugins are only supported on Linux, FreeBSD, and macOS.

[conall88]

@briancappello Thanks for the response, golang is totally new territory for me, but wanted to confirm.

It's weird, I set some WSL2 memory limits, and now its only using ~ 3GB of a 12GB limit, so perhaps something environmental was to blame. all good now :)

I'm seeing ondiskagg going nuts as you mentioned,

{"level":"error","timestamp":"2021-01-31T01:17:42.381Z","msg":"recovering from integer overflow\ngoroutine 1895894 [running]:\nruntime/debug.Stack(0xc000021418, 0x1414f40, 0x7f6e7fc63940)\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x9d\ngithub.com/alpacahq/marketstore/v4/executor.fire.func1()\n\t/go/src/github.com/alpacahq/marketstore/executor/written.go:67 +0x70\npanic(0x1414f40, 0x7f6e7fc63940)\n\t/usr/local/go/src/runtime/panic.go:969 +0x16a\ngithub.com/alpacahq/marketstore/v4/contrib/ondiskagg/aggtrigger/functions.SumFloat64(0xc003a2c000, 0x51, 0x218, 0xc0004776c0)\n\t/go/src/github.com/alpacahq/marketstore/contrib/ondiskagg/aggtrigger/functions/sum.go:17 +0x81\ngithub.com/alpacahq/marketstore/v4/contrib/ondiskagg/aggtrigger.(*accumulator).apply(0xc003622750, 0x0, 0x51)\n\t/go/src/github.com/alpacahq/marketstore/contrib/ondiskagg/aggtrigger/accum.go:284 +0x131f\ngithub.com/alpacahq/marketstore/v4/contrib/ondiskagg/aggtrigger.(*accumGroup).apply(0xc003622780, 0x0, 0x51)\n\t/go/src/github.com/alpacahq/marketstore/contrib/ondiskagg/aggtrigger/accum.go:39 +0x5f\ngithub.com/alpacahq/marketstore/v4/contrib/ondiskagg/aggtrigger.aggregate(0xc000021b10, 0xc003167830, 0xc0031674b0, 0xc00014c5a6, 0xc, 0x1)\n\t/go/src/github.com/alpacahq/marketstore/contrib/ondiskagg/aggtrigger/aggtrigger.go:348 +0x1233\ngithub.com/alpacahq/marketstore/v4/contrib/ondiskagg/aggtrigger.(*OnDiskAggTrigger).writeAggregates(0xc00021c540, 0xc003167830, 0xc0031674b0, 0x0, 0x0, 0xc002cd8e70, 0xc000250100, 0x6, 0x8, 0xc002cd8ea0, ...)\n\t/go/src/github.com/alpacahq/marketstore/contrib/ondiskagg/aggtrigger/aggtrigger.go:279 +0x715\ngithub.com/alpacahq/marketstore/v4/contrib/ondiskagg/aggtrigger.(*OnDiskAggTrigger).write(0xc00021c540, 0xc0031674b0, 0xc001f792c0, 0x0, 0xed15a3e00, 0x0, 0x0, 0xed1525500, 0x0, 0xc001f79180, ...)\n\t/go/src/github.com/alpacahq/marketstore/contrib/ondiskagg/aggtrigger/aggtrigger.go:200 +0x2b6\ngithub.com/alpacahq/marketstore/v4/contrib/ondiskagg/aggtrigger.(*OnDiskAggTrigger).Fire(0xc00021c540, 0xc00014c5a6, 0x21, 0xc002a8a000, 0x433, 0x555)\n\t/go/src/github.com/alpacahq/marketstore/contrib/ondiskagg/aggtrigger/aggtrigger.go:180 +0x66b\ngithub.com/alpacahq/marketstore/v4/executor.fire(0x7f6e7fc6e540, 0xc00021c540, 0xc00014c5a6, 0x21, 0xc002a8a000, 0x433, 0x555)\n\t/go/src/github.com/alpacahq/marketstore/executor/written.go:70 +0x8c\ncreated by github.com/alpacahq/marketstore/v4/executor.run\n\t/go/src/github.com/alpacahq/marketstore/executor/written.go:57 +0x160\n"}

I'l try to the timezone setting, thanks for the tip.