OpenEnergyDashboard
test data
We should generate test data that looks like actual data and be fed into OED. This will be used for testing and also for demos instead of using the more limited real data. For testing the scaling/performance of the system, we should generate 10 years of data for 1000 meters on a minute interval (for now). I estimate this will be in the 2GB range of data. The meter names can be something like M0001, M0002, etc. and the data can be random in some range (such as 0-20 KWh).
I'd like to generate some meters which have data whose waveforms are:
- Sinusoidal with high frequency Perlin component (this is how most campus meters currently look)
- Constant with high frequency Perlin component (Mauer looks like this)
- Square with ultra-low frequency (seasonal) sinusoidal component and high frequency Perlin component
- Perlin noise with ultra-low frequency (seasonal) sinusoidal component
I think these will provide both a pleasing visual demonstration and a good exercise of the compression system.
I would imagine that a completed #54 would really help with this, as we can just generate a CSV and import it. On the other hand, we could always just put it into the DB directly.
There is now some standard development data that is documented on the wiki page. It is expected to expand the set of data in the future. Also, when we allow CSV via a URL, the adding of test data will be even easier (#370). We also want to start using some of this test data for Travis testing.
We can now generate sine/cosine data. We are in the process of using those functions to generate different types of data for developer usage and testing.
The test data generation was completed with PR #614. We still will probably do more types of data in the future. We still need to integrate this data into the testings code.
At this point in time, the one new type of test data that is desired is something that looks like real data from a meter. This would allow for automated data generation for the website and developers. Some ideas are above. We want it to vary with time (maybe season of the year) and have some quasi-random looking component where we can easily generate the same data over and over. See src/server/data/ for the current data generated and the generate... scripts listed in package.json.
OED now loads some real (or real-like) data for the website via CSV. Given real data has not been important for testing or development and it can be generated outside OED and added, this issue is no longer of great importance. If anyone feels otherwise or has thoughts then let OED know. I'm closing this issue based on this comment.