ai4cr
ai4cr copied to clipboard
Add simple RNN functionality
Add simple RNN functionality (See Issue: https://github.com/drhuffman12/ai4cr/issues/15 )
- [x] Phase 1: initial prototype/skeleton (w/ basic structure calc's) (See PR: https://github.com/drhuffman12/ai4cr/pull/46 )
- [x] Phase 2: expand prototype/skeleton (to include the associated 'mini_net' objects) (See PR: https://github.com/drhuffman12/ai4cr/pull/47 )
- [x] Phase 3: expand prototype/skeleton (to include the methods to 'guess' and 'train') (See PR: https://github.com/drhuffman12/ai4cr/pull/47 )
- [x] Phase 4: FIX to/from json (use
type
key?) (in a later PR, at least partly in PR: #47) - [x] Add code to: (See PR: https://github.com/drhuffman12/ai4cr/pull/51 )
- [x] Collect (split?) a 'sequence of input and output data' into small chunks for use by
RnnSimple#train(..)
- [x] Add a variation of
RnnSimple#train(..)
that loops thru the 'sequence of input and output data'
- [x] Collect (split?) a 'sequence of input and output data' into small chunks for use by
- [x] add initial
BreederUtils
andBreeder
(See PR: https://github.com/drhuffman12/ai4cr/pull/54) - [x] Consolidate
error_distance
related code intoErrorStats
(See PR: https://github.com/drhuffman12/ai4cr/pull/56 and https://github.com/crystal-lang/crystal/tree/master/.github/workflows) - [x] Convert misc classes to use
ErrorStats
(See PR: https://github.com/drhuffman12/ai4cr/pull/56) - [x] Reorg/refactor props and inits (See PR: https://github.com/drhuffman12/ai4cr/pull/57)
- [ ] Add a net mixer
- [ ] (?) ~~
BreedParent
- [ ] Breeder class(es) (renamed to
*Manager
classes)- [x] Abstract
Breeder(T)
(See PR: https://github.com/drhuffman12/ai4cr/pull/56) - [x]
MiniNetBreeder < Breeder(MiniNet)
(See PR: https://github.com/drhuffman12/ai4cr/pull/57) - [ ]
ChainBreeder < Breeder(Chain)
- [x]
RnnSimpleBreeder < Breeder(RnnSimple)
(See PR: https://github.com/drhuffman12/ai4cr/pull/58) - [ ] (?)
BackproagationBreeder < Breeder(Backproagation)
- [x] Add misc methods for picking delta (See PR: https://github.com/drhuffman12/ai4cr/pull/57)
- [x] random (default)
- [x] based on some mix of both parent's error scores (ideal)
- [x] switch to using 'CounterSafe'
- [x] Abstract
- [x] (?)
TeamUtils
(See PR: https://github.com/drhuffman12/ai4cr/pull/59)- [x] Should mix misc combo's of nets
- [x] Should use delta based on some mix of both parent's error scores
- [x] re MiniNetManager
- [x] re RnnSimpleManager
- [x] Mod app and test code in
spec_bench/ai4cr/neural_network/rnn/rnn_simple_manager_spec.cr
re training a team of RNN nets on a text file.- See: examples/rnn_simple_manager_example.cr
- [x] Mod text util so that when converting float bits to chars, it uses: (a) 0.0 when <= 0.0 (b) 1.0 when > =1.0 (c) rounds bits [to 0.0 or 1.0]
- [x] Add
#(un)certainty
methods foriod
data - [x] Add code/tests/benches for utf text files (for RNN usage)
- [x] test: errors should decrease (kinda depends on net and training data structure/size)
- [x] update this app and shards to be Crystal v1.0 compatible
- [ ] (?) ~~
- [ ] Implement Bi-directional RNN (i.e.: RnnSimple pulls from inputs and previous time column.)
- [ ] Switch to (or add associated classes to use)
BigFloat
instead ofFloat64
- Reasons: (a) Outputs are getting too big when using RELU for larger networks. (b) I tried handling nan and infinity values via forcing to 0,1,etc or by breeding and purging, but that starts to fail for larger networks. (c) I tried auto-scaling down the initial weights (based on network params) to avoid (a), but when scaled down too much (i.e.: larger networks), I get arithmetic errors. (d) The things I have tried (or upgrade to Crystal 1.0) seems to now tend to lead to more memory usage (sometimes maxing out my memory).
- [ ] Refactor
Chain
?); e.g.: - [ ] Fix
spec_bench
(reChain
?); e.g.:-------- plot: '' error_stats.history: '[]'
- [ ] CI should test builds for:
- [ ] Linux
- [x] Ubuntu (x86) (from the beginning!)
- [ ] Alpine (aarch64)
- [ ] Mac
- [ ] Windows
- See: https://github.com/crystal-lang/shards/blob/master/.github/workflows/ci.yml
- [ ] Linux
- [ ] Update https://drhuffman12.github.io/ai4cr/
- [ ] (?) Phase 5: Add more examples and benchmarking (e.g.: simple wave form learning)
- [ ] WHY is the last ti's value in outputs_guessed reset to '0.0' after training (but NOT after eval'ing)??? (and NOT reset to '0.0' after next round of training???)
- [ ] (?) Convert RnnSimple's validation
@errors
toHash(<Enum>, String)
. (I switched it fromHash(Symbol, String)
toHash(String, String)
, due to issues w/#from_json
. An Enum would probably be more efficient than a String.) - [ ] Convert all specs to
Spectator
format. - [ ] Likewise, split up
Chain
intochain_concerns
and fix where applicable.- Check for cleanup in app and tests
- [ ] (re)Update docs (written to https://drhuffman12.github.io/ai4cr/)
- [ ] Code cleanup!
- [ ] Update version!!!