ai4cr icon indicating copy to clipboard operation
ai4cr copied to clipboard

Add simple RNN functionality

Open drhuffman12 opened this issue 4 years ago • 0 comments

Add simple RNN functionality (See Issue: https://github.com/drhuffman12/ai4cr/issues/15 )

  • [x] Phase 1: initial prototype/skeleton (w/ basic structure calc's) (See PR: https://github.com/drhuffman12/ai4cr/pull/46 )
  • [x] Phase 2: expand prototype/skeleton (to include the associated 'mini_net' objects) (See PR: https://github.com/drhuffman12/ai4cr/pull/47 )
  • [x] Phase 3: expand prototype/skeleton (to include the methods to 'guess' and 'train') (See PR: https://github.com/drhuffman12/ai4cr/pull/47 )
  • [x] Phase 4: FIX to/from json (use type key?) (in a later PR, at least partly in PR: #47)
  • [x] Add code to: (See PR: https://github.com/drhuffman12/ai4cr/pull/51 )
    • [x] Collect (split?) a 'sequence of input and output data' into small chunks for use by RnnSimple#train(..)
    • [x] Add a variation of RnnSimple#train(..) that loops thru the 'sequence of input and output data'
  • [x] add initial BreederUtils and Breeder (See PR: https://github.com/drhuffman12/ai4cr/pull/54)
  • [x] Consolidate error_distance related code into ErrorStats (See PR: https://github.com/drhuffman12/ai4cr/pull/56 and https://github.com/crystal-lang/crystal/tree/master/.github/workflows)
  • [x] Convert misc classes to use ErrorStats (See PR: https://github.com/drhuffman12/ai4cr/pull/56)
  • [x] Reorg/refactor props and inits (See PR: https://github.com/drhuffman12/ai4cr/pull/57)
  • [ ] Add a net mixer
    • [ ] (?) ~~BreedParent
    • [ ] Breeder class(es) (renamed to *Manager classes)
      • [x] Abstract Breeder(T) (See PR: https://github.com/drhuffman12/ai4cr/pull/56)
      • [x] MiniNetBreeder < Breeder(MiniNet) (See PR: https://github.com/drhuffman12/ai4cr/pull/57)
      • [ ] ChainBreeder < Breeder(Chain)
      • [x] RnnSimpleBreeder < Breeder(RnnSimple) (See PR: https://github.com/drhuffman12/ai4cr/pull/58)
      • [ ] (?) BackproagationBreeder < Breeder(Backproagation)
      • [x] Add misc methods for picking delta (See PR: https://github.com/drhuffman12/ai4cr/pull/57)
        • [x] random (default)
        • [x] based on some mix of both parent's error scores (ideal)
      • [x] switch to using 'CounterSafe'
    • [x] (?) TeamUtils (See PR: https://github.com/drhuffman12/ai4cr/pull/59)
      • [x] Should mix misc combo's of nets
      • [x] Should use delta based on some mix of both parent's error scores
      • [x] re MiniNetManager
      • [x] re RnnSimpleManager
    • [x] Mod app and test code in spec_bench/ai4cr/neural_network/rnn/rnn_simple_manager_spec.cr re training a team of RNN nets on a text file.
      • See: examples/rnn_simple_manager_example.cr
      • [x] Mod text util so that when converting float bits to chars, it uses: (a) 0.0 when <= 0.0 (b) 1.0 when > =1.0 (c) rounds bits [to 0.0 or 1.0]
      • [x] Add #(un)certainty methods for iod data
      • [x] Add code/tests/benches for utf text files (for RNN usage)
      • [x] test: errors should decrease (kinda depends on net and training data structure/size)
      • [x] update this app and shards to be Crystal v1.0 compatible
  • [ ] Implement Bi-directional RNN (i.e.: RnnSimple pulls from inputs and previous time column.)
  • [ ] Switch to (or add associated classes to use) BigFloat instead of Float64
    • Reasons: (a) Outputs are getting too big when using RELU for larger networks. (b) I tried handling nan and infinity values via forcing to 0,1,etc or by breeding and purging, but that starts to fail for larger networks. (c) I tried auto-scaling down the initial weights (based on network params) to avoid (a), but when scaled down too much (i.e.: larger networks), I get arithmetic errors. (d) The things I have tried (or upgrade to Crystal 1.0) seems to now tend to lead to more memory usage (sometimes maxing out my memory).
  • [ ] Refactor Chain?); e.g.:
  • [ ] Fix spec_bench (re Chain?); e.g.:
    --------
      plot: ''
      error_stats.history: '[]'
    
  • [ ] CI should test builds for:
    • [ ] Linux
      • [x] Ubuntu (x86) (from the beginning!)
      • [ ] Alpine (aarch64)
    • [ ] Mac
    • [ ] Windows
    • See: https://github.com/crystal-lang/shards/blob/master/.github/workflows/ci.yml
  • [ ] Update https://drhuffman12.github.io/ai4cr/
  • [ ] (?) Phase 5: Add more examples and benchmarking (e.g.: simple wave form learning)
  • [ ] WHY is the last ti's value in outputs_guessed reset to '0.0' after training (but NOT after eval'ing)??? (and NOT reset to '0.0' after next round of training???)
  • [ ] (?) Convert RnnSimple's validation @errors to Hash(<Enum>, String). (I switched it from Hash(Symbol, String) to Hash(String, String), due to issues w/ #from_json. An Enum would probably be more efficient than a String.)
  • [ ] Convert all specs to Spectator format.
  • [ ] Likewise, split up Chain into chain_concerns and fix where applicable.
    • Check for cleanup in app and tests
  • [ ] (re)Update docs (written to https://drhuffman12.github.io/ai4cr/)
  • [ ] Code cleanup!
  • [ ] Update version!!!

drhuffman12 avatar Dec 17 '20 06:12 drhuffman12