Codi stops working at around 1000 characters on BSD

[metakirby5]

This is an issue with BSD's script, with no known workaround.

[metakirby5]

Plan to fix this with Python's pty module.

[lukelbd]

Sorry to bombard your inbox but are you still planning to address this? I'm trying to use codi for scientific calculations for my research, which means lots and lots of comments, so the 1000 character limit is unfortunately a dealbreaker :/.

[metakirby5]

No worries! I unfortunately don't have available time to fix this issue myself. I am more than happy to review and take in pull requests, though.

IMO, Codi really shouldn't be used for long scripts, since the implementation re-runs the entire script with every keystroke. I would consider something more geared towards research like Jupyter for your use case instead.

[lukelbd]

No worries. Just so you know I tried to make this work with python's pty module and it appears to have the same 1000-character limit. Though I just found a python "replwrap" module that might be promising. Maybe will try re-writing my fork around this module some day.

[lukelbd]

Ok I just figured out a much much simpler workaround... simply define a rephraser that removes the comment characters. This covers many of my use cases -- I just need lots of comments to explain the calculations but very little actual code.

Here's a sample .vimrc configuration:

" Return the searchable comment character
function! s:comment_char() abort
  let string = substitute(&commentstring, '%s.*', '', '')  " leading comment indicator
  let string = substitute(string, '\s\+', '', 'g')  " ignore spaces
  return escape(string, '[]\.*$~')  " escape magic characters
endfunction

" Pre-processor fixes escapes returned by interpreters. For the
" escape issue see: https://github.com/metakirby5/codi.vim/issues/120
function! s:codi_preprocess(line) abort
  return substitute(a:line, '�[?2004l', '', '')
endfunction

" Rephraser to remove comment characters before passing to interpreter. For the
" 1000 char limit issue see: https://github.com/metakirby5/codi.vim/issues/88
" Warning: Vim substitute() function works differently fron :substitute command, with
" escape characters disallowed in [] (requires literals) and '.' matching newlines.
" Also codi silently fials if rephrased input lines don't match original line count.
function! s:codi_rephrase(text) abort
  let text = substitute(a:text, s:comment_char() . "[^\n]*\\(\n\\|$\\)", '\1', 'g')
  if len(text) > 1000
    echohl WarningMsg
    echom 'Warning: Too many characters for calculator. Truncating the input.'
    echohl None
  endif
  return text
endfunction

" Declare interpreters with preprocessors/rephraser
let g:codi#interpreters = {
  \ 'python': {
      \ 'bin': ['python3', '-i', '-c', "'import readline; readline.set_auto_history(False)'"],
      \ 'prompt': '^\(>>>\|\.\.\.\) ',
      \ 'quitcmd': 'exit()',
      \ 'preprocess': function('s:codi_preprocess'),
      \ 'rephrase': function('s:codi_rephrase'),
      \ },
  \ 'julia': {
      \ 'bin': ['julia', '-q', '-i', '--color=no', '--history-file=no'],
      \ 'prompt': '^\(julia>\|      \)',
      \ 'quitcmd': 'exit()',
      \ 'preprocess': function('s:codi_preprocess'),
      \ 'rephrase': function('s:codi_rephrase'),
      \ },
  \ }

Note codi silently fails if I change the number of newlines so I take care to preserve them. A better version of this function might successively truncate lines beyond the 1000 character limit but will cross that bridge when I come to it.

[lukelbd]

Ran into the 1000 character problem quickly so here's a function that auto-truncates to 1000 characters, preserving newlines and deleting code fragments. When this happens I can usually just split the file in two. This is really helpful because it's now really obvious when the file is too long rather than this silent failure where some other thing could be wrong.

function! s:codi_rephrase(text) abort
  let pat = '\s*' . utils#comment_char() . '[^\n]*\(\n\|$\)'  " remove comments
  let text = substitute(a:text, pat, '\1', 'g')
  let maxlen = 950  " too close to 1000 gets risky
  let cutoff = maxlen
  while len(text) > maxlen
    let cutoff -= count(text[cutoff:], "\n")
    if exists('prevlen') && prevlen == len(text) || cutoff < 0 | break | endif
    let prevlen = len(text)
    let text = ''
      \ . substitute(text[:cutoff - 1], '\(^\|\n\)[^\n]*$', '\n', '')
      \ . substitute(text[cutoff:], '[^\n]', '', 'g')
  endwhile
  return text
endfunction