Ability to add new keywords to existing languages

[bobbylight]

Copied over from SourceForge:

It would be handy to be able to add new keywords to existing language scanners, such as Java. This would allow users to leverage existing TokenMaker implementations with minimal fuss, assuming all they want is new keywords. This would be accomplished by checking each normal identifier parsed against the "new keyword" list. If found, it's highlighted as a keyword. One possible issue is performance, although there are TokenMakers in RSTA (WindowsBatchTokenMaker, UnixShellTokenMaker) that already work in the manner suggested above and don't seem to have performance issues. Another thing to consider is making ALL keywords done in this manner (e.g. none hard-coded). This would allow users to replace a language scanner's keywords, not just append to it. This same technique could also be applied to the "Function" token type as well as Keywords.

[ricardojlrufino]

I do not know if it's ideal way, but is an example for those who need.

import org.fife.ui.rsyntaxtextarea.TokenMap;
import org.fife.ui.rsyntaxtextarea.TokenTypes;
import org.fife.ui.rsyntaxtextarea.modes.CPlusPlusTokenMaker;

public class PdeTokenMaker extends CPlusPlusTokenMaker {

  static TokenMap extraTokens;

  public PdeTokenMaker() {
      extraTokens = getKeywords();
  }

  @Override
  public void addToken(char[] array, int start, int end, int tokenType, int startOffset, boolean hyperlink) {
      // This assumes all of your extra tokens would normally be scanned as IDENTIFIER.
      if (tokenType == TokenTypes.IDENTIFIER) {
          int newType = extraTokens.get(array, start, end);
          if (newType>-1) {
              tokenType = newType;
          }
      }
      super.addToken(array, start, end, tokenType, startOffset, hyperlink);
  }

  public void addKeyword(String keyword, int type) {
      extraTokens.put(keyword, type);
  }

  public void clear() {
      extraTokens = new TokenMap();
  }


  /**
   * Handles loading of keywords file.
   * <P>
   * It is recommended that a # sign be used for comments
   * inside keywords.txt.
   */
  static public TokenMap getKeywords() {
    if (extraTokens == null) {
      try {
        extraTokens = new TokenMap(false);

        HashMap<String, Integer> keywords = PdeKeywords.get();
        Set<String> keys = keywords.keySet();
        for (String key : keys) {
          extraTokens.put(key, keywords.get(key));
        }

      } catch (Exception e) {
        Base.showError("Problem loading keywords",
                          "Could not load keywords.txt,\n" +
                          "please re-install Arduino.", e);
        System.exit(1);
      }
    }
    return extraTokens;
  }

}

[BullyWiiPlaza]

This code example isn't complete, Base and PdeKeywords are classes which cannot be resolved. Where do they come from? They are also not mentioned in the imports just like java.util.HashMap and java.util.Set but the latter two are obviously easily added.

[JamzTheMan]

So has there been any more thought on this? I created a base tokenMaker class using TMM which worked well but we have a need to dynamically add new Keywords.

The above code workes decently except if I add a keyword Identifier with a . like "json.get" but these worked if I used TMM.

Is there a specific way to escape special characters added to TokenMap or am I at a loss?