Code folding/Indentation marking in DecompilerPanel

[Enigmatrix]

Is your feature request related to a problem? Please describe. For functions that are long and have many nested statements, it is hard to see the scope where a line of code is being executed, since there are so many different scopes exiting and being entered into.

Describe the solution you'd like It would be nice to have indentation markings, to know which level of scope you are in, just like here: Furthermore, if we could 'fold' certain scopes so that the view is less cluttered, it will also be useful.

[astrelsky]

I think this is a fantastic idea.

[Wall-AF]

Not a perfect solution, but this works:

1. add a FieldInputListener to the DecompilerPanel to pick up keyboard input (see https://github.com/NationalSecurityAgency/ghidra/issues/4264#issuecomment-1133584444);
2. add a property called collapsedToken to class ClangToken together with its getter and setter;
3. modify the new `DecompilerPanel.keyPressed(...) method to respond to two additional keys (one to collapse a code section and another to reopen it), e.g.:

	public static final KeyStroke SELECT = KeyStroke.getKeyStroke(KeyEvent.VK_ENTER, 0);
	public static final KeyStroke HIDE = KeyStroke.getKeyStroke(KeyEvent.VK_MINUS, 0);
	public static final KeyStroke SHOW = KeyStroke.getKeyStroke(KeyEvent.VK_EQUALS, 0);

	@Override
	public void keyPressed(KeyEvent ev, BigInteger index, int fieldNum, int row, int col, Field field) {
		KeyStroke keyStroke = KeyStroke.getKeyStrokeForEvent(ev);
		FieldLocation location = getCursorPosition();
		ClangTextField textField = (ClangTextField) field;
		ClangToken token = textField.getToken(location);

		if (SELECT.equals(keyStroke)) {
			if (DockingUtils.isControlModifier(ev)) {
				tryToGoto(location, field, ev, true);
			}
			else {
				tryToGoto(location, field, ev, false);
			}
		}
		else if (SHOW.equals(keyStroke)) {
			if (token instanceof ClangSyntaxToken) {
				toggleCollapseToken((ClangSyntaxToken) token, false);
			}
		}
		else if (HIDE.equals(keyStroke)) {
			if (token instanceof ClangSyntaxToken) {
				toggleCollapseToken((ClangSyntaxToken) token, true);
			}
		}
	}

	private void toggleCollapseToken(ClangSyntaxToken startToken, boolean isCollapsed) {
		if ("{".equals(startToken.getText())) {
			ClangSyntaxToken closingBrace = DecompilerUtils.getMatchingBrace(startToken);
			if (closingBrace == null) {
				return;
			}

			ClangNode parent = startToken.Parent();
			List<ClangNode> list = new ArrayList<>();
			parent.flatten(list);

			boolean inSection = false;
			for (ClangNode element : list) {
				ClangToken token = (ClangToken) element;
				if (inSection) {
					if ((token instanceof ClangSyntaxToken)) {
						inSection = (closingBrace != token);
					}
					if (inSection) {
						token.setCollapsedToken(isCollapsed);
					}
				}
				else if ((token instanceof ClangSyntaxToken)) {
					inSection |= (startToken == token);
				}
			}

			setDecompileData(decompileData);
		}
	}

4. modify the private ClangLayoutController.createFieldElementsForLine(...) method to NOT add a token if it has its collapsedToken property set, e.g.:

	private FieldElement[] createFieldElementsForLine(List<ClangToken> tokens) {
		List<FieldElement> elements = new ArrayList<>();
		int columnPosition = 0;
		for (int i = 0; i < tokens.size(); ++i) {
			ClangToken token = tokens.get(i);
			if (token.isCollapsedToken()) {
				continue;
			}

			Color color = syntax_color[token.getSyntaxType()];
			FieldElement el;
			if (token instanceof ClangCommentToken) {
				AttributedString prototype = new AttributedString("prototype", color, metrics);
				Program program = decompilerPanel.getProgram();
				el = CommentUtils.parseTextForAnnotations(token.getText(), program, prototype, 0);
				columnPosition += el/*ements[i]*/.length();
			}
			else {
				AttributedString as = new AttributedString(token.getText(), color, metrics);
				el = new ClangFieldElement(token, as, columnPosition);
				columnPosition += as.length();
			}
			elements.add(el);
		}
		return elements.toArray(FieldElement[]::new);
	}

As this uses the { token as the trigger for collapsing and subsequently reshowing a code block, there is always at least a couple of lines to return.

[Mgamerz]

Would love to see this implemented. Right now I have to deal with fun things like:

[orenbenya1]

Not a perfect solution, but this works:

1. add a FieldInputListener to the DecompilerPanel to pick up keyboard input (see Keyboard shortcut for moving to other brace - "Decompile:" panel #4264 (comment));
2. add a property called collapsedToken to class ClangToken together with its getter and setter;
3. modify the new `DecompilerPanel.keyPressed(...) method to respond to two additional keys (one to collapse a code section and another to reopen it), e.g.:

	public static final KeyStroke SELECT = KeyStroke.getKeyStroke(KeyEvent.VK_ENTER, 0);
	public static final KeyStroke HIDE = KeyStroke.getKeyStroke(KeyEvent.VK_MINUS, 0);
	public static final KeyStroke SHOW = KeyStroke.getKeyStroke(KeyEvent.VK_EQUALS, 0);

	@Override
	public void keyPressed(KeyEvent ev, BigInteger index, int fieldNum, int row, int col, Field field) {
		KeyStroke keyStroke = KeyStroke.getKeyStrokeForEvent(ev);
		FieldLocation location = getCursorPosition();
		ClangTextField textField = (ClangTextField) field;
		ClangToken token = textField.getToken(location);

		if (SELECT.equals(keyStroke)) {
			if (DockingUtils.isControlModifier(ev)) {
				tryToGoto(location, field, ev, true);
			}
			else {
				tryToGoto(location, field, ev, false);
			}
		}
		else if (SHOW.equals(keyStroke)) {
			if (token instanceof ClangSyntaxToken) {
				toggleCollapseToken((ClangSyntaxToken) token, false);
			}
		}
		else if (HIDE.equals(keyStroke)) {
			if (token instanceof ClangSyntaxToken) {
				toggleCollapseToken((ClangSyntaxToken) token, true);
			}
		}
	}

	private void toggleCollapseToken(ClangSyntaxToken startToken, boolean isCollapsed) {
		if ("{".equals(startToken.getText())) {
			ClangSyntaxToken closingBrace = DecompilerUtils.getMatchingBrace(startToken);
			if (closingBrace == null) {
				return;
			}

			ClangNode parent = startToken.Parent();
			List<ClangNode> list = new ArrayList<>();
			parent.flatten(list);

			boolean inSection = false;
			for (ClangNode element : list) {
				ClangToken token = (ClangToken) element;
				if (inSection) {
					if ((token instanceof ClangSyntaxToken)) {
						inSection = (closingBrace != token);
					}
					if (inSection) {
						token.setCollapsedToken(isCollapsed);
					}
				}
				else if ((token instanceof ClangSyntaxToken)) {
					inSection |= (startToken == token);
				}
			}

			setDecompileData(decompileData);
		}
	}

4. modify the private ClangLayoutController.createFieldElementsForLine(...) method to NOT add a token if it has its collapsedToken property set, e.g.:

	private FieldElement[] createFieldElementsForLine(List<ClangToken> tokens) {
		List<FieldElement> elements = new ArrayList<>();
		int columnPosition = 0;
		for (int i = 0; i < tokens.size(); ++i) {
			ClangToken token = tokens.get(i);
			if (token.isCollapsedToken()) {
				continue;
			}

			Color color = syntax_color[token.getSyntaxType()];
			FieldElement el;
			if (token instanceof ClangCommentToken) {
				AttributedString prototype = new AttributedString("prototype", color, metrics);
				Program program = decompilerPanel.getProgram();
				el = CommentUtils.parseTextForAnnotations(token.getText(), program, prototype, 0);
				columnPosition += el/*ements[i]*/.length();
			}
			else {
				AttributedString as = new AttributedString(token.getText(), color, metrics);
				el = new ClangFieldElement(token, as, columnPosition);
				columnPosition += as.length();
			}
			elements.add(el);
		}
		return elements.toArray(FieldElement[]::new);
	}

As this uses the { token as the trigger for collapsing and subsequently reshowing a code block, there is always at least a couple of lines to return.

Can you share an updated version of this please? I think it is a necessary feature to have during the reversing process.

[Wall-AF]

My change in https://github.com/NationalSecurityAgency/ghidra/issues/1294#issuecomment-1133644447 hasn't changed.

[orenbenya1]

My change in #1294 (comment) hasn't changed.

Thanks for the quick response, while trying to fold the code I got the following error (which is whyI assumed it needed to be updated):

NullPointerException - Cannot invoke "docking.widgets.fieldpanel.field.FieldElement.getStringWidth()" because 
"fieldElement" is null java.lang.NullPointerException: Cannot invoke "docking.widgets.fieldpanel.field.FieldElement.getStringWidth()" because "fieldElement" is null
	at docking.widgets.fieldpanel.field.CompositeFieldElement.getStringWidth(CompositeFieldElement.java:176)
	at docking.widgets.fieldpanel.support.FieldUtils.wrap(FieldUtils.java:87)
	at docking.widgets.fieldpanel.field.WrappingVerticalLayoutTextField.<init>(WrappingVerticalLayoutTextField.java:53)
	at ghidra.app.decompiler.component.ClangTextField.<init>(ClangTextField.java:36)
	at ghidra.app.decompiler.component.ClangLayoutController.createTextFieldForLine(ClangLayoutController.java:174)
	at ghidra.app.decompiler.component.ClangLayoutController.buildLayoutInternal(ClangLayoutController.java:245)
	at ghidra.app.decompiler.component.ClangLayoutController.buildLayouts(ClangLayoutController.java:328)
	at ghidra.app.decompiler.component.DecompilerPanel.setDecompileData(DecompilerPanel.java:415)
	at ghidra.app.decompiler.component.DecompilerPanel.toggleCollapseToken(DecompilerPanel.java:740)
	at ghidra.app.decompiler.component.DecompilerPanel.keyPressed(DecompilerPanel.java:707)
	at docking.widgets.fieldpanel.FieldPanel$CursorHandler.notifyInputListeners(FieldPanel.java:2472)
	at docking.widgets.fieldpanel.FieldPanel$FieldPanelKeyAdapter.keyPressed(FieldPanel.java:1722)
	at java.desktop/java.awt.Component.processKeyEvent(Component.java:6584)
	at java.desktop/javax.swing.JComponent.processKeyEvent(JComponent.java:2896)
	at java.desktop/java.awt.Component.processEvent(Component.java:6403)
	at java.desktop/java.awt.Container.processEvent(Container.java:2266)
	at java.desktop/java.awt.Component.dispatchEventImpl(Component.java:5001)
	at java.desktop/java.awt.Container.dispatchEventImpl(Container.java:2324)
	at java.desktop/java.awt.Component.dispatchEvent(Component.java:4833)
	at java.desktop/java.awt.KeyboardFocusManager.redispatchEvent(KeyboardFocusManager.java:1952)
	at java.desktop/java.awt.DefaultKeyboardFocusManager.dispatchKeyEvent(DefaultKeyboardFocusManager.java:883)
	at java.desktop/java.awt.DefaultKeyboardFocusManager.preDispatchKeyEvent(DefaultKeyboardFocusManager.java:1150)
	at java.desktop/java.awt.DefaultKeyboardFocusManager.typeAheadAssertions(DefaultKeyboardFocusManager.java:1020)
	at java.desktop/java.awt.DefaultKeyboardFocusManager.dispatchEvent(DefaultKeyboardFocusManager.java:848)
	at java.desktop/java.awt.Component.dispatchEventImpl(Component.java:4882)
	at java.desktop/java.awt.Container.dispatchEventImpl(Container.java:2324)
	at java.desktop/java.awt.Window.dispatchEventImpl(Window.java:2780)
	at java.desktop/java.awt.Component.dispatchEvent(Component.java:4833)
	at java.desktop/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:775)
	at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:720)
	at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:714)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:86)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:97)
	at java.desktop/java.awt.EventQueue$5.run(EventQueue.java:747)
	at java.desktop/java.awt.EventQueue$5.run(EventQueue.java:745)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:86)
	at java.desktop/java.awt.EventQueue.dispatchEvent(EventQueue.java:744)
	at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:203)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:124)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:113)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:109)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
	at java.desktop/java.awt.EventDispatchThread.run(EventDispatchThread.java:90)

[Wall-AF]

Humble apologies, I've found a change to DecompilePanel.toggleCollapsedToken(...). See below:

	private void toggleCollapseToken(ClangSyntaxToken firstToken, boolean isCollapsed) {
		if (DecompilerUtils.isBrace(firstToken)) {
			ClangSyntaxToken closingBrace = DecompilerUtils.getMatchingBrace(firstToken);
			if (closingBrace == null) {
				return;
			}
			ClangSyntaxToken openingBrace = firstToken;
			if ("}".equals(firstToken.getText())) {
				openingBrace = closingBrace;
				closingBrace = firstToken;
			}

			ClangNode parent = firstToken.Parent();
			List<ClangNode> list = new ArrayList<>();
			parent.flatten(list);

			boolean inSection = false;
			for (ClangNode element : list) {
				ClangToken token = (ClangToken) element;
				if (inSection) {
					if ((token instanceof ClangSyntaxToken)) {
						inSection = (!token.equals(closingBrace));
					}
					if (inSection) {
						token.setCollapsedToken(isCollapsed);
					}
				}
				else if ((token instanceof ClangSyntaxToken)) {
					inSection |= (token.equals(openingBrace));
				}
			}

			// IMPORTANT: to trigger redisplay
			setDecompileData(decompileData);
		}
	}

[orenbenya1]

Humble apologies, I've found a change to DecompilePanel.toggleCollapsedToken(...). See below:

	private void toggleCollapseToken(ClangSyntaxToken firstToken, boolean isCollapsed) {
		if (DecompilerUtils.isBrace(firstToken)) {
			ClangSyntaxToken closingBrace = DecompilerUtils.getMatchingBrace(firstToken);
			if (closingBrace == null) {
				return;
			}
			ClangSyntaxToken openingBrace = firstToken;
			if ("}".equals(firstToken.getText())) {
				openingBrace = closingBrace;
				closingBrace = firstToken;
			}

			ClangNode parent = firstToken.Parent();
			List<ClangNode> list = new ArrayList<>();
			parent.flatten(list);

			boolean inSection = false;
			for (ClangNode element : list) {
				ClangToken token = (ClangToken) element;
				if (inSection) {
					if ((token instanceof ClangSyntaxToken)) {
						inSection = (!token.equals(closingBrace));
					}
					if (inSection) {
						token.setCollapsedToken(isCollapsed);
					}
				}
				else if ((token instanceof ClangSyntaxToken)) {
					inSection |= (token.equals(openingBrace));
				}
			}

			// IMPORTANT: to trigger redisplay
			setDecompileData(decompileData);
		}
	}

Thanks again, yet this didnt fix the bug i mentioned before. After looking a bit, I decided that instead modifing the code at CLangLayoutController.createFieldElementsForLine as you mentioned before, I modified the function that called it CLangLayoutController.createTextFieldForLine , and added the next code:

private ClangTextField createTextFieldForLine(ClangLine line, int lineCount,
			boolean paintLineNumbers) {
		List<ClangToken> tokens = line.getAllTokens();
		
// ============ MY MODIFICATION ===================
		// Using an iterator to remove tokens with a collapsed flag
                Iterator<ClangToken> iterator = tokens.iterator();
                while (iterator.hasNext()) {
        	        ClangToken tk = iterator.next();
                    if (tk.getCollapsedToken()) {
                        iterator.remove();
                    }
                }
// =====================================

		FieldElement[] elements = createFieldElementsForLine(tokens);

		int indent = line.getIndent() * indentWidth;
		int updatedMaxWidth = maxWidth;
		return new ClangTextField(tokens, elements, indent, line.getLineNumber(), updatedMaxWidth,
			hlFactory);
}

Also added the next line to the function DecompilerUtils.findAddressBefore:

	public static Address findAddressBefore(Field[] lines, ClangToken token) {
		ClangLine lineParent = token.getLineParent();
		int lineNumber = lineParent.getLineNumber();
		for (int i = lineNumber - 1; i >= 0; i--) {
			ClangTextField textLine = (ClangTextField) lines[i];
			List<ClangToken> tokens = textLine.getTokens();
			
			// ============= ADDED LINES =======
			if (tokens.size() == 0)
				continue;
			// ===========================
			
			ClangToken addressedToken = findClosestAddressedToken(tokens.get(0));
			if (addressedToken != null) {
				return addressedToken.getMinAddress();
			}
		}
		return null;
	}

This way it worked pretty well and I was indeed able to fold the code

[Wall-AF]

Turns out I modified DecompilerUtils.toLines(...) and added

			if (tok.isCollapsedToken()) {
				continue;
			}

in the for loop effectively doing something similar. Soz I forgot!

[idan-h]

This is very needed! some times there are 20 indentations, and it is impossible to follow