grammars-v4 icon indicating copy to clipboard operation
grammars-v4 copied to clipboard

Published 20 hours ago • verified publisher icon antlr

[PLSQL java target] Java heap space error

Open wegoodwin opened this issue 3 years ago • 6 comments

I decided to run a test on the PLSQL grammar that has about 9000 insert statements. When I try to build a parse tree, I get the following error:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at java.lang.Integer.toString(Integer.java:401)
	at java.lang.String.valueOf(String.java:3099)
	at plsqlparser.PLSQLListner.exitEveryRule(PLSQLListner.java:145)
	at org.antlr.v4.runtime.tree.ParseTreeWalker.exitRule(ParseTreeWalker.java:63)
	at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:38)
	at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:36)
	at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:36)
	at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:36)
	at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:36)
	at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:36)
	at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:36)
	at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:36)
	at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:36)
	at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:36)
	at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:36)
	at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:36)
	at plsqlparser.PLSQLCodeReview.buildSRCAST(PLSQLCodeReview.java:122)
	at plsqlparser.PLSQLCodeReview.start(PLSQLCodeReview.java:50)
	at CreateAST.main(CreateAST.java:111)

If I take the number of records in the file down to about 1000 it seems to work fine. Any help would be greatly appreciated.

LargeTest.log

wegoodwin avatar Sep 01 '22 11:09 wegoodwin

In my experience with it, you have two options as it is now.

  1. increase the heap size for Java when you run the parser
  2. reduce the size of each file you pass in like you already did.

Commodore68 avatar Sep 03 '22 19:09 Commodore68

I was able to increase the size of the heap and reduce the size of the file to get around the issue. It just seems like such a strange thing to have to do to process 10K lines of code.

From: Commodore68 @.> Sent: Saturday, September 3, 2022 3:00 PM To: antlr/grammars-v4 @.> Cc: wegoodwin @.>; Author @.> Subject: Re: [antlr/grammars-v4] [PLSQL java target] Java heap space error (Issue #2791)

In my experience with it, you have two options as it now.

  1. increase the heap size for Java when you run the parser
  2. reduce the size of each file you pass in like you already did.

— Reply to this email directly, view it on GitHubhttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fantlr%2Fgrammars-v4%2Fissues%2F2791%23issuecomment-1236182573&data=05%7C01%7C%7C98730bd00fa24efea15408da8dde950f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637978284369989644%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=KJJXJE2MgM6%2Bmq7zKcdJO4SM%2FiLUWhNw%2FjvXq5ulQi4%3D&reserved=0, or unsubscribehttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAKTMT3CJ3I5P6OWGYA5GEU3V4ON5FANCNFSM6AAAAAAQCIFJXM&data=05%7C01%7C%7C98730bd00fa24efea15408da8dde950f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637978284369989644%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=TGENlWiy8DBS4ARFpuhsWD1ZFAohy1U2Y%2F%2Bi0%2B3E32k%3D&reserved=0. You are receiving this because you authored the thread.Message ID: @.***>

wegoodwin avatar Sep 04 '22 01:09 wegoodwin

Have you tried to use parse tree free processing strategy (buildParseTrees = false)? It allows creating AST during parsing using Listener without creating full parse tree. It saves a lot of memory and increased performance.

KvanTTT avatar Sep 04 '22 11:09 KvanTTT

Hi Ivan,

How do I do this? I've not heard of this.

From: Ivan Kochurkin @.> Sent: Sunday, September 4, 2022 7:15 AM To: antlr/grammars-v4 @.> Cc: wegoodwin @.>; Author @.> Subject: Re: [antlr/grammars-v4] [PLSQL java target] Java heap space error (Issue #2791)

Have you tried to use parse tree free processing strategy? It allows creating AST during parsing without creating full parse tree.

— Reply to this email directly, view it on GitHubhttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fantlr%2Fgrammars-v4%2Fissues%2F2791%23issuecomment-1236314362&data=05%7C01%7C%7Cd6cade3d57804d7c7c7408da8e66b7fe%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637978869074788867%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ENfQzj0d%2BbpXn9YTgkVACzyi0lVbR49Fr3ur5BA53%2Fs%3D&reserved=0, or unsubscribehttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAKTMT3DGJ4FNZDLQUWQU5UDV4SADRANCNFSM6AAAAAAQCIFJXM&data=05%7C01%7C%7Cd6cade3d57804d7c7c7408da8e66b7fe%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637978869074788867%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=CbOEZgIRzZu%2BA6OsVBSJW8wjLrtjLkRwB%2BXm1bjCwXQ%3D&reserved=0. You are receiving this because you authored the thread.Message ID: @.***>

wegoodwin avatar Sep 04 '22 12:09 wegoodwin

How do I do this? I've not heard of this.

ANTLR generates Parse Tree instead of AST, but the latest also can be generated by hand without generating Parse Tree using listeners and bottom-up AST building. See PlSqlAntlrListenerConverter for example. To skip Parse Tree building, the option BuildParseTree should be set up to false.

Such an approach may reduce memory pressure and improve performance especially for huge files (1KK lines) because in general, AST has much fewer nodes than Parse Tree.

In my opinion, it's an underestimated piece of advice about performance because I've never read about it in ANTLR articles.

It looks like you already use listeners, thus my advice is not actual.

KvanTTT avatar Sep 05 '22 15:09 KvanTTT

OK,

Thanks. I will research this.

From: Ivan Kochurkin @.> Sent: Monday, September 5, 2022 11:19 AM To: antlr/grammars-v4 @.> Cc: wegoodwin @.>; Author @.> Subject: Re: [antlr/grammars-v4] [PLSQL java target] Java heap space error (Issue #2791)

ANTLR generates Parse Tree instead of AST, but the latest also can be generated by hand without generating Parse Tree using listeners and bottom-up AST building. See PlSqlAntlrListenerConverterhttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FKvanTTT%2FPT.PM%2Fblob%2Fdev%2FSources%2FPT.PM.PlSqlParseTreeUst%2FPlSqlAntlrListenerConverter.cs&data=05%7C01%7C%7Cb4a6eec0b6a24e8e8be208da8f52152f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637979879946322047%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=44Zz%2FVfvDeVu19Zq0s4Y8zCzIm4uiSe4Co3itwJddD4%3D&reserved=0 for example. To skip Parse Tree building, the option BuildParseTree should be set up to false.

Such an approach may reduce memory pressure and improve performance especially for huge files (1KK lines) because in general, AST has much fewer nodes than Parse Tree.

In my opinion, it's an underestimated piece of advice about performance because I've never read about it in ANTLR articles.

— Reply to this email directly, view it on GitHubhttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fantlr%2Fgrammars-v4%2Fissues%2F2791%23issuecomment-1237197824&data=05%7C01%7C%7Cb4a6eec0b6a24e8e8be208da8f52152f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637979879946322047%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=rs5e%2B2tS110Ba0wGaPV4Rjj%2FpysdjjMH3O2l5qJJGiE%3D&reserved=0, or unsubscribehttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAKTMT3HB2MR7QH5O5OHEUL3V4YFRRANCNFSM6AAAAAAQCIFJXM&data=05%7C01%7C%7Cb4a6eec0b6a24e8e8be208da8f52152f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637979879946478282%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=YcrzjYFVIaPXt%2B15JxRHE56cP%2BIqb1VqMlCiaXRVLqA%3D&reserved=0. You are receiving this because you authored the thread.Message ID: @.***>

wegoodwin avatar Sep 06 '22 19:09 wegoodwin