node-html-to-text

node-html-to-text copied to clipboard

[44605:0x103ef2000] 45258 ms: Mark-sweep 2049.2 (2054.1) -> 2042.0 (2046.9) MB, 5.7 / 0.0 ms (+ 3.3 ms in 1 steps since start of marking, biggest step 3.3 ms, walltime since start of marking 11 ms) (average mu = 0.386, current mu = 0.389) allocation f[44605:0x103ef2000] 45261 ms: Scavenge 2045.6 (2050.5) -> 2045.6 (2050.5) MB, 1.1 / 0.0 ms (average mu = 0.386, current mu = 0.389) allocation failure

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x178b8c0c0921 <JSObject> 0: builtin exit frame: lastIndexOf(this=0x178bb0c00411 <Very long string[1893525]>,0x178b7030db49 <String[#1]: \n>,0x178bb0c00411 <Very long string[1893525]>)

1: /* anonymous */ [0x178ba71e0941] [/Library/WebServer/Documents/fraud-detection/node_modules/html-to-text/lib/html-to-text.js:181] [bytecode=0x178b7de7d141 offset=873](this=0x178b31382239 <JSGlobal Object>,0x178bf8437579 <Objec...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory 1: 0x100b9f982 node::Abort() (.cold.1) [/usr/local/bin/node] 2: 0x100084338 node::FatalError(char const*, char const*) [/usr/local/bin/node] 3: 0x100084460 node::OnFatalError(char const*, char const*) [/usr/local/bin/node] 4: 0x10017dfc1 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node] 5: 0x10017df6b v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node] 6: 0x100297f23 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/usr/local/bin/node] 7: 0x1002992a8 v8::internal::Heap::MarkCompactPrologue() [/usr/local/bin/node] 8: 0x100296e76 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/local/bin/node] 9: 0x100295955 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node] 10: 0x10029d2f4 v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node] 11: 0x10029d34a v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node] 12: 0x10027a4f5 v8::internal::Factory::AllocateRawWithImmortalMap(int, v8::internal::AllocationType, v8::internal::Map, v8::internal::AllocationAlignment) [/usr/local/bin/node] 13: 0x10027ca47 v8::internal::Factory::NewRawTwoByteString(int, v8::internal::AllocationType) [/usr/local/bin/node] 14: 0x10043f552 v8::internal::String::SlowFlatten(v8::internal::Isolate*, v8::internal::Handlev8::internal::ConsString, v8::internal::AllocationType) [/usr/local/bin/node] 15: 0x1004438c0 v8::internal::String::LastIndexOf(v8::internal::Isolate*, v8::internal::Handlev8::internal::Object, v8::internal::Handlev8::internal::Object, v8::internal::Handlev8::internal::Object) [/usr/local/bin/node] 16: 0x100202b7f v8::internal::Builtin_Impl_StringPrototypeLastIndexOf(v8::internal::BuiltinArguments, v8::internal::Isolate*) [/usr/local/bin/node] 17: 0x10074e6d9 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_BuiltinExit [/usr/local/bin/node] 18: 0x1006da77b Builtins_InterpreterEntryTrampoline [/usr/local/bin/node] Abort trap: 6

This is pretty useless without the input string that caused it.

(You might not need to share the exact data you working with. Depersonalized data that still triggers the issue - will suffice.)

@KillyMXI The data was confidential, so I didn't share. I am sharing another snippet with which i've replicated the issue.

Text: 7247 | Text: 3814-08 | Text: text Textttt 2.0 Texttt - Ztextt | Seottttt | error: Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool.  This may have occurred because all pooled connections were in use and max pool size was reached.\n<br />\n   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)\n   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)\n   at System.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)\n   at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)\n   at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)\n   at System.Data.SqlClient.SqlConnection.Open()\n   at ServiceStack.OrmLite.OrmLiteConnection.Open()\n   at ServiceStack.OrmLite.OrmLiteConnectionFactory.OpenDbConnection()\n   at Erp.External.ServiceStack.OrmLite.OrmLiteBase.get_ErpDb() in C:\\Users\\Wesley.PACOLIBA\\Documents\\## Code\\ERP\\intershoe\\Erp.External.ServiceStack.OrmLite\\OrmLiteBase.cs:line 31\n   at Erp.Service.Catalog.SpecificationAttributeService.GetProductSpecificationAttributesByProductId(Int32 productId) in C:\\Users\\Wesley.PACOLIBA\\Documents\\## Code\\ERP\\intershoe\\Erp.Service\\Catalog\\SpecificationAttributeService.cs:line 144\n   at Erp.External.Shopware.ShopwareSyncService.<ProductToWebshopAsync>d__27.MoveNext() in C:\\Users\\Wesley.PACOLIBA\\Documents\\## Code\\ERP\\intershoe\\Erp.External.Shopware\\ShopwareSyncService.cs:line 1377\n--- End of stack trace from previous location where exception was thrown ---\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\n   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()\n   at Erp.Web.Api.Jobs.SyncJob.<ProductSync>d__14.MoveNext() in C:\\Users\\Wesley.PACOLIBA\\Documents\\## Code\\ERP\\intershoe\\Erp.Web.Api\\Jobs\\SyncJob.cs:line 125\n<br />\n*******************************\n<br />\nFile: C:\\Users\\Wesley.PACOLIBA\\Documents\\## Code\\ERP\\intershoe\\Erp.External.ServiceStack.OrmLite\\OrmLiteBase.cs\n<br />\nMethod:get_ErpDb\n<br />\nLineNumber: 31\n<br />\n*******************************\n*******************************\n<br />\nFile: C:\\Users\\Wesley.PACOLIBA\\Documents\\## Code\\ERP\\intershoe\\Erp.Service\\Catalog\\SpecificationAttributeService.cs\n<br />\nMethod:GetProductSpecificationAttributesByProductId\n<br />\nLineNumber: 144\n<br />\n*******************************\n*******************************\n<br />\nFile: C:\\Users\\Wesley.PACOLIBA\\Documents\\## Code\\ERP\\intershoe\\Erp.External.Shopware\\ShopwareSyncService.cs\n<br />\nMethod:MoveNext\n<br />\nLineNumber: 1377\n<br />\n*******************************\n*******************************\n<br />\nFile: C:\\Users\\Wesley.PACOLIBA\\Documents\\## Code\\ERP\\intershoe\\Erp.Web.Api\\Jobs\\SyncJob.cs\n<br />\nMethod:MoveNext\n<br />\nLineNumber: 125\n<br />\n*******************************\n\n<br />\n-----------------------------------------------------------------------------------------\n<br />\n-----------------------------------------------------------------------------------------\n<br />\n<hr/>

When I've passed 1000 times of this text like this

var content; //above content
var html; //same content, just 1000 times bigger
for (var i = 0; i < 1000; i++) {
        html += content; 
}
var plainTextContent = htmlToText.fromString(html, {preserveNewlines: true});

Ok. First of all, I can't see why you trying to use html-to-text on this - it doesn't look any close to valid html. To make things worse, it has stack trace lines with fragments resembling html tags that should've been html-escaped before mixing with html tags. If all you need is to get your <br />s and <hr />s transformed - you are better off with just a replace().

We have a known related issue - #172. When solved - it will make possible to avoid fatal error by the cost of incompletely processed input. And with this particular input it will still produce garbage. Remember - Garbage In - Garbage Out!

I think we need a configurable behavior for what to do with unknown tags, but that will have to wait.