parse-server
parse-server copied to clipboard
parse-community
Fetch large data for reports slow
I'm trying to fetch more than 100,000 rows for report generation. I tried these approaches,
- Recursive calls with 1,000 limit to fetch all rows
- Using query.each() and push row in array
Both the calls are slow,
For cloud code query
query = //new query query.equalTo("SOMEKEY","SOMEVALUE")
For total of 1,500,000 rows In cloud code,
In recursive call approach, there is approx 1 second waiting time per 1000 rows.
In query.each() approach this query takes approx 60 seconds to process, http waiting time
I assume this is fastest parse can work.
Can i see some improvement by indexing the mongodb objectId column?
Is there a better approach for processing such huge data.
One alternative I can think of is parse aggregate but this approach doesn't respect parse authentication, security, acl etc. and doesn't work with existing query logic. Will this show any performance improvement and worth a try?
This should not be slow. The most obvious reasons why it may be slow:
- 'SOMEKEY' is not properly indexed
- Each object is very large
Considering your question, I suspect that the problem is the item 1.
Trying properly indexing your collection.
@omkar-tenkale You can look at the logs of your database to find out how long the query takes on the DB side.
With mongo profiling i can see 58 queries are run for 58000 rows
The last 2 blocks of log
{
"op" : "getmore",
"ns" : "dev.Product",
"cursorid" : 39369748207,
"ntoreturn" : 1000,
"keyUpdates" : 0,
"numYield" : 0,
"lockStats" : {
"timeLockedMicros" : {
"r" : NumberLong(1288),
"w" : NumberLong(0)
},
"timeAcquiringMicros" : {
"r" : NumberLong(3),
"w" : NumberLong(3)
}
},
"nreturned" : 1000,
"responseLength" : 671635,
"millis" : 1,
"execStats" : {
},
"ts" : ISODate("2020-03-30T10:53:02.885Z"),
"client" : "127.0.0.1",
"allUsers" : [ ],
"user" : ""
}
{
"op" : "getmore",
"ns" : "dev.Product",
"cursorid" : 39369748207,
"ntoreturn" : 1000,
"keyUpdates" : 0,
"numYield" : 0,
"lockStats" : {
"timeLockedMicros" : {
"r" : NumberLong(1552),
"w" : NumberLong(0)
},
"timeAcquiringMicros" : {
"r" : NumberLong(2),
"w" : NumberLong(3)
}
},
"nreturned" : 1000,
"responseLength" : 682352,
"millis" : 1,
"execStats" : {
},
"ts" : ISODate("2020-03-30T10:53:02.918Z"),
"client" : "127.0.0.1",
"allUsers" : [ ],
"user" : ""
}
Type "it" for more
>
Parse is still taking upto 10 seconds for 58000 items without any query constraints
The query i simple,this is the cloud codee
async function indexTest(req,res){ console.time("timelog");
var Product = Parse.Object.extend("Product"); var query = new Parse.Query(Product); query.limit(1000000) var res = await query.find(); console.timeEnd("timelog"); console.time("timelog"); console.log("resp size=="+res.length) console.timeEnd("timelog"); return res.length; }
This is the log (pm2 logs 0)
0|app1 | timelog: 10.097s 0|app1 | resp size==58110 0|app1 | timelog: 0.145ms 0|app1 | info: Ran cloud function indexTest for user undefined with: 0|app1 | Input: {} 0|app1 | Result: 58110 {"functionName":"indexTest","params":{}}
In your mongodb logs I don't see any excessive execution times.
Can you turn on profiling to only log queries that take longer than 100ms?
Actually, the execStats are empty in the logs you posted here. The stats should give you more hints about the execution time and whether an index was used for the scan, etc.
Here's what i could find Cloud code request from parse-dashboard console
> db.setProfilingLevel(2)
{ "was" : 0, "slowms" : -1, "ok" : 1 }
> db.system.profile.find().pretty()
{
"op" : "query",
"ns" : "dev.system.indexes",
"query" : {
"expireAfterSeconds" : {
"$exists" : true
}
},
"ntoreturn" : 0,
"ntoskip" : 0,
"nscanned" : 29,
"nscannedObjects" : 29,
"keyUpdates" : 0,
"numYield" : 0,
"lockStats" : {
"timeLockedMicros" : {
"r" : NumberLong(92),
"w" : NumberLong(0)
},
"timeAcquiringMicros" : {
"r" : NumberLong(3),
"w" : NumberLong(2)
}
},
"nreturned" : 0,
"responseLength" : 20,
"millis" : 0,
"execStats" : {
"type" : "COLLSCAN",
"works" : 31,
"yields" : 0,
"unyields" : 0,
"invalidates" : 0,
"advanced" : 0,
"needTime" : 30,
"needFetch" : 0,
"isEOF" : 1,
"docsTested" : 29,
"children" : [ ]
},
"ts" : ISODate("2020-03-31T08:01:25.597Z"),
"client" : "0.0.0.0",
"allUsers" : [
{
"user" : "__system",
"db" : "local"
}
],
"user" : "__system@local"
}
>
>
>
>
> db.system.profile.find().pretty()
{
"op" : "query",
"ns" : "dev.system.indexes",
"query" : {
"expireAfterSeconds" : {
"$exists" : true
}
},
"ntoreturn" : 0,
"ntoskip" : 0,
"nscanned" : 29,
"nscannedObjects" : 29,
"keyUpdates" : 0,
"numYield" : 0,
"lockStats" : {
"timeLockedMicros" : {
"r" : NumberLong(92),
"w" : NumberLong(0)
},
"timeAcquiringMicros" : {
"r" : NumberLong(3),
"w" : NumberLong(2)
}
},
"nreturned" : 0,
"responseLength" : 20,
"millis" : 0,
"execStats" : {
"type" : "COLLSCAN",
"works" : 31,
"yields" : 0,
"unyields" : 0,
"invalidates" : 0,
"advanced" : 0,
"needTime" : 30,
"needFetch" : 0,
"isEOF" : 1,
"docsTested" : 29,
"children" : [ ]
},
"ts" : ISODate("2020-03-31T08:01:25.597Z"),
"client" : "0.0.0.0",
"allUsers" : [
{
"user" : "__system",
"db" : "local"
}
],
"user" : "__system@local"
}
{
"op" : "query",
"ns" : "dev.system.profile",
"query" : {
},
"ntoreturn" : 0,
"ntoskip" : 0,
"nscanned" : 1,
"nscannedObjects" : 1,
"keyUpdates" : 0,
"numYield" : 0,
"lockStats" : {
"timeLockedMicros" : {
"r" : NumberLong(96),
"w" : NumberLong(0)
},
"timeAcquiringMicros" : {
"r" : NumberLong(3),
"w" : NumberLong(4)
}
},
"nreturned" : 1,
"responseLength" : 649,
"millis" : 0,
"execStats" : {
"type" : "COLLSCAN",
"works" : 3,
"yields" : 0,
"unyields" : 0,
"invalidates" : 0,
"advanced" : 1,
"needTime" : 1,
"needFetch" : 0,
"isEOF" : 1,
"docsTested" : 1,
"children" : [ ]
},
"ts" : ISODate("2020-03-31T08:01:28.613Z"),
"client" : "157.33.212.99",
"allUsers" : [ ],
"user" : ""
}
{
"op" : "query",
"ns" : "dev._SCHEMA",
"query" : {
"$query" : {
}
},
"ntoreturn" : 1000,
"ntoskip" : 0,
"nscanned" : 21,
"nscannedObjects" : 21,
"keyUpdates" : 0,
"numYield" : 0,
"lockStats" : {
"timeLockedMicros" : {
"r" : NumberLong(132),
"w" : NumberLong(0)
},
"timeAcquiringMicros" : {
"r" : NumberLong(2),
"w" : NumberLong(3)
}
},
"nreturned" : 21,
"responseLength" : 6562,
"millis" : 0,
"execStats" : {
"type" : "COLLSCAN",
"works" : 23,
"yields" : 0,
"unyields" : 0,
"invalidates" : 0,
"advanced" : 21,
"needTime" : 1,
"needFetch" : 0,
"isEOF" : 1,
"docsTested" : 21,
"children" : [ ]
},
"ts" : ISODate("2020-03-31T08:02:05.054Z"),
"client" : "127.0.0.1",
"allUsers" : [ ],
"user" : ""
}
{
"op" : "query",
"ns" : "dev._Session",
"query" : {
"$orderby" : {
},
"$query" : {
"_session_token" : "r:69bf2af111cc47a67c6d3119f282e5a1"
}
},
"ntoreturn" : 1,
"ntoskip" : 0,
"nscanned" : 125,
"nscannedObjects" : 125,
"keyUpdates" : 0,
"numYield" : 0,
"lockStats" : {
"timeLockedMicros" : {
"r" : NumberLong(176),
"w" : NumberLong(0)
},
"timeAcquiringMicros" : {
"r" : NumberLong(2),
"w" : NumberLong(2)
}
},
"nreturned" : 1,
"responseLength" : 324,
"millis" : 0,
"execStats" : {
"type" : "COLLSCAN",
"works" : 126,
"yields" : 0,
"unyields" : 0,
"invalidates" : 0,
"advanced" : 1,
"needTime" : 125,
"needFetch" : 0,
"isEOF" : 0,
"docsTested" : 125,
"children" : [ ]
},
"ts" : ISODate("2020-03-31T08:02:05.056Z"),
"client" : "127.0.0.1",
"allUsers" : [ ],
"user" : ""
}
{
"op" : "query",
"ns" : "dev._User",
"query" : {
"$orderby" : {
},
"$query" : {
"_id" : "MoEjAOXpF4"
}
},
"ntoreturn" : 1000,
"ntoskip" : 0,
"nscanned" : 1,
"nscannedObjects" : 1,
"idhack" : true,
"keyUpdates" : 0,
"numYield" : 0,
"lockStats" : {
"timeLockedMicros" : {
"r" : NumberLong(42),
"w" : NumberLong(0)
},
"timeAcquiringMicros" : {
"r" : NumberLong(1),
"w" : NumberLong(2)
}
},
"nreturned" : 1,
"responseLength" : 419,
"millis" : 0,
"execStats" : {
},
"ts" : ISODate("2020-03-31T08:02:05.057Z"),
"client" : "127.0.0.1",
"allUsers" : [ ],
"user" : ""
}
{
"op" : "query",
"ns" : "dev._Session",
"query" : {
"$orderby" : {
},
"$query" : {
"_session_token" : "r:69bf2af111cc47a67c6d3119f282e5a1"
}
},
"ntoreturn" : 1000,
"ntoskip" : 0,
"nscanned" : 219,
"nscannedObjects" : 219,
"keyUpdates" : 0,
"numYield" : 0,
"lockStats" : {
"timeLockedMicros" : {
"r" : NumberLong(281),
"w" : NumberLong(0)
},
"timeAcquiringMicros" : {
"r" : NumberLong(4),
"w" : NumberLong(4)
}
},
"nreturned" : 1,
"responseLength" : 324,
"millis" : 0,
"execStats" : {
"type" : "COLLSCAN",
"works" : 221,
"yields" : 1,
"unyields" : 1,
"invalidates" : 0,
"advanced" : 1,
"needTime" : 219,
"needFetch" : 0,
"isEOF" : 1,
"docsTested" : 219,
"children" : [ ]
},
"ts" : ISODate("2020-03-31T08:02:05.066Z"),
"client" : "127.0.0.1",
"allUsers" : [ ],
"user" : ""
}
{
"op" : "query",
"ns" : "dev._SCHEMA",
"query" : {
"$query" : {
}
},
"ntoreturn" : 1000,
"ntoskip" : 0,
"nscanned" : 21,
"nscannedObjects" : 21,
"keyUpdates" : 0,
"numYield" : 0,
"lockStats" : {
"timeLockedMicros" : {
"r" : NumberLong(80),
"w" : NumberLong(0)
},
"timeAcquiringMicros" : {
"r" : NumberLong(3),
"w" : NumberLong(2)
}
},
"nreturned" : 21,
"responseLength" : 6562,
"millis" : 0,
"execStats" : {
"type" : "COLLSCAN",
"works" : 23,
"yields" : 0,
"unyields" : 0,
"invalidates" : 0,
"advanced" : 21,
"needTime" : 1,
"needFetch" : 0,
"isEOF" : 1,
"docsTested" : 21,
"children" : [ ]
},
"ts" : ISODate("2020-03-31T08:02:05.070Z"),
"client" : "127.0.0.1",
"allUsers" : [ ],
"user" : ""
}
{
"op" : "query",
"ns" : "dev._Join:users:_Role",
"query" : {
"$orderby" : {
},
"$query" : {
"relatedId" : {
"$in" : [
"MoEjAOXpF4"
]
}
}
},
"ntoreturn" : 1000,
"ntoskip" : 0,
"nscanned" : 0,
"nscannedObjects" : 0,
"keyUpdates" : 0,
"numYield" : 0,
"lockStats" : {
"timeLockedMicros" : {
"r" : NumberLong(151),
"w" : NumberLong(0)
},
"timeAcquiringMicros" : {
"r" : NumberLong(2),
"w" : NumberLong(3)
}
},
"nreturned" : 0,
"responseLength" : 20,
"millis" : 0,
"execStats" : {
},
"ts" : ISODate("2020-03-31T08:02:05.074Z"),
"client" : "127.0.0.1",
"allUsers" : [ ],
"user" : ""
}
{
"op" : "query",
"ns" : "dev._Role",
"query" : {
"$orderby" : {
"_id" : 1
},
"$query" : {
"_id" : {
"$in" : [ ]
}
}
},
"ntoreturn" : 100,
"ntoskip" : 0,
"nscanned" : 0,
"nscannedObjects" : 0,
"keyUpdates" : 0,
"numYield" : 0,
"lockStats" : {
"timeLockedMicros" : {
"r" : NumberLong(183),
"w" : NumberLong(0)
},
"timeAcquiringMicros" : {
"r" : NumberLong(2),
"w" : NumberLong(2)
}
},
"nreturned" : 0,
"responseLength" : 20,
"millis" : 0,
"execStats" : {
"type" : "FETCH",
"works" : 1,
"yields" : 0,
"unyields" : 0,
"invalidates" : 0,
"advanced" : 0,
"needTime" : 0,
"needFetch" : 0,
"isEOF" : 1,
"alreadyHasObj" : 0,
"forcedFetches" : 0,
"matchTested" : 0,
"children" : [
{
"type" : "IXSCAN",
"works" : 1,
"yields" : 0,
"unyields" : 0,
"invalidates" : 0,
"advanced" : 0,
"needTime" : 0,
"needFetch" : 0,
"isEOF" : 1,
"keyPattern" : "{ _id: 1 }",
"isMultiKey" : 0,
"boundsVerbose" : "field #0['_id']: ",
"yieldMovedCursor" : 0,
"dupsTested" : 0,
"dupsDropped" : 0,
"seenInvalidated" : 0,
"matchTested" : 0,
"keysExamined" : 0,
"children" : [ ]
}
]
},
"ts" : ISODate("2020-03-31T08:02:05.075Z"),
"client" : "127.0.0.1",
"allUsers" : [ ],
"user" : ""
}
{
"op" : "query",
"ns" : "dev.userMaster_Test",
"query" : {
"$orderby" : {
},
"$query" : {
"_rperm" : {
"$in" : [
null,
"*",
"*",
"MoEjAOXpF4"
]
}
}
},
"ntoreturn" : 1000,
"ntoskip" : 0,
"nscanned" : 1,
"nscannedObjects" : 1,
"keyUpdates" : 0,
"numYield" : 0,
"lockStats" : {
"timeLockedMicros" : {
"r" : NumberLong(90),
"w" : NumberLong(0)
},
"timeAcquiringMicros" : {
"r" : NumberLong(2),
"w" : NumberLong(1)
}
},
"nreturned" : 1,
"responseLength" : 169,
"millis" : 0,
"execStats" : {
"type" : "COLLSCAN",
"works" : 3,
"yields" : 0,
"unyields" : 0,
"invalidates" : 0,
"advanced" : 1,
"needTime" : 1,
"needFetch" : 0,
"isEOF" : 1,
"docsTested" : 1,
"children" : [ ]
},
"ts" : ISODate("2020-03-31T08:02:05.076Z"),
"client" : "127.0.0.1",
"allUsers" : [ ],
"user" : ""
}
{
"op" : "query",
"ns" : "dev._SCHEMA",
"query" : {
"$query" : {
}
},
"ntoreturn" : 1000,
"ntoskip" : 0,
"nscanned" : 21,
"nscannedObjects" : 21,
"keyUpdates" : 0,
"numYield" : 0,
"lockStats" : {
"timeLockedMicros" : {
"r" : NumberLong(91),
"w" : NumberLong(0)
},
"timeAcquiringMicros" : {
"r" : NumberLong(3),
"w" : NumberLong(4)
}
},
"nreturned" : 21,
"responseLength" : 6562,
"millis" : 0,
"execStats" : {
"type" : "COLLSCAN",
"works" : 23,
"yields" : 0,
"unyields" : 0,
"invalidates" : 0,
"advanced" : 21,
"needTime" : 1,
"needFetch" : 0,
"isEOF" : 1,
"docsTested" : 21,
"children" : [ ]
},
"ts" : ISODate("2020-03-31T08:02:05.151Z"),
"client" : "127.0.0.1",
"allUsers" : [ ],
"user" : ""
}
{
"op" : "command",
"ns" : "dev.$cmd",
"command" : {
"findAndModify" : "_Session",
"query" : {
"_id" : "SJhEylfnb3",
"_wperm" : {
"$in" : [
null,
"*",
"MoEjAOXpF4"
]
}
},
"new" : true,
"remove" : false,
"upsert" : false,
"update" : {
"$set" : {
"_updated_at" : ISODate("2020-03-31T08:02:05.150Z")
}
}
},
"updateobj" : {
"$set" : {
"_updated_at" : ISODate("2020-03-31T08:02:05.150Z")
}
},
"nscanned" : 1,
"nscannedObjects" : 1,
"nMatched" : 1,
"nModified" : 1,
"fastmod" : true,
"keyUpdates" : 0,
"numYield" : 0,
"lockStats" : {
"timeLockedMicros" : {
"r" : NumberLong(0),
"w" : NumberLong(278)
},
"timeAcquiringMicros" : {
"r" : NumberLong(0),
"w" : NumberLong(5)
}
},
"responseLength" : 395,
"millis" : 0,
"execStats" : {
},
"ts" : ISODate("2020-03-31T08:02:05.153Z"),
"client" : "127.0.0.1",
"allUsers" : [ ],
"user" : ""
}
{
"op" : "query",
"ns" : "dev._SCHEMA",
"query" : {
"$query" : {
}
},
"ntoreturn" : 1000,
"ntoskip" : 0,
"nscanned" : 21,
"nscannedObjects" : 21,
"keyUpdates" : 0,
"numYield" : 0,
"lockStats" : {
"timeLockedMicros" : {
"r" : NumberLong(240),
"w" : NumberLong(0)
},
"timeAcquiringMicros" : {
"r" : NumberLong(5),
"w" : NumberLong(3)
}
},
"nreturned" : 21,
"responseLength" : 6562,
"millis" : 0,
"execStats" : {
"type" : "COLLSCAN",
"works" : 23,
"yields" : 0,
"unyields" : 0,
"invalidates" : 0,
"advanced" : 21,
"needTime" : 1,
"needFetch" : 0,
"isEOF" : 1,
"docsTested" : 21,
"children" : [ ]
},
"ts" : ISODate("2020-03-31T08:02:16.592Z"),
"client" : "127.0.0.1",
"allUsers" : [ ],
"user" : ""
}
{
"op" : "query",
"ns" : "dev.Product",
"query" : {
"$orderby" : {
},
"$query" : {
"_rperm" : {
"$in" : [
null,
"*",
"*"
]
}
}
},
"cursorid" : 82313479072,
"ntoreturn" : 1000,
"ntoskip" : 0,
"nscanned" : 1000,
"nscannedObjects" : 1000,
"keyUpdates" : 0,
"numYield" : 0,
"lockStats" : {
"timeLockedMicros" : {
"r" : NumberLong(2778),
"w" : NumberLong(0)
},
"timeAcquiringMicros" : {
"r" : NumberLong(3),
"w" : NumberLong(3)
}
},
"nreturned" : 1000,
"responseLength" : 687471,
"millis" : 2,
"execStats" : {
"type" : "COLLSCAN",
"works" : 1001,
"yields" : 7,
"unyields" : 7,
"invalidates" : 0,
"advanced" : 1000,
"needTime" : 1,
"needFetch" : 0,
"isEOF" : 0,
"docsTested" : 1000,
"children" : [ ]
},
"ts" : ISODate("2020-03-31T08:02:16.598Z"),
"client" : "127.0.0.1",
"allUsers" : [ ],
"user" : ""
}
{
"op" : "getmore",
"ns" : "dev.Product",
"cursorid" : 82313479072,
"ntoreturn" : 1000,
"keyUpdates" : 0,
"numYield" : 0,
"lockStats" : {
"timeLockedMicros" : {
"r" : NumberLong(2151),
"w" : NumberLong(0)
},
"timeAcquiringMicros" : {
"r" : NumberLong(5),
"w" : NumberLong(5)
}
},
"nreturned" : 1000,
"responseLength" : 689544,
"millis" : 2,
"execStats" : {
},
"ts" : ISODate("2020-03-31T08:02:16.654Z"),
"client" : "127.0.0.1",
"allUsers" : [ ],
"user" : ""
}
{
"op" : "getmore",
"ns" : "dev.Product",
"cursorid" : 82313479072,
"ntoreturn" : 1000,
"keyUpdates" : 0,
"numYield" : 0,
"lockStats" : {
"timeLockedMicros" : {
"r" : NumberLong(2144),
"w" : NumberLong(0)
},
"timeAcquiringMicros" : {
"r" : NumberLong(2),
"w" : NumberLong(4)
}
},
"nreturned" : 1000,
"responseLength" : 697344,
"millis" : 2,
"execStats" : {
},
"ts" : ISODate("2020-03-31T08:02:16.690Z"),
"client" : "127.0.0.1",
"allUsers" : [ ],
"user" : ""
}
{
"op" : "getmore",
"ns" : "dev.Product",
"cursorid" : 82313479072,
"ntoreturn" : 1000,
"keyUpdates" : 0,
"numYield" : 0,
"lockStats" : {
"timeLockedMicros" : {
"r" : NumberLong(1619),
"w" : NumberLong(0)
},
"timeAcquiringMicros" : {
"r" : NumberLong(3),
"w" : NumberLong(4)
}
},
"nreturned" : 1000,
"responseLength" : 694691,
"millis" : 1,
"execStats" : {
},
"ts" : ISODate("2020-03-31T08:02:16.763Z"),
"client" : "127.0.0.1",
"allUsers" : [ ],
"user" : ""
}
{
"op" : "getmore",
"ns" : "dev.Product",
"cursorid" : 82313479072,
"ntoreturn" : 1000,
"keyUpdates" : 0,
"numYield" : 0,
"lockStats" : {
"timeLockedMicros" : {
"r" : NumberLong(1525),
"w" : NumberLong(0)
},
"timeAcquiringMicros" : {
"r" : NumberLong(3),
"w" : NumberLong(3)
}
},
"nreturned" : 1000,
"responseLength" : 682413,
"millis" : 1,
"execStats" : {
},
"ts" : ISODate("2020-03-31T08:02:16.797Z"),
"client" : "127.0.0.1",
"allUsers" : [ ],
"user" : ""
}
{
"op" : "getmore",
"ns" : "dev.Product",
"cursorid" : 82313479072,
"ntoreturn" : 1000,
"keyUpdates" : 0,
"numYield" : 0,
"lockStats" : {
"timeLockedMicros" : {
"r" : NumberLong(2501),
"w" : NumberLong(0)
},
"timeAcquiringMicros" : {
"r" : NumberLong(4),
"w" : NumberLong(3)
}
},
"nreturned" : 1000,
"responseLength" : 681485,
"millis" : 2,
"execStats" : {
},
"ts" : ISODate("2020-03-31T08:02:16.837Z"),
"client" : "127.0.0.1",
"allUsers" : [ ],
"user" : ""
}
{
"op" : "getmore",
"ns" : "dev.Product",
"cursorid" : 82313479072,
"ntoreturn" : 1000,
"keyUpdates" : 0,
"numYield" : 0,
"lockStats" : {
"timeLockedMicros" : {
"r" : NumberLong(1579),
"w" : NumberLong(0)
},
"timeAcquiringMicros" : {
"r" : NumberLong(3),
"w" : NumberLong(3)
}
},
"nreturned" : 1000,
"responseLength" : 691719,
"millis" : 1,
"execStats" : {
},
"ts" : ISODate("2020-03-31T08:02:16.883Z"),
"client" : "127.0.0.1",
"allUsers" : [ ],
"user" : ""
}
Type "it" for more
>
it looks like the getmores are occurring in just a few milis.
there is a bottleneck of some sort.
-
If your run a query against the database form the server the iterates through all the objects in you parouct tables, how long does it take?
-
what does a product document 'look like' (i.e. how big is the json text).
-
what is the network topology between the database server and the parse-server? Are they on the same network? What is latency and bandwidth between the two?
If your run a query against the database form the server the iterates through all the objects in you parouct tables, how long does it take?
In mongoshell its quiet fast,1-2 secs
what does a product document 'look like' (i.e. how big is the json text).
Closely similar to { "_id": "y2IEREsCJR", "_p_createdBy": "Employee$nqzq73IJZn", "_p_company": "Company$othercmpny", "_p_brand": "Brand$5KUm1qj624", "name": "SOME NAM NAME NAME", "parentName": "SOME NAM NAME NAME", "ERPName": "SOME NAM NAME NAME", "description": "SOME NAM NAME NAME", "quantity": "28", "season": "SOME", "sleeve": "", "basicRate": 1899, "ERPCode": "2423423523532", "pattern": "", "subCategory2": "SLIM FIT", "HSCode": 0, "content": "SOME NAM NAME NAME", "source": "", "category": "JEANS", "exciseItem": false, "subCategory": "SLIM FIT", "EANCode": 24332543232532, "neck": "", "unitOfMeasurement": "SOME NAM NAME NAME", "SellingPrice": 1899, "_created_at": "2020-03-13T23:30:32.098Z", "_updated_at": "2020-03-27T11:36:35.814Z", }
what is the network topology between the database server and the parse-server? Are they on the same network? What is latency and bandwidth between the two?
Parse server specs npm list [email protected]
Server version 3.9.0 (From Parse dashboard app list)
db.version() 2.6.10 <<Can be an issue?
System Ubuntu 16.04 Mongodb at localhost:27017 same server Mongouri (parse default)
Tried accessing db directly
Included in index.js
const Config = require('./node_modules/parse-server/lib/Config');
const config = Config.get(Parse.applicationId);
const mongoAdapter = config.database.adapter;
app.get('/mongotest',async function(req, res) {
console.time("connect")
await mongoAdapter.connect();
const collection = await mongoAdapter.database.collection("Product");
console.timeEnd("connect")
console.time("find")
var response = await collection.find().toArray();
console.timeEnd("find")
console.log("totals="+response.length)
res.status(200).send(response);
});
Processed in under 3 seconds, Console log
0|app1 | connect: 0.375ms
0|app1 | find: 1.701s
0|app1 | totals=58110
-
Monitor your server resources during the slow request. Do CPU, IOPS, network connections, etc max out?
-
Are there any other apps running on Node that could be blocking the Node thread?
-
Profile Node app. For example, https://nodejs.org/en/docs/guides/simple-profiling/
-
Is the response also slow when you deploy the latest version of parse server and mongodb?
-
Debug parse server while running the request and inspect the code to see at which point it is slow.
Sent with GitHawk
Monitor your server resources during the slow request. Do CPU, IOPS, network connections, etc max out?
CPU > There's a visible spike but cpu usage doesn't become 100% IOPS > Ram usage is increased but still no 100% usage. RAM normally uses 1.1gb when api is hit it goes to around 400mb then gradually increases till 1.9gb then drops to original 1.1gb. Strange?Please refer image
There may be a memory leak somewhere but this doesn't constitute to our problem as even a freshly restarted server with 86 mb usage gives slow response pm2 restart > 86mb > cloudcodehit > 1.9gb gradual > response send > 1.2 gb
network connections > No idea on this,doesnt seem an issue as response is slow when no other client interacting with server
4 times api hit usage>
Are there any other apps running on Node that could be blocking the Node thread
There are other pm2 parse instances but even stopping them doesn't solve the issue
db.version() 2.6.10 <<Can be an issue?
Is it possible to upgrade the database server? That is definitely an unsupported db version.
Can you test with >= Mongo 3.6?
Is the response also slow when you deploy the latest version of parse server and mongodb?
After updating
Parse server specs npm list [email protected] [email protected]
Server version 4.2.0 (From Parse dashboard app list)
db.version() 4.2.5
Updating has not solved the issue
Seems to be related with mongoObjectToParseObject ,
Generated using node --prof index.js ...
This is the full log processlog.txt
If you set a breakpoint in that function of parse server and debug step-by-step you should be able to find the line that causes the delay.
Sent with GitHawk
Please share more info and tools on how i can do that as i never done this before
@flovilmart @dplewis @acinader @davimacedo @mtrezza I've tried many approaches and it finally seems the issue is in parse server code, I'm unable to debug the server being unfamiliar with its internal workings.
Can anyone shed some light on this issue? I'll research more in it if there's still scope for the issue being caused by outside code or my mistakes