@plasmicapp/cli fails to sync new projects

[zvictor]

I have been trying to setup a new Plasmic project, but plasmic sync insists on failing. The command works as expected up to the creation of the plasmic.json file, but after that it hangs forever.

$ DEBUG=* plasmic sync -p [redacted] --all-files

winston:create-logger: Define prototype method for "error"
winston:create-logger: Define prototype method for "warn"
winston:create-logger: Define prototype method for "info"
winston:create-logger: Define prototype method for "http"
winston:create-logger: Define prototype method for "verbose"
winston:create-logger: Define prototype method for "debug"
winston:create-logger: Define prototype method for "silly"
winston:create-logger: Define prototype method for "error"
winston:create-logger: Define prototype method for "warn"
winston:create-logger: Define prototype method for "info"
winston:create-logger: Define prototype method for "http"
winston:create-logger: Define prototype method for "verbose"
winston:create-logger: Define prototype method for "debug"
winston:create-logger: Define prototype method for "silly"
  follow-redirects options {
  maxRedirects: 21,
  maxBodyLength: 10485760,
  protocol: 'https:',
  path: '/api/v1/auth/self',
  method: 'GET',
  headers: {
    Accept: 'application/json, text/plain, */*',
    'x-plasmic-api-user': '[redacted]',
    'x-plasmic-api-token': '[redacted]',
    'User-Agent': 'axios/0.21.4'
  },
  agent: undefined,
  agents: { http: undefined, https: undefined },
  auth: undefined,
  hostname: 'studio.plasmic.app',
  port: null,
  nativeProtocols: {
    'http:': {
      _connectionListener: [Function: connectionListener],
      METHODS: [Array],
      STATUS_CODES: [Object],
      Agent: [Function],
      ClientRequest: [Function: ClientRequest],
      IncomingMessage: [Function: IncomingMessage],
      OutgoingMessage: [Function: OutgoingMessage],
      Server: [Function: Server],
      ServerResponse: [Function: ServerResponse],
      createServer: [Function: createServer],
      validateHeaderName: [Function: __node_internal_],
      validateHeaderValue: [Function: __node_internal_],
      get: [Function: get],
      request: [Function: request],
      maxHeaderSize: [Getter],
      globalAgent: [Getter/Setter]
    },
    'https:': {
      Agent: [Function: Agent],
      globalAgent: [Agent],
      Server: [Function: Server],
      createServer: [Function: createServer],
      get: [Function: get],
      request: [Function: request]
    }
  }
} +0ms
  follow-redirects options {
  maxRedirects: 21,
  maxBodyLength: 10485760,
  protocol: 'https:',
  path: '/api/v1/code/required-packages',
  method: 'POST',
  headers: {
    Accept: 'application/json, text/plain, */*',
    'Content-Type': 'application/json',
    'x-plasmic-api-user': '[redacted]',
    'x-plasmic-api-token': '[redacted]',
    'User-Agent': 'axios/0.21.4',
    'Content-Length': 2
  },
  agent: undefined,
  agents: { http: undefined, https: undefined },
  auth: undefined,
  hostname: 'codegen.plasmic.app',
  port: null,
  nativeProtocols: {
    'http:': {
      _connectionListener: [Function: connectionListener],
      METHODS: [Array],
      STATUS_CODES: [Object],
      Agent: [Function],
      ClientRequest: [Function: ClientRequest],
      IncomingMessage: [Function: IncomingMessage],
      OutgoingMessage: [Function: OutgoingMessage],
      Server: [Function: Server],
      ServerResponse: [Function: ServerResponse],
      createServer: [Function: createServer],
      validateHeaderName: [Function: __node_internal_],
      validateHeaderValue: [Function: __node_internal_],
      get: [Function: wrappedMethod],
      request: [Function: wrappedMethod],
      maxHeaderSize: [Getter],
      globalAgent: [Getter/Setter]
    },
    'https:': {
      Agent: [Function: Agent],
      globalAgent: [Agent],
      Server: [Function: Server],
      createServer: [Function: createServer],
      get: [Function: wrappedMethod],
      request: [Function: wrappedMethod]
    }
  }
} +351ms

It hangs there for a couple of minutes without any new message or any code synced. If I wait long enough, an OOM error shows up:

<--- Last few GCs --->

[54580:0x7f8f5f02b000]   156507 ms: Mark-sweep 4051.2 (4138.7) -> 4038.1 (4141.7) MB, 1335.9 / 0.0 ms  (average mu = 0.251, current mu = 0.196) allocation failure; scavenge might not succeed
[54580:0x7f8f5f02b000]   158922 ms: Mark-sweep 4054.0 (4141.7) -> 4041.8 (4145.5) MB, 2092.5 / 0.0 ms  (average mu = 0.184, current mu = 0.134) allocation failure; scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 0x110634d85 node::Abort() (.cold.1) [/Users/zvictor/.volta/tools/image/node/18.6.0/bin/node]
 2: 0x10f0f9f89 node::Abort() [/Users/zvictor/.volta/tools/image/node/18.6.0/bin/node]
 3: 0x10f0fa0ff node::OnFatalError(char const*, char const*) [/Users/zvictor/.volta/tools/image/node/18.6.0/bin/node]
 4: 0x10f282527 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/zvictor/.volta/tools/image/node/18.6.0/bin/node]
 5: 0x10f44ab35 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/zvictor/.volta/tools/image/node/18.6.0/bin/node]
 6: 0x10f449512 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/zvictor/.volta/tools/image/node/18.6.0/bin/node]
 7: 0x10f43b84a v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/zvictor/.volta/tools/image/node/18.6.0/bin/node]
 8: 0x10f43c1c5 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/zvictor/.volta/tools/image/node/18.6.0/bin/node]
 9: 0x10f41de0a v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [/Users/zvictor/.volta/tools/image/node/18.6.0/bin/node]
10: 0x10f417829 v8::internal::MaybeHandle<v8::internal::SeqTwoByteString> v8::internal::FactoryBase<v8::internal::Factory>::NewRawStringWithMap<v8::internal::SeqTwoByteString>(int, v8::internal::Map, v8::internal::AllocationType) [/Users/zvictor/.volta/tools/image/node/18.6.0/bin/node]
11: 0x10f764b5f v8::internal::String::SlowFlatten(v8::internal::Isolate*, v8::internal::Handle<v8::internal::ConsString>, v8::internal::AllocationType) [/Users/zvictor/.volta/tools/image/node/18.6.0/bin/node]
12: 0x10f86df21 v8::internal::Runtime_StringCharCodeAt(int, unsigned long*, v8::internal::Isolate*) [/Users/zvictor/.volta/tools/image/node/18.6.0/bin/node]
13: 0x10fc4ee79 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/Users/zvictor/.volta/tools/image/node/18.6.0/bin/node]

Things I have tried:

• Switching from PNPM to NPM
• Customised and default plasmic.json files
• With and without --all-files
• Connected to a VPN in order to bypass any possible network issue
• Waited 4 hours

[IcaroG]

Hello @zvictor. Can you send me a message at our community slack with your project id?

[zvictor]

This issue is caused by this line: https://github.com/plasmicapp/plasmic/blob/95aaf8f7ce29378844e1078fd99089ab30694ea4/packages/cli/src/utils/npm-utils.ts#L172

The problem with this glob is that node_modules can contain all sorts of things inside it.

E.g. In my case, as I am the author of Brainyduck, I do not install it from the registry but from my local disk. As result, the **/node_modules glob will traverse all the node_modules folders inside all the other folders. That can also lead to circular references, such as ${rootDir}/node_modules/@brainy/domain/node_modules/brainyduck/examples/modularized-esbuild-bundle/node_modules/brainyduck/examples/modularized-esbuild-bundle/node_modules/brainyduck/examples/modularized/node_modules/brainyduck/node_modules/log-symbols/package.json (the circular dependency comes from here)

But even if circular dependencies are not a problem, this glob will suffer drastically from performance the heavier the node_modules folder is.

Please try running the script below in your projects and notice the speed of it. Then run the script again but with the line const pkg = '*' uncommented. Notice how many unnecessary searching it does in your directories?

const fg = require('fast-glob');

const rootDir = process.cwd();
const pkg = `@plasmicapp/react-web`
// const pkg = '*'

fg.stream(`${rootDir}/**/node_modules/${pkg}/package.json`, {
    // ignore:`**/node_modules/**/node_modules/**`
})
    .on('data', (file) => {
        console.log(file);
    })
    .on('end', () => {
        console.log('done');
    })
    .on('error', (err) => {
        throw err
    })
    .on('close', () => {
        console.log('closed');
    })

[yang]

Thank you for the patch! We will handle it once some folks are back in the office. :-)

[IcaroG]

Fixed with https://github.com/plasmicapp/plasmic/pull/48