Compiled Schema Stored In Config Cache

[pmclain]

The compiled schema is currently stored with the CONFIG cache tag. This mean a store admin clearing the cache would require the schema stitcher to rebuild the schema from the filesystem. This is fine during development, but would cause a similar lag following cache flush as the interception cache prior to 2.3.0 see magento/magento2#17680 (fixed in magento/magento2#18648). I would propose compiling the generated schema and storing in generated/metadata/graphql.php. This would move the stitched schema out of the config cache and allow allow stores in production mode to fetch the stitched schema from pure php which would be faster than cache especially when opcache is enabled.

[pmclain]

Did some rough benchmarking and prototyping on my laptop. I setup a local instance of the current 2.3-develop branch in production mode using redis for the cache backend. Local php setup:

Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies
    with Zend OPcache v7.2.15, Copyright (c) 1999-2018, by Zend Technologies
    with blackfire v1.24.4~mac-x64-non_zts72, https://blackfire.io, by Blackfire

All tests were done using this mutation:

mutation {
  createEmptyCart
}

I first wanted to gauge the penalty to re-stitch the schema on a config cache clean. To simulate I executed the mutation to warm the cache, then made five requests clearing the Magento_Framework_GraphQlSchemaStitching_Config_Data key from redis before each request.

Schema Stitch Time (ms)
1850
1781
1867
1822
1783
AVG 1821

Should a store admin clear the config cache on a production site the first request following the clear will include ~1.8s of load time regenerating the graphql schema.

Next I recorded the time required for fetching the cached data from redis and unserializing. The following requests were made after warming the cache with schema data and executing the mutation.

Fetch From Redis and Unserialize Schema (ms)
6.787
7.371
7.201
6.608
7.280
AVG 7.049

Roughly 7ms of every request is spent fetching the schema from the backend cache and unserializing.

Finally I moved the schema out of redis and into a php file that returns the schema in the form of an array. I very crudely hacked the config reader to load the config from this file using an include and executed the mutation once to warm the file in the opcache. The output is already an array and does not require being unserialized.

Load from OPCache (ms)
0.0281
0.0288
0.0231
0.0100
0.0110
AVG 0.0202

The result was a reduction in time spent loading the schema of ~99% (7ms) and would apply to every graphql request.

[pmclain]

Since the product and category attribute schema is generated dynamically the schema wouldn't be able generate with setup:di:compile https://github.com/magento/graphql-ce/blob/65a806a02b9ca72379017b33da990ca536429e54/app/code/Magento/CatalogGraphQl/etc/di.xml#L17-L24

[naydav]

@pmclain Great job, Pat!

[m2-assistant[bot]]

Hi @pmclain. Thank you for your report. To help us process this issue please make sure that you provided the following information:

• [ ] Summary of the issue
• [ ] Information on your environment
• [ ] Steps to reproduce
• [ ] Expected and actual results

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento give me 2.4-develop instance - upcoming 2.4.x release

For more details, please, review the Magento Contributor Assistant documentation.

@pmclain do you confirm that you were able to reproduce the issue on vanilla Magento instance following steps to reproduce?

• [ ] yes
• [ ] no

---

• Join Magento Community Engineering Slack and ask your questions in #github channel.

[Andrii-Antoniuk]

@sivaschenko Hello! Is this ticket still in progress? This is a huge performance improvement, don't we want it to be released?

[jonathanribas]

Hello guys, on our side we have developped this non Best Practice patch but it really makes the job.

As we don't create customer or product EAV attributes directly from backoffice, it works for us, we don't need to regenerate Schema all the time, when we deploy, it is generated and serves till next deploy.

We run on Kubernetes, we have done a fake GQL call on startup probe so it generates this file automatically. Then Opcache takes care of it.

It's running live for months now and no more Schema Stitching!

generate-gql-cache-in-file.patch

[Andrii-Antoniuk]

@jonathanribas thanks, I'll try this out