codeql-action icon indicating copy to clipboard operation
codeql-action copied to clipboard
verified publisher icon github

GitHub Code Quality: IOException "No space left on device"

Open karlhorky opened this issue 1 month ago • 4 comments

Hi, first of all, thanks for your continued work on GitHub infrastructure including Actions and CodeQL, really great!

I just enabled GitHub Code Quality in a medium-sized monorepo of ours, and the new action immediately failed during the Perform CodeQL Analysis step of the Analyze (javascript-typescript) job, running into low disk space issues.

This is a well-known issue with GitHub Actions runners, seemingly because of a reduction in available disk space on the runners in 2024:

  • https://github.com/actions/runner-images/issues/10386
  • https://github.com/actions/runner-images/issues/9344
  • https://github.com/orgs/community/discussions/26488
  • https://github.com/orgs/community/discussions/26493

Errors reported in the logs below include:

  1. Severe disk cache trouble (corruption or out of space)
  2. Failed to write item to disk
  3. IOException "No space left on device"
  4. Warning: The Actions runner is running low on disk space (664.8 MB available).

Job: Analyze (javascript-typescript) Step: Perform CodeQL Analysis

Run github/codeql-action/analyze@v3
  with:
    category: /language:javascript-typescript
    upload-database: false
    output: ../results
    upload: always
    skip-queries: false
    ...
    wait-for-processing: true
    token: ***
    matrix: {
    "language": "javascript-typescript",
    "category": "/language:javascript-typescript",
    "build-mode": "none",
    "runner": "[\"ubuntu-latest\"]"
  }
    expect-error: false
  env:
    CODE_SCANNING_REF: refs/heads/main
    CODE_SCANNING_BASE_BRANCH: 
    CODE_SCANNING_IS_ANALYZING_DEFAULT_BRANCH: true
    CODE_SCANNING_WORKFLOW_FILE: ...
    CODEQL_ACTION_FEATURE_MULTI_LANGUAGE: false
    CODEQL_ACTION_FEATURE_SANDWICH: false
    CODEQL_ACTION_FEATURE_SARIF_COMBINE: true
    CODEQL_ACTION_FEATURE_WILL_UPLOAD: true
    CODEQL_ACTION_VERSION: 3.31.2
    JOB_RUN_UUID: f2ed4aa8-134b-4876-9080-a782525ac55e
    CODEQL_ACTION_INIT_HAS_RUN: true
    CODEQL_ACTION_ANALYSIS_KEY: dynamic/github-code-scanning/codeql:analyze
    CODEQL_WORKFLOW_STARTED_AT: 2025-11-01T07:54:35.059Z
    CODEQL_RAM: 6920
    CODEQL_THREADS: 2
    CODEQL_PROXY_HOST: 
    CODEQL_PROXY_PORT: 
    CODEQL_PROXY_CA_CERTIFICATE: 
    CODEQL_PROXY_URLS: 
While resolving threads, found a cgroup CPUs file with 2 CPUs in /sys/fs/cgroup/cpuset.cpus.effective.
Not performing diff-informed analysis because we are not analyzing a pull request.
Extracting javascript
Finalizing javascript
Running queries for javascript

  ...

  Starting evaluation of codeql/javascript-queries/AngularJS/DuplicateDependency.ql.
  Starting evaluation of codeql/javascript-queries/AngularJS/DependencyMismatch.ql.
  Starting evaluation of codeql/javascript-queries/AngularJS/IncompatibleService.ql.
  Starting evaluation of codeql/javascript-queries/AngularJS/MissingExplicitInjection.ql.
  Starting evaluation of codeql/javascript-queries/AngularJS/RepeatedInjection.ql.
  Starting evaluation of codeql/javascript-queries/AngularJS/UseNgSrc.ql.
  Starting evaluation of codeql/javascript-queries/DOM/DuplicateAttributes.ql.
  Starting evaluation of codeql/javascript-queries/DOM/MalformedIdAttribute.ql.
  [1/99 eval 10m19s] Evaluation done; writing results to codeql/javascript-queries/AngularJS/DependencyMismatch.bqrs.
  [2/99 eval 10m19s] Evaluation done; writing results to codeql/javascript-queries/AngularJS/IncompatibleService.bqrs.
  [3/99 eval 10m19s] Evaluation done; writing results to codeql/javascript-queries/AngularJS/DuplicateDependency.bqrs.
  [4/99 eval 12m16s] Evaluation done; writing results to codeql/javascript-queries/AngularJS/MissingExplicitInjection.bqrs.
  [5/99 eval 12m16s] Evaluation done; writing results to codeql/javascript-queries/AngularJS/RepeatedInjection.bqrs.
  Starting evaluation of codeql/javascript-queries/DOM/PseudoEval.ql.
  [6/99 eval 4m21s] Evaluation done; writing results to codeql/javascript-queries/DOM/PseudoEval.bqrs.
  Starting evaluation of codeql/javascript-queries/Declarations/ArgumentsRedefined.ql.
  [7/99 eval 23.4s] Evaluation done; writing results to codeql/javascript-queries/Declarations/ArgumentsRedefined.bqrs.
  Starting evaluation of codeql/javascript-queries/Declarations/AssignmentToConst.ql.
  Oops! A fatal internal error occurred. Details:
  com.semmle.util.exception.CatastrophicError: An error occurred while evaluating num#AbstractPropertiesImpl::MkAbstractProperty#c6cb71fa/3@i19#1ab1e1wp
  Severe disk cache trouble (corruption or out of space) at /home/runner/work/_temp/codeql_databases/javascript/db-javascript/default/cache/pages/38/6c.pack: Failed to write item to disk
  The RA to evaluate was:
  
      {3} r1 = INVOKE HIGHER-ORDER RELATION construct<AbstractPropertiesImpl#5654241c::TAbstractProperty,0> ON <AbstractPropertiesImpl::MkAbstractProperty#dom#d09da573#prev_delta>
      return r1
  
  (eventual cause: IOException "No space left on device")
  	at com.semmle.inmemory.pipeline.MetaPipelineInstance.wrapWithRaDump(MetaPipelineInstance.java:212)
  	at com.semmle.inmemory.pipeline.MetaPipelineInstance.exceptionCaught(MetaPipelineInstance.java:182)
  	at com.semmle.inmemory.scheduler.execution.ThreadableWork.handleAndLog(ThreadableWork.java:593)
  	at com.semmle.inmemory.scheduler.execution.ThreadableWork.doSomeWork(ThreadableWork.java:410)
  	at com.semmle.inmemory.scheduler.RecursiveLayer$RecursiveWork.doWork(RecursiveLayer.java:514)
  	at com.semmle.inmemory.scheduler.execution.ThreadableWork.doSomeWork(ThreadableWork.java:396)
  	at com.semmle.inmemory.scheduler.execution.ExecutionScheduler.runnerMain(ExecutionScheduler.java:710)
  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
  	at java.base/java.lang.Thread.run(Unknown Source)
  Caused by: Severe disk cache trouble (corruption or out of space) at /home/runner/work/_temp/codeql_databases/javascript/db-javascript/default/cache/pages/38/6c.pack: Failed to write item to disk
  (eventual cause: IOException "No space left on device")
  	at com.semmle.inmemory.caching.RelationCacheImpl.lambda$create$0(RelationCacheImpl.java:87)
  	at com.semmle.inmemory.caching.byhash.disk.OnDiskStore.put(OnDiskStore.java:136)
  	at com.semmle.inmemory.caching.byhash.interfaces.HashBasedCache.putIfPresent(HashBasedCache.java:29)
  	at com.semmle.inmemory.caching.byhash.evict.Evictor$ItemHandle.writeToDisk(Evictor.java:723)
  	at java.base/java.util.ArrayList.forEach(Unknown Source)
  	at com.semmle.inmemory.caching.byhash.evict.Evictor.writeSelectedItems(Evictor.java:1059)
  	at com.semmle.inmemory.caching.byhash.evict.Evictor.reduceMemoryUsage(Evictor.java:459)
  	at com.semmle.inmemory.alloc.MemoryManager.reduceArraySpace(MemoryManager.java:342)
  	at com.semmle.inmemory.alloc.RigidArrayAllocator.allocateArrays(RigidArrayAllocator.java:178)
  	at com.semmle.inmemory.alloc.RigidArrayAllocator$1.<init>(RigidArrayAllocator.java:216)
  	at com.semmle.inmemory.alloc.RigidArrayAllocator.preallocate(RigidArrayAllocator.java:211)
  	at com.semmle.inmemory.alloc.MemoryManager.preallocate(MemoryManager.java:491)
  	at com.semmle.inmemory.caching.PagePrimitives.parseItem(PagePrimitives.java:90)
  	at com.semmle.inmemory.caching.byhash.disk.OnDiskStore.prepareLoading(OnDiskStore.java:275)
  	at com.semmle.inmemory.caching.byhash.disk.OnDiskStore.getItem(OnDiskStore.java:193)
  	at com.semmle.inmemory.caching.byhash.evict.CacheChainPartialMetadata.getItem(CacheChainPartialMetadata.java:62)
  	at com.semmle.inmemory.caching.byhash.interfaces.HashBasedCache.getItem(HashBasedCache.java:44)
  	at com.semmle.inmemory.caching.RelationCacheImpl.getPage(RelationCacheImpl.java:291)
  	at com.semmle.inmemory.caching.PagedRelation.getPage(PagedRelation.java:93)
  	at com.semmle.inmemory.relations.IntArrayMatcher.switchPageTo(IntArrayMatcher.java:158)
  	at com.semmle.inmemory.relations.IntArrayMatcher.locate(IntArrayMatcher.java:429)
  	at com.semmle.inmemory.relations.IntArrayMatcher.locate(IntArrayMatcher.java:447)
  	at com.semmle.inmemory.relations.IntArrayMatcher.matchTuple(IntArrayMatcher.java:522)
  	at com.semmle.inmemory.relations.UnionRelation$UnionMatchCache.matchTuple(UnionRelation.java:135)
  	at com.semmle.inmemory.caching.TuplePool$ComplexNumberer$NumberSink.addTuple(TuplePool.java:658)
  	at com.semmle.inmemory.relations.BaseIntArrayRelation.map(BaseIntArrayRelation.java:88)
  	at com.semmle.inmemory.relations.AbstractRelation.deduplicateMap(AbstractRelation.java:138)
  	at com.semmle.inmemory.caching.TuplePool$ComplexNumberer.lambda$doNumbering$0(TuplePool.java:444)
  	at com.semmle.inmemory.relations.writers.RelationWriter.withSink(RelationWriter.java:72)
  	at com.semmle.inmemory.caching.TuplePool$ComplexNumberer.doNumbering(TuplePool.java:440)
  	at com.semmle.inmemory.caching.TuplePool.lambda$doNumbering$1(TuplePool.java:245)
  	at com.semmle.inmemory.caching.TuplePool$ComplexNumberer.doWithBranchLock(TuplePool.java:540)
  	at com.semmle.inmemory.caching.TuplePool.doNumbering(TuplePool.java:233)
  	at com.semmle.inmemory.ast.IMBConstructDatatype.doNumberingStreaming(IMBConstructDatatype.java:165)
  	at com.semmle.inmemory.ast.IMBConstructDatatype$1.evaluateSink(IMBConstructDatatype.java:142)
  	at com.semmle.inmemory.pipeline.HOPStep.generateTuples(HOPStep.java:57)
  	at com.semmle.inmemory.pipeline.SimpleHeadStep.lambda$forwardInitialize$0(SimpleHeadStep.java:32)
  	at com.semmle.inmemory.pipeline.HeadEndDispatcher.headEndWork(HeadEndDispatcher.java:75)
  	at com.semmle.inmemory.pipeline.PipelineState.doSomeWork(PipelineState.java:88)
  	at com.semmle.inmemory.pipeline.PipelineInstance.doWork(PipelineInstance.java:79)
  	at com.semmle.inmemory.scheduler.execution.ThreadableWork.doSomeWork(ThreadableWork.java:396)
  	at com.semmle.inmemory.scheduler.execution.ChainingWork.doWork(ChainingWork.java:30)
  	at com.semmle.inmemory.scheduler.execution.ThreadableWork.doSomeWork(ThreadableWork.java:396)
  	... 6 more
  Caused by: java.io.IOException: No space left on device
  	at java.base/sun.nio.ch.UnixFileDispatcherImpl.write0(Native Method)
  	at java.base/sun.nio.ch.UnixFileDispatcherImpl.write(Unknown Source)
  	at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)
  	at java.base/sun.nio.ch.IOUtil.write(Unknown Source)
  	at java.base/sun.nio.ch.IOUtil.write(Unknown Source)
  	at java.base/sun.nio.ch.FileChannelImpl.write(Unknown Source)
  	at com.semmle.inmemory.alloc.IntArray.writeToChannel(IntArray.java:223)
  	at com.semmle.inmemory.alloc.IntArray.writeTo(IntArray.java:149)
  	at com.semmle.inmemory.relations.ConcreteIntArrayRelation.serialiseToDisk(ConcreteIntArrayRelation.java:51)
  	at com.semmle.inmemory.caching.PagePrimitives.serialiseItem(PagePrimitives.java:75)
  	at com.semmle.inmemory.caching.PagePrimitives.serialiseItem(PagePrimitives.java:26)
  	at com.semmle.inmemory.caching.byhash.disk.OnDiskPack$Appender.writeItemToSelectedLocation(OnDiskPack.java:337)
  	at com.semmle.inmemory.caching.byhash.disk.OnDiskPack$Appender.selectLocationAndWriteItem(OnDiskPack.java:310)
  	at com.semmle.inmemory.caching.byhash.disk.OnDiskPack$Appender.writeFromRam(OnDiskPack.java:255)
  	at com.semmle.inmemory.caching.byhash.disk.OnDiskPack.store(OnDiskPack.java:224)
  	at com.semmle.inmemory.caching.byhash.disk.OnDiskStore.put(OnDiskStore.java:126)
  	... 47 more
  
  Error: Error running analysis for javascript: Encountered a fatal error while running "/opt/hostedtoolcache/CodeQL/2.23.3/x64/codeql/codeql database run-queries --ram=6920 --threads=2 --expect-discarded-cache /home/runner/work/_temp/codeql_databases/javascript --min-disk-free=1024 -v". Exit code was 100 and error was: Oops! A fatal internal error occurred. Details:
  com.semmle.util.exception.CatastrophicError: An error occurred while evaluating num#AbstractPropertiesImpl::MkAbstractProperty#c6cb71fa/3@i19#1ab1e1wp
  Severe disk cache trouble (corruption or out of space) at /home/runner/work/_temp/codeql_databases/javascript/db-javascript/default/cache/pages/38/6c.pack: Failed to write item to disk
  The RA to evaluate was:
  
      {3} r1 = INVOKE HIGHER-ORDER RELATION construct<AbstractPropertiesImpl#5654241c::TAbstractProperty,0> ON <AbstractPropertiesImpl::MkAbstractProperty#dom#d09da573#prev_delta>
      return r1
  
  (eventual cause: IOException "No space left on device")
  	at com.semmle.inmemory.pipeline.MetaPipelineInstance.wrapWithRaDump(MetaPipelineInstance.java:212)
  	at com.semmle.inmemory.pipeline.MetaPipelineInstance.exceptionCaught(MetaPipelineInstance.java:182)
  	at com.semmle.inmemory.scheduler.execution.ThreadableWork.handleAndLog(ThreadableWork.java:593)
  	at com.semmle.inmemory.scheduler.execution.ThreadableWork.doSomeWork(ThreadableWork.java:410)
  	at com.semmle.inmemory.scheduler.RecursiveLayer$RecursiveWork.doWork(RecursiveLayer.java:514)
  	at com.semmle.inmemory.scheduler.execution.ThreadableWork.doSomeWork(ThreadableWork.java:396)
  	at com.semmle.inmemory.scheduler.execution.ExecutionScheduler.runnerMain(ExecutionScheduler.java:710)
  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
  	at java.base/java.lang.Thread.run(Unknown Source)
  Caused by: Severe disk cache trouble (corruption or out of space) at /home/runner/work/_temp/codeql_databases/javascript/db-javascript/default/cache/pages/38/6c.pack: Failed to write item to disk
  (eventual cause: IOException "No space left on device")
  	at com.semmle.inmemory.caching.RelationCacheImpl.lambda$create$0(RelationCacheImpl.java:87)
  	at com.semmle.inmemory.caching.byhash.disk.OnDiskStore.put(OnDiskStore.java:136)
  	at com.semmle.inmemory.caching.byhash.interfaces.HashBasedCache.putIfPresent(HashBasedCache.java:29)
  	at com.semmle.inmemory.caching.byhash.evict.Evictor$ItemHandle.writeToDisk(Evictor.java:723)
  	at java.base/java.util.ArrayList.forEach(Unknown Source)
  	at com.semmle.inmemory.caching.byhash.evict.Evictor.writeSelectedItems(Evictor.java:1059)
  	at com.semmle.inmemory.caching.byhash.evict.Evictor.reduceMemoryUsage(Evictor.java:459)
  	at com.semmle.inmemory.alloc.MemoryManager.reduceArraySpace(MemoryManager.java:342)
  	at com.semmle.inmemory.alloc.RigidArrayAllocator.allocateArrays(RigidArrayAllocator.java:178)
  	at com.semmle.inmemory.alloc.RigidArrayAllocator$1.<init>(RigidArrayAllocator.java:216)
  	at com.semmle.inmemory.alloc.RigidArrayAllocator.preallocate(RigidArrayAllocator.java:211)
  	at com.semmle.inmemory.alloc.MemoryManager.preallocate(MemoryManager.java:491)
  	at com.semmle.inmemory.caching.PagePrimitives.parseItem(PagePrimitives.java:90)
  	at com.semmle.inmemory.caching.byhash.disk.OnDiskStore.prepareLoading(OnDiskStore.java:275)
  	at com.semmle.inmemory.caching.byhash.disk.OnDiskStore.getItem(OnDiskStore.java:193)
  	at com.semmle.inmemory.caching.byhash.evict.CacheChainPartialMetadata.getItem(CacheChainPartialMetadata.java:62)
  	at com.semmle.inmemory.caching.byhash.interfaces.HashBasedCache.getItem(HashBasedCache.java:44)
  	at com.semmle.inmemory.caching.RelationCacheImpl.getPage(RelationCacheImpl.java:291)
  	at com.semmle.inmemory.caching.PagedRelation.getPage(PagedRelation.java:93)
  	at com.semmle.inmemory.relations.IntArrayMatcher.switchPageTo(IntArrayMatcher.java:158)
  	at com.semmle.inmemory.relations.IntArrayMatcher.locate(IntArrayMatcher.java:429)
  	at com.semmle.inmemory.relations.IntArrayMatcher.locate(IntArrayMatcher.java:447)
  	at com.semmle.inmemory.relations.In
  Warning: The Actions runner is running low on disk space (664.8 MB available).

Suggested solutions

The github/codeql-action could either / both:

  1. Use less disk space
  2. Clean up unused tools to recover disk space eg:
    • https://github.com/actions/runner-images/issues/2875#issuecomment-1163363045
    • https://github.com/actions/runner-images/issues/10386#issuecomment-2269439585

karlhorky avatar Nov 01 '25 10:11 karlhorky

Hi @karlhorky 👋🏻

Thank you for giving the public preview of Code Quality a go!

Sorry to hear that you have run into this issue. Note that, while Code Quality is a new product, the CodeQL engine that powers it is not. The CodeQL Action just allows running CodeQL in GitHub Actions -- it doesn't in itself perform the analysis.

For CodeQL generally, we have a lot of documentation, including on hardware requirements.

I am not sure whether there is currently a way to customise the runner type that is used for Code Quality analysis, but I'll check with the relevant team to pass on your report to see what's possible now during the public preview and what's on their roadmap.

Also, if this happened on a public repo, would you be able to share a link to the repo/workflow?

mbg avatar Nov 03 '25 10:11 mbg

It's a private repo - the public repos that I enabled it on worked out of the box.

I am not sure whether there is currently a way to customise the runner type that is used for Code Quality analysis, but I'll check with the relevant team to pass on your report to see what's possible now during the public preview and what's on their roadmap.

Yeah from the interface it looks like this is possible:

Image

But for ordinary end users, should this be the workflow? It seems like a feature of GitHub like Code Quality should work "out of the box" for users instead of requiring custom configuration?

karlhorky avatar Nov 03 '25 10:11 karlhorky

Thanks, yes -- the team has also confirmed that this option should be available for Code Quality analyses.

We do strive to provide an out-of-the-box experience for as many repos as possible, but CodeQL performs fairly comprehensive analyses of code, which can require more powerful hardware for larger repos. You can read about how CodeQL works at https://codeql.github.com/docs/codeql-overview/about-codeql/.

That said, we are of course happy to look into the details of why the analysis ran out of disk space to see if there's a bug or something else we can improve on our end, especially since this feature is in preview. Since it is a private repo, would you be able to get in touch with me via email at [email protected] with the full workflow logs?

mbg avatar Nov 03 '25 11:11 mbg

Sure, reaching out privately with the full workflow logs.

karlhorky avatar Nov 03 '25 11:11 karlhorky