zfs
zfs copied to clipboard
openzfs
Cannot free space in full pool
System information
| Type | Version/Name |
|---|---|
| Distribution Name | Debian |
| Distribution Version | 10 |
| Kernel Version | 4.19.0-21 |
| Architecture | amd64 |
| OpenZFS Version | 2.1.99-1335_ga582d5299 |
Describe the problem you're observing
Freshly created pool became full and now i'm unable to free any space so it's effectively unusable. This would be amusing if it wasn't disappointing:
root@linux:~# df /home
Filesystem 1K-blocks Used Available Use% Mounted on
rpool/home 33280 33280 0 100% /home
root@linux:~# rm -fv /home/nginx-1.19.3.tar.gz
rm: cannot remove '/home/nginx-1.19.3.tar.gz': No space left on device
root@linux:~# zfs destroy rpool/home
cannot destroy 'rpool/home': out of space
root@linux:~# zfs set test:test=test rpool/home
cannot set property for 'rpool/home': out of space
root@linux:~#
Describe how to reproduce the problem
Fill zfs pool then try to free some space
Include any warning/errors/backtraces from the system logs
This kind of bugs are the reason i'm not going to upgrade zfs on any of my systems for the time being, see #13755
Nope
root@linux:~# truncate -s0 /home/nginx-1.19.3.tar.gz
truncate: failed to truncate '/home/nginx-1.19.3.tar.gz' at 0 bytes: No space left on device
root@linux:~#
I tried this and I had no issue issue to delete the file.
cp xubuntu-21.10-desktop-amd64.iso /ztest/
cp: error writing '/ztest/xubuntu-21.10-desktop-amd64.iso': No space left on device
# zfs list ztest
NAME USED AVAIL REFER MOUNTPOINT
ztest 1.88G 0B 1.87G /ztest
# df .
Filesystem Type Size Used Avail Use% Mounted on
ztest zfs 1,9G 1,9G 0 100% /ztest
# rm xubuntu-21.10-desktop-amd64.iso
# df .
Filesystem Type Size Used Avail Use% Mounted on
ztest zfs 1,9G 128K 1,9G 1% /ztest
This is with EndeavourOS zfs-2.1.5-1 zfs-kmod-2.1.5-1 kernel 5.15.64-1-lts
For obvious reasons i am not in a position to share this particular pool containing my home directory: i'll see if i can get another zfs pool to crap itself.
It would probably be useful to share any details of the pool and dataset(s) that you can, like the outputs from zpool get all [pool] and zfs get all [dataset], and the current state of all the module tunables (the simplest way to do this is probably to get them from the long tail of arc_summary's output...or something like grep . /sys/module/{spl,zfs}/parameters/*, I suppose, on a new enough git that it's all welded into a couple modules...)
Nope
root@linux:~# truncate -s0 /home/nginx-1.19.3.tar.gz truncate: failed to truncate '/home/nginx-1.19.3.tar.gz' at 0 bytes: No space left on device root@linux:~#
Is that file part of a snapshot? Find a decent sized file (let’s say 10MB) that is not part of a snapshot and try truncating it.
I had this issue happen on my system a few weeks ago and truncating a decently sized file fixed it.
It would probably be useful to share any details of the pool and dataset(s) that you can, like the outputs from
zpool get all [pool]andzfs get all [dataset], and the current state of all the module tunables (the simplest way to do this is probably to get them from the long tail ofarc_summary's output...or something likegrep . /sys/module/{spl,zfs}/parameters/*, I suppose, on a new enough git that it's all welded into a couple modules...)
root@linux:~# zpool get all rpool
NAME PROPERTY VALUE SOURCE
rpool size 7.50G -
rpool capacity 99% -
rpool altroot - default
rpool health ONLINE -
rpool guid 13424868433598158628 -
rpool version - default
rpool bootfs - default
rpool delegation on default
rpool autoreplace off default
rpool cachefile - default
rpool failmode wait default
rpool listsnapshots off default
rpool autoexpand off default
rpool dedupratio 1.00x -
rpool free 14.5M -
rpool allocated 7.49G -
rpool readonly off -
rpool ashift 0 default
rpool comment - default
rpool expandsize - -
rpool freeing 0 -
rpool fragmentation 93% -
rpool leaked 0 -
rpool multihost off default
rpool checkpoint - -
rpool load_guid 8200461303034091991 -
rpool autotrim off default
rpool compatibility off default
rpool feature@async_destroy enabled local
rpool feature@empty_bpobj active local
rpool feature@lz4_compress active local
rpool feature@multi_vdev_crash_dump enabled local
rpool feature@spacemap_histogram active local
rpool feature@enabled_txg active local
rpool feature@hole_birth active local
rpool feature@extensible_dataset active local
rpool feature@embedded_data active local
rpool feature@bookmarks enabled local
rpool feature@filesystem_limits enabled local
rpool feature@large_blocks enabled local
rpool feature@large_dnode enabled local
rpool feature@sha512 enabled local
rpool feature@skein enabled local
rpool feature@edonr enabled local
rpool feature@userobj_accounting active local
rpool feature@encryption enabled local
rpool feature@project_quota active local
rpool feature@device_removal enabled local
rpool feature@obsolete_counts enabled local
rpool feature@zpool_checkpoint enabled local
rpool feature@spacemap_v2 active local
rpool feature@allocation_classes enabled local
rpool feature@resilver_defer enabled local
rpool feature@bookmark_v2 enabled local
rpool feature@redaction_bookmarks enabled local
rpool feature@redacted_datasets enabled local
rpool feature@bookmark_written enabled local
rpool feature@log_spacemap active local
rpool feature@livelist enabled local
rpool feature@device_rebuild enabled local
rpool feature@zstd_compress enabled local
rpool feature@draid enabled local
rpool feature@zilsaxattr enabled local
rpool feature@head_errlog active local
rpool feature@blake3 enabled local
root@linux:~# zfs get all rpool/home
NAME PROPERTY VALUE SOURCE
rpool/home type filesystem -
rpool/home creation Tue Sep 6 11:17 2022 -
rpool/home used 111M -
rpool/home available 0B -
rpool/home referenced 111M -
rpool/home compressratio 1.00x -
rpool/home mounted yes -
rpool/home quota none default
rpool/home reservation none default
rpool/home recordsize 128K default
rpool/home mountpoint /home local
rpool/home sharenfs off default
rpool/home checksum on default
rpool/home compression off inherited from rpool
rpool/home atime on default
rpool/home devices on default
rpool/home exec on default
rpool/home setuid on default
rpool/home readonly off default
rpool/home zoned off default
rpool/home snapdir hidden default
rpool/home aclmode discard default
rpool/home aclinherit restricted default
rpool/home createtxg 6 -
rpool/home canmount on default
rpool/home xattr on default
rpool/home copies 1 default
rpool/home version 5 -
rpool/home utf8only off -
rpool/home normalization none -
rpool/home casesensitivity sensitive -
rpool/home vscan off default
rpool/home nbmand off default
rpool/home sharesmb off default
rpool/home refquota none default
rpool/home refreservation none default
rpool/home guid 5000977005295121824 -
rpool/home primarycache all default
rpool/home secondarycache all default
rpool/home usedbysnapshots 0B -
rpool/home usedbydataset 111M -
rpool/home usedbychildren 0B -
rpool/home usedbyrefreservation 0B -
rpool/home logbias latency default
rpool/home objsetid 70 -
rpool/home dedup off default
rpool/home mlslabel none default
rpool/home sync standard default
rpool/home dnodesize legacy default
rpool/home refcompressratio 1.00x -
rpool/home written 111M -
rpool/home logicalused 111M -
rpool/home logicalreferenced 111M -
rpool/home volmode default default
rpool/home filesystem_limit none default
rpool/home snapshot_limit none default
rpool/home filesystem_count none default
rpool/home snapshot_count none default
rpool/home snapdev hidden default
rpool/home acltype off default
rpool/home context none default
rpool/home fscontext none default
rpool/home defcontext none default
rpool/home rootcontext none default
rpool/home relatime on default
rpool/home redundant_metadata all default
rpool/home overlay on default
rpool/home encryption off default
rpool/home keylocation none default
rpool/home keyformat none default
rpool/home pbkdf2iters 0 default
rpool/home special_small_blocks 0 default
root@linux:~# grep . /sys/module/{spl,zfs}/parameters/*
/sys/module/spl/parameters/spl_hostid:0
/sys/module/spl/parameters/spl_hostid_path:/etc/hostid
/sys/module/spl/parameters/spl_kmem_alloc_max:1048576
/sys/module/spl/parameters/spl_kmem_alloc_warn:65536
/sys/module/spl/parameters/spl_kmem_cache_kmem_threads:4
/sys/module/spl/parameters/spl_kmem_cache_magazine_size:0
/sys/module/spl/parameters/spl_kmem_cache_max_size:32
/sys/module/spl/parameters/spl_kmem_cache_obj_per_slab:8
/sys/module/spl/parameters/spl_kmem_cache_reclaim:0
/sys/module/spl/parameters/spl_kmem_cache_slab_limit:16384
/sys/module/spl/parameters/spl_max_show_tasks:512
/sys/module/spl/parameters/spl_panic_halt:0
/sys/module/spl/parameters/spl_schedule_hrtimeout_slack_us:0
/sys/module/spl/parameters/spl_taskq_kick:0
/sys/module/spl/parameters/spl_taskq_thread_bind:0
/sys/module/spl/parameters/spl_taskq_thread_dynamic:1
/sys/module/spl/parameters/spl_taskq_thread_priority:1
/sys/module/spl/parameters/spl_taskq_thread_sequential:4
/sys/module/zfs/parameters/dbuf_cache_hiwater_pct:10
/sys/module/zfs/parameters/dbuf_cache_lowater_pct:10
/sys/module/zfs/parameters/dbuf_cache_max_bytes:18446744073709551615
/sys/module/zfs/parameters/dbuf_cache_shift:5
/sys/module/zfs/parameters/dbuf_metadata_cache_max_bytes:18446744073709551615
/sys/module/zfs/parameters/dbuf_metadata_cache_shift:6
/sys/module/zfs/parameters/dmu_object_alloc_chunk_shift:7
/sys/module/zfs/parameters/dmu_prefetch_max:134217728
/sys/module/zfs/parameters/icp_aes_impl:cycle [fastest] generic x86_64 aesni
/sys/module/zfs/parameters/icp_blake3_impl:cycle [fastest] generic sse2 sse41 avx2
/sys/module/zfs/parameters/icp_gcm_avx_chunk_size:32736
/sys/module/zfs/parameters/icp_gcm_impl:cycle [fastest] avx generic pclmulqdq
/sys/module/zfs/parameters/ignore_hole_birth:1
/sys/module/zfs/parameters/l2arc_exclude_special:0
/sys/module/zfs/parameters/l2arc_feed_again:1
/sys/module/zfs/parameters/l2arc_feed_min_ms:200
/sys/module/zfs/parameters/l2arc_feed_secs:1
/sys/module/zfs/parameters/l2arc_headroom:2
/sys/module/zfs/parameters/l2arc_headroom_boost:200
/sys/module/zfs/parameters/l2arc_meta_percent:33
/sys/module/zfs/parameters/l2arc_mfuonly:0
/sys/module/zfs/parameters/l2arc_noprefetch:1
/sys/module/zfs/parameters/l2arc_norw:0
/sys/module/zfs/parameters/l2arc_rebuild_blocks_min_l2size:1073741824
/sys/module/zfs/parameters/l2arc_rebuild_enabled:1
/sys/module/zfs/parameters/l2arc_trim_ahead:0
/sys/module/zfs/parameters/l2arc_write_boost:8388608
/sys/module/zfs/parameters/l2arc_write_max:8388608
/sys/module/zfs/parameters/metaslab_aliquot:1048576
/sys/module/zfs/parameters/metaslab_bias_enabled:1
/sys/module/zfs/parameters/metaslab_debug_load:0
/sys/module/zfs/parameters/metaslab_debug_unload:0
/sys/module/zfs/parameters/metaslab_df_max_search:16777216
/sys/module/zfs/parameters/metaslab_df_use_largest_segment:0
/sys/module/zfs/parameters/metaslab_force_ganging:16777217
/sys/module/zfs/parameters/metaslab_fragmentation_factor_enabled:1
/sys/module/zfs/parameters/metaslab_lba_weighting_enabled:1
/sys/module/zfs/parameters/metaslab_preload_enabled:1
/sys/module/zfs/parameters/metaslab_unload_delay:32
/sys/module/zfs/parameters/metaslab_unload_delay_ms:600000
/sys/module/zfs/parameters/send_holes_without_birth_time:1
/sys/module/zfs/parameters/spa_asize_inflation:24
/sys/module/zfs/parameters/spa_config_path:/etc/zfs/zpool.cache
/sys/module/zfs/parameters/spa_load_print_vdev_tree:0
/sys/module/zfs/parameters/spa_load_verify_data:1
/sys/module/zfs/parameters/spa_load_verify_metadata:1
/sys/module/zfs/parameters/spa_load_verify_shift:4
/sys/module/zfs/parameters/spa_slop_shift:5
/sys/module/zfs/parameters/spa_upgrade_errlog_limit:0
/sys/module/zfs/parameters/vdev_file_logical_ashift:9
/sys/module/zfs/parameters/vdev_file_physical_ashift:9
/sys/module/zfs/parameters/vdev_removal_max_span:32768
/sys/module/zfs/parameters/vdev_validate_skip:0
/sys/module/zfs/parameters/zap_iterate_prefetch:1
/sys/module/zfs/parameters/zfetch_array_rd_sz:1048576
/sys/module/zfs/parameters/zfetch_max_distance:67108864
/sys/module/zfs/parameters/zfetch_max_idistance:67108864
/sys/module/zfs/parameters/zfetch_max_sec_reap:2
/sys/module/zfs/parameters/zfetch_max_streams:8
/sys/module/zfs/parameters/zfetch_min_distance:4194304
/sys/module/zfs/parameters/zfetch_min_sec_reap:1
/sys/module/zfs/parameters/zfs_abd_scatter_enabled:1
/sys/module/zfs/parameters/zfs_abd_scatter_max_order:10
/sys/module/zfs/parameters/zfs_abd_scatter_min_size:1536
/sys/module/zfs/parameters/zfs_admin_snapshot:0
/sys/module/zfs/parameters/zfs_allow_redacted_dataset_mount:0
/sys/module/zfs/parameters/zfs_arc_average_blocksize:8192
/sys/module/zfs/parameters/zfs_arc_dnode_limit:0
/sys/module/zfs/parameters/zfs_arc_dnode_limit_percent:10
/sys/module/zfs/parameters/zfs_arc_dnode_reduce_percent:10
/sys/module/zfs/parameters/zfs_arc_evict_batch_limit:10
/sys/module/zfs/parameters/zfs_arc_eviction_pct:200
/sys/module/zfs/parameters/zfs_arc_grow_retry:0
/sys/module/zfs/parameters/zfs_arc_lotsfree_percent:10
/sys/module/zfs/parameters/zfs_arc_max:0
/sys/module/zfs/parameters/zfs_arc_meta_adjust_restarts:4096
/sys/module/zfs/parameters/zfs_arc_meta_limit:0
/sys/module/zfs/parameters/zfs_arc_meta_limit_percent:75
/sys/module/zfs/parameters/zfs_arc_meta_min:0
/sys/module/zfs/parameters/zfs_arc_meta_prune:10000
/sys/module/zfs/parameters/zfs_arc_meta_strategy:1
/sys/module/zfs/parameters/zfs_arc_min:0
/sys/module/zfs/parameters/zfs_arc_min_prefetch_ms:0
/sys/module/zfs/parameters/zfs_arc_min_prescient_prefetch_ms:0
/sys/module/zfs/parameters/zfs_arc_pc_percent:0
/sys/module/zfs/parameters/zfs_arc_p_dampener_disable:1
/sys/module/zfs/parameters/zfs_arc_p_min_shift:0
/sys/module/zfs/parameters/zfs_arc_prune_task_threads:1
/sys/module/zfs/parameters/zfs_arc_shrinker_limit:10000
/sys/module/zfs/parameters/zfs_arc_shrink_shift:0
/sys/module/zfs/parameters/zfs_arc_sys_free:0
/sys/module/zfs/parameters/zfs_async_block_max_blocks:18446744073709551615
/sys/module/zfs/parameters/zfs_autoimport_disable:1
/sys/module/zfs/parameters/zfs_checksum_events_per_second:20
/sys/module/zfs/parameters/zfs_commit_timeout_pct:5
/sys/module/zfs/parameters/zfs_compressed_arc_enabled:1
/sys/module/zfs/parameters/zfs_condense_indirect_commit_entry_delay_ms:0
/sys/module/zfs/parameters/zfs_condense_indirect_obsolete_pct:25
/sys/module/zfs/parameters/zfs_condense_indirect_vdevs_enable:1
/sys/module/zfs/parameters/zfs_condense_max_obsolete_bytes:1073741824
/sys/module/zfs/parameters/zfs_condense_min_mapping_bytes:131072
/sys/module/zfs/parameters/zfs_dbgmsg_enable:1
/sys/module/zfs/parameters/zfs_dbgmsg_maxsize:4194304
/sys/module/zfs/parameters/zfs_dbuf_state_index:0
/sys/module/zfs/parameters/zfs_ddt_data_is_special:1
/sys/module/zfs/parameters/zfs_deadman_checktime_ms:60000
/sys/module/zfs/parameters/zfs_deadman_enabled:1
/sys/module/zfs/parameters/zfs_deadman_failmode:wait
/sys/module/zfs/parameters/zfs_deadman_synctime_ms:600000
/sys/module/zfs/parameters/zfs_deadman_ziotime_ms:300000
/sys/module/zfs/parameters/zfs_dedup_prefetch:0
/sys/module/zfs/parameters/zfs_delay_min_dirty_percent:60
/sys/module/zfs/parameters/zfs_delay_scale:500000
/sys/module/zfs/parameters/zfs_delete_blocks:20480
/sys/module/zfs/parameters/zfs_dirty_data_max:413803724
/sys/module/zfs/parameters/zfs_dirty_data_max_max:1034509312
/sys/module/zfs/parameters/zfs_dirty_data_max_max_percent:25
/sys/module/zfs/parameters/zfs_dirty_data_max_percent:10
/sys/module/zfs/parameters/zfs_dirty_data_sync_percent:20
/sys/module/zfs/parameters/zfs_disable_ivset_guid_check:0
/sys/module/zfs/parameters/zfs_dmu_offset_next_sync:1
/sys/module/zfs/parameters/zfs_embedded_slog_min_ms:64
/sys/module/zfs/parameters/zfs_expire_snapshot:300
/sys/module/zfs/parameters/zfs_fallocate_reserve_percent:110
/sys/module/zfs/parameters/zfs_flags:0
/sys/module/zfs/parameters/zfs_fletcher_4_impl:[fastest] scalar superscalar superscalar4 sse2 ssse3 avx2
/sys/module/zfs/parameters/zfs_free_bpobj_enabled:1
/sys/module/zfs/parameters/zfs_free_leak_on_eio:0
/sys/module/zfs/parameters/zfs_free_min_time_ms:1000
/sys/module/zfs/parameters/zfs_history_output_max:1048576
/sys/module/zfs/parameters/zfs_immediate_write_sz:32768
/sys/module/zfs/parameters/zfs_initialize_chunk_size:1048576
/sys/module/zfs/parameters/zfs_initialize_value:16045690984833335022
/sys/module/zfs/parameters/zfs_keep_log_spacemaps_at_export:0
/sys/module/zfs/parameters/zfs_key_max_salt_uses:400000000
/sys/module/zfs/parameters/zfs_livelist_condense_new_alloc:0
/sys/module/zfs/parameters/zfs_livelist_condense_sync_cancel:0
/sys/module/zfs/parameters/zfs_livelist_condense_sync_pause:0
/sys/module/zfs/parameters/zfs_livelist_condense_zthr_cancel:0
/sys/module/zfs/parameters/zfs_livelist_condense_zthr_pause:0
/sys/module/zfs/parameters/zfs_livelist_max_entries:500000
/sys/module/zfs/parameters/zfs_livelist_min_percent_shared:75
/sys/module/zfs/parameters/zfs_lua_max_instrlimit:100000000
/sys/module/zfs/parameters/zfs_lua_max_memlimit:104857600
/sys/module/zfs/parameters/zfs_max_async_dedup_frees:100000
/sys/module/zfs/parameters/zfs_max_dataset_nesting:50
/sys/module/zfs/parameters/zfs_max_logsm_summary_length:10
/sys/module/zfs/parameters/zfs_max_log_walking:5
/sys/module/zfs/parameters/zfs_max_missing_tvds:0
/sys/module/zfs/parameters/zfs_max_nvlist_src_size:0
/sys/module/zfs/parameters/zfs_max_recordsize:16777216
/sys/module/zfs/parameters/zfs_metaslab_find_max_tries:100
/sys/module/zfs/parameters/zfs_metaslab_fragmentation_threshold:70
/sys/module/zfs/parameters/zfs_metaslab_max_size_cache_sec:3600
/sys/module/zfs/parameters/zfs_metaslab_mem_limit:25
/sys/module/zfs/parameters/zfs_metaslab_segment_weight_enabled:1
/sys/module/zfs/parameters/zfs_metaslab_switch_threshold:2
/sys/module/zfs/parameters/zfs_metaslab_try_hard_before_gang:0
/sys/module/zfs/parameters/zfs_mg_fragmentation_threshold:95
/sys/module/zfs/parameters/zfs_mg_noalloc_threshold:0
/sys/module/zfs/parameters/zfs_min_metaslabs_to_flush:1
/sys/module/zfs/parameters/zfs_multihost_fail_intervals:10
/sys/module/zfs/parameters/zfs_multihost_history:0
/sys/module/zfs/parameters/zfs_multihost_import_intervals:20
/sys/module/zfs/parameters/zfs_multihost_interval:1000
/sys/module/zfs/parameters/zfs_multilist_num_sublists:0
/sys/module/zfs/parameters/zfs_nocacheflush:0
/sys/module/zfs/parameters/zfs_nopwrite_enabled:1
/sys/module/zfs/parameters/zfs_no_scrub_io:0
/sys/module/zfs/parameters/zfs_no_scrub_prefetch:0
/sys/module/zfs/parameters/zfs_object_mutex_size:64
/sys/module/zfs/parameters/zfs_obsolete_min_time_ms:500
/sys/module/zfs/parameters/zfs_override_estimate_recordsize:0
/sys/module/zfs/parameters/zfs_pd_bytes_max:52428800
/sys/module/zfs/parameters/zfs_per_txg_dirty_frees_percent:5
/sys/module/zfs/parameters/zfs_prefetch_disable:0
/sys/module/zfs/parameters/zfs_read_history:0
/sys/module/zfs/parameters/zfs_read_history_hits:0
/sys/module/zfs/parameters/zfs_rebuild_max_segment:1048576
/sys/module/zfs/parameters/zfs_rebuild_scrub_enabled:1
/sys/module/zfs/parameters/zfs_rebuild_vdev_limit:33554432
/sys/module/zfs/parameters/zfs_reconstruct_indirect_combinations_max:4096
/sys/module/zfs/parameters/zfs_recover:0
/sys/module/zfs/parameters/zfs_recv_best_effort_corrective:0
/sys/module/zfs/parameters/zfs_recv_queue_ff:20
/sys/module/zfs/parameters/zfs_recv_queue_length:16777216
/sys/module/zfs/parameters/zfs_recv_write_batch_size:1048576
/sys/module/zfs/parameters/zfs_removal_ignore_errors:0
/sys/module/zfs/parameters/zfs_removal_suspend_progress:0
/sys/module/zfs/parameters/zfs_remove_max_segment:16777216
/sys/module/zfs/parameters/zfs_resilver_disable_defer:0
/sys/module/zfs/parameters/zfs_resilver_min_time_ms:3000
/sys/module/zfs/parameters/zfs_scan_blkstats:0
/sys/module/zfs/parameters/zfs_scan_checkpoint_intval:7200
/sys/module/zfs/parameters/zfs_scan_fill_weight:3
/sys/module/zfs/parameters/zfs_scan_ignore_errors:0
/sys/module/zfs/parameters/zfs_scan_issue_strategy:0
/sys/module/zfs/parameters/zfs_scan_legacy:0
/sys/module/zfs/parameters/zfs_scan_max_ext_gap:2097152
/sys/module/zfs/parameters/zfs_scan_mem_lim_fact:20
/sys/module/zfs/parameters/zfs_scan_mem_lim_soft_fact:20
/sys/module/zfs/parameters/zfs_scan_strict_mem_lim:0
/sys/module/zfs/parameters/zfs_scan_suspend_progress:0
/sys/module/zfs/parameters/zfs_scan_vdev_limit:4194304
/sys/module/zfs/parameters/zfs_scrub_min_time_ms:1000
/sys/module/zfs/parameters/zfs_send_corrupt_data:0
/sys/module/zfs/parameters/zfs_send_no_prefetch_queue_ff:20
/sys/module/zfs/parameters/zfs_send_no_prefetch_queue_length:1048576
/sys/module/zfs/parameters/zfs_send_queue_ff:20
/sys/module/zfs/parameters/zfs_send_queue_length:16777216
/sys/module/zfs/parameters/zfs_send_unmodified_spill_blocks:1
/sys/module/zfs/parameters/zfs_slow_io_events_per_second:20
/sys/module/zfs/parameters/zfs_snapshot_history_enabled:1
/sys/module/zfs/parameters/zfs_spa_discard_memory_limit:16777216
/sys/module/zfs/parameters/zfs_special_class_metadata_reserve_pct:25
/sys/module/zfs/parameters/zfs_sync_pass_deferred_free:2
/sys/module/zfs/parameters/zfs_sync_pass_dont_compress:8
/sys/module/zfs/parameters/zfs_sync_pass_rewrite:2
/sys/module/zfs/parameters/zfs_sync_taskq_batch_pct:75
/sys/module/zfs/parameters/zfs_traverse_indirect_prefetch_limit:32
/sys/module/zfs/parameters/zfs_trim_extent_bytes_max:134217728
/sys/module/zfs/parameters/zfs_trim_extent_bytes_min:32768
/sys/module/zfs/parameters/zfs_trim_metaslab_skip:0
/sys/module/zfs/parameters/zfs_trim_queue_limit:10
/sys/module/zfs/parameters/zfs_trim_txg_batch:32
/sys/module/zfs/parameters/zfs_txg_history:100
/sys/module/zfs/parameters/zfs_txg_timeout:5
/sys/module/zfs/parameters/zfs_unflushed_log_block_max:131072
/sys/module/zfs/parameters/zfs_unflushed_log_block_min:1000
/sys/module/zfs/parameters/zfs_unflushed_log_block_pct:400
/sys/module/zfs/parameters/zfs_unflushed_log_txg_max:1000
/sys/module/zfs/parameters/zfs_unflushed_max_mem_amt:1073741824
/sys/module/zfs/parameters/zfs_unflushed_max_mem_ppm:1000
/sys/module/zfs/parameters/zfs_unlink_suspend_progress:0
/sys/module/zfs/parameters/zfs_user_indirect_is_special:1
/sys/module/zfs/parameters/zfs_vdev_aggregate_trim:0
/sys/module/zfs/parameters/zfs_vdev_aggregation_limit:1048576
/sys/module/zfs/parameters/zfs_vdev_aggregation_limit_non_rotating:131072
/sys/module/zfs/parameters/zfs_vdev_async_read_max_active:3
/sys/module/zfs/parameters/zfs_vdev_async_read_min_active:1
/sys/module/zfs/parameters/zfs_vdev_async_write_active_max_dirty_percent:60
/sys/module/zfs/parameters/zfs_vdev_async_write_active_min_dirty_percent:30
/sys/module/zfs/parameters/zfs_vdev_async_write_max_active:10
/sys/module/zfs/parameters/zfs_vdev_async_write_min_active:2
/sys/module/zfs/parameters/zfs_vdev_cache_bshift:16
/sys/module/zfs/parameters/zfs_vdev_cache_max:16384
/sys/module/zfs/parameters/zfs_vdev_cache_size:0
/sys/module/zfs/parameters/zfs_vdev_default_ms_count:200
/sys/module/zfs/parameters/zfs_vdev_default_ms_shift:29
/sys/module/zfs/parameters/zfs_vdev_initializing_max_active:1
/sys/module/zfs/parameters/zfs_vdev_initializing_min_active:1
/sys/module/zfs/parameters/zfs_vdev_max_active:1000
/sys/module/zfs/parameters/zfs_vdev_max_auto_ashift:16
/sys/module/zfs/parameters/zfs_vdev_min_auto_ashift:9
/sys/module/zfs/parameters/zfs_vdev_min_ms_count:16
/sys/module/zfs/parameters/zfs_vdev_mirror_non_rotating_inc:0
/sys/module/zfs/parameters/zfs_vdev_mirror_non_rotating_seek_inc:1
/sys/module/zfs/parameters/zfs_vdev_mirror_rotating_inc:0
/sys/module/zfs/parameters/zfs_vdev_mirror_rotating_seek_inc:5
/sys/module/zfs/parameters/zfs_vdev_mirror_rotating_seek_offset:1048576
/sys/module/zfs/parameters/zfs_vdev_ms_count_limit:131072
/sys/module/zfs/parameters/zfs_vdev_nia_credit:5
/sys/module/zfs/parameters/zfs_vdev_nia_delay:5
/sys/module/zfs/parameters/zfs_vdev_queue_depth_pct:1000
/sys/module/zfs/parameters/zfs_vdev_raidz_impl:cycle [fastest] original scalar sse2 ssse3 avx2
/sys/module/zfs/parameters/zfs_vdev_read_gap_limit:32768
/sys/module/zfs/parameters/zfs_vdev_rebuild_max_active:3
/sys/module/zfs/parameters/zfs_vdev_rebuild_min_active:1
/sys/module/zfs/parameters/zfs_vdev_removal_max_active:2
/sys/module/zfs/parameters/zfs_vdev_removal_min_active:1
/sys/module/zfs/parameters/zfs_vdev_scheduler:unused
/sys/module/zfs/parameters/zfs_vdev_scrub_max_active:3
/sys/module/zfs/parameters/zfs_vdev_scrub_min_active:1
/sys/module/zfs/parameters/zfs_vdev_sync_read_max_active:10
/sys/module/zfs/parameters/zfs_vdev_sync_read_min_active:10
/sys/module/zfs/parameters/zfs_vdev_sync_write_max_active:10
/sys/module/zfs/parameters/zfs_vdev_sync_write_min_active:10
/sys/module/zfs/parameters/zfs_vdev_trim_max_active:2
/sys/module/zfs/parameters/zfs_vdev_trim_min_active:1
/sys/module/zfs/parameters/zfs_vdev_write_gap_limit:4096
/sys/module/zfs/parameters/zfs_vnops_read_chunk_size:1048576
/sys/module/zfs/parameters/zfs_wrlog_data_max:827607448
/sys/module/zfs/parameters/zfs_xattr_compat:0
/sys/module/zfs/parameters/zfs_zevent_len_max:512
/sys/module/zfs/parameters/zfs_zevent_retain_expire_secs:900
/sys/module/zfs/parameters/zfs_zevent_retain_max:2000
/sys/module/zfs/parameters/zfs_zil_clean_taskq_maxalloc:1048576
/sys/module/zfs/parameters/zfs_zil_clean_taskq_minalloc:1024
/sys/module/zfs/parameters/zfs_zil_clean_taskq_nthr_pct:100
/sys/module/zfs/parameters/zfs_zil_saxattr:1
/sys/module/zfs/parameters/zil_maxblocksize:131072
/sys/module/zfs/parameters/zil_nocacheflush:0
/sys/module/zfs/parameters/zil_replay_disable:0
/sys/module/zfs/parameters/zil_slog_bulk:786432
/sys/module/zfs/parameters/zio_deadman_log_all:0
/sys/module/zfs/parameters/zio_dva_throttle_enabled:1
/sys/module/zfs/parameters/zio_requeue_io_start_cut_in_line:1
/sys/module/zfs/parameters/zio_slow_io_ms:30000
/sys/module/zfs/parameters/zio_taskq_batch_pct:80
/sys/module/zfs/parameters/zio_taskq_batch_tpq:0
/sys/module/zfs/parameters/zstd_abort_size:131072
/sys/module/zfs/parameters/zstd_earlyabort_pass:1
/sys/module/zfs/parameters/zvol_blk_mq_blocks_per_thread:8
/sys/module/zfs/parameters/zvol_blk_mq_queue_depth:128
/sys/module/zfs/parameters/zvol_inhibit_dev:0
/sys/module/zfs/parameters/zvol_major:230
/sys/module/zfs/parameters/zvol_max_discard_blocks:16384
/sys/module/zfs/parameters/zvol_prefetch_bytes:131072
/sys/module/zfs/parameters/zvol_request_sync:0
/sys/module/zfs/parameters/zvol_threads:0
/sys/module/zfs/parameters/zvol_use_blk_mq:0
/sys/module/zfs/parameters/zvol_volmode:1
root@linux:~#
Nope
root@linux:~# truncate -s0 /home/nginx-1.19.3.tar.gz truncate: failed to truncate '/home/nginx-1.19.3.tar.gz' at 0 bytes: No space left on device root@linux:~#Is that file part of a snapshot? Find a decent sized file (let’s say 10MB) that is not part of a snapshot and try truncating it.
I had this issue happen on my system a few weeks ago and truncating a decently sized file fixed it.
There are no snapshot on rpool/home:
root@linux:~# zfs list -tall -r rpool/home
NAME USED AVAIL REFER MOUNTPOINT
rpool/home 111M 0B 111M /home
root@linux:~# rm -f /home/*
rm: cannot remove '/home/big.bin': No space left on device
rm: cannot remove '/home/medium.bin': No space left on device
rm: cannot remove '/home/small.bin': No space left on device
root@linux:~# truncate -s0 /home/*
truncate: failed to truncate '/home/big.bin' at 0 bytes: No space left on device
truncate: failed to truncate '/home/medium.bin' at 0 bytes: No space left on device
truncate: failed to truncate '/home/small.bin' at 0 bytes: No space left on device
root@linux:~# zfs destroy rpool/home
cannot destroy 'rpool/home': out of space
root@linux:~#
My guess for what's happening goes like this: ZFS used to, once upon a time, have problems like this sometimes, where you could get the pool so full that it was impossible to free space, because ZFS basically can never overwrite something in-place, so you always need space to do things.
The way this was worked around was by reserving enough space out of the pool's free space to always be able to make progress on operations that shouldn't be able to fail or are expected to free space, and only allow those operations to use space from that reserved set.
That calculation is a percentage of the total space on the pool, with a minimum (128 MB) and more recently, a maximum (I think 128 GB?)
At a pool of 8 GB, I still wouldn't expect you to be hitting the minimum (1/32 of 8GB would be 256 MB), but if zpool list is reporting only 14 MB, I believe that's including slop space, so...something has gone very awry. I would assume something was erroneously allowed to use that space, or the worst-case calculations for how much an operation might use were missing some edge case and the result went over (I'm wondering about gang blocks swelling size requirements, from that free space fragmentation, but with so little free space, data unclear).
I took a shot at reproducing this on a testbed, and didn't immediately find a way to do it...
One maybe working way out of this situation:
echo 10 >> /sys/module/zfs/parameters/spa_slop_shift to reduce the slop (free space in pool that is not made available to the datasets) calculation, this could return the dataset to normal working condition (allowed to make allocations, removing the 'No space left on device' condition) so you can rm some bigger files.
Revert the change by echo 5 >> /sys/module/zfs/parameters/spa_slop_shift as soon as you have made some free space.
This approach will work the better, the bigger the pool.
Maybe rm or truncate on some smaller files could work better, reasoning being the amount of modified metadata for freeing a large file (multiple space maps have to be rewritten, the space for them needs to be available) possibly blocking the atomic update... should that be the case smaller files (fewer or only one space map touched) might work.
One maybe working way out of this situation:
echo 10 >> /sys/module/zfs/parameters/spa_slop_shiftto reduce the slop (free space in pool that is not made available to the datasets) calculation, this could return the dataset to normal working condition (allowed to make allocations, removing the 'No space left on device' condition) so you can rm some bigger files. Revert the change byecho 5 >> /sys/module/zfs/parameters/spa_slop_shiftas soon as you have made some free space.This approach will work the better, the bigger the pool.
Maybe rm or truncate on some smaller files could work better, reasoning being the amount of modified metadata for freeing a large file (multiple space maps have to be rewritten, the space for them needs to be available) possibly blocking the atomic update... should that be the case smaller files (fewer or only one space map touched) might work.
Well it did not work, it's all ogre now.
root@linux:~# ls -lh /home/small.bin
-rw-r--r-- 1 root root 1.0M Sep 6 11:17 /home/small.bin
root@linux:~# echo 10 >> /sys/module/zfs/parameters/spa_slop_shift
root@linux:~# rm -fv /home/small.bin
rm: cannot remove '/home/small.bin': No space left on device
root@linux:~#
@tuider Would you contact me at 1pm EST tomorrow on the OpenZFS slack server?
https://openzfs.slack.com/
I am willing to try to help you fix this in real-time. My schedule has become very tight this week, but I should have some time then to try to help.
@ryao thanks for the offer, but i don't have a slack account. Fortunately this pool does not contain critical data which still sits on older zfs installations (lucky me!).
Well it did not work, it's all ogre now.
As the pool is just 8GB: in case you have space on a zfs enabled system you could try to dd the device to a file, truncate that to some GB more, and try to expand the pool (zpool online -e ...).
As the pool is just 8GB: in case you have space on a zfs enabled system you could try to dd the device to a file, truncate that to some GB more, and try to expand the pool (
zpool online -e ...).
I can try all the workarounds at this point but the solution to what is clearly a software bug should not be to just throw more hardware at it; i reported this issue so you guys are aware of it and can hopefully fix it. Like i said this pool is not critical, i have my data stored on "good old zfs" which does not behave like this. I am more inclined to try troubleshooting steps to understand the problem, not workarounds to it.
@tuider If you could give me access to the pool, I might be able to understand what went wrong. Unfortunately, my free time keeps disappearing, so I would need to look on either the weekend or a week day next week if you are willing to give me access.
@ryao this is one of the test pools that crapped itself, compressed and encrypted with your ryao[at]gentoo.org public key (which expired in 2021 btw): https://a.tmp.ninja/klTXZNlU.gpg
sha256sum: e61f4b1b0523939a7b787a47c14ee36cd9e45463db5bf83f55fd57f57816df4d
Link should last 48 hours.
@tuider Thanks. The unencrypted and decompressed file has a sha256 checksum of b0d7a310e3d911b5b559a5569a1f9163a45bb623e8c39ff06b803ad4c7a4b5ea. I assume that is what you had at your end.
You are the first to send me data this way. I was pleasantly surprised to see someone use GPG to make a public file download only understandable to me. :)
Also, thanks for reminding me to refresh my key. It has been a while since I have done it, so I will need to look up how to do it. I will do it when my schedule is more free.
The unencrypted and decompressed file has a sha256 checksum of b0d7a310e3d911b5b559a5569a1f9163a45bb623e8c39ff06b803ad4c7a4b5ea. I assume that is what you had at your end.
Yes. Thank you for looking into this issue.
This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.
![stale[bot] avatar](https://avatars.githubusercontent.com/in/1724?v=4 "Published by a pub.dev verified publisher"){kind=small}