tiup icon indicating copy to clipboard operation
tiup copied to clipboard

ERROR 8256 (HY000): Check ingest environment failed: no enough space in /tmp/tidb/tmp_ddl-4000

Open yahonda opened this issue 1 year ago • 2 comments

Bug Report

Please answer these questions before submitting your issue. Thanks!

  1. What did you do?
% tiup -v
1.16.0 tiup
Go Version: go1.21.11
Git Ref: v1.16.0
GitHash: e543145831d44a863366ff8c379d25980730bbd1
% tiup playground
Note: Version constraint  is resolved to v8.1.0. If you'd like to use other versions:

    Use exact version:      tiup playground v7.1.0
    Use version range:      tiup playground ^5
    Use nightly:            tiup playground nightly

Start pd instance:v8.1.0
Start tikv instance:v8.1.0
Start tidb instance:v8.1.0
Waiting for tidb instances ready
127.0.0.1:4000 ... Done
Start tiflash instance:v8.1.0
Waiting for tiflash instances ready
127.0.0.1:3930 ... Done

🎉 TiDB Playground Cluster is started, enjoy!

Connect TiDB:    mysql --comments --host 127.0.0.1 --port 4000 -u root
TiDB Dashboard:  http://127.0.0.1:2379/dashboard
Grafana:         http://127.0.0.1:3000
  • Connect to the TiDB Playground Cluster using mysql client
% mysql --comments --host 127.0.0.1 --port 4000 -u root
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1421869062
Server version: 8.0.11-TiDB-v8.1.0 TiDB Server (Apache License 2.0) Community Edition, MySQL 8.0 compatible

Copyright (c) 2000, 2024, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> create database mydatabase;
Query OK, 0 rows affected (0.17 sec)

mysql> use mydatabase;
Database changed
mysql> CREATE TABLE `django_content_type` (`id` integer AUTO_INCREMENT NOT NULL PRIMARY KEY, `name` varchar(100) NOT NULL, `app_label` varchar(100) NOT NULL, `model` varchar(100) NOT NULL);
Query OK, 0 rows affected (0.12 sec)

mysql> ALTER TABLE `django_content_type` ADD CONSTRAINT `django_content_type_app_label_model_76bd3d3b_uniq` UNIQUE (`app_label`, `model`);
ERROR 8256 (HY000): Check ingest environment failed: no enough space in /tmp/tidb/tmp_ddl-4000
mysql>
  1. What did you expect to see? This DDL should run successfully.
ALTER TABLE `django_content_type` ADD CONSTRAINT `django_content_type_app_label_model_76bd3d3b_uniq` UNIQUE (`app_label`, `model`);
  1. What did you see instead? It gets ERROR 8256 (HY000): Check ingest environment failed: no enough space in /tmp/tidb/tmp_ddl-4000.
mysql> ALTER TABLE `django_content_type` ADD CONSTRAINT `django_content_type_app_label_model_76bd3d3b_uniq` UNIQUE (`app_label`, `model`);
ERROR 8256 (HY000): Check ingest environment failed: no enough space in /tmp/tidb/tmp_ddl-4000
mysql>
  1. What version of TiUP are you using (tiup --version)?
 % tiup --version
1.16.0 tiup
Go Version: go1.21.11
Git Ref: v1.16.0
GitHash: e543145831d44a863366ff8c379d25980730bbd1

yahonda avatar Jul 09 '24 04:07 yahonda

Here is a workaround found for this error. I assume this is related to DXF.

% mysql --comments --host 127.0.0.1 --port 4000 -u root
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1421869066
Server version: 8.0.11-TiDB-v8.1.0 TiDB Server (Apache License 2.0) Community Edition, MySQL 8.0 compatible

Copyright (c) 2000, 2024, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use mydatabase;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> set global tidb_ddl_enable_fast_reorg = off;
Query OK, 0 rows affected (0.05 sec)

mysql> set global tidb_enable_dist_task = off;
Query OK, 0 rows affected (0.02 sec)

mysql> ALTER TABLE `django_content_type` ADD CONSTRAINT `django_content_type_app_label_model_76bd3d3b_uniq` UNIQUE (`app_label`, `model`);
Query OK, 0 rows affected (0.59 sec)

mysql>

yahonda avatar Jul 09 '24 04:07 yahonda

This box has enough /tmp space available.

% df -h /tmp
Filesystem      Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/disk1s2   466Gi   408Gi    41Gi    91%    7.3M  425M    2%   /System/Volumes/Data
%

yahonda avatar Jul 09 '24 04:07 yahonda

According to the codebase, this check will fail with available space is < 10%. Try free more space?

breezewish avatar Dec 11 '24 10:12 breezewish

Thanks for the update. Closing this issue because it does not reproduce anymore.

  • Filesystem usage
 % df
Filesystem     512-blocks      Used Available Capacity iused      ifree %iused  Mounted on
/dev/disk3s1s1  965595304  21875352 726730680     3%  412140 3633653400    0%   /
devfs                 397       397         0   100%     688          0  100%   /dev
/dev/disk3s6    965595304        40 726730680     1%       0 3633653400    0%   /System/Volumes/VM
/dev/disk3s2    965595304  13447792 726730680     2%    1414 3633653400    0%   /System/Volumes/Preboot
/dev/disk3s4    965595304      4744 726730680     1%      53 3633653400    0%   /System/Volumes/Update
/dev/disk1s2      1024000     12328    988240     2%       1    4941200    0%   /System/Volumes/xarts
/dev/disk1s1      1024000     11376    988240     2%      33    4941200    0%   /System/Volumes/iSCPreboot
/dev/disk1s3      1024000      2336    988240     1%     102    4941200    0%   /System/Volumes/Hardware
/dev/disk3s5    965595304 201201944 726730680    22% 1798501 3633653400    0%   /System/Volumes/Data
map auto_home           0         0         0   100%       0          0     -   /System/Volumes/Data/home
/dev/disk3s3    965595304   2021544 726730680     1%      41 3633653400    0%   /Volumes/Recovery
%
  • Steps to reproduce
% tiup --version
1.16.1 tiup
Go Version: go1.21.13
Git Ref: v1.16.1
GitHash: 775384c48a95f7047c016cbf2905eec1e267f629
% tiup playground
Checking updates for component playground...
Note: Version constraint  is resolved to v8.4.0. If you'd like to use other versions:

    Use exact version:      tiup playground v7.1.0
    Use version range:      tiup playground ^5
    Use nightly:            tiup playground nightly

Start pd instance: v8.4.0
Start tikv instance: v8.4.0
Start tidb instance: v8.4.0
Waiting for tidb instances ready
127.0.0.1:4000 ... Done
Start tiflash instance: v8.4.0
Waiting for tiflash instances ready
127.0.0.1:3930 ... Done

🎉 TiDB Playground Cluster is started, enjoy!

Connect TiDB:    mysql --host 127.0.0.1 --port 4000 -u root
TiDB Dashboard:  http://127.0.0.1:2379/dashboard
Grafana:         http://127.0.0.1:56688
% mysql --comments --host 127.0.0.1 --port 4000 -u root -p

mysql> create database mydatabase;
Query OK, 0 rows affected (0.03 sec)

mysql> use mydatabase;
Database changed
mysql> CREATE TABLE `django_content_type` (`id` integer AUTO_INCREMENT NOT NULL PRIMARY KEY, `name` varchar(100) NOT NULL, `app_label` varchar(100) NOT NULL, `model` varchar(100) NOT NULL);
Query OK, 0 rows affected (0.02 sec)

mysql> ALTER TABLE `django_content_type` ADD CONSTRAINT `django_content_type_app_label_model_76bd3d3b_uniq` UNIQUE (`app_label`, `model`);
Query OK, 0 rows affected (1.35 sec)

mysql> select tidb_version();
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| tidb_version()                                                                                                                                                                                                                               |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Release Version: v8.4.0
Edition: Community
Git Commit Hash: 1a9f0fa25580907d3266dc3edf000ba24379a600
Git Branch: HEAD
UTC Build Time: 2024-11-07 15:20:21
GoVersion: go1.23.2
Race Enabled: false
Check Table Before Drop: false
Store: tikv |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

mysql> select  @@global.tidb_ddl_enable_fast_reorg;
+-------------------------------------+
| @@global.tidb_ddl_enable_fast_reorg |
+-------------------------------------+
|                                   1 |
+-------------------------------------+
1 row in set (0.00 sec)

mysql>

yahonda avatar Dec 13 '24 03:12 yahonda