Docker volume inspect showing incorrect size after increasing volume size

[tanvir-ahmed-m4]

Hello,

I have created a thin volume of 150G. Then extended the thin volume by 10G and the file system (as per #17). However, docker volume inspect still shows 150G. How can it be fixed?

Thanks.

Before

$ sudo lvs 
  LV              VG     Attr       LSize    Pool    Origin Data%  Meta%  Move Log Cpy%Sync Convert
  mydb_master     docker Vwi-aotz--  150.00g volumes        97.11                                  
  example1        docker Vwi-a-tz--  100.00m volumes        3.50                                   
  storage         docker twi-aotz--   <3.50g                19.54  34.18                           
  volumes         docker twi-aotz-- <172.04g                84.67  82.79                           

$ docker volume inspect mydb_master
[
    {
        "CreatedAt": "0001-01-01T00:00:00Z",
        "Driver": "lvm",
        "Labels": {
            "com.docker.stack.namespace": "mydb"
        },
        "Mountpoint": "/var/lib/docker-lvm-plugin/mydb_master",
        "Name": "mydb_master",
        "Options": {
            "size": "150G",
            "thinpool": "volumes"
        },
        "Scope": "local"
    }
]

After

$ sudo lvextend -L+10G /dev/mapper/docker-mydb_master 
  Size of logical volume docker/mydb_master changed from 150.00 GiB (38400 extents) to 160.00 GiB (40960 extents).
  Logical volume docker/mydb_master successfully resized.

$ sudo lvs
  LV              VG     Attr       LSize   Pool    Origin Data%  Meta%  Move Log Cpy%Sync Convert
  mydb_master     docker Vwi-aotz-- 160.00g volumes        91.05                                  
  example1        docker Vwi-a-tz-- 100.00m volumes        3.50                                   
  storage         docker twi-aotz--  <3.50g                19.54  34.18                           
  volumes         docker twi-aotz-- 189.21g                76.99  82.87                           

$ sudo xfs_growfs /var/lib/docker-lvm-plugin/mydb_master
meta-data=/dev/mapper/docker-mydb_master isize=512    agcount=16, agsize=2457584 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0 spinodes=0
data     =                       bsize=4096   blocks=39321344, imaxpct=25
         =                       sunit=16     swidth=16 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal               bsize=4096   blocks=19200, version=2
         =                       sectsz=512   sunit=16 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
data blocks changed from 39321344 to 41943040

$ docker volume inspect mydb_master
[
    {
        "CreatedAt": "0001-01-01T00:00:00Z",
        "Driver": "lvm",
        "Labels": {
            "com.docker.stack.namespace": "mydb"
        },
        "Mountpoint": "/var/lib/docker-lvm-plugin/mydb_master",
        "Name": "mydb_master",
        "Options": {
            "size": "150G",
            "thinpool": "volumes"
        },
        "Scope": "local"
    }
]

$ sudo lvs
  LV              VG     Attr       LSize   Pool    Origin Data%  Meta%  Move Log Cpy%Sync Convert
  mydb_master     docker Vwi-aotz-- 160.00g volumes        91.05                                  
  example1        docker Vwi-a-tz-- 100.00m volumes        3.50                                   
  storage         docker twi-aotz--  <3.50g                19.54  34.18                           
  volumes         docker twi-aotz-- 189.21g                77.00  82.88                           

$ sudo df -h /var/lib/docker-lvm-plugin/mydb_master
Filesystem                      Size  Used Avail Use% Mounted on
/dev/mapper/docker-mydb_master  160G  117G   44G  73% /var/lib/docker-lvm-plugin/mydb_master

$ docker info
Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 2
Server Version: 18.03.0-ce
Storage Driver: devicemapper
 Pool Name: docker-storage
 Pool Blocksize: 65.54kB
 Base Device Size: 10.74GB
 Backing Filesystem: xfs
 Udev Sync Supported: true
 Data Space Used: 733.7MB
 Data Space Total: 3.754GB
 Data Space Available: 3.02GB
 Metadata Space Used: 1.434MB
 Metadata Space Total: 4.194MB
 Metadata Space Available: 2.761MB
 Thin Pool Minimum Free Space: 375.4MB
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.140-RHEL7 (2017-05-03)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local lvm
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: ntob6bn2c4jslyphyhmlohfv8
 Is Manager: false
 Node Address: 10.0.3.61
 Manager Addresses:
  10.0.3.213:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfd04396dc68220d1cecbe686a6cc3aa5ce3667c
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-693.21.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.51GiB
Name: xxxx.xx.xx
ID: QJWV:ST3L:MPLI:VPLG:EY5Q:ZCSJ:KPV5:Z57F:QSWP:BVMS:7VG2:377U
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: bridge-nf-call-ip6tables is disabled

[shishir-a412ed]

@tanvir-ahmed-m4

Can you provide me:

1. docker volume create command which you used to create the thin volume ?
2. I created a thin volume thin_vol and when I do docker volume inspect thin_vol I see this:

[root@localhost volume]# docker volume inspect thin_vol 
[
    {
        "Name": "thin_vol",
        "Driver": "lvm",
        "Mountpoint": "/var/lib/docker-lvm-plugin/thin_vol",
        "Labels": {},
        "Scope": "local"
    }
]

I don't see options listed in my output, however I see options {size, thinpool} listed in your output. How did you get docker-lvm-plugin ? Did you build it from master branch from our github repo ? or downloaded the RPM ?

3. Output of cat /etc/os-release

[tanvir-ahmed-m4]

Hi @shishir-a412ed

1. I have created the volume using docker stack deploy -c mydb.yml. So, I don't know the exact command used to create the volume. from mydb.yml:

volumes:
    master:
        driver: lvm
        driver_opts:
            size: "150G"
            thinpool: "volumes"

2. I built it from the master branch from your github repo. I have created a volume for testing the setup. Here are the details:

$ docker volume create --driver lvm --name example1 --opt size=100M --opt thinpool=volumes
$ docker volume inspect example1 
[
    {
        "CreatedAt": "0001-01-01T00:00:00Z",
        "Driver": "lvm",
        "Labels": {},
        "Mountpoint": "/var/lib/docker-lvm-plugin/example1",
        "Name": "example1",
        "Options": {
            "size": "100M",
            "thinpool": "volumes"
        },
        "Scope": "local"
    }
]

3. Contents of /etc/os-release

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

[shishir-a412ed]

@tanvir-ahmed-m4 Can you tell me a little bit more about your setup ?

1. docker stack deploy -c mydb.yml Is this part of a docker swarm cluster or some other docker tooling ?
2. Can you share your mydb.yml file. It would help me to reproduce this problem.

Shishir

[shishir-a412ed]

Closing due to inactivity.

[aston-r]

@shishir-a412ed

I have the same issue.

STR:

• create the volume (docker volume create -d lvm --opt size=5G --name jenkins_home)
• run container (docker run -d -v jenkins_home:/var/jenkins_home -p 8080:8080 -p 50000:50000 jenkins/jenkins:lts)
• sudo lvextend -L+265G /dev/vgsystem/jenkins_home
• sudo xfs_growfs /var/lib/docker-lvm-plugin/jenkins_home

Actual Result:

[root@jenkins ]# lvs | grep jenkins_home
  jenkins_home    vgsystem -wi-ao---- 270,00g    

[root@jenkins ]# docker volume inspect jenkins_home
[
    {
        "CreatedAt": "0001-01-01T00:00:00Z",
        "Driver": "lvm",
        "Labels": {},
        "Mountpoint": "/var/lib/docker-lvm-plugin/jenkins_home",
        "Name": "jenkins_home",
        "Options": {
            "size": "5G"
        },
        "Scope": "local"
    }
]

[angermueller]

I would also like to know if there is a preferred way to resize existing volumes.

PS: Restarting the docker service does not help

[aol-nnov]

any recommendations yet?

[aol-nnov]

https://github.com/moby/moby/pull/30109

We discussed this in the maintainers meeting, and we're ok with changing volume labels, but not comfortable with changing driver options / volume options, as that opens a whole can of worms.

sad to read that... so, what't the proposed method though? I'm stuck now... Just resized my volume and now I'm unable to start my stack due to ERROR: Configuration for volume data specifies "size" driver_opt 50G, but a volume with the same name uses a different "size" driver_opt (5G). If you wish to use the new configuration, please remove the existing volume "cloud_data" first:

But I do not want to remove it! There is useful data on it..

[L4rS6]

Is there still no solution available for this problem?

I also resized my XFS volume manually and now I want to adjust the metadata size of the volume.

[shishir-a412ed]

Is there still no solution available for this problem?

@L4rS6 This is not on my immediate priority list, but I will try to investigate when I get a chance. PR's are welcome 🙂 if you wanna propose a PR with the fix.