Is disk latency really getting injected?

[db-vmw]

Hello Team, I have mounted charybdefs on my ubuntu machine. I have written a small python code to simulate slow write (disk latency). The python code will connect to thrift module as documented in the charybdefs blogpost. I have run the python script with the following parameters

python inject_fault.py -o disklatency -l 2000000 -p 9090

Here, I was expecting 20seconds of delay for the write operation. But, the below dd command has finished in 6.6 seconds. Before injecting the above fault (20 seconds delay) the same dd command had completed with 165MB/s (6.3s). I don't know if the charybdefs is really injecting fault here. Can you please confirm if any thing wrong here?

$ sudo dd if=/dev/zero of=/var/data-dir/file_1gb_after_20s_delay_2 bs=1G count=1
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 6.60714 s, 163 MB/s

Python client code:

import sys

sys.path.append('gen-py')
import errno

from server import server
from server.ttypes import *

from thrift import Thrift
from thrift.transport import TSocket
from thrift.transport import TTransport
from thrift.protocol import TBinaryProtocol
from argparse import ArgumentParser

if __name__ == "__main__":
    parser = ArgumentParser()
    parser.add_argument("-o", "--operation", dest="operation", help="the type of the operation that has to be executed",
                    choices=["disklatency", "ioError", "randomError", "NOSPCError", "QUOTAExceed", "remediate"])
    parser.add_argument("-l", "--latency", dest="latency", help="Disk Latency in milliseconds")
    parser.add_argument("-p", "--port", dest="socketPort", help="socket Port for client connection")
try:
    args = parser.parse_args()
except:
    parser.error("Error parsing arguments..")
    sys.exit()

try:

    transport = TSocket.TSocket('127.0.0.1', int(args.socketPort))
    transport = TTransport.TBufferedTransport(transport)
    protocol = TBinaryProtocol.TBinaryProtocol(transport)
    client = server.Client(protocol)
    transport.open()


    if args.operation == "ioError":
         print('Executing the injection of ioError fault')
         client.set_all_fault(False, errno.EIO, 0, "", False, 0, False)
    if args.operation == "disklatency":
         print('Executing the injection of DiskLatency fault')
         client.set_all_fault(False, 0, 0, "", False, int(args.latency), True)
except Thrift.TException as tx:
    print('%s' % tx.message)

[db-vmw]

@lmr , @bentsi can you please help me here? Note: Including top contributors of this project to comment on my observation.

[bentsi]

@db-vmw what is the command that you used to mount CharybdeFS?

[db-vmw]

Hi @bentsi , I'm using the following command:

./charybdefs /var/data-dir -omodules=subdir,subdir=/var/skpscyla/data-dir -o nonempty -o allow_other

I'm injecting the disk latency (slow disk) using charybdefs on to /var/data-dir

Thanks, Dinesh

[db-vmw]

Hi @bentsi , any comments on the observation?

[bentsi]

Looking in the C++ code again, the latency is implemented as a delay before executing the syscall, you can fix this and fire a PR