yas3fs icon indicating copy to clipboard operation
yas3fs copied to clipboard

Published 20 hours ago • verified publisher icon danilop

NoSuchKey issue stops FS syncing

Open robmoore opened this issue 10 years ago • 2 comments

We are running yas3fs and ran into an instance where we deleted a file on one server and it never is deleted on the other server. Once this occurs, attempts to create new files in either node do not result in syncing. Looking at the logs, it appears that we are seeing a NoSuchKey response in S3 and then some errors about 'could not clear'. The SQS queues have messages that are stuck as well.

2015-04-02 19:22:34,139 ERROR S3ResponseError: 404 Not Found
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>qa-ar/xyz/551d9650e4b0fb56d15de7f9/AAA/xyzConcurrentIn00050_V2.4.0_1.zip</Key><RequestId>AB2EEED64423F131</RequestId><HostId>UotyXmWFM8Y/kdq5ooYeomlDe7xM/IOLBbc9eqzdal/PFs1mOAMtphsjc8SdzpSYfBcrzJ0sTMk=</HostId></Error>
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 2063, in do_cmd_on_s3_now
    key.copy(*args, **kargs)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 500, in copy
    src_version_id=self.version_id)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/bucket.py", line 888, in copy_key
    response.reason, body)
S3ResponseError: S3ResponseError: 404 Not Found
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>qa-ar/xyz/551d9650e4b0fb56d15de7f9/AAA/xyzConcurrentIn00050_V2.4.0_1.zip</Key><RequestId>AB2EEED64423F131</RequestId><HostId>UotyXmWFM8Y/kdq5ooYeomlDe7xM/IOLBbc9eqzdal/PFs1mOAMtphsjc8SdzpSYfBcrzJ0sTMk=</HostId></Error>
2015-04-02 19:22:35,186 ERROR S3ResponseError: 404 Not Found
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>qa-ar/xyz/551d9650e4b0fb56d15de7f9/AAA/xyzConcurrentIn00050_V2.4.0_1.zip</Key><RequestId>B2906E7C00BCC8DB</RequestId><HostId>sIeC03bqJ0SKEgXq8NW4jKUhE+bMYVwkQEabSOrjwW5HkZftFMZqYsEUwJn+ZuuqP0JHQsfPeDc=</HostId></Error>
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 2063, in do_cmd_on_s3_now
    key.copy(*args, **kargs)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 500, in copy
    src_version_id=self.version_id)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/bucket.py", line 888, in copy_key
    response.reason, body)
S3ResponseError: S3ResponseError: 404 Not Found
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>qa-ar/xyz/551d9650e4b0fb56d15de7f9/AAA/xyzConcurrentIn00050_V2.4.0_1.zip</Key><RequestId>B2906E7C00BCC8DB</RequestId><HostId>sIeC03bqJ0SKEgXq8NW4jKUhE+bMYVwkQEabSOrjwW5HkZftFMZqYsEUwJn+ZuuqP0JHQsfPeDc=</HostId></Error>
2015-04-02 19:22:36,228 ERROR S3ResponseError: 404 Not Found
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>qa-ar/xyz/551d9650e4b0fb56d15de7f9/AAA/xyzConcurrentIn00050_V2.4.0_1.zip</Key><RequestId>F4AE30E29459FD7A</RequestId><HostId>KQZ1H+OgqOGH2sF/yOpvn5wJszZr9vDH5O5JrxGyea6KH8dnqpLPjhMShNFWE3+dTXLR/3dbgpc=</HostId></Error>
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 2063, in do_cmd_on_s3_now
    key.copy(*args, **kargs)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 500, in copy
    src_version_id=self.version_id)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/bucket.py", line 888, in copy_key
    response.reason, body)
S3ResponseError: S3ResponseError: 404 Not Found
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>qa-ar/xyz/551d9650e4b0fb56d15de7f9/AAA/xyzConcurrentIn00050_V2.4.0_1.zip</Key><RequestId>F4AE30E29459FD7A</RequestId><HostId>KQZ1H+OgqOGH2sF/yOpvn5wJszZr9vDH5O5JrxGyea6KH8dnqpLPjhMShNFWE3+dTXLR/3dbgpc=</HostId></Error>
2015-04-02 19:22:36,229 ERROR do_cmd_on_s3_now_w_retries FAILED 'copy' key 'qa-ar/xyz/551d9650e4b0fb56d15de7f9/AAA/xyzConcurrentIn00050_V2.4.0_1.zip' args '['xyz-sftpserver', u'qa-ar/xyz/551d9650e4b0fb56d15de7f9/AAA/xyzConcurrentIn00050_V2.4.0_1.zip.reading', {'Content-Type': 'binary/octet-stream', 'attr': u'{"st_ctime": 1428002389.0, "st_mtime": 1428002389.382, "st_nlink": 1, "st_mode": 33188, "st_size": 646730, "st_gid": 1003, "st_uid": 1002, "st_atime": 1428002389.0}'}]' kargs '{'encrypt_key': True, 'preserve_acl': False}'
2015-04-02 19:22:36,229 ERROR Uncaught Exception in Thread
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 2894, in run
    super(TracebackLoggingThread, self).run()
  File "/usr/lib/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 2033, in get_to_do_on_s3
    self.do_on_s3_now(key, pub, cmds)
  File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 958, in fn_wrapper
    return fn(*arg, **karg)
  File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 2165, in do_on_s3_now
    pub = self.do_cmd_on_s3_now_w_retries(key, pub, action, args, kargs, self.s3_retries)
  File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 958, in fn_wrapper
    return fn(*arg, **karg)
  File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 2151, in do_cmd_on_s3_now_w_retries
    raise last_exception
S3ResponseError: S3ResponseError: 404 Not Found
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>qa-ar/xyz/551d9650e4b0fb56d15de7f9/AAA/xyzConcurrentIn00050_V2.4.0_1.zip</Key><RequestId>F4AE30E29459FD7A</RequestId><HostId>KQZ1H+OgqOGH2sF/yOpvn5wJszZr9vDH5O5JrxGyea6KH8dnqpLPjhMShNFWE3+dTXLR/3dbgpc=</HostId></Error>
2015-04-02 19:22:44,073 ERROR wait_until_cleared s3_busy could not clear '/xyz/551d9650e4b0fb56d15de7f9/AAA/xyzConcurrentIn00050_V2.4.0_1.zip.reading'
2015-04-02 19:22:44,256 ERROR wait_until_cleared deleting could not clear '/xyz/551d9650e4b0fb56d15de7f9/AAA/xyzConcurrentIn00050_V2.4.0_1.zip'
2015-04-02 19:22:44,256 ERROR Uncaught Exception in Thread
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 2894, in run
    super(TracebackLoggingThread, self).run()
  File "/usr/lib/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 1140, in listen_for_messages_over_sqs
    self.process_message(message)
  File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 1182, in process_message
    self.delete_cache(c[2])
  File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 1165, in delete_cache
    with self.cache.get_lock(path):
  File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 443, in get_lock
    self.is_ready(path, proplist = wait_until_cleared_proplist)
  File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 406, in is_ready
    return self.wait_until_cleared(path, proplist = proplist)
  File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 438, in wait_until_cleared
    raise Exception("Path has not yet been cleared but operation wants to happen on it '%s' '%s'"%(prop, path))
Exception: Path has not yet been cleared but operation wants to happen on it 'deleting' '/xyz/551d9650e4b0fb56d15de7f9/AAA/xyzConcurrentIn00050_V2.4.0_1.zip'
2015-04-02 19:28:22,863 ERROR S3ResponseError: 404 Not Found
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>qa-ar/xyz/5518f7c6e4b0cc1d691dfc00/AAA/PerfTestAPI_v1.5_Adapter00036_V1.6.0_1.zip.reading</Key><RequestId>3B3DF49B21802F59</RequestId><HostId>0eaoHWbZelu1abtpK9GaLgTa2DsOheX8eGKZENNRIR8Wfq34/cg7vUxsDhWP/BoA</HostId></Error>
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 2063, in do_cmd_on_s3_now
    key.copy(*args, **kargs)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 500, in copy
    src_version_id=self.version_id)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/bucket.py", line 888, in copy_key
    response.reason, body)
S3ResponseError: S3ResponseError: 404 Not Found
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>qa-ar/xyz/5518f7c6e4b0cc1d691dfc00/AAA/PerfTestAPI_v1.5_Adapter00036_V1.6.0_1.zip.reading</Key><RequestId>3B3DF49B21802F59</RequestId><HostId>0eaoHWbZelu1abtpK9GaLgTa2DsOheX8eGKZENNRIR8Wfq34/cg7vUxsDhWP/BoA</HostId></Error>
2015-04-02 19:28:23,920 ERROR S3ResponseError: 404 Not Found
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>qa-ar/xyz/5518f7c6e4b0cc1d691dfc00/AAA/PerfTestAPI_v1.5_Adapter00036_V1.6.0_1.zip.reading</Key><RequestId>691C8F2DF998B9E3</RequestId><HostId>7Pjr5tCCopqTHCYXJrK+c07aBM9CQx+hyr2x3x7s80ChmoSoa2XrtQ63cYjn0CRL</HostId></Error>
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 2063, in do_cmd_on_s3_now
    key.copy(*args, **kargs)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 500, in copy
    src_version_id=self.version_id)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/bucket.py", line 888, in copy_key
    response.reason, body)
S3ResponseError: S3ResponseError: 404 Not Found
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>qa-ar/xyz/5518f7c6e4b0cc1d691dfc00/AAA/PerfTestAPI_v1.5_Adapter00036_V1.6.0_1.zip.reading</Key><RequestId>691C8F2DF998B9E3</RequestId><HostId>7Pjr5tCCopqTHCYXJrK+c07aBM9CQx+hyr2x3x7s80ChmoSoa2XrtQ63cYjn0CRL</HostId></Error>
2015-04-02 19:28:24,980 ERROR S3ResponseError: 404 Not Found
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>qa-ar/xyz/5518f7c6e4b0cc1d691dfc00/AAA/PerfTestAPI_v1.5_Adapter00036_V1.6.0_1.zip.reading</Key><RequestId>9B8B08DE4C40E63D</RequestId><HostId>hSf7TdrwCNgKlZu7Ud1m4zmYw4td02FUbfMbwRNDn7jOTltfgQmcKls1mn/5ftZe</HostId></Error>
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 2063, in do_cmd_on_s3_now
    key.copy(*args, **kargs)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 500, in copy
    src_version_id=self.version_id)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/bucket.py", line 888, in copy_key
    response.reason, body)
S3ResponseError: S3ResponseError: 404 Not Found
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>qa-ar/xyz/5518f7c6e4b0cc1d691dfc00/AAA/PerfTestAPI_v1.5_Adapter00036_V1.6.0_1.zip.reading</Key><RequestId>9B8B08DE4C40E63D</RequestId><HostId>hSf7TdrwCNgKlZu7Ud1m4zmYw4td02FUbfMbwRNDn7jOTltfgQmcKls1mn/5ftZe</HostId></Error>
2015-04-02 19:28:24,980 ERROR do_cmd_on_s3_now_w_retries FAILED 'copy' key 'qa-ar/xyz/5518f7c6e4b0cc1d691dfc00/AAA/PerfTestAPI_v1.5_Adapter00036_V1.6.0_1.zip.reading' args '['xyz-sftpserver', u'qa-ar/xyz/5518f7c6e4b0cc1d691dfc00/AAA/PerfTestAPI_v1.5_Adapter00036_V1.6.0_1.zip.reading', {'attr': '{"st_ctime": 1427704430.0, "st_mtime": 1428002902.0, "st_nlink": 1, "st_gid": 1003, "st_size": 1223217, "st_mode": 33188, "st_uid": 1002, "st_atime": 1428002902.0}'}]' kargs '{'encrypt_key': True, 'preserve_acl': False}'
2015-04-02 19:28:24,980 ERROR Uncaught Exception in Thread
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 2894, in run
    super(TracebackLoggingThread, self).run()
  File "/usr/lib/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 2033, in get_to_do_on_s3
    self.do_on_s3_now(key, pub, cmds)
  File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 958, in fn_wrapper
    return fn(*arg, **karg)
  File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 2165, in do_on_s3_now
    pub = self.do_cmd_on_s3_now_w_retries(key, pub, action, args, kargs, self.s3_retries)
  File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 958, in fn_wrapper
    return fn(*arg, **karg)
  File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 2151, in do_cmd_on_s3_now_w_retries
    raise last_exception
S3ResponseError: S3ResponseError: 404 Not Found
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>qa-ar/xyz/5518f7c6e4b0cc1d691dfc00/AAA/PerfTestAPI_v1.5_Adapter00036_V1.6.0_1.zip.reading</Key><RequestId>9B8B08DE4C40E63D</RequestId><HostId>hSf7TdrwCNgKlZu7Ud1m4zmYw4td02FUbfMbwRNDn7jOTltfgQmcKls1mn/5ftZe</HostId></Error>
2015-04-02 19:29:28,455 ERROR wait_until_cleared s3_busy could not clear '/xyz/551d9650e4b0fb56d15de7f9/AAA/xyzConcurrentIn00050_V2.4.0_1.zip.reading'
2015-04-02 19:29:38,466 ERROR wait_until_cleared s3_busy could not clear '/xyz/551d9650e4b0fb56d15de7f9/AAA/xyzConcurrentIn00050_V2.4.0_1.zip.reading'
2015-04-02 19:36:39,388 ERROR wait_until_cleared s3_busy could not clear '/xyz/551d9650e4b0fb56d15de7f9/AAA/xyzConcurrentIn00050_V2.4.0_1.zip.reading'
2015-04-02 19:43:33,379 ERROR wait_until_cleared s3_busy could not clear '/xyz/551d9650e4b0fb56d15de7f9/AAA/xyzConcurrentIn00050_V2.4.0_1.zip.reading'
2015-04-02 19:50:49,305 ERROR wait_until_cleared s3_busy could not clear '/xyz/551d9650e4b0fb56d15de7f9/AAA/xyzConcurrentIn00050_V2.4.0_1.zip.reading'
2015-04-02 19:57:46,975 ERROR wait_until_cleared s3_busy could not clear '/xyz/551d9650e4b0fb56d15de7f9/AAA/xyzConcurrentIn00050_V2.4.0_1.zip.reading'
2015-04-02 19:58:02,453 ERROR wait_until_cleared s3_busy could not clear '/xyz/551d9650e4b0fb56d15de7f9/AAA/xyzConcurrentIn00050_V2.4.0_1.zip.reading'

robmoore avatar Apr 02 '15 20:04 robmoore

Were you ever able to figure it out? We are experiencing a similar issue.

#107

Jaykah avatar Sep 25 '15 21:09 Jaykah

Hello,

We are facing the same issue. Could someone provide a solution/workaround for this? Thanks.

bjoerndalen avatar Mar 30 '17 09:03 bjoerndalen