yas3fs icon indicating copy to clipboard operation
yas3fs copied to clipboard
verified publisher icon danilop

Use the New Event Notifications for Amazon S3 to sync the local cache

Open danilop opened this issue 11 years ago • 10 comments

Any volunteer? :)

It think that it'll be awesome to use notifications coming straight from S3:

http://aws.amazon.com/blogs/aws/s3-event-notification/

Best, dp

danilop avatar Nov 13 '14 19:11 danilop

How do you suppose you will achieve this? I've just performed a basic test (setting an extended attribute) using an email subscription. The only interesting items within the message seem to be the eventName (ObjectCreated:Copy), the s3 object key, and the x-amz-request-id in the response elements. Note that S3 event notifications don't yet support object removed/deleted.

AFAIK the request ID is for Amazon debugging assistance. There isn't anything in there that seems to suggest exactly what was updated, only that there was an update. That may be sufficient for you if you go and query S3 for the keys that have changed.

mcassaniti avatar Feb 20 '15 03:02 mcassaniti

You are right, without object delete notification it is not possible to use notifications. But notifications include the bucket and the key of the modified object (e.g. see http://docs.aws.amazon.com/AmazonS3/latest/dev/notification-content-structure.html ), so if / when that feature is available this issue could be checked again.

danilop avatar Feb 23 '15 18:02 danilop

I naively thought we were supposed to use the s3 notifications. I was getting an error and realized why. I am posting this error for reference in case someone else does the same.

2015-07-07 14:42:00,289 ERROR Uncaught Exception in Thread
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/yas3fs/__init__.py", line 2901, in run
    super(TracebackLoggingThread, self).run()
  File "/usr/lib64/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.6/site-packages/yas3fs/__init__.py", line 1144, in listen_for_messages_over_sqs
    self.process_message(message)
  File "/usr/lib/python2.6/site-packages/yas3fs/__init__.py", line 1176, in process_message
    if c[0] == self.unique_id:
KeyError: 0

pearcec avatar Jul 07 '15 19:07 pearcec

Now delete notifications from S3 are available, what should we do? Replace the previous "node-generated" notification system with the native S3 one, or keep both, letting the user decide which to use with different startup parameters?

danilop avatar Aug 02 '15 15:08 danilop

At a minimum we should build the code, test how it performs compared to the current process and go from there. Since changing the way ya3fs get notified about s3 object changes is essentially an API change, making s3 direct notifications the only option would require a major version change. If you make it the default alongside the SNS messages, then that isn't quite so bad.

AFAIK you need a delete notification, and a create/update notification. That is now covered by the S3 event notificaitons, so from that end you've got all you need right?

mcassaniti avatar Aug 03 '15 01:08 mcassaniti

not quite yet... it doesn't metadata change notification.

ewah avatar May 12 '16 23:05 ewah

What is the status of S3 Event Notification in yas3fs? I have configured a bucket to send notifications to a SNS topic but it seems that event messages are not properly processed by yas3fs. Here's the log of what happens:

ectory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
  'Supervisord is running as root and it is searching '
2016-10-11 07:23:27,728 CRIT Supervisor running as root (no user in config file)
2016-10-11 07:23:27,735 INFO RPC interface 'supervisor' initialized
2016-10-11 07:23:27,735 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2016-10-11 07:23:27,735 INFO supervisord started with pid 1
2016-10-11 07:23:28,738 INFO spawned: 'stdout' with pid 8
2016-10-11 07:23:28,742 INFO spawned: 'yas3fs' with pid 9
2016-10-11 07:23:28,746 INFO spawned: 'gunicorn' with pid 10
[2016-10-11 07:23:28 +0000] [10] [INFO] Starting gunicorn 19.6.0
[2016-10-11 07:23:28 +0000] [10] [INFO] Listening at: http://0.0.0.0:80 (10)
[2016-10-11 07:23:28 +0000] [10] [INFO] Using worker: eventlet
[2016-10-11 07:23:28 +0000] [21] [INFO] Booting worker with pid: 21
2016-10-11 07:23:28,933 DEBUG options = Namespace(aws_managed_encryption=False, buffer_prefetch=0, buffer_size=10240, cache_check=5, cache_disk_size=1024, cache_entries=100000, cache_mem_size=128, cache_on_disk=0, cache_path='', debug=True, download_num=4, download_retries_num=60, download_retries_sleep=1, expiration=2592000, foreground=True, gid=None, hostname=None, id=None, log=None, log_backup_count=10, log_backup_gzip=False, log_mb_size=100, mkdir=False, mountpoint='/mnt/data', mp_num=4, mp_retries=3, mp_size=100, new_queue=True, new_queue_with_hostname=False, no_allow_other=False, no_metadata=True, nonempty=False, port=None, prefetch=False, prefetch_num=2, queue='hello-world', queue_polling=0, queue_wait=20, read_only=True, read_retries_num=10, read_retries_sleep=1, recheck_s3=False, region='eu-central-1', requester_pays=False, s3_endpoint='s3-eu-central-1.amazonaws.com', s3_num=32, s3_retries=3, s3_retries_sleep=1, s3_use_sigv4=True, s3path='s3://hello-world/instances', st_blksize=None, topic='arn:aws:sns:eu-central-1:102573368484:hello-world', uid=None, umask=None, use_ec2_hostname=False, with_plugin_class=None, with_plugin_file=None)
2016-10-11 07:23:28,933 INFO Version: 2.3.2
2016-10-11 07:23:28,933 INFO s3-retries: '3'
2016-10-11 07:23:28,933 INFO s3-retries-sleep: '1' seconds
2016-10-11 07:23:28,933 INFO S3 bucket: 'hello-world'
2016-10-11 07:23:28,933 INFO S3 prefix (can be empty): 'instances'
2016-10-11 07:23:28,933 INFO AWS region for SNS and SQS: 'eu-central-1'
2016-10-11 07:23:28,933 INFO SNS topic ARN: 'arn:aws:sns:eu-central-1:102573368484:hello-world'
2016-10-11 07:23:28,933 INFO SQS queue name: 'hello-world'
2016-10-11 07:23:28,933 INFO SQS queue wait time (in seconds): '20'
2016-10-11 07:23:28,933 INFO SQS queue polling interval (in seconds): '0'
2016-10-11 07:23:28,933 INFO Cache entries: '100000'
2016-10-11 07:23:28,933 INFO Cache memory size (in bytes): '134217728'
2016-10-11 07:23:28,934 INFO Cache disk size (in bytes): '1073741824'
2016-10-11 07:23:28,934 INFO Cache on disk if file size greater than (in bytes): '0'
2016-10-11 07:23:28,934 INFO Cache check interval (in seconds): '5'
2016-10-11 07:23:28,934 INFO Cache ENOENT rechecks S3: False
2016-10-11 07:23:28,934 INFO AWS Managed Encryption enabled: False
2016-10-11 07:23:28,934 INFO AWS Managed Encryption enabled: False
2016-10-11 07:23:28,934 INFO Number of parallel S3 threads (0 to disable writeback): '32'
2016-10-11 07:23:28,934 INFO Number of parallel downloading threads: '4'
2016-10-11 07:23:28,934 INFO Number download retry attempts: '60'
2016-10-11 07:23:28,934 INFO Download retry sleep time seconds: '1'
2016-10-11 07:23:28,934 INFO Number read retry attempts: '10'
2016-10-11 07:23:28,934 INFO Read retry sleep time seconds: '1'
2016-10-11 07:23:28,934 INFO Number of parallel prefetching threads: '2'
2016-10-11 07:23:28,934 INFO Download buffer size (in KB, 0 to disable buffering): '10485760'
2016-10-11 07:23:28,934 INFO Number of buffers to prefetch: '0'
2016-10-11 07:23:28,934 INFO Write metadata (file system attr/xattr) on S3: 'False'
2016-10-11 07:23:28,935 INFO Download prefetch: 'False'
2016-10-11 07:23:28,935 INFO Multipart size: '104857600'
2016-10-11 07:23:28,935 INFO Multipart maximum number of parallel threads: '4'
2016-10-11 07:23:28,935 INFO Multipart maximum number of retries per part: '3'
2016-10-11 07:23:28,935 INFO Default expiration for signed URLs via xattrs: '2592000'
2016-10-11 07:23:28,935 INFO S3 Request Payer: 'False'
2016-10-11 07:23:28,935 INFO Cache path (on disk): '/tmp/yas3fs/hello-world/instances'
2016-10-11 07:23:28,939 INFO Unique node ID: '3e740572-4b24-4cd5-8530-c370b4177a08'
[2016-10-11 07:23:28 +0000] [22] [INFO] Booting worker with pid: 22
[2016-10-11 07:23:29 +0000] [23] [INFO] Booting worker with pid: 23
[2016-10-11 07:23:29 +0000] [24] [INFO] Booting worker with pid: 24
2016-10-11 07:23:29,060 INFO Attempting to create SQS queue: yas3fs-helloworld-instances-3e740572-4b24-4cd5-8530-c370b4177a08
[2016-10-11 07:23:29 +0000] [25] [INFO] Booting worker with pid: 25
[2016-10-11 07:23:29 +0000] [26] [INFO] Booting worker with pid: 26
[2016-10-11 07:23:29 +0000] [27] [INFO] Booting worker with pid: 27
[2016-10-11 07:23:29 +0000] [28] [INFO] Booting worker with pid: 28
2016-10-11 07:23:29,302 INFO SQS queue name (new): 'yas3fs-helloworld-instances-3e740572-4b24-4cd5-8530-c370b4177a08'
2016-10-11 07:23:29,303 DEBUG init '/'
2016-10-11 07:23:29,303 DEBUG check_threads 'True'
2016-10-11 07:23:29,303 DEBUG Starting S3 thread #0
2016-10-11 07:23:29,303 DEBUG Starting S3 thread #1
2016-10-11 07:23:29,304 DEBUG Starting S3 thread #2
2016-10-11 07:23:29,304 DEBUG Starting S3 thread #3
2016-10-11 07:23:29,305 DEBUG Starting S3 thread #4
2016-10-11 07:23:29,306 DEBUG Starting S3 thread #5
2016-10-11 07:23:29,307 DEBUG Starting S3 thread #6
2016-10-11 07:23:29,307 DEBUG Starting S3 thread #7
2016-10-11 07:23:29,308 DEBUG Starting S3 thread #8
2016-10-11 07:23:29,308 DEBUG Starting S3 thread #9
2016-10-11 07:23:29,308 DEBUG Starting S3 thread #10
2016-10-11 07:23:29,309 DEBUG Starting S3 thread #11
2016-10-11 07:23:29,309 DEBUG Starting S3 thread #12
2016-10-11 07:23:29,309 DEBUG Starting S3 thread #13
2016-10-11 07:23:29,310 DEBUG Starting S3 thread #14
2016-10-11 07:23:29,310 DEBUG Starting S3 thread #15
2016-10-11 07:23:29,312 DEBUG Starting S3 thread #16
2016-10-11 07:23:29,313 DEBUG Starting S3 thread #17
2016-10-11 07:23:29,314 DEBUG Starting S3 thread #18
2016-10-11 07:23:29,314 DEBUG Starting S3 thread #19
2016-10-11 07:23:29,314 DEBUG Starting S3 thread #20
2016-10-11 07:23:29,315 DEBUG Starting S3 thread #21
2016-10-11 07:23:29,318 DEBUG Starting S3 thread #22
2016-10-11 07:23:29,319 DEBUG Starting S3 thread #23
2016-10-11 07:23:29,319 DEBUG Starting S3 thread #24
2016-10-11 07:23:29,320 DEBUG Starting S3 thread #25
2016-10-11 07:23:29,320 DEBUG Starting S3 thread #26
2016-10-11 07:23:29,320 DEBUG Starting S3 thread #27
2016-10-11 07:23:29,321 DEBUG Starting S3 thread #28
2016-10-11 07:23:29,321 DEBUG Starting S3 thread #29
2016-10-11 07:23:29,322 DEBUG Starting S3 thread #30
2016-10-11 07:23:29,322 DEBUG Starting S3 thread #31
2016-10-11 07:23:29,323 DEBUG Starting download thread #0
2016-10-11 07:23:29,323 DEBUG Starting download thread #1
2016-10-11 07:23:29,323 DEBUG Starting download thread #2
2016-10-11 07:23:29,326 DEBUG Starting download thread #3
2016-10-11 07:23:29,326 DEBUG Starting prefetch thread #0
2016-10-11 07:23:29,327 DEBUG Starting prefetch thread #1
2016-10-11 07:23:29,328 DEBUG Starting publish thread
2016-10-11 07:23:29,328 DEBUG Starting queue listen thread
2016-10-11 07:23:29,328 DEBUG Starting check cache thread
2016-10-11 07:23:29,328 INFO Listening on queue: 'yas3fs-helloworld-instances-3e740572-4b24-4cd5-8530-c370b4177a08'
2016-10-11 07:23:29,329 DEBUG check_cache_size
2016-10-11 07:23:29,330 DEBUG check_cache_size get_memory_usage
2016-10-11 07:23:29,329 DEBUG Subscribing 'yas3fs-helloworld-instances-3e740572-4b24-4cd5-8530-c370b4177a08' to 'arn:aws:sns:eu-central-1:102573368484:hello-world'
2016-10-11 07:23:29,331 DEBUG check_status
2016-10-11 07:23:29,331 INFO entries, mem_size, disk_size, download_queue, prefetch_queue, s3_queue: 0, 0, 0, 0, 0, 0
2016-10-11 07:23:29,331 INFO multipart_uploads_in_progress = 0
2016-10-11 07:23:29,331 DEBUG new_locks, unused_locks: 0, 0
2016-10-11 07:23:29,332 DEBUG gc count0/threshold0, count1/threshold1, count2/threshold2: 400/700, 8/10, 4/10
2016-10-11 07:23:29,332 DEBUG check_threads 'False'
2016-10-11 07:23:29,720 DEBUG SNS SQS subscription = arn:aws:sns:eu-central-1:102573368484:hello-world:2371faa3-5ea9-4e2b-895e-7a2bb0419851
2016-10-11 07:23:30,721 INFO success: stdout entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-10-11 07:23:30,721 INFO success: yas3fs entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-10-11 07:23:30,721 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
gunicorn stdout | [2016-10-11 07:23:28 +0000] [10] [INFO] Starting gunicorn 19.6.0
gunicorn stdout | [2016-10-11 07:23:28 +0000] [10] [INFO] Listening at: http://0.0.0.0:80 (10)
gunicorn stdout | [2016-10-11 07:23:28 +0000] [10] [INFO] Using worker: eventlet
gunicorn stdout | [2016-10-11 07:23:28 +0000] [21] [INFO] Booting worker with pid: 21
yas3fs stdout | 2016-10-11 07:23:28,933 DEBUG options = Namespace(aws_managed_encryption=False, buffer_prefetch=0, buffer_size=10240, cache_check=5, cache_disk_size=1024, cache_entries=100000, cache_mem_size=128, cache_on_disk=0, cache_path='', debug=True, download_num=4, download_retries_num=60, download_retries_sleep=1, expiration=2592000, foreground=True, gid=None, hostname=None, id=None, log=None, log_backup_count=10, log_backup_gzip=False, log_mb_size=100, mkdir=False, mountpoint='/mnt/data', mp_num=4, mp_retries=3, mp_size=100, new_queue=True, new_queue_with_hostname=False, no_allow_other=False, no_metadata=True, nonempty=False, port=None, prefetch=False, prefetch_num=2, queue='hello-world', queue_polling=0, queue_wait=20, read_only=True, read_retries_num=10, read_retries_sleep=1, recheck_s3=False, region='eu-central-1', requester_pays=False, s3_endpoint='s3-eu-central-1.amazonaws.com', s3_num=32, s3_retries=3, s3_retries_sleep=1, s3_use_sigv4=True, s3path='s3://hello-world/instances', st_blksize=None, topic='arn:aws:sns:eu-central-1:102573368484:hello-world', uid=None, umask=None, use_ec2_hostname=False, with_plugin_class=None, with_plugin_file=None)
yas3fs stdout | 2016-10-11 07:23:28,933 INFO Version: 2.3.2
2016-10-11 07:23:28,933 INFO s3-retries: '3'
2016-10-11 07:23:28,933 INFO s3-retries-sleep: '1' seconds
2016-10-11 07:23:28,933 INFO S3 bucket: 'hello-world'
2016-10-11 07:23:28,933 INFO S3 prefix (can be empty): 'instances'
2016-10-11 07:23:28,933 INFO AWS region for SNS and SQS: 'eu-central-1'
yas3fs stdout | 2016-10-11 07:23:28,933 INFO SNS topic ARN: 'arn:aws:sns:eu-central-1:102573368484:hello-world'
2016-10-11 07:23:28,933 INFO SQS queue name: 'hello-world'
2016-10-11 07:23:28,933 INFO SQS queue wait time (in seconds): '20'
2016-10-11 07:23:28,933 INFO SQS queue polling interval (in seconds): '0'
2016-10-11 07:23:28,933 INFO Cache entries: '100000'
2016-10-11 07:23:28,933 INFO Cache memory size (in bytes): '134217728'
2016-10-11 07:23:28,934 INFO Cache disk size (in bytes): '1073741824'
2016-10-11 07:23:28,934 INFO Cache on disk if file size greater than (in bytes): '0'
2016-10-11 07:23:28,934 INFO Cache check interval (in seconds): '5'
yas3fs stdout | 2016-10-11 07:23:28,934 INFO Cache ENOENT rechecks S3: False
2016-10-11 07:23:28,934 INFO AWS Managed Encryption enabled: False
2016-10-11 07:23:28,934 INFO AWS Managed Encryption enabled: False
2016-10-11 07:23:28,934 INFO Number of parallel S3 threads (0 to disable writeback): '32'
yas3fs stdout | 2016-10-11 07:23:28,934 INFO Number of parallel downloading threads: '4'
2016-10-11 07:23:28,934 INFO Number download retry attempts: '60'
yas3fs stdout | 2016-10-11 07:23:28,934 INFO Download retry sleep time seconds: '1'
2016-10-11 07:23:28,934 INFO Number read retry attempts: '10'
2016-10-11 07:23:28,934 INFO Read retry sleep time seconds: '1'
yas3fs stdout | 2016-10-11 07:23:28,934 INFO Number of parallel prefetching threads: '2'
2016-10-11 07:23:28,934 INFO Download buffer size (in KB, 0 to disable buffering): '10485760'
yas3fs stdout | 2016-10-11 07:23:28,934 INFO Number of buffers to prefetch: '0'
yas3fs stdout | 2016-10-11 07:23:28,934 INFO Write metadata (file system attr/xattr) on S3: 'False'
yas3fs stdout | 2016-10-11 07:23:28,935 INFO Download prefetch: 'False'
2016-10-11 07:23:28,935 INFO Multipart size: '104857600'
yas3fs stdout | 2016-10-11 07:23:28,935 INFO Multipart maximum number of parallel threads: '4'
2016-10-11 07:23:28,935 INFO Multipart maximum number of retries per part: '3'
yas3fs stdout | 2016-10-11 07:23:28,935 INFO Default expiration for signed URLs via xattrs: '2592000'
2016-10-11 07:23:28,935 INFO S3 Request Payer: 'False'
yas3fs stdout | 2016-10-11 07:23:28,935 INFO Cache path (on disk): '/tmp/yas3fs/hello-world/instances'
yas3fs stdout | 2016-10-11 07:23:28,939 INFO Unique node ID: '3e740572-4b24-4cd5-8530-c370b4177a08'
gunicorn stdout | [2016-10-11 07:23:28 +0000] [22] [INFO] Booting worker with pid: 22
gunicorn stdout | [2016-10-11 07:23:29 +0000] [23] [INFO] Booting worker with pid: 23
gunicorn stdout | [2016-10-11 07:23:29 +0000] [24] [INFO] Booting worker with pid: 24
yas3fs stdout | 2016-10-11 07:23:29,060 INFO Attempting to create SQS queue: yas3fs-helloworld-instances-3e740572-4b24-4cd5-8530-c370b4177a08
gunicorn stdout | [2016-10-11 07:23:29 +0000] [25] [INFO] Booting worker with pid: 25
gunicorn stdout | [2016-10-11 07:23:29 +0000] [26] [INFO] Booting worker with pid: 26
gunicorn stdout | [2016-10-11 07:23:29 +0000] [27] [INFO] Booting worker with pid: 27
gunicorn stdout | [2016-10-11 07:23:29 +0000] [28] [INFO] Booting worker with pid: 28
yas3fs stdout | 2016-10-11 07:23:29,302 INFO SQS queue name (new): 'yas3fs-helloworld-instances-3e740572-4b24-4cd5-8530-c370b4177a08'
yas3fs stdout | 2016-10-11 07:23:29,303 DEBUG init '/'
yas3fs stdout | 2016-10-11 07:23:29,303 DEBUG check_threads 'True'
2016-10-11 07:23:29,303 DEBUG Starting S3 thread #0
2016-10-11 07:23:29,303 DEBUG Starting S3 thread #1
2016-10-11 07:23:29,304 DEBUG Starting S3 thread #2
yas3fs stdout | 2016-10-11 07:23:29,304 DEBUG Starting S3 thread #3
2016-10-11 07:23:29,305 DEBUG Starting S3 thread #4
yas3fs stdout | 2016-10-11 07:23:29,306 DEBUG Starting S3 thread #5
yas3fs stdout | 2016-10-11 07:23:29,307 DEBUG Starting S3 thread #6
yas3fs stdout | 2016-10-11 07:23:29,307 DEBUG Starting S3 thread #7
yas3fs stdout | 2016-10-11 07:23:29,308 DEBUG Starting S3 thread #8
yas3fs stdout | 2016-10-11 07:23:29,308 DEBUG Starting S3 thread #9
yas3fs stdout | 2016-10-11 07:23:29,308 DEBUG Starting S3 thread #10
yas3fs stdout | 2016-10-11 07:23:29,309 DEBUG Starting S3 thread #11
yas3fs stdout | 2016-10-11 07:23:29,309 DEBUG Starting S3 thread #12
yas3fs stdout | 2016-10-11 07:23:29,309 DEBUG Starting S3 thread #13
yas3fs stdout | 2016-10-11 07:23:29,310 DEBUG Starting S3 thread #14
yas3fs stdout | 2016-10-11 07:23:29,310 DEBUG Starting S3 thread #15
yas3fs stdout | 2016-10-11 07:23:29,312 DEBUG Starting S3 thread #16
yas3fs stdout | 2016-10-11 07:23:29,313 DEBUG Starting S3 thread #17
yas3fs stdout | 2016-10-11 07:23:29,314 DEBUG Starting S3 thread #18
yas3fs stdout | 2016-10-11 07:23:29,314 DEBUG Starting S3 thread #19
yas3fs stdout | 2016-10-11 07:23:29,314 DEBUG Starting S3 thread #20
yas3fs stdout | 2016-10-11 07:23:29,315 DEBUG Starting S3 thread #21
yas3fs stdout | 2016-10-11 07:23:29,318 DEBUG Starting S3 thread #22
yas3fs stdout | 2016-10-11 07:23:29,319 DEBUG Starting S3 thread #23
yas3fs stdout | 2016-10-11 07:23:29,319 DEBUG Starting S3 thread #24
yas3fs stdout | 2016-10-11 07:23:29,320 DEBUG Starting S3 thread #25
yas3fs stdout | 2016-10-11 07:23:29,320 DEBUG Starting S3 thread #26
yas3fs stdout | 2016-10-11 07:23:29,320 DEBUG Starting S3 thread #27
yas3fs stdout | 2016-10-11 07:23:29,321 DEBUG Starting S3 thread #28
yas3fs stdout | 2016-10-11 07:23:29,321 DEBUG Starting S3 thread #29
yas3fs stdout | 2016-10-11 07:23:29,322 DEBUG Starting S3 thread #30
yas3fs stdout | 2016-10-11 07:23:29,322 DEBUG Starting S3 thread #31
yas3fs stdout | 2016-10-11 07:23:29,323 DEBUG Starting download thread #0
yas3fs stdout | 2016-10-11 07:23:29,323 DEBUG Starting download thread #1
yas3fs stdout | 2016-10-11 07:23:29,323 DEBUG Starting download thread #2
yas3fs stdout | 2016-10-11 07:23:29,326 DEBUG Starting download thread #3
yas3fs stdout | 2016-10-11 07:23:29,326 DEBUG Starting prefetch thread #0
yas3fs stdout | 2016-10-11 07:23:29,327 DEBUG Starting prefetch thread #1
yas3fs stdout | 2016-10-11 07:23:29,328 DEBUG Starting publish thread
yas3fs stdout | 2016-10-11 07:23:29,328 DEBUG Starting queue listen thread
yas3fs stdout | 2016-10-11 07:23:29,328 DEBUG Starting check cache thread
yas3fs stdout | 2016-10-11 07:23:29,328 INFO Listening on queue: 'yas3fs-helloworld-instances-3e740572-4b24-4cd5-8530-c370b4177a08'
yas3fs stdout | 2016-10-11 07:23:29,329 DEBUG check_cache_size
yas3fs stdout | 2016-10-11 07:23:29,330 DEBUG check_cache_size get_memory_usage
2016-10-11 07:23:29,329 DEBUG Subscribing 'yas3fs-helloworld-instances-3e740572-4b24-4cd5-8530-c370b4177a08' to 'arn:aws:sns:eu-central-1:102573368484:hello-world'
yas3fs stdout | 2016-10-11 07:23:29,331 DEBUG check_status
yas3fs stdout | 2016-10-11 07:23:29,331 INFO entries, mem_size, disk_size, download_queue, prefetch_queue, s3_queue: 0, 0, 0, 0, 0, 0
yas3fs stdout | 2016-10-11 07:23:29,331 INFO multipart_uploads_in_progress = 0
2016-10-11 07:23:29,331 DEBUG new_locks, unused_locks: 0, 0
yas3fs stdout | 2016-10-11 07:23:29,332 DEBUG gc count0/threshold0, count1/threshold1, count2/threshold2: 400/700, 8/10, 4/10
2016-10-11 07:23:29,332 DEBUG check_threads 'False'
yas3fs stdout | 2016-10-11 07:23:29,720 DEBUG SNS SQS subscription = arn:aws:sns:eu-central-1:102573368484:hello-world:2371faa3-5ea9-4e2b-895e-7a2bb0419851
2016-10-11 07:23:34,334 DEBUG check_cache_size get_memory_usage
yas3fs stdout | 2016-10-11 07:23:34,334 DEBUG check_cache_size get_memory_usage
2016-10-11 07:23:34,337 INFO entries, mem_size, disk_size, download_queue, prefetch_queue, s3_queue: 0, 0, 0, 0, 0, 0
yas3fs stdout | 2016-10-11 07:23:34,337 INFO entries, mem_size, disk_size, download_queue, prefetch_queue, s3_queue: 0, 0, 0, 0, 0, 0
2016-10-11 07:23:34,337 INFO multipart_uploads_in_progress = 0
2016-10-11 07:23:34,337 DEBUG new_locks, unused_locks: 0, 0
yas3fs stdout | 2016-10-11 07:23:34,337 INFO multipart_uploads_in_progress = 0
2016-10-11 07:23:34,337 DEBUG new_locks, unused_locks: 0, 0
2016-10-11 07:23:34,338 DEBUG gc count0/threshold0, count1/threshold1, count2/threshold2: 631/700, 8/10, 4/10
2016-10-11 07:23:34,339 DEBUG check_threads 'False'
yas3fs stdout | 2016-10-11 07:23:34,338 DEBUG gc count0/threshold0, count1/threshold1, count2/threshold2: 631/700, 8/10, 4/10
2016-10-11 07:23:34,339 DEBUG check_threads 'False'
2016-10-11 07:23:39,340 DEBUG check_cache_size get_memory_usage
yas3fs stdout | 2016-10-11 07:23:39,340 DEBUG check_cache_size get_memory_usage
2016-10-11 07:23:39,342 INFO entries, mem_size, disk_size, download_queue, prefetch_queue, s3_queue: 0, 0, 0, 0, 0, 0
yas3fs stdout | 2016-10-11 07:23:39,342 INFO entries, mem_size, disk_size, download_queue, prefetch_queue, s3_queue: 0, 0, 0, 0, 0, 0
2016-10-11 07:23:39,342 INFO multipart_uploads_in_progress = 0
2016-10-11 07:23:39,342 DEBUG new_locks, unused_locks: 0, 0
2016-10-11 07:23:39,342 DEBUG gc count0/threshold0, count1/threshold1, count2/threshold2: 632/700, 8/10, 4/10
2016-10-11 07:23:39,342 DEBUG check_threads 'False'
yas3fs stdout | 2016-10-11 07:23:39,342 INFO multipart_uploads_in_progress = 0
2016-10-11 07:23:39,342 DEBUG new_locks, unused_locks: 0, 0
2016-10-11 07:23:39,342 DEBUG gc count0/threshold0, count1/threshold1, count2/threshold2: 632/700, 8/10, 4/10
2016-10-11 07:23:39,342 DEBUG check_threads 'False'
2016-10-11 07:23:44,245 DEBUG Got 1 messages from SQS
yas3fs stdout | 2016-10-11 07:23:44,245 DEBUG Got 1 messages from SQS
2016-10-11 07:23:44,245 DEBUG process_message '{"Records":[{"eventVersion":"2.0","eventSource":"aws:s3","awsRegion":"eu-central-1","eventTime":"2016-10-11T07:23:43.798Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"A37LZVIZWDA39U"},"requestParameters":{"sourceIPAddress":"54.239.6.33"},"responseElements":{"x-amz-request-id":"975F17563D3F603F","x-amz-id-2":"/Zqx7JBGPifuXNliITenR4e9So/rAiWnKMlUW45hU2hfYjB/PFa+P+Y92zjGJCbtZ9FmI7Kq8wA="},"s3":{"s3SchemaVersion":"1.0","configurationId":"3cd3d867-aeb3-4ad3-8884-2b0b0da28c62","bucket":{"name":"hello-world","ownerIdentity":{"principalId":"A37LZVIZWDA39U"},"arn":"arn:aws:s3:::hello-world"},"object":{"key":"instances/index.html","size":465,"eTag":"f54571dd875ab56b0c55d5afbf9764f1","sequencer":"0057FC937FBB9CB285"}}}]}'
yas3fs stdout | 2016-10-11 07:23:44,245 DEBUG process_message '{"Records":[{"eventVersion":"2.0","eventSource":"aws:s3","awsRegion":"eu-central-1","eventTime":"2016-10-11T07:23:43.798Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"A37LZVIZWDA39U"},"requestParameters":{"sourceIPAddress":"54.239.6.33"},"responseElements":{"x-amz-request-id":"975F17563D3F603F","x-amz-id-2":"/Zqx7JBGPifuXNliITenR4e9So/rAiWnKMlUW45hU2hfYjB/PFa+P+Y92zjGJCbtZ9FmI7Kq8wA="},"s3":{"s3SchemaVersion":"1.0","configurationId":"3cd3d867-aeb3-4ad3-8884-2b0b0da28c62","bucket":{"name":"hello-world","ownerIdentity":{"principalId":"A37LZVIZWDA39U"},"arn":"arn:aws:s3:::hello-world"},"object":{"key":"instances/index.html","size":465,"eTag":"f54571dd875ab56b0c55d5afbf9764f1","sequencer":"0057FC937FBB9CB285"}}}]}'
2016-10-11 07:23:44,245 ERROR Uncaught Exception in Thread
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 2906, in run
    super(TracebackLoggingThread, self).run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 1147, in listen_for_messages_over_sqs
    self.process_message(message)
  File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 1179, in process_message
    if c[0] == self.unique_id:
KeyError: 0
yas3fs stdout | 2016-10-11 07:23:44,245 ERROR Uncaught Exception in Thread
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 2906, in run
    super(TracebackLoggingThread, self).run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 1147, in listen_for_messages_over_sqs
    self.process_message(message)
  File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 1179, in process_message
    if c[0] == self.unique_id:
KeyError: 0
Exception in thread Thread-41:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 2906, in run
    super(TracebackLoggingThread, self).run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 1147, in listen_for_messages_over_sqs
    self.process_message(message)
  File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 1179, in process_message
    if c[0] == self.unique_id:
KeyError: 0

yas3fs stdout | Exception in thread Thread-41:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 2906, in run
    super(TracebackLoggingThread, self).run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 1147, in listen_for_messages_over_sqs
    self.process_message(message)
  File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 1179, in process_message
    if c[0] == self.unique_id:
KeyError: 0

2016-10-11 07:23:44,344 DEBUG check_cache_size get_memory_usage
yas3fs stdout | 2016-10-11 07:23:44,344 DEBUG check_cache_size get_memory_usage
2016-10-11 07:23:44,347 INFO entries, mem_size, disk_size, download_queue, prefetch_queue, s3_queue: 0, 0, 0, 0, 0, 0
yas3fs stdout | 2016-10-11 07:23:44,347 INFO entries, mem_size, disk_size, download_queue, prefetch_queue, s3_queue: 0, 0, 0, 0, 0, 0
2016-10-11 07:23:44,347 INFO multipart_uploads_in_progress = 0
2016-10-11 07:23:44,347 DEBUG new_locks, unused_locks: 0, 0
2016-10-11 07:23:44,347 DEBUG gc count0/threshold0, count1/threshold1, count2/threshold2: 668/700, 8/10, 4/10
2016-10-11 07:23:44,347 DEBUG check_threads 'False'
yas3fs stdout | 2016-10-11 07:23:44,347 INFO multipart_uploads_in_progress = 0
2016-10-11 07:23:44,347 DEBUG new_locks, unused_locks: 0, 0
2016-10-11 07:23:44,347 DEBUG gc count0/threshold0, count1/threshold1, count2/threshold2: 668/700, 8/10, 4/10
2016-10-11 07:23:44,347 DEBUG check_threads 'False'
2016-10-11 07:23:44,347 DEBUG Restarting queue listen thread
yas3fs stdout | 2016-10-11 07:23:44,347 DEBUG Restarting queue listen thread
2016-10-11 07:23:44,347 INFO Listening on queue: 'yas3fs-helloworld-instances-3e740572-4b24-4cd5-8530-c370b4177a08'
yas3fs stdout | 2016-10-11 07:23:44,347 INFO Listening on queue: 'yas3fs-helloworld-instances-3e740572-4b24-4cd5-8530-c370b4177a08'

Is that a bug in yas3fs or is it simply a non supported functionality?

dalbani avatar Oct 11 '16 07:10 dalbani

It is (still) not implemented.

danilop avatar Oct 11 '16 08:10 danilop

@danilop: thanks for your quick response (I'm silly really, @pearcec had just mentioned the same error above...) Given that this "cluster feature" is the most important one in YAS3FS for me, what options are available?

  • Try to help implement the functionality: how complex do you evaluate the work to be done?
  • Use a workaround: should I manually send a notification via SNS then, once I have made a change to a S3 object?

dalbani avatar Oct 11 '16 09:10 dalbani

One option is to use a lambda function to convert the native S3 notification to what YAS3FS expects and send to its SNS.

bnusunny avatar Nov 13 '16 04:11 bnusunny