rabbithub
rabbithub copied to clipboard
can't enable plugin
This is the output of the command line. rabbithub plug-in is expecting a plugin named "rabbitmq_mochiweb", which I can't find on my rabbitmq distribution. I've tried enabling similar plugins (see below), with no success.
root@ubuntu:/mnt/hgfs/Shared# sudo rabbitmq-plugins enable rabbithub Error: The following dependencies could not be found: rabbitmq_mochiweb root@ubuntu:/mnt/hgfs/Shared# sudo rabbitmq-plugins enable mochiweb Plugin configuration unchanged. root@ubuntu:/mnt/hgfs/Shared# sudo rabbitmq-plugins enable webmachine Plugin configuration unchanged. root@ubuntu:/mnt/hgfs/Shared# sudo rabbitmq-plugins enable rabbithub Error: The following dependencies could not be found: rabbitmq_mochiweb
Is there a way to solve this ?
I'm afraid I haven't been keeping the rabbithub codebase up-to-date with respect to RabbitMQ. It probably needs some work to run with recent releases; in particular, to run with the current RabbitMQ HTTP-server APIs.
It seems like replacing rabbitmq_mochiweb by "mochiweb", allowed me to enable the plugin. now I'm gonna try if it actually works. BTW, are you planning to evolve this plugin ? or is this project kind of "idle" ?do you know if there is another alternative for working with webhooks ?
On 8 April 2013 14:51, sonoman [email protected] wrote:
It seems like replacing rabbitmq_mochiweb by "mochiweb", allowed me to enable the plugin.
That's good news!
now I'm gonna try if it actually works.
I fear that it will not...
BTW, are you planning to evolve this plugin ? or is this project kind of "idle" ?do you know if there is another alternative for working with webhooks ?
Both: I do intend to come back to it someday, and for now it is "idle". I recommend asking on the RabbitMQ mailing list for help with webhooks/PSHB/HTTP-access-to-messaging.
Regards, Tony
I helped Tony bring the code up to date to work with 2.8.x, but I must admit that I've not tried it with anything higher than 2.8.7. What version of RabbitMQ are you using?
Also, you might find some of the junk in http://assortedrambles.blogspot.co.nz/2012/11/the-polygot-rabbit.htmlvaguely useful
On Tue, Apr 9, 2013 at 7:11 AM, Tony Garnock-Jones <[email protected]
wrote:
On 8 April 2013 14:51, sonoman [email protected] wrote:
It seems like replacing rabbitmq_mochiweb by "mochiweb", allowed me to enable the plugin.
That's good news!
now I'm gonna try if it actually works.
I fear that it will not...
BTW, are you planning to evolve this plugin ? or is this project kind of "idle" ?do you know if there is another alternative for working with webhooks ?
Both: I do intend to come back to it someday, and for now it is "idle". I recommend asking on the RabbitMQ mailing list for help with webhooks/PSHB/HTTP-access-to-messaging.
Regards, Tony
— Reply to this email directly or view it on GitHubhttps://github.com/tonyg/rabbithub/issues/3#issuecomment-16071806 .
Sadly i got some errors during startup of the broker after enabling rabbithub. I tried recompiling (a recommendation i found in some blog for the same error on another component), but it got worst...that time it didn't even start the broker. Bad luck :(
=ERROR REPORT==== 8-Apr-2013::14:26:23 === Loading of /var/lib/rabbitmq/mnesia/rabbit@ubuntu-plugins-expand/rabbithub/ebin/ rabbithub_subscription.beam failed: badfile
=ERROR REPORT==== 8-Apr-2013::14:26:23 === beam/beam_load.c(1365): Error loading module rabbithub_subscription_sup: use of opcode 153; this emulator supports only up to 152
=ERROR REPORT==== 8-Apr-2013::14:26:23 === Loading of /var/lib/rabbitmq/mnesia/rabbit@ubuntu-plugins-expand/rabbithub/ebin/ rabbithub_subscription_sup.beam failed: badfile
=ERROR REPORT==== 8-Apr-2013::14:26:23 === beam/beam_load.c(1365): Error loading module rabbithub_sup: use of opcode 153; this emulator supports only up to 152
This suggests that the RabbitHub plugin was built with a higher version of Erlang than the one your using with RabbitMQ... whatever version(s) are you using?
On Tue, Apr 9, 2013 at 9:30 AM, sonoman [email protected] wrote:
Sadly i got some errors during startup of the broker after enabling rabbithub. I tried recompiling (a recommendation i found in some blog for the same error on another component), but it got worst...that time it didn't even start the broker. Bad luck :(
=ERROR REPORT==== 8-Apr-2013::14:26:23 === Loading of /var/lib/rabbitmq/mnesia/rabbit@ubuntu-plugins-expand /rabbithub/ebin/ rabbithub_subscription.beam failed: badfile
=ERROR REPORT==== 8-Apr-2013::14:26:23 === beam/beam_load.c(1365): Error loading module rabbithub_subscription_sup: use of opcode 153; this emulator supports only up to 152
=ERROR REPORT==== 8-Apr-2013::14:26:23 === Loading of /var/lib/rabbitmq/mnesia/rabbit@ubuntu-plugins-expand /rabbithub/ebin/ rabbithub_subscription_sup.beam failed: badfile
=ERROR REPORT==== 8-Apr-2013::14:26:23 === beam/beam_load.c(1365): Error loading module rabbithub_sup: use of opcode 153; this emulator supports only up to 152
— Reply to this email directly or view it on GitHubhttps://github.com/tonyg/rabbithub/issues/3#issuecomment-16080117 .
This is the output when I enter erlang's interactive shell:
root@ubuntu:/mnt/hgfs/Shared/aws-dev# erl Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:2:2] [rq:2] [async-threads:0] [kernel-poll:false]
Eshell V5.8.5 (abort with ^G)
I've successfully started rabbitmq with erlang R16B, but again, when I enable rabbithub, rabbitmq stops working. this is the log with the error at startup (after enabling rabbithub).
=INFO REPORT==== 9-Apr-2013::11:41:53 === Error description: undef
Log files (may contain more information): /var/log/rabbitmq/[email protected] /var/log/rabbitmq/[email protected]
Stack trace: [{rabbit_mochiweb,register_context_handler, ["rabbithub",[],#Fun<rabbithub_web.0.64600142>, "RabbitHub"], []}, {rabbit,'-run_boot_step/1-lc$^1/1-1-',1,[]}, {rabbit,run_boot_step,1,[]}, {rabbit,'-start/2-lc$^0/1-0-',1,[]}, {rabbit,start,2,[]}, {application_master,start_it_old,4, [{file,"application_master.erl"},{line,274}]}]
=INFO REPORT==== 9-Apr-2013::11:41:54 === stopped TCP Listener on [::]:5672
=INFO REPORT==== 9-Apr-2013::11:41:54 === application: rabbit exited: {bad_return,{{rabbit,start,[normal,[]]}, {'EXIT',{rabbit,failure_during_boot,undef}}}} type: temporary
Right, this is the failure I expected. Certain modules have been renamed recent(ish)ly I think. Some code changes in Rabbithub will be required. They will be minor (I hope) but they are necessary.
Help me understand and I will give a try (though I know nothing about erlang :) )... This "application_master" that doesn't exist anymore, is a module from rabbit_mochiweb that rabbithub requires ?
It's actually the other end of the stack trace that's the innermost: rabbit_mochiweb:register_context_handler/4
seems to no longer be present.
If you just want to play around and experiment with the RabbitHub plugin, I can give you internet access to an instance. If this would be of interest, ping be an email directly ([email protected]) and I'll send you details.
Brett
On Wed, Apr 10, 2013 at 8:57 AM, Tony Garnock-Jones < [email protected]> wrote:
It's actually the other end of the stack trace that's the innermost: rabbit_mochiweb:register_context_handler/4 seems to no longer be present.
— Reply to this email directly or view it on GitHubhttps://github.com/tonyg/rabbithub/issues/3#issuecomment-16139695 .
actually I'm trying to execute it in our rabbitmq environment....I think I'll try to finish with the re-compilation...running into some issues now (like for example, i can't find the xmerl.hrl include file anywhere in my system :( )
Hi sonoman, have you had any success in your attempts to re-align rabbithub with the latest rabbitmq distribution? I'm using rabbitmq-server-3.1.4 and running into the same problem with mochiweb at the moment :(
Try https://github.com/brc859844/rabbithub. I updated the code a few months back for RabbitMQ 3.1.0, but have not tried it yet with 3.1.4. In theory it should be okay... maybe.
Brett
On Thu, Aug 15, 2013 at 11:53 AM, redsquare [email protected]:
Hi sonoman, have you had any success in your attempts to re-align rabbithub with the latest rabbitmq distribution? I'm using rabbitmq-server-3.1.4 and running into the same problem with mochiweb at the moment :(
— Reply to this email directly or view it on GitHubhttps://github.com/tonyg/rabbithub/issues/3#issuecomment-22676440 .
Brett, I picked up a binary you dropped on https://region-a.geo1.objects.hpcloudsvc.com:443/v1/66395568832599/RabbitHub/rabbithub-3.1.0.ez. Is this compiled from the source on https://github.com/brc859844/rabbithub? I was able to enable rabbithub-3.1.0.ez with 3.1.4 but immediately after I restarted RabbitMQ service it terminated with "RabbitMQ: Erlang machine stopped instantly (distribution name conflict?). The service is not restarted as OnFail is set to ignore."
When I disable rabbithub, RabbitMQ service is all happy and running again. Not sure I understand the problem but here is log entries:
=INFO REPORT==== 15-Aug-2013::11:01:02 ===Error description:
{noproc,{gen_server,call,
[rabbit_web_dispatch_registry,
{add,"rabbithub",
[{port,15670}],
#Fun<rabbit_web_dispatch.1.31447083>,
#Fun<rabbithub_web.0.108453991>,
{[],"RabbitHub"}},
infinity]}}
Log files (may contain more information):.....
Stack trace: [{gen_server,call,3,[{file,"gen_server.erl"},{line,188}]}, {rabbit_web_dispatch,register_context_handler,5,[]}, {rabbit,'-run_boot_step/1-lc$^1/1-1-',1,[]}, {rabbit,run_boot_step,1,[]}, {rabbit,'-start/2-lc$^0/1-0-',1,[]}, {rabbit,start,2,[]}, {application_master,start_it_old,4, [{file,"application_master.erl"},{line,269}]}]
=INFO REPORT==== 15-Aug-2013::11:01:03 ===stopped TCP Listener on 0.0.0.0:5672 =INFO REPORT==== 15-Aug-2013::11:01:03 ===stopped TCP Listener on [::]:5672 =INFO REPORT==== 15-Aug-2013::11:01:03 ===Error description: {could_not_start,rabbit, {bad_return, {{rabbit,start,[normal,[]]}, {'EXIT', {rabbit,failure_during_boot, {noproc, {gen_server,call, [rabbit_web_dispatch_registry, {add,"rabbithub", [{port,15670}], #Fun<rabbit_web_dispatch.1.31447083>, #Fun<rabbithub_web.0.108453991>, {[],"RabbitHub"}}, infinity]}}}}}}}
=CRASH REPORT==== 15-Aug-2013::11:01:03 === crasher: initial call: application_master:init/4 pid: <0.148.0> registered_name: [] exception exit: {bad_return, {{rabbit,start,[normal,[]]}, {'EXIT', {rabbit,failure_during_boot, {noproc, {gen_server,call, [rabbit_web_dispatch_registry, {add,"rabbithub", [{port,15670}], #Fun<rabbit_web_dispatch.1.31447083>, #Fun<rabbithub_web.0.108453991>, {[],"RabbitHub"}}, infinity]}}}}}} in function application_master:init/4 (application_master.erl, line 133) ancestors: [<0.147.0>] messages: [{'EXIT',<0.149.0>,normal}] links: [<0.147.0>,<0.7.0>] dictionary: [] trap_exit: true status: running heap_size: 2586 stack_size: 27 reductions: 289 neighbours:
Tried downgrading to RabbitMQ 3.1.0, but still ran into similar (same?) problem when enabling plugin:
=CRASH REPORT==== 15-Aug-2013::14:04:46 === crasher: initial call: application_master:init/4 pid: <0.148.0> registered_name: [] exception exit: {bad_return, {{rabbit,start,[normal,[]]}, {'EXIT', {rabbit,failure_during_boot, {noproc, {gen_server,call, [rabbit_web_dispatch_registry, {add,"rabbithub", [{port,15670}], #Fun<rabbit_web_dispatch.1.31447083>, #Fun<rabbithub_web.0.108453991>, {[],"RabbitHub"}}, infinity]}}}}}} in function application_master:init/4 (application_master.erl, line 133) ancestors: [<0.147.0>] messages: [{'EXIT',<0.149.0>,normal}] links: [<0.147.0>,<0.7.0>] dictionary: [] trap_exit: true status: running heap_size: 2586 stack_size: 27 reductions: 288 neighbours:
=CRASH REPORT==== 15-Aug-2013::14:05:04 === crasher: initial call: application_master:init/4 pid: <0.148.0> registered_name: [] exception exit: {bad_return, {{rabbit,start,[normal,[]]}, {'EXIT', {rabbit,failure_during_boot, {noproc, {gen_server,call, [rabbit_web_dispatch_registry, {add,"rabbithub", [{port,15670}], #Fun<rabbit_web_dispatch.1.31447083>, #Fun<rabbithub_web.0.108453991>, {[],"RabbitHub"}}, infinity]}}}}}} in function application_master:init/4 (application_master.erl, line 133) ancestors: [<0.147.0>] messages: [{'EXIT',<0.149.0>,normal}] links: [<0.147.0>,<0.7.0>] dictionary: [] trap_exit: true status: running heap_size: 2586 stack_size: 27 reductions: 288 neighbours:
I should have mentioned that the instance running on Windows Server 2012.
Natalya
Okay. I'll try to have a look at it over the weekend - it will hopefully not be a big deal! I might try it with RabbitMQ 3.1.5, which has just been released (the guys introduced a couple of bugs into 3.1.4 which are resolved in 3.1.5).
Regards, Brett
On Fri, Aug 16, 2013 at 4:40 AM, redsquare [email protected] wrote:
Brett, I picked up a binary you dropped on https://region-a.geo-1.objects.hpcloudsvc.com:443/v1/66395568832599/RabbitHub/rabbithub-3.1.0.ez. Is this compiled from the source on https://github.com/brc859844/rabbithub? I was able to enable it with 3.1.4 but immediately after I restarted RabbitMQ service it terminated with "RabbitMQ: Erlang machine stopped instantly (distribution name conflict?). The service is not restarted as OnFail is set to ignore." Doing some digging now as to what the cause might be...
— Reply to this email directly or view it on GitHubhttps://github.com/tonyg/rabbithub/issues/3#issuecomment-22712458 .
Awesome! I noticed 3.1.5 was released in the last couple of days, will definitely want it if it works with rabbithub. Please please let me know what you find as far as the cause of a crash.
Thank you! Natalya
Okay, give https://region-a.geo-1.objects.hpcloudsvc.com:443/v1/66395568832599/RabbitHub/rabbithub-3.1.5.eza spin with RabbitMQ 3.1.5, or use the latest sources from https://github.com/brc859844/rabbithub.
Brett
On Sat, Aug 17, 2013 at 3:41 AM, redsquare [email protected] wrote:
Awesome! I noticed 3.1.5 was released in the last couple of days, will definitely want it if it works with rabbithub. Please please let me know what you find as far as the cause of a crash.
Thank you! Natalya
— Reply to this email directly or view it on GitHubhttps://github.com/tonyg/rabbithub/issues/3#issuecomment-22774545 .
Brett, thank you for tweaking the plugin! I feel I'm so close but no luck :( I disabled rabbithub-3.1.0.ez and uninstalled rabbitmq 3.1.0. I then installed 3.1.5 and enabled rabbithub-3.1.5.ez. But got the same error again:
=CRASH REPORT==== 19-Aug-2013::11:22:31 === crasher: initial call: application_master:init/4 pid: <0.148.0> registered_name: [] exception exit: {bad_return, {{rabbit,start,[normal,[]]}, {'EXIT', {rabbit,failure_during_boot, {noproc, {gen_server,call, [rabbit_web_dispatch_registry, {add,"rabbithub", [{port,15670}], #Fun<rabbit_web_dispatch.1.31447083>, #Fun<rabbithub_web.0.63518153>, {[],"RabbitHub"}}, infinity]}}}}}} in function application_master:init/4 (application_master.erl, line 133) ancestors: [<0.147.0>] messages: [{'EXIT',<0.149.0>,normal}] links: [<0.147.0>,<0.7.0>] dictionary: [] trap_exit: true status: running heap_size: 2586 stack_size: 27 reductions: 294
I am not sure how to debug this. But I thought it might be related to a port number since rabbitmq_management was configured to use 15672 and rabbithub was trying to run on 15670. So I changed the config and put rabbitmq_management on 15670, restarted and confirmed that the mgmt web console ran on 15670, all looked good. Then enabled rabbithub and it crashed the service again with the error that looked identical to the above. I've never used Erlang and never tried the plugin development for RabbitMQ. Would I need to install the dev tools and run the source code on the machine where it crashes to get to the bottom of this? Dunno if it matters but it's a 64-bit Windows Server 2012 machine.
That's just a bit frustrating!
When I updated and tested the plugin over the weekend I was using Erlang 15B01 and things were working fine; however upon switching to Erlang 16B01 just now I have managed to reproduce the problem. I can see where it's breaking and will see what can be done to fix the problem.
Just out of interest, what version of Erlang are you using?
Brett
On Tue, Aug 20, 2013 at 7:09 AM, redsquare [email protected] wrote:
Brett, thank you for tweaking the plugin! I feel I'm so close but no luck :( I disabled rabbithub-3.1.0.ez and uninstalled rabbitmq 3.1.0. I then installed 3.1.5 and enabled rabbithub-3.1.5.ez. But got the same error again:
=CRASH REPORT==== 19-Aug-2013::11:22:31 === crasher: initial call: application_master:init/4 pid: registered_name: [] exception exit: {bad_return,
{{rabbit,start,[normal,[]]}, {'EXIT', {rabbit,failure_during_boot, {noproc, {gen_server,call, [rabbit_web_dispatch_registry, {add,"rabbithub", [{port,15670}], #Fun, #Fun, {[],"RabbitHub"}}, infinity]}}}}}} in function application_master:init/4 (application_master.erl, line 133) ancestors: [] messages: [{'EXIT',,normal}] links: [,] dictionary: [] trap_exit: true status: running heap_size: 2586 stack_size: 27 reductions: 294
I am not sure how to debug this. But I thought it might be related to a port number since rabbitmq_management was configured to use 15672 and rabbithub was trying to run on 15670. So I changed the config and put rabbitmq_management on 15670, restarted and confirmed that the mgmt web console ran on 15670, all looked good. Then enabled rabbithub and it crashed the service again with the error that looked identical to the above. I've never used Erlang and never tried the plugin development for RabbitMQ. Would I need to install the dev tools and run the source code on the machine where it crashes to get to the bottom of this?
— Reply to this email directly or view it on GitHubhttps://github.com/tonyg/rabbithub/issues/3#issuecomment-22896620 .
Heh, otp_win64_R16B01.exe :)
That's great news - the Erlang version dependency you found! The worst case, I can try to downgrade :)
With a bit of frantic hacking I've got things working with Erlang 16B01.
Give https://region-a.geo-1.objects.hpcloudsvc.com:443/v1/66395568832599/RabbitHub/rabbithub-3.1.5.eza spin and let me know how you get on!
Tony, FYI I'm not entirely sure what the guys have done that caused what appears to be an Erlang version compatibility issue, but to get around it I've dropped the boot_step's from the plugin and essentially moved those steps into rabbithub:start/2, etc.
Brett
On Tue, Aug 20, 2013 at 10:10 AM, redsquare [email protected]:
Heh, otp_win64_R16B01.exe :)
— Reply to this email directly or view it on GitHubhttps://github.com/tonyg/rabbithub/issues/3#issuecomment-22908515 .
Yay! It's definitely much happier now and no errors on startup :) Thank you so much for tweaking the code! As far as implementation goes, there is no support for hub.secret, correct? Also, the rabbithub doc notes that the exchange type can be specified with a query string param, is it also possible to mark queue/exchange/messages as durable using query string? Thank you, Brett, this is great stuff!
No problem.
You are correct; there is currently no support for hub.secret. I'll try to find some time to look into this and support for durable/persistent queues/exchanges/messages.
Regards, Brett
On Wed, Aug 21, 2013 at 4:51 AM, redsquare [email protected] wrote:
Yay! It's definitely much happier now and no errors on startup :) Thank you so much for tweaking the code! As far as implementation goes, there is no support for hub.secret, correct? Also, the rabbithub doc notes that the exchange type can be specified with a query string param, is it also possible to mark queue/exchange/messages as durable using query string? Thank you, Brett, this is great stuff!
— Reply to this email directly or view it on GitHubhttps://github.com/tonyg/rabbithub/issues/3#issuecomment-22959509 .
hub.secret is one of those nice-to-have things but I was hoping the persistence was already there since RabbitMQ supports it. Of course it may not be easy to leverage it from the plugin side, I just don't know. Will wait for the updates and try to learn Erlang meantime :) Thanks, Brett!
Brett, it shouldn't be a problem if one or both of the URLs (pub and sub) are HTTPS? For whatever reason everything works for me as long as I don't use SSL. But I see no attempts to post to a HTTPS URL when publishing from HTTP URL. Although I always get get "Accepted" 202 response from the hub in all cases. Then nothing happens and nothing in IIS or RabbitMQ logs (not sure if rabbit logs would even show anything). Just curious.
In terms of HTTPS, for calls to RabbitHub it would be necessary to configure SSL along similar lines to the RabbitMQ management UI, as it uses essentially the same web server interface, but I'd need to verify the details. As far as subscription callbacks go, a bit more work would be required. The code currently uses a simple hand-crafted HTTP client that Tony crafted when he put the original code together some years back. This would need to be changed to use the Erlang/OTP HTTP client, which does support HTTPS - probably not a big job to change; just a bit fiddly.
Over the weekend I was planning to start looking at the durable/persistent stuff you mentioned previously, and then maybe I can find some time to look at HTTPS (it's actually something I'd been thinking about too).
Regards, Brett
On Fri, Aug 23, 2013 at 4:54 AM, redsquare [email protected] wrote:
Brett, it shouldn't be a problem if one or both of the URLs (pub and sub) are HTTPS? For whatever reason everything works for me as long as I don't use SSL. But I see no attempts to post to a HTTPS URL when publishing from HTTP URL. Although I always get get "Accepted" 202 response from the hub in all cases. Then nothing happens and nothing in IIS or RabbitMQ logs (not sure if rabbit logs would even show anything). Just curious.
— Reply to this email directly or view it on GitHubhttps://github.com/tonyg/rabbithub/issues/3#issuecomment-23107028 .
Ah, I see now. The SSL question was more out of curiosity than anything else as a result of a test I ran. I think that for purely RabbitHub purposes hub.secret should do just fine without an overhead of maintaining SSL certificates. I was contemplating including the hmac digest of the entire content into the request body in the absence of hub.secret implementation. Passing the key from sub to pub would become a somewhat manual process. But I may re-think all that. On a different subject, durable queues/messages sound fantastic! :)
UPDATE: just noticed when I create a queue via RabbitHub url, it gets "durable" set to True when I inspect it in web management console. Is this reflective of the actual setting? Also, it doesn't automatically mean that messages are durable, correct?
Brett, I was wondering if you had any luck configuring durable queues/messages through RabbitHub? I noticed that the queues created using hub URLs were marked durable in the management console. So after creating two queues I wanted to see what happens if I restart the service. One of them had one unprocessed message in it but no subscribers. Another one has been passing messages to a subscriber. I couldn't start the service after stopping it. It seemed to have trouble with the queue that had a subscriber:
=CRASH REPORT==== 27-Aug-2013::13:42:51 === crasher: initial call: application_master:init/4 pid: <0.304.0> registered_name: [] exception exit: {bad_return, {{rabbithub,start,[normal,[]]}, {'EXIT', {noproc, {gen_server,call, [rabbithub_subscription_sup, {start_child, [{rabbithub_lease, {rabbithub_subscription, {resource,<<"/">>,queue,<<"Group3">>}, "Group3", "http://localhost/callback.ashx"}, 1692656779662140}]}, infinity]}}}}} in function application_master:init/4 (application_master.erl, line 133) ancestors: [<0.303.0>] messages: [{'EXIT',<0.305.0>,normal}] links: [<0.303.0>,<0.7.0>] dictionary: [] trap_exit: true status: running heap_size: 610 stack_size: 27 reductions: 117 neighbours:
An observation from the log above, the rabbithub_lease in the log above seems to be set to a high value of 1692656779662140 although I passed 315360000 seconds (roughly 10 years) when subscribing. Not sure how that got messed up...