errbot icon indicating copy to clipboard operation
errbot copied to clipboard
verified publisher icon errbotio

HTML-templating not being interpreted by ejabber/Pidgin on later errbot versions

Open zaughon opened this issue 8 years ago • 8 comments

I am...

  • [ ] Reporting a bug
  • [ ] Suggesting a new feature
  • [ ] Requesting help with running my bot
  • [x] Requesting help writing plugins
  • [ ] Here about something else

I am running...

  • Errbot version: 5.1.2
  • OS version: Ubuntu 14.04 LTS (Retested on Ubuntu 16.04.3 LTS)
  • Python version: 3.4
  • Using a virtual environment: yes
  • Server: EJabber (XMPP)
  • Client: Pidgin for Ubuntu

Issue description

I currently have a virtual environment running errbot 2.2.1 (really old yes), so i've created a new virtual environment running errbot 5.1.2, but whenever i use this, html-templating breaks.

I've created a command for testing templating (code is pasted under "Steps to reproduce"). When i use the command, the message is returned as uninterpreted html-code on Errbot 5.1.2, but is interpreted fine on 2.2.1:

ErrBot 5.1.2: (15:22:08) Zaubot-test: (Line break added due to github)

 <html xmlns="http://jabber.org/protocol/xhtml-im">
<body>
<b>Hello test</b>
 </body>
</html>

ErrBot 2.2.1: (15:22:45) Zaubot-test: Hello test

I really can't figure out why this happens.

  • Have HTML-templating been removed? Documentation shows Markdown templating, but the Jinja documentation shows HTML, so i'd assume this is still supported?
  • Any ideas on why it's not interpreted? Any ideas on how to troubleshoot it?
  • Have something changed in regards to it?
  • If it have been removed. Why?

Steps to reproduce

Plugin: test.py

from errbot import BotPlugin, botcmd, version
from errbot.templating import tenv

class Test(BotPlugin):
    errbotVersion = version.VERSION
    @botcmd()
    def vtest(self, msg, args):
        test = "test"
        response = tenv().get_template("test.html").render(user = test)
        if self.errbotVersion == "2.2.1":  # If old ErrBot version
            self.send(str(msg.frm), response, message_type = msg.type)
        else:
            self.send(self.build_identifier(str(msg.frm)), response)

test.html

{%- extends "base.html" -%}
{%- block body -%}

<b>Hello {{user}}</b>

{% endblock %}

base.html

<!-- look here to see what is supported : http://xmpp.org/extensions/xep-0071.html -->
<html xmlns="http://jabber.org/protocol/xhtml-im">
<body>
{% block body %}{% endblock %}
</body>
</html>

Additional info

Running Jinja2==2.9.6 on both old and new virtual environment.

I have 2 different bots, and i can reproduce it on both bots. (One of them uses errbot==5.0.1)

Feel free to ask if you need more into :)

zaughon avatar Dec 29 '17 15:12 zaughon

Everything is markdown now. Try !render test to see if it works correctly with your service and client. If it is adapt your template likehttps://github.com/errbotio/errbot/blob/master/errbot/core_plugins/test.md

I hope it helps

Guillaume

gbin-argo avatar Dec 29 '17 15:12 gbin-argo

Yea that doesn't work at all. The only thing that's interpreted correctly is the list. Everything else is just a mess.

Any way to get HTML supported?

Why was it removed in the first place? It seems rather sad basically ruling out one of the major XMPP clients on Linux when it doesn't know Markdown.

Do you know when it was removed? I can't find anything in the changelog. All i can find is XHTML-IM output can now be enabled on XMPP again. for version 4.0

zaughon avatar Dec 29 '17 16:12 zaughon

Ha you need to activate it now : XMPP_XHTML_IM: True in the config

gbin-argo avatar Dec 29 '17 17:12 gbin-argo

Ah, thanks! I completely missed that option.

It's odd though. Whenever i turn that on, i can't start the bot anymore. Judging by the traceback, and the "warn_admins"-part, i think it's failing on sending the message that "VersionChecker plugin is blacklisted".

19:03:01 ERROR    sleekxmpp.xmlstream.xmlst Error processing event handler: <bound method XMPPBackend.connected of <yapsy_loaded_plugin_XMPP_0.XMPPBackend object at 0x7f8469bc2a58>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/sleekxmpp/xmlstream/xmlstream.py", line 1710, in _event_runner
    func(*args)
  File "/usr/local/lib/python3.5/dist-packages/errbot/backends/xmpp.py", line 492, in connected
    self.connect_callback()
  File "/usr/local/lib/python3.5/dist-packages/errbot/core.py", line 648, in connect_callback
    self.warn_admins(errors)
  File "/usr/local/lib/python3.5/dist-packages/errbot/core.py", line 579, in warn_admins
    self.send(self.build_identifier(admin), warning)
  File "/usr/local/lib/python3.5/dist-packages/errbot/core.py", line 143, in send
    self.split_and_send_message(msg)
  File "/usr/local/lib/python3.5/dist-packages/errbot/core.py", line 165, in split_and_send_message
    self.send_message(partial_message)
  File "/usr/local/lib/python3.5/dist-packages/errbot/backends/xmpp.py", line 509, in send_message
    mtype='chat' if msg.is_direct else 'groupchat')
  File "/usr/local/lib/python3.5/dist-packages/sleekxmpp/basexmpp.py", line 544, in send_message
    mhtml, mfrom, mnick).send()
  File "/usr/local/lib/python3.5/dist-packages/sleekxmpp/basexmpp.py", line 499, in make_message
    message['html']['body'] = mhtml
TypeError: 'str' object does not support item assignment
19:03:01 ERROR    sleekxmpp.basexmpp        'str' object does not support item assignment
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/sleekxmpp/xmlstream/xmlstream.py", line 1710, in _event_runner
    func(*args)
  File "/usr/local/lib/python3.5/dist-packages/errbot/backends/xmpp.py", line 492, in connected
    self.connect_callback()
  File "/usr/local/lib/python3.5/dist-packages/errbot/core.py", line 648, in connect_callback
    self.warn_admins(errors)
  File "/usr/local/lib/python3.5/dist-packages/errbot/core.py", line 579, in warn_admins
    self.send(self.build_identifier(admin), warning)
  File "/usr/local/lib/python3.5/dist-packages/errbot/core.py", line 143, in send
    self.split_and_send_message(msg)
  File "/usr/local/lib/python3.5/dist-packages/errbot/core.py", line 165, in split_and_send_message
    self.send_message(partial_message)
  File "/usr/local/lib/python3.5/dist-packages/errbot/backends/xmpp.py", line 509, in send_message
    mtype='chat' if msg.is_direct else 'groupchat')
  File "/usr/local/lib/python3.5/dist-packages/sleekxmpp/basexmpp.py", line 544, in send_message
    mhtml, mfrom, mnick).send()
  File "/usr/local/lib/python3.5/dist-packages/sleekxmpp/basexmpp.py", line 499, in make_message
    message['html']['body'] = mhtml

Any idea why this is happening? If i comment out that setting, or set it to False, it starts up just fine.

zaughon avatar Dec 30 '17 02:12 zaughon

It seems to happen on all messages it attempts to send, not just warn_admin messages. The bot i tested on there is my own bot at home using python3.5. The one i have at work is running python3.4. It happens on this as well, and it doesn't send the warn_admin-message.

15:36:13 ERROR    sleekxmpp.xmlstream.xmlst Error processing event handler: <bound method XMPPBackend.connected of <yapsy_loaded_plugin_XMPP_0.XMPPBackend object at 0x7f58a34c8940>>
Traceback (most recent call last):
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/sleekxmpp/xmlstream/xmlstream.py", line 1710, in _event_runner
    func(*args)
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/errbot/backends/xmpp.py", line 492, in connected
    self.connect_callback()
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/errbot/core.py", line 648, in connect_callback
    self.warn_admins(errors)
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/errbot/core.py", line 579, in warn_admins
    self.send(self.build_identifier(admin), warning)
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/errbot/core.py", line 143, in send
    self.split_and_send_message(msg)
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/errbot/core.py", line 165, in split_and_send_message
    self.send_message(partial_message)
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/errbot/backends/xmpp.py", line 509, in send_message
    mtype='chat' if msg.is_direct else 'groupchat')
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/sleekxmpp/basexmpp.py", line 544, in send_message
    mhtml, mfrom, mnick).send()
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/sleekxmpp/basexmpp.py", line 499, in make_message
    message['html']['body'] = mhtml
TypeError: 'str' object does not support item assignment
15:36:13 ERROR    sleekxmpp.basexmpp        'str' object does not support item assignment
Traceback (most recent call last):
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/sleekxmpp/xmlstream/xmlstream.py", line 1710, in _event_runner
    func(*args)
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/errbot/backends/xmpp.py", line 492, in connected
    self.connect_callback()
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/errbot/core.py", line 648, in connect_callback
    self.warn_admins(errors)
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/errbot/core.py", line 579, in warn_admins
    self.send(self.build_identifier(admin), warning)
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/errbot/core.py", line 143, in send
    self.split_and_send_message(msg)
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/errbot/core.py", line 165, in split_and_send_message
    self.send_message(partial_message)
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/errbot/backends/xmpp.py", line 509, in send_message
    mtype='chat' if msg.is_direct else 'groupchat')
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/sleekxmpp/basexmpp.py", line 544, in send_message
    mhtml, mfrom, mnick).send()
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/sleekxmpp/basexmpp.py", line 499, in make_message
    message['html']['body'] = mhtml
TypeError: 'str' object does not support item assignment

Same error on that bot. Seems to be an issue between ErrBot and sleekxmpp. Any hints on how to fix it?

zaughon avatar Jan 04 '18 14:01 zaughon

@gbin-argo Adding self.client.register_plugin('xep_0071') # For sending HTML-messages to: venv-new/lib/python3.4/site-packages/errbot/backends/xmpp.py > line 319 seems to sort of semi-fix that issue. Inspired by https://gist.github.com/legastero/4497613#gistcomment-2110918

I don't know if you have a better way, as i'm now getting errors like this instead:

13:30:00 ERROR    errbot.core               An error happened while processing a message ("?vtest"): Traceback (most recent call last):
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/errbot/core.py", line 459, in _execute_and_send
    reply = method(msg, match) if match else method(msg, args)
  File "/home/maf/dev/lebowski/errbot/extra/Test/test.py", line 15, in vtest
    self.send(self.build_identifier(str(msg.frm)), response)
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/errbot/botplugin.py", line 529, in send
    return self._bot.send(identifier, text, in_reply_to, groupchat_nick_reply)
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/errbot/core.py", line 143, in send
    self.split_and_send_message(msg)
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/errbot/core.py", line 165, in split_and_send_message
    self.send_message(partial_message)
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/errbot/backends/xmpp.py", line 510, in send_message
    mtype='chat' if msg.is_direct else 'groupchat')
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/sleekxmpp/basexmpp.py", line 544, in send_message
    mhtml, mfrom, mnick).send()
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/sleekxmpp/basexmpp.py", line 499, in make_message
    message['html']['body'] = mhtml
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/sleekxmpp/xmlstream/stanzabase.py", line 778, in __setitem__
    getattr(self, set_method)(value, **kwargs)
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/sleekxmpp/plugins/xep_0071/stanza.py", line 42, in set_body
    xhtml = ET.fromstring(content)
  File "/usr/lib/python3.4/xml/etree/ElementTree.py", line 1325, in XML
    parser.feed(text)
  File "<string>", line None
xml.etree.ElementTree.ParseError: mismatched tag: line 5, column 19
"
Traceback (most recent call last):
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/errbot/core.py", line 459, in _execute_and_send
    reply = method(msg, match) if match else method(msg, args)
  File "/home/maf/dev/lebowski/errbot/extra/Test/test.py", line 15, in vtest
    self.send(self.build_identifier(str(msg.frm)), response)
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/errbot/botplugin.py", line 529, in send
    return self._bot.send(identifier, text, in_reply_to, groupchat_nick_reply)
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/errbot/core.py", line 143, in send
    self.split_and_send_message(msg)
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/errbot/core.py", line 165, in split_and_send_message
    self.send_message(partial_message)
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/errbot/backends/xmpp.py", line 510, in send_message
    mtype='chat' if msg.is_direct else 'groupchat')
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/sleekxmpp/basexmpp.py", line 544, in send_message
    mhtml, mfrom, mnick).send()
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/sleekxmpp/basexmpp.py", line 499, in make_message
    message['html']['body'] = mhtml
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/sleekxmpp/xmlstream/stanzabase.py", line 778, in __setitem__
    getattr(self, set_method)(value, **kwargs)
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/sleekxmpp/plugins/xep_0071/stanza.py", line 42, in set_body
    xhtml = ET.fromstring(content)
  File "/usr/lib/python3.4/xml/etree/ElementTree.py", line 1325, in XML
    parser.feed(text)
  File "<string>", line None
xml.etree.ElementTree.ParseError: mismatched tag: line 5, column 19

When trying to send:

<!-- look here to see what is supported : http://xmpp.org/extensions/xep-0071.html -->
<html xmlns="http://jabber.org/protocol/xhtml-im">
<body>
<b>Hello test</b>


</body>
</html>

Seems to happen on most HTML i try to send, but on different lines/columns. It seems to be struggling with the </b> tag on the <b>Hello test</b> line. There's 1 html-command that works fine though, which is surprising as that's the most advanced one.

Update: Apparently it works if you remove all empty lines in the HTML-files. Like:

{%- extends "base.html" -%}
{%- block body -%}
<b>Hello {{ user }}</b>

{% endblock %}

should be:

{%- extends "base.html" -%}
{%- block body -%}
<b>Hello {{ user }}</b>
{% endblock %}

Not sure why empty lines suddenly aren't accepted. It makes everything kinda clumped together and ugly when you remove them in larger templates :slightly_frowning_face:

zaughon avatar Jan 08 '18 12:01 zaughon

@gbin / @gbin-argo Do you have any feedback on my latest comment? I've been rather stuck for a while.

As soon as i enable XMPP_XHTML_IM, it will start giving errors on empty lines in templates. It even happens on the build-in help-command:

12:16:07 ERROR    errbot.core               An error happened while processing a message ("?help"): Traceback (most recent call last):
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/errbot/core.py", line 461, in _execute_and_send
    self.send_simple_reply(msg, self.process_template(template_name, reply), private, threaded)
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/errbot/core.py", line 201, in send_simple_reply
    self.split_and_send_message(reply)
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/errbot/core.py", line 165, in split_and_send_message
    self.send_message(partial_message)
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/errbot/backends/xmpp.py", line 509, in send_message
    mtype='chat' if msg.is_direct else 'groupchat')
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/sleekxmpp/basexmpp.py", line 545, in send_message
    mhtml, mfrom, mnick).send()
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/sleekxmpp/basexmpp.py", line 500, in make_message
    message['html']['body'] = mhtml
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/sleekxmpp/xmlstream/stanzabase.py", line 770, in __setitem__
    getattr(self, set_method)(value, **kwargs)
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/sleekxmpp/plugins/xep_0071/stanza.py", line 42, in set_body
    xhtml = ET.fromstring(content)
  File "/usr/lib/python3.4/xml/etree/ElementTree.py", line 1325, in XML
    parser.feed(text)
  File "<string>", line None
xml.etree.ElementTree.ParseError: mismatched tag: line 73, column 97
"
Traceback (most recent call last):
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/errbot/core.py", line 461, in _execute_and_send
    self.send_simple_reply(msg, self.process_template(template_name, reply), private, threaded)
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/errbot/core.py", line 201, in send_simple_reply
    self.split_and_send_message(reply)
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/errbot/core.py", line 165, in split_and_send_message
    self.send_message(partial_message)
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/errbot/backends/xmpp.py", line 509, in send_message
    mtype='chat' if msg.is_direct else 'groupchat')
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/sleekxmpp/basexmpp.py", line 545, in send_message
    mhtml, mfrom, mnick).send()
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/sleekxmpp/basexmpp.py", line 500, in make_message
    message['html']['body'] = mhtml
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/sleekxmpp/xmlstream/stanzabase.py", line 770, in __setitem__
    getattr(self, set_method)(value, **kwargs)
  File "/home/maf/dev/lebowski/venv-new/lib/python3.4/site-packages/sleekxmpp/plugins/xep_0071/stanza.py", line 42, in set_body
    xhtml = ET.fromstring(content)
  File "/usr/lib/python3.4/xml/etree/ElementTree.py", line 1325, in XML
    parser.feed(text)
  File "<string>", line None
xml.etree.ElementTree.ParseError: mismatched tag: line 73, column 97

The help command worked at one point, but everything was on 1 line when XMPP_XHTML_IM was enabled. My guess is \n is used to create the help-output, and \n isn't supported in HTML.

I've been trying to upgrade the Errbot version for about a month now, but the templating is giving me a headache. At the moment i either need to go over to Text-templating, which isn't very pretty, or everything breaks when using HTML-templating.

I could also try going through the errbot/tenv/sleekxmpp files and change things there, but that seems rather bad practice.

zaughon avatar Jan 30 '18 11:01 zaughon

@zaughon Is this still an issue? Could you try the latest version and see if anything changes.

sijis avatar Jun 19 '19 05:06 sijis