Clean OS, clean docker image

[y0d4a]

Hi, i just spins new vm (ubuntu 20.04) and installed docker 20.10. Pulled this image and on first start i got in problems, grafana is up but without dashboard and data. In log i see:

parsedmarc_1 | urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f49ef6c3890>: Failed to establish a new connection: [Errno 111] Connection refused parsedmarc_1 | elasticsearch.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7ffb51ceba90>: Failed to establish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7ffb51ceba90>: Failed to establish a new connection: [Errno 111] Connection refused) parsedmarc_1 | FileNotFoundError: [Errno 2] No such file or directory: '/output/aggregate.json'

and somewhere i saw error that he had permission denied for dashboard file..

where i am wrong? i do like is in tutorial

[manpreets]

Hey @y0d4a

I am having a similar issue in a Mac running Docker 4.15.0 (93002) on engine 20.10.21

dmarc-visualizer-master-parsedmarc-1 | Traceback (most recent call last): dmarc-visualizer-master-parsedmarc-1 | File "/usr/local/bin/parsedmarc", line 8, in dmarc-visualizer-master-parsedmarc-1 | sys.exit(_main()) dmarc-visualizer-master-parsedmarc-1 | ^^^^^^^ dmarc-visualizer-master-parsedmarc-1 | File "/usr/local/lib/python3.11/site-packages/parsedmarc/cli.py", line 981, in _main dmarc-visualizer-master-parsedmarc-1 | process_reports(results) dmarc-visualizer-master-parsedmarc-1 | File "/usr/local/lib/python3.11/site-packages/parsedmarc/cli.py", line 81, in process_reports dmarc-visualizer-master-parsedmarc-1 | save_output(results, output_directory=opts.output, dmarc-visualizer-master-parsedmarc-1 | File "/usr/local/lib/python3.11/site-packages/parsedmarc/init.py", line 1325, in save_output dmarc-visualizer-master-parsedmarc-1 | append_json(os.path.join(output_directory, aggregate_json_filename), dmarc-visualizer-master-parsedmarc-1 | File "/usr/local/lib/python3.11/site-packages/parsedmarc/init.py", line 1265, in append_json dmarc-visualizer-master-parsedmarc-1 | with open(filename, "r+", newline="\n", encoding="utf-8") as output: dmarc-visualizer-master-parsedmarc-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ dmarc-visualizer-master-parsedmarc-1 | FileNotFoundError: [Errno 2] No such file or directory: '/output/aggregate.json' dmarc-visualizer-master-parsedmarc-1 exited with code 1

[erikrandom]

Hi, i just spins new vm (ubuntu 20.04) and installed docker 20.10. Pulled this image and on first start i got in problems, grafana is up but without dashboard and data. In log i see:

parsedmarc_1 | urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f49ef6c3890>: Failed to establish a new connection: [Errno 111] Connection refused parsedmarc_1 | elasticsearch.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7ffb51ceba90>: Failed to establish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7ffb51ceba90>: Failed to establish a new connection: [Errno 111] Connection refused) parsedmarc_1 | FileNotFoundError: [Errno 2] No such file or directory: '/output/aggregate.json'

and somewhere i saw error that he had permission denied for dashboard file..

where i am wrong? i do like is in tutorial

Hello,

I had to create the following empty files in the /OUTPUT folder mounted by the parsedmarc container: aggregate.csv aggregate.json forensic.csv forensic.json

..and the subfolder: sample

Try that if you, like me, run out of patience solving the underlying permission issue.

Now I'm having problems with the grafana graphs from the elastic data. Something about an elastic interval setting that might be deprecated. Let me know if you figure that one out! "Invalid interval specified, must be non-null and non-empty"

[y0d4a]

i created but he did not bring up anything (no grafana, no elastic...) not sure how this project is working after all....

[jburgui]

Hi @y0d4a @erikrandom , I solved the problem adding this to grafana-dockerfile: run sed 's/fixed_interval/interval/' -i /var/lib/grafana/dashboards/grafana-dmarc_reports.json Cheers ;)

[rtlong]

I have it working, though I encountered these errors too.

The connection errors come when ElasticSearch is not yet available (it takes a short time to initialize and start listening), but parsedmarc will continue trying to connect for bit. Once it connects and starts processing, it goes silent!

I just opened a PR #42 which is the code I have working and most helpfully it adds the --verbose flag so you get some output during what may be a long process of importing your reports.

Spin it up, let it run, and eventually the parsedmarc container will stop on it's own. Then you can load the dashboard and see it working.

[erikrandom]

That did the trick! ..I had to use case sensitive file name though: /var/lib/grafana/dashboards/Grafana-DMARC_Reports.json

Thank you

[phi0x]

I'm still having issues. I cloned the git, tried to run stock and it fails. I did tweak some ES settings to my liking which is fine, ES seems to run perfectly fine and grafana also appears to be running the parsedmarc fails to establish a connection with ES.

Many failed tried attempts:

elasticsearch.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7fb781086110>: Failed to establish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7fb781086110>: Failed to establish a new connection: [Errno 111] Connection refused) INFO:cli.py:753:Starting parsedmarc

the docker container exits after this and if i restart the container it just loops through these errors and exits again instantly.

I tried the potential workarounds people mentioned such as creating blank files in the output directory, updating the grafana Dockerfile to include both: run sed 's/fixed_interval/interval/' -i /var/lib/grafana/dashboards/grafana-dmarc_reports.json run sed 's/fixed_interval/interval/' -i /var/lib/grafana/dashboards/Grafana-DMARC_Reports.json

Nothing appears to work to fix enabling the ES connection between parsedmarc and ES. thoughts?...

[tsvane]

I had the "Invalid interval specified, must be non-null and non-empty"-problem when moving the project to a new server. The sed command did the trick for me, though i had to format it a bit differently.

Added the following command to the end of /dmarc-visualizer/grafana/Dockerfile: RUN sed -i 's/fixed_interval/interval/g' /var/lib/grafana/dashboards/Grafana-DMARC_Reports.json

[phi0x]

I tried now installing on windows 11 WSL ubuntu, same problem. Seems it's broken right out of the box. Seems parsedmarc can't connect to ES yet ES logs seem to be fine for showing the DB is up. any suggestions?

[Erklin]

@phi0x Have you checked the details of the parsedmarc container? Are the /input and /output mounted as expected and do you have any actual files to process in the input folder?

[phi0x]

I managed to get it working by setting my ES host configuration to 127.0.0.1:9200 instead of what's in the config as 'elasticsearch:9200'. Along with as you note, needing some files for it to parse. When I fixed the connection issue between parsedmarc and ES, I saw the container was quiting right away and the logs weren't saying much that was helpful. I decided to try adding files to the files folder and then restarted the container for parsedmarc and it worked. I then realized that the container must be restarted each time as it doesn't just stay running in the background watching the files folder continuously.

[GIYItalk]

I managed to get it working by setting my ES host configuration to 127.0.0.1:9200 instead of what's in the config as 'elasticsearch:9200'. Along with as you note, needing some files for it to parse. When I fixed the connection issue between parsedmarc and ES, I saw the container was quiting right away and the logs weren't saying much that was helpful. I decided to try adding files to the files folder and then restarted the container for parsedmarc and it worked. I then realized that the container must be restarted each time as it doesn't just stay running in the background watching the files folder continuously.

I encountered the same error as you, did you solve the problem after changing the IP of ES to 127.0.0.1? I did the same, but there is still no data in Grafana. The error is still going on, have you solved this BUG

[skjbulcher]

Hi @y0d4a @erikrandom , I solved the problem adding this to grafana-dockerfile: run sed 's/fixed_interval/interval/' -i /var/lib/grafana/dashboards/grafana-dmarc_reports.json Cheers ;)

This is kind of the fix, based on what I used to get it working. For my part, I only changed fixed_interval to interval when it fell under bucketAggs. I did not change it when it fell directly under panels.

[ton-An]

run sed 's/fixed_interval/interval/' -i /var/lib/grafana/dashboards/grafana-dmarc_reports.json

That issue seems to stem from the fact that Grafana-DMARC_Reports.json (in grafna/Dockerfile) is pulled directly from the master branch of parsedmarc which has been updated a few times in the last year or so.

I fixed it by specifying the most recent version of the json before the last edit to grafna/Dockerfile in this repo

The link to that version is: https://raw.githubusercontent.com/domainaware/parsedmarc/77331b55c54cb3269205295bd57d0ab680638964/grafana/Grafana-DMARC_Reports.json

[djorkaef]

I spent a full day on getting this to work, then it stopped working again, did a full OS reinstall and tried to make the docker compose work from the first time without any edits. Then it worked. These are my steps

• Clean Ubuntu Server image with SSH access
• Download the .zip
• Extract the dmarc-visualiser-master.zip to /home/user/Downloads/dmarc-visualiser-master
• Create a subfolder "files" and a subfolder "output_files"
• Put some dmarc reports (zipped as received) in the "files" folder
• Adjust these files:

Grafana\dockerfile:

• Change the version of grafana to 10.2.3

FROM grafana/grafana:10.2.3

ADD --chown=grafana:root https://raw.githubusercontent.com/domainaware/parsedmarc/master/grafana/Grafana-DMARC_Reports.json /var/lib/grafana/dashboards/
RUN chmod 644 /etc/grafana/provisioning

COPY grafana-provisioning/ /etc/grafana/provisioning/

Parsedmarc.sample.ini rename to parsedmarc.ini The content of this file remains the same:

[general]
save_aggregate = True
save_forensic = True
output = /output/

[elasticsearch]
hosts = elasticsearch:9200
ssl = False

docker-compose.yml file

• adjust the volume for elasticsearch

elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.5
    environment:
      - discovery.type=single-node
    volumes:
      - ./elastic_data:/var/lib/elasticsearch/data

Next run sudo docker-compose up. It will give a few timeouts but once the elasticsearch is fully running you will see success:

dmarc-visualizer-master-parsedmarc-1     |     INFO:cli.py:1018:Starting parsedmarc
dmarc-visualizer-master-parsedmarc-1     | /usr/local/lib/python3.9/site-packages/elasticsearch/connection/base.py:208: ElasticsearchWarning: Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html to enable security.
dmarc-visualizer-master-parsedmarc-1     |   warnings.warn(message, category=ElasticsearchWarning)
dmarc-visualizer-master-parsedmarc-1     |    DEBUG:__init__.py:1173:Parsing /input/***.zip

• Use the webbrowser to go to http://ipaddress:3000
• Log into grafana with username: admin password:admin, adjust your password
• Go to Dashboards, select DMARC Summary

If you mess up: reinstall the complete OS, set all settings correctly and try again :)

I was also able to make it work with msgraph, amazing INFO:cli.py:1313:Watching for email - Quit with ctrl-c

One thing to know is that when you change the parsedmarc.ini file, you need to run docker-compose up --build to include the changes.