Need to do R&D on possibilities to rearchitect LME

[llwaterhouse]

• [ ] #252
• [ ] #253
• [ ] #254
• [ ] #255
• [ ] #256

[llwaterhouse]

Need to verify that we have documented all the requirements necessary for forwarding events to WEC (or anything else in the future that we use). Based on https://github.com/cisagov/LME/issues/146, users have had issues where they have issues like "COM+ needs to be allowed on the WEC for it to receive that connection".

Are there any other instructions that we can include to help users?

[llwaterhouse]

Also refer to #11

[llwaterhouse]

From @aarz-snl you can still do a cluster with docker compose will just take some extra steps with config files

keep in mind that the data is retained based on docker compose volume location. So, you can't just put to a different location during the script or something

[aarz-snl]

Using the same framework seen here:

https://github.com/peasead/elastic-container

.env file:

This is used to set variables for both the install.sh or .py, and for the docker-compose.yml

deploy.sh / .py

This would be used for:

1. running the docker compose command
2. After containers are up you can use this script to run curl commands to configure elasticsearch (set retention size, configure fleet settings, etc) Same as we do with deploy.sh

docker-compose.yml

Speaks for itself. This is the configuration of the containers you will be running with docker compose up. Certain capabilities youll see used in the ECP docker compose is using an isntance.yml and the elasticsearch-certutil. elastic comes with a cert generator built in... something we don't currently use.

some concerns I've heard:

1. What about backups? Nothing has to change here. In our environment we use a bind mount:

      - type: bind
        source: /opt/lme/backups
        target: /usr/share/elasticsearch/backups

We would just include the same logic in our docker compose yaml.

2. What if they want to do a multi node install?

I personally don't think this meets the nature of LME. You should only need multiple nodes if you're talking about thousands of hosts. That being said the idea to prepare for this would be:

1. create 2 docker-compose templates. One for the master node, and one for additional nodes.

2. Edit the .env to contain a variable where users can set a single node or multi node deployment.

3. edit the .sh script to dynamically create the 'master' docker-compose.yml and the nodes docker-compose.yml. Using something like 'yq' 'sed' or yaml python package you can parse the template and insert / remove environment variables as needed. This is what we currently do already with deploy.sh

4. The most lifting in theory the customer would have to do here is copy of the docker compose and do another docker-compose up on the secondary server.

5. Will they lose data?

No.. Our docker-compose.yml will just mount to the same bind mounts as before. It's still the same thing.

Overall, this is very close to what we are doing already. However, it removes a lot of the logic from the install script, and install of logstash + wec+ dc + certificates etcetc, we are just installing a fleetserver and an agent on each host. There's like 300+ lines of code for generating certs in deploy.sh and the elasticsearch cert util can handle it with one instances.yml and a few lines of code. Like so:

> config/certs/instances.yml;
          bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
          unzip config/certs/certs.zip -d config/certs;
          cat config/certs/elasticsearch/elasticsearch.crt config/certs/ca/ca.crt > config/certs/elasticsearch/elasticsearch.chain.pem

I just configured elasticsearches certificates. Point to this pem in the docker compose file.

[llwaterhouse]

In the re-architecture discussions, we should talk about whether we need to still include logstash. I spoke with Adam and he questioned whether we need the additional complexity. It would make it easier when we upgrade winlogbeat.

[dcernoch]

Add Children:

• [ ] Determine target audience and goals
• [ ] Secure by default architecture (podman/secure storage/compose)
• [ ] Easy Deployment and Stable/Sane Upgrade/Management
• [ ] Q/A: unit tests AND threat emulation
• [ ] Architecture: (podman/(elastic or opensearch)+(agent or wazuh), os support, agnostic architecture?

[aarz-snl]

In the re-architecture discussions, we should talk about whether we need to still include logstash. I spoke with Adam and he questioned whether we need the additional complexity. It would make it easier when we upgrade winlogbeat.

I do not think we need logstash with 2.0. Sounds like we want to go agent based -- agents send the logs which is really all we are using logstash for right now (input from winlogbeat, output to elasticsearch)

There's some enrichment happening with the logs as well -- but im not sure how important this would be in 2.0

[llwaterhouse]

Is your comment valid only if we continue using Elastic? In one of our discussions about switching to OpenSearch Mikey said we would still need to use logstash in that case.

[safiuddinr]

@mreeve-snl adding as a watcher.

[safiuddinr]

This has been taken care of during the TEM.