(com.aws.greengrass.lifecyclemanager): Separate directory for all write operations

[moro-si]

Feature Description Greengrass v2 is deployed under a single root directory. This feature add the possibility to configure to use a separate directory for all write operations, including creating directories and files.

Use Case We have a use case where our device has limited storage to store the FW at manufacturing but additional space on a different device (e.g. SD card). We also have a requirement to configure in read-only mode the FW partition.

We would like to keep a minimal version of Greengrass (just the nucleus component) bundled with the FW and use the additional storage for everything else where Greengrass needs write-permission (package, log, etc).

Proposed Solution This feature is available in Greengrass v1. In the nucleus configuration we should have a parameter to define the write-directory path.

[MikeDombo]

Hi, The entire Greengrass root directory needs to be writable, so there's no reason to separate anything out. If you really wanted to have the jar file in RO then you can already do that; Greengrass doesn't care where the jar file is located. However, if you do this, any deployment which updates the Nucleus version will write into the writable area and start using that newer version.

[MikeDombo]

Can you please let me know some more about your use case and why the current setup is not sufficient to solve the use case?

[kennu]

Didn't notice this issue :) I wrote about our use case in https://github.com/aws-greengrass/aws-greengrass-nucleus/issues/986 and how we need the Greengrass V1 writeDirectory setting in V2 too.

[kennu]

I might add that we are using https://github.com/aws/meta-aws/tree/master/recipes-iot/aws-iot-greengrass to build the Linux image with Greengrass. (Still working on backporting it to Yocto Linux Zeus branch.)

[kennu]

@MikeDombo > In your use case you don't want to use Greengrass to handle deployments? Why not, and what do you use Greengrass for?

We use Greengrass to deploy applications to our IoT Devices. We use Mender to deploy the system software (Linux, Docker, Greengrass) that runs on the devices. As I described in the other issue, the system software is distributed as Linux images, allowing transactional updates and rollbacks in case of failure.

[MikeDombo]

So Greengrass is still used to deploy code onto the device? If that's the case then the root directory must be writable, otherwise how can we download your updated code?

If you want the Greengrass jar to be in a read only partition, then that's certainly possible for you to do.

[kennu]

@MikeDombo In Greengrass V1 this works simply by setting writeDirectory to point under /data. Greengrass will then write all its downloads there and run the Lambda functions etc. in there. Greengrass Core is built into the read-only, updateable system image using this recipe: https://github.com/aws/meta-aws/tree/zeus/recipes-greengrass/greengrass-core. I would like V2 to somehow work in a similar way.

[MikeDombo]

I understand what V1 does, however these are quite different systems. I'm saying that you can already achieve the same thing by treating the Greengrass root as the "writeDirectory" from V1. If you want the Greengrass jar and certificates to be in a read only partition, then that is possible to do.

[kennu]

Okay thanks, sounds good. Sounds like it would be basically left for the https://github.com/aws/meta-aws/tree/master/recipes-iot/aws-iot-greengrass recipe (and @rpcme :)) to support a deployment structure where the Greengrass V2 system files are installed under a different path than where GG_BASENAME/GG_ROOT points to.

It would probably also be very helpful to document this use case in https://docs.aws.amazon.com/greengrass/v2/developerguide so that there is a supported way to run Greengrass V2 on read-only filesystems. Same issues has also been discussed here https://forums.aws.amazon.com/thread.jspa?messageID=985413.

[MikeDombo]

So the only difference is that you need to configure the alts/current directory as a symlink into the RO file system, that should be all that needs to be done.

The system service should be calling <root>/alts/current/distro/bin/loader. Loader then reads the jar file from <root>/alts/current/distro/lib/Greengrass.jar

[kennu]

Thanks, I'm going to hack on the recipe and see what I can do myself. There are probably some challenges in that we need to also edit the Greengrass configuration files locally when devices auto-register to AWS on first boot. Hopefully it's doable with some symlinks.

In the future it is also important that Greengrass V2 has some kind of commitment to not suddently start writing files outside its own root path. I already had a problem with Greengrass V2 trying to execute usermod, which wouldn't work with a read-only filesystem. We need to be sure that once we have pre-configured and baked everything in the Yocto image, Greengrass V2 will only makes changes under its own root path.

[MikeDombo]

Greengrass only writes to the root (except for /tmp which is used by the AWS IoT Device SDK).

When using --provision we will create users and groups automatically, which is why you would have seen the usermod. If you manually provision, then that won't happen.

[kennu]

Btw, the usermod issue actually happens regardless of --provision in here: https://github.com/aws-greengrass/aws-greengrass-nucleus/blob/59a89f19590a51af3dd02f5c7b13f5b5052153ef/src/main/java/com/aws/greengrass/easysetup/GreengrassSetup.java#L312

For some reason setComponentDefaultUserAndGroup() tries to call usermod anyway. although we already have ggc_user in /etc/passwd with default group ggc_group in /etc/group.

[MikeDombo]

If you provide the user as a command line or config argument then we won't try to do anything.

[kennu]

Yeah looks like there's two separate problems:

(1) The meta-aws recipe is not passing our configuration properly to Greengrass (where we set posixUser: ggc_user:ggc_group). I will try to fix this.

(2) setComponentDefaultUserAndGroup() fails to autodetect our existing user in if (!platform.userExists(GGC_USER)) {...}

This question is a bit off-topic already, but I wonder how to make Greengrass V2 / Nucleus output debug level logs? I'd like to figure out why id is failing. Our /usr/bin/id is from GNU coreutils.

[MikeDombo]

There is a configuration under the nucleus service in the config file. Have a look for "logging" under https://docs.aws.amazon.com/greengrass/v2/developerguide/greengrass-nucleus-component.html#greengrass-nucleus-component-configuration

[kennu]

Got it thanks. I wonder if I can just edit effectiveConfig.yaml and restart? The Developer Guide says I should create a deployment to change the config, but I'm not quite that far yet.

[MikeDombo]

Editing effective config doesn't do anything, however in your yocto setup I believe that it uses an initial config file, so you can edit that file.

[kennu]

Hmm. I wonder how we are going to solve this in our 1st boot autoregistration, which is a shell script that generates the certificates and keys and the configuration for Greengrass (endpoints etc). How can we insert that configuration into Greengrass V2, before it's able to connect to the cloud.. Previously we just replaced config.json.

[MikeDombo]

Edit this file: https://github.com/aws/meta-aws/blob/master/recipes-iot/aws-iot-greengrass/files/greengrassv2-init.yaml

[kennu]

Will that file be reloaded every time Greengrass starts? We don't have the right settings when building the Yocto image; we need to perform a 1st boot to get them from the cloud.

[MikeDombo]

No, it is not. Greengrass will read from config.tlog. If config.tlog does not exist it will read from config.yaml file which has the same format as the linked file.

[kennu]

Okey. I wonder if this process involving config.tlog is documented somewhere... We probably need to take advantage of it for our auto-registration. Or more generally, I wish Developer Guide had some instructions on how to change the configuration locally when necessary.

[rpcme]

@kennu this is really great feedback for us to improve our prototype-to-production level documentation. Super important!

[kennu]

I got rid of my usermod problem by upgrading meta-aws greengrass-bin to use Greengrass 2.1.0 instead of 2.0.3!

Now I can get Greengrass V2 on our development device to connect to AWS IoT and it's looking good. This is on a read-only root filesystem, although it's still running Greengrass.jar from a data partition.

I also looked inside greengrass-2.1.0.zip for the first time and realized it's a very simple distribution compared to V1. So I'm pretty confident that I can just move the files to the read-only filesystem in our deployment, while configuring the root directory to point inside /data.

Thanks again @MikeDombo @rpcme for all your help, today was very productive for me. 🍺