Use a unified shared directory

Please describe the problem you are trying to solve.

Establishing a communication in preCICE requires an exchange directory, which is configurable. Each M2N as well as the IntraComm of each participant can use a different exchange directory. It is generally only useful for a single exchange directory to exist, as it simplifies cleaning up stale connectivity files.

Furthermore, additional output, such as exports or profiling data pollute the runtime directories of the solver. This can become tedious when running on a cluster. These files could be stored in the exchange directory too, but there is no way to access this directory.

Describe the solution you propose.

Use a single exchange directory for the entire simulation. Each M2N and IntraComm then uses this directory to establish communication. Furthermore, it can now also be used by other components, such as profiling.

Example:

<precice-configuration>
  <solver-interface dimensions="3">

   <shared-dir location="." name="precice-run"/>

   <data .../>
   <mesh .../>
   <participant .../>
   <m2n .../>
   <coupling-scheme .../>
 </solver-interface>
</precice-configuration>

Open questions:

How should we call the tag?

<shared-dir ... />
<temp-dir ... />
<shared ... />
<tmp ... />
<scratch-dir ... />

Do we want to provide an override in the m2n and intra-comm tags?

How should exports behave?

<export:vtu directory="name" />

The current default is the working directory of the solver. Relative paths are relative to that directory. Should we change this to the shared-directory?

Describe alternatives you've considered Add further configurable directories for separate components.

Additional context

Related to https://github.com/precice/precice/pull/1419#issuecomment-1237972146

Context (because I was confused for a second): This only makes sense in cases with more than one coupling schemes.

It is generally only useful for a single exchange directory to exist, as it simplifies cleaning up stale connectivity files.

I agree.

Furthermore, additional output, such as exports or profiling data pollute the runtime directories of the solver. This can become tedious when running on a cluster. These files could be stored in the exchange directory too, but there is no way to access this directory.

These are also often useful result files, so I don't see it as a problem that they are stored in the runtime directory of each solver.

Note that we currently treat precice-run as a completely temporary directory that should always be deleted after a simulation, if still there.

How should we call the tag?

Let's agree on the scope of this directory first, to avoid renaming it again soon. But shared-dir sounds self-descriptive enough (though inconsistent with output-directory).

The current default is the working directory of the solver. Relative paths are relative to that directory. Should we change this to the shared-directory?

Similarly: I am not sure if this is a good idea.

I am not sure I understand the motivation for this issue, and I tend to see it as "high effort / low benefit" at the moment.

This only makes sense in cases with more than one coupling schemes.

This is currently possible:

<precice-configuration>
  <solver-interface dimensions="3">
    <participant name="SolverOne">
      <intra-comm:sockets exchange-directory="one"/>
    </participant>

    <participant name="SolverTwo">
      <intra-comm:mpi exchange-directory="two"/>
    </participant>

    <m2n:sockets from="SolverOne" to="SolverTwo" exchange-directory="three" />

    <coupling-scheme />
  </solver-interface>
</precice-configuration>

Leading to directories one, two and three, each containing a precice-run folder.

I am not sure I understand the motivation for this issue, and I tend to see it as "high effort / low benefit" at the moment.

The motivation is to have a single well-defined exchange-directory for the simulation. It can then also be used to save other files that could be interesting in some cases. Let's say that you want to write the network of connected ranks to a file for inspection. Then that shared directory could be a good location for that.

If we save all run files to this shared directory, then we can even create tools that only need this shared directory as input and can automatically locate all files of a run. Think convergence measures, profiling data.

Ah, I did not have this case in mind. Thanks for clarifying. Makes much more sense now.

I agree that having a single shared directory is better than the multiple ones we could have right now.

But shared-dir sounds self-descriptive enough (though inconsistent with output-directory).

Agree on both.

Do we want to provide an override in the m2n and intra-comm tags?

I would say no. I don't see a use case.

How should exports behave? The current default is the working directory of the solver. Relative paths are relative to that directory. Should we change this to the shared-directory?

That's a tricky one. I actually tend towards moving everything into the shared directory. For certain files (convergence and iterations file), it actually makes not much sense to write them twice as we do now. I also think that user does not care much about which participant writes a watchpoint etc. But I have not yet thought this through completely.#

Important will be to have good defaults. We probably want that 95% of users don't use these options. The complete tag shared-dir should be optional pointing to .. probably (what we currently use in the tutorials). Do we need the name attribute at all?

<shared-dir location="." name="precice-run"/>

Maybe a problem (maybe not): If location is relative, to what should it be relative? Working directory of all participants? We would need that these are consistent (we also do now). Or maybe relative to the location of precice-config.xml, which would/should be unique. The first option would probably be less disruptive.

I actually tend towards moving everything into the shared directory. For certain files (convergence and iterations file), it actually makes not much sense to write them twice as we do now. I also think that user does not care much about which participant writes a watchpoint etc. But I have not yet thought this through completely.

Moving everything into a shared directory will also simplify the implementation of advanced functionality in the future preCICE controller™.

Do we need the name attribute at all?

Not really. I just wanted to make clear that we always create a directory inside the exchange directory. We could of course ignore that.

Maybe a problem (maybe not): If location is relative, to what should it be relative? Working directory of all participants? We would need that these are consistent (we also do now).

I would stick to the current strict requirement that all participants need to be on the same nesting level.

Or maybe relative to the location of precice-config.xml, which would/should be unique. The first option would probably be less disruptive.

This will require us to keep the config path and forward it to the configuration logic. Not an issue, just an additional requirement.

Maybe the solution to make absolute paths easier to handle is to expand environment variables in the location. Then one could adjust this without major headache.

I just wanted to make clear that we always create a directory inside the exchange directory.

Yes, I would still do that. But we could just always call it precice-run. I don't see a use case where renaming this folder makes sense.

Conclusion from the discussion:

• This will break all configuration files that use the example from the xmlreference as it includes the exchange-dir.
• There is barely any upside for this change and even takes flexibility away.
• It is unintuitive for all exports to go into a single directory. This could also lead to performance problems due to the common destination directory. We could add subdirectories, but this requires to create them, which is again problematic for filesystems.
• It could become tricky to implement correctly depending on the order in the configuration file.

Hence, medium work for questionable gain

Discussing in person once more.

We decided to not apply this as it is hard to implement and there are no clear upsides.