Implicitly link processes with "public" variables

[ethho]

TL;DR

Add a variable type xsimlab.public that allows for implicit process linking.

---

I am a relatively new user of this package. When I first started learning about it, I noticed that there was one aspect of xsimlab design that was unexpected:

from another_class import AnotherClass

xsimlab.foreign(AnotherClass, 'my_var')

In other words, the user must explicitly link two processes. This approach has many benefits as explained in the docs. A major detriment to this approach, however, is significantly reduced modularity and reusability of the user-defined processes. For example, let's consider the minimal use case:

@xs.process
class DefaultInput:
    some_input = xs.foreign(UseInput, 'some_input', intent='out')

    def initialize(self):
        self.some_input = 'foo'

@xs.process
class UseInput:
    some_input = xs.variable(intent='in')

Replacing the DefaultInput process with a InputFromFile process requires updates to every other linked process. In this simple case, we only need to do this for UseInput, but for larger connected networks of processes this is non-trivial.

My proposal (a mere point of discussion at this stage) is addition of a new public variable type:

@xs.process
class DefaultInput:
    # Very similar to xs.foreign!
    some_input = xs.public('unique_name', intent='out')

    def initialize(self):
        self.some_input = 'foo'

@xs.process
class UseInput:
    some_input = xs.variable(name='unique_name', public=True, intent='in')

Replacing DefaultInput with InputFromFile is now trivial, even for large scale use cases. A public variable is processed at the model scope, so it can be read or provided by any other process in the model. The group variable type already behaves this way (if I'm reading the package code right), except that intent='in' is required for group variables only. A group variable would be a special case of public variable.

[ethho]

Usage Constraints

• The name of a public variable must be unique in the model scope. This implies that only one process in the model can define a public variable with a given name.
• Multiple (or zero) processes can ingest the variable using xs.public, just like xs.foreign.
• Other obvious violations during DAG construction are disallowed, such as:
  • Some process calls xs.public(name), but no process defines a public variable with name
  • Cycle that contains a public edge is detected in the DAG

[ethho]

Alternatives & Workarounds

1. group variables already exhibit similar behavior. One could make a separate group for each public variable, or preferably...
2. ...define all public variables in a single group, then access them using group_dict (group_dict variable #140). This is a viable alternative that does not deviate significantly from status quo.
   • One downside of this approach is that a process that only requires one public variable ingests every public variable.
3. Subclassing - User can define a set of static, lightweight base processes that are often subclassed but never replaced. One could implement InputFromFile this way, thanks to v0.3.0:

@xs.process
class DefaultInput:
    some_input = xs.foreign(UseInput, 'some_input', intent='out')

    def initialize(self):
        self.some_input = 'foo'

@xs.process
class InputFromFile(DefaultInput):

    def initialize(self):
        self.some_input = self.read_from_file()

@xs.process
class UseInput:
    some_input = xs.variable(intent='in')

The major downside, of course, is that subclassing is bad for you. It seems antithetical to the declarative, functional programming that xsimlab and attr encourage. It also requires the user to categorize their processes into two types: "static and inheritable" or "modular and mutable."

4. A number of unspeakable workarounds that I have not and do not want to test. Calling xs.foreign inside a process method, paired with importlib, for instance.

[benbovy]

Thanks for your detailed feedback @eho-tacc.

I admit that hard-linked process classes with xs.foreign limits the overall flexibility of the framework.

I've also been thinking about using variable names, which is an approach used in other frameworks, but it has it's limits too:

• Users who want to build compatible models/processes in separate code repositories need to agree on some standard names. My experience with standard names is that it can be very rigid/hard too.

• Especially with this framework, using variable names to link processes would require a lot of logic to ensure that the DAG of processes is consistent (like you mentioned here above). Those automatically generated DAGs is a key feature of xarray-simlab, so I'd prefer explicit over implicit links.

The major downside, of course, is that subclassing is bad for you. It seems antithetical to the declarative, functional programming that xsimlab and attr encourage.

I wouldn't say that subclassing is bad. I find it very useful here. I often use it to define base processes + different specializations and I'm quite happy with it (note: I'm not trying to use complex things like multiple inheritance, mixins, etc... just simple subclasses).

While Xarray-simlab's model setup/runs interface based on xarray Datasets encourage some functional style, the modeling framework (small process classes) is still heavily object-oriented. I like to leverage the advantages of those different paradigms, which is possible with a language like Python.

[benbovy]

To be clear, I'm not against adding some mechanism like proposed here to circumvent hard-coded links between process classes. I need to think a bit more on that.

Regarding the API:

• I would rather use xs.global() instead of xs.public().
• We could use only one argument in the original variable declaration, e.g., xs.variable(global_name='foo').

The proposal here looks very similar to Python's global variables, with the additional intent constraint and the model scope. Like for Python's global variables, I would recommend using such feature with care, though.

Other questions arising:

• Which name to use for model I/O: <process>__<variable> or <global_name>?

[ethho]

Thanks again for letting me pick your brain on this.

Users who want to build compatible models/processes in separate code repositories need to agree on some standard names.

I agree that this (and the corresponding naming issue) is a cost of implementing xs.public. A follow-up question: is "building compatible models/processes across separate repos" even possible using xs.foreign alone? If one must explicitly link processes within a repo, it stands to reason that Variables passed between repos must also have explicit links, no?

The major downside, of course, is that subclassing is bad for you

This was a hasty conflation of personal preference and best practice 😆 . A less provocative version of the above statement:

• Yes, users should be able to choose inheritance if they want to. We have you to thank for that. I'm currently using "basic" subclassing and it works as expected!
• A nontrivial (growing?) fraction of the Python community would prefer a more functional paradigm. One example is my use case, in which subclassing got a bit messy, but that's another story...

Regarding the API:

xs.global and xs.variable(global_name="foo") (not necessarily mutually exclusive!) have a nice ring. One benefit of exposing as a kwarg is that users are nudged towards the default global=False behavior, because I agree that a bunch of xs.global calls could land users in trouble.

• Which name to use for model I/O: <process>__<variable> or <global_name>?

I feel like I'm venturing into the realm of over-engineering with this idea, but it's worth putting on paper: <namespace>__<name>. A namespace could be a simple mapping to global vars: xs.variable(name="foo", namespace="parameters"), where the default is namespace='global'. As a user, I already think about groups this way. I admit that I don't know if this would be breaking or non-trivial lift. Also, I/O logic doesn't immediately know whether <namespace> is a namespace or a process, and would probably have to do a little digging.

Thanks for listening, and let me know how I can help!

[benbovy]

"building compatible models/processes across separate repos" even possible using xs.foreign alone?

It's certainly not easier than using "global" xarray-simlab variables with some standardized names, but I'd say it's not really harder either.

One benefit of exposing as a kwarg is that users are nudged towards the default global=False behavior

I think global_name=None by default would have a clear meaning too.

A namespace could be a simple mapping to global vars: xs.variable(name="foo", namespace="parameters")

The keys of the dictionary given in xs.Model already provide some namespace, so I would keep it "simple" and not provide another namespace mechanism. Users can use the global name of that, e.g., xs.variable(global_name='parameters_foo'). We could also allow any hashable type, e.g., a tuple: xs.variable(global_name=('parameters', 'foo'))

For some reasons I have in mind, I would be in favor of sticking with <process>__<variable> for model I/O. For example, in #52 it would be nice to allow user-defined process order with a variable value that is passed and updated in chain from one process to another (i.e., multiple intent 'inout'): <process>__<variable> would allow saving multiple snapshots unambiguously in this chain.

let me know how I can help!

PRs are welcome! The hardest part of the work here is to implement in the _ModelBuilder class all the logic needed to reconstruct explicit links from global names and ensure that those links don't break the DAG of processes.

Unfortunately, I only have limited time to spend on this project for the moment. I'll do my best to help but I might be slow to respond.