|
8 | 8 | - [Host Request Processing](#host-request-processing) |
9 | 9 | - [Intuitive User Experience](#intuitive-user-experience) |
10 | 10 | - [Concurrent Invocation of Functions](#concurrent-invocation-of-functions) |
| 11 | +- [Managed Dependencies](#managed-dependencies) |
11 | 12 | - [Durable Functions Prototype](#durable-functions-prototype) |
12 | 13 |
|
13 | 14 | ## Introduction |
@@ -459,6 +460,56 @@ Once a PowerShell Manager instance is checked out, the worker's main thread will |
459 | 460 |
|
460 | 461 | Note that, checking out a PowerShell Manager instance from the pool is a blocking operation. When the number of concurrent invocations reaches the pool size, the worker's main thread will block on the checkout operation for the next invocation request until a PowerShell Manager instance becomes available. The initial design was to let the checkout operation happen in the thread-pool thread too, so the main thread would just create a task and go back processing the next request. However, it turned out to result in high lock contention due to too many tasks competing during the checkout operation. Therefore, the design was changed to make the checkout operation happen only in main thread, and a task gets created only after a PowerShell Manager instance becomes available. |
461 | 462 |
|
| 463 | +## Managed Dependencies |
| 464 | + |
| 465 | +### Problem |
| 466 | + |
| 467 | +The goal is to let the user declare the dependencies required by functions, and rely on the service automatically locating and installing the dependencies from the PowerShell Gallery or other sources, taking care of selecting the proper versions, and automatically upgrading the dependencies to the latest versions (if allowed by the version specifications provided by the user). |
| 468 | + |
| 469 | +Dependencies are declared in the _requirements.psd1_ file (_manifest_) as a collection of pairs (<_name_>, <_version specification_>). Currently, the version specification should either be an exact and complete version, or strictly match the following pattern: `<major version>.*`. So, a typical manifest may look like this: |
| 470 | + |
| 471 | +``` PowerShell |
| 472 | +@{ |
| 473 | + 'Az' = '2.*' |
| 474 | + 'PSDepend' = '0.*' |
| 475 | + 'Pester' = '5.0.0-alpha3' |
| 476 | +} |
| 477 | +``` |
| 478 | + |
| 479 | +When the `<major version>.*` format is used, the worker will retrieve the latest available module version (within the specified major version) from the PowerShell Gallery, ignoring prerelease versions. |
| 480 | + |
| 481 | +When the exact version is specified, the worker will retrieve the specified version only, ignoring any other version. Prerelease versions are allowed in this case. |
| 482 | + |
| 483 | +The number of entries in the _requirements.psd1_ file should not exceed **10**. This limit is not user-configurable. |
| 484 | + |
| 485 | +Installing and upgrading dependencies should be performed automatically, without requiring any interaction with the user, and without interfering with the currently running functions. This represents an important design challenge. In a different context, dependencies could be stored on a single location on the file system, managed by regular PowerShell tools (`Install-Module`/`Save-Module`, `PSDepend`, etc.), while having the same file system location added to _PSModulePath_ to make all the modules available to scripts running on this machine. This is what PowerShell users normally do, and this approach looks attractive because it is simple and conventional. However, in the contexts where multiple independent workers load modules and execute scripts concurrently, and at the same time some module versions are being added, upgraded, or removed, this simple approach causes many known problems. The root causes of these problems are in the fundamentals of PowerShell and PowerShell modules design. The managed dependencies design in Azure Functions must take this into account. The problems will be solved if we satisfy the following conditions: |
| 486 | + |
| 487 | +- **Only one writer at a time**. No concurrent executions of `Save-Module`, `PSDepend`, or anything else that could perform changes _on the same target file path_*_. |
| 488 | +- **Atomic updates**. All workers executing a PowerShell script should always observe a state of dependency files that is a result of a _successful and complete_ execution of `Save-Module` or a similar tool. The workers should never observe any partial results. |
| 489 | +- **Immutable view**. As soon as a set of dependency files is exposed to a worker for loading module purposes for the first time, this set of files should never change _during the life time of this worker._ Deletions, additions, or modifications are not acceptable. |
| 490 | + |
| 491 | +### Solution |
| 492 | + |
| 493 | +The main design idea is to partition the installed dependencies in such a way that every PowerShell worker gets exactly one complete and immutable set of dependencies for the lifetime of this worker (until restart). The same set of dependencies can be shared by multiple PowerShell workers, but each worker is strictly tied to a single set. |
| 494 | + |
| 495 | +On the first function load request, the PowerShell worker reads the manifest and installs all the required dependencies into a dedicated folder on a file storage shared between all PowerShell workers within the function app. The subsequent function invocation requests are blocked until the installation of _all_ the dependencies is _completed successfully_. After the successful installation, the PowerShell worker insert the path of the folder to the '_PSModulePath_' variable of the PowerShell worker process, so that they become available to all functions running on this worker, and allows function invocation requests to proceed. The path to this folder is inserted into '_PSModulePath_' _before_ the `functionAppRoot\Modules` path, so that the managed dependencies folder is discovered first when modules are imported. This entire set of dependencies becomes an immutable _snapshot_: once created and exposed to PowerShell workers, it never changes: no modules or module versions are ever added to this snapshot or removed from this snapshot. |
| 496 | + |
| 497 | +On the next function load request (normally received on a worker start), the worker reads the manifest and checks if the latest snapshot contains the dependencies satisfying the manifest. If the latest snapshot is _acceptable_, the worker makes '_PSModulePath_' point to this snapshot folder and allows the next function invocation proceed immediately. At the same time, the worker starts a background installation of all the dependencies into a new folder, which after successful completion becomes the latest snapshot. At this point, other starting workers will be able to find and use the new snapshot. The workers that were already running when the new snapshot was installed will be able to use it after restart. |
| 498 | + |
| 499 | +A snapshot is considered _acceptable_ if it contains any version _allowed_ by the manifest for each required dependency. For example, Az 2.1 is allowed by the manifest containing `'Az' = '2.*'`, so the snapshot containing Az 2.1 will be considered acceptable, even if Az 2.2 is published on the PowerShell Gallery. As a result, the next function invocation will be allowed to proceed with Az 2.1 without waiting for any other Az version to be installed, and without even trying to contact the PowerShell Gallery for discovering the latest module version. All these activities can be performed in the background, without blocking function invocations. |
| 500 | + |
| 501 | +However, if the latest snapshot is _not acceptable_ (i.e. it does not contain module versions required by the manifest), the worker starts installing the dependencies into a new snapshot, and all the subsequent function invocation requests are blocked, waiting for the new snapshot installation to complete. |
| 502 | + |
| 503 | +When a snapshot installation starts, the dependencies are first installed into a folder with a name following a special pattern (`*i`), so that this snapshot is not picked up by any worker prematurely, before the installation is complete. After _successful_ completion, the snapshot is _atomically promoted_ by renaming the folder to follow a different pattern (`*r`), which indicates to other workers that this snapshot is ready to use. If the installation fails or cannot complete for any reason (for example, the worker restarts, crashes, or gets decommissioned), the folder stays in the installing state until removed. |
| 504 | + |
| 505 | +Incomplete and old snapshots that are no longer in use are periodically removed from the file storage. In order to allow detecting unused snapshots, each PowerShell worker keeps "touching" a file named `.used` at the root of the used snapshot folder every `MDHeartbeatPeriod` minutes. Before and after installing any new snapshot, every PowerShell worker looks for unused snapshots by checking the folder creation time and the `.used` file modification time. If both these time values are older than (`MDHeartbeatPeriod` + `MDOldSnapshotHeartbeatMargin`) minutes, the snapshot is considered unused, so the PowerShell worker removes it. The latest `MDMinNumberOfSnapshotsToKeep` snapshots will never be removed, regardless of usage. |
| 506 | + |
| 507 | +`MDHeartbeatPeriod`, `MDOldSnapshotHeartbeatMargin`, and `MDMinNumberOfSnapshotsToKeep` are environment variables configurable via Application Settings of a Function App. `MDHeartbeatPeriod` and `MDOldSnapshotHeartbeatMargin` are expected to contain strings in the format that can be parsed by `System.TimeSpan.Parse` method. |
| 508 | + |
| 509 | +No other changes are ever performed to the snapshots that were once installed. |
| 510 | + |
| 511 | +In this design, upgrading dependencies is conceptually decoupled from executing functions. Upgrading dependencies can be performed on any schedule by one or multiple agents, (almost) without any coordination between each other or with the workers executing functions. This allows us to make independent decisions on whether to run it from a separate service or keep it in PowerShell workers, and schedule the upgrade as often as we want. For now, upgrading dependencies is still performed by the PowerShell workers, just to avoid the overhead of deploying and maintaining a separate service. However, the design keeps this option open. |
| 512 | + |
462 | 513 | ## Durable Functions Prototype |
463 | 514 |
|
464 | 515 | The [Durable Functions](https://docs.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-concepts) is essentially the stateful workflow in a serverless environment. It consists of an orchestrator function and one or more activity functions. Durable Functions runs in a completely asynchronous way: the orchestrator function will stop after triggering an activity function, and later gets restarted once the activity function finishes execution. The actions of triggering activity functions and the activity function results will be saved along the way. After the orchestrator function is restarted, it replays the execution from the very beginning, but will skip the actions that are already done and use the results directly from the saved logs. |
|
0 commit comments