docs: dataflows: Improve docs

[johnandersen777]

These are notes and scratch work around the purpose and future of the project.

Mission: Provide a clear, meticulously validated, ubiquitously adopted reference architecture for an egalitarian Artificial General Intelligence (AGI) which respects the first law of robotics.

To do so we must enable the AGI with the ability to act in response to the current system context where it understands how to predict possible future system contexts and understands which future system contexts it wishes to pursue are acceptable according to guiding strategic plans (such as do no harm). We must also ensure that human and machine can interact via a shared language, the universal blueprint.

AI has the potential to do many great things. However, it also has the potential to to terrible things too. Recently there was an example of scientists who used a model that was good a generating life saving drugs, in reverse, to generate deadly poisons. GPU manufacturers recently implemented anti-crypto mining features. Since the ubiquitous unit of parallel compute is a GPU, this stops people from buying up GPUs for what we as a community at large have deemed undesirable behavior (hogging all the GPUs). There is nothing stopping those people from buying for building their own ASICs to mine crypto. However, the market for that is a subset of the larger GPU market. Cost per unit goes up, multi-use capabilities go down. GPU manufacturers are effectively able to ensure that the greater good is looked after because GPUs are the ubiquitous facilitator of parallel compute. If we prove out an architecture for an AGI that is robust, easy to adopt, and integrates with the existing open source ecosystem, we can bake in this looking after the greater good.

As we democratize AI, we must be careful not to democratize AI that will do harm. We must think secure by default in terms of architecture which has facilities for guard rails, baking safety into AI.

Failure to achieve ubiquitous adoption of an open architecture with meticulously audited safety controls will result in further consolidation of wealth and widening inequality.

[johnandersen777]

By convention operations which have a single output we usually name that output result

[johnandersen777]

• DataFlows are all effectively typed streams
• Complete state transition of system

[johnandersen777]

• Manifest is a domain specific way to describe system state
• Manifests to describe pipelines, BOMs, tests
  • We need this so that we can do our polyrepo setup
• Manifest contains enough information to reproduce the run
  • Decouples intent from implementation
    • Reduces lock in
    • In absence of a tool (EOL) one could implement the operations to reproduce the execution of the manifest
• Human and machine editable text files
• Will be checked in to version control
  • Added to PR comments to allow for version overrides

[johnandersen777]

• You could have a manifests for anything
  • Any CLI tool could have a manifest which could be converted into it’s CLI args
    • See subprocess orchestrator branch
• Convert from manifest into data flow description
  • Ideally well defined machine readable (auditable) and writable
  • consoletest even plays with documentation as manifest / dataflow
• Manifests are a problem space specific way of defining a dataflow
  • We convert to data flow so people have common way to understand implementations for different problem spaces

[johnandersen777]

• Data Flow is generic representation of program flow
  • Implement executors or synthesizers based off dataflow (Orchestrators)
    • Instead of directly from manifest, can be reused across manifest formats
    • Allows us to template all of our plugins in our polyrepo setup
  • Data Flow maps well to concepts like a Jenkins’s Pipeline, GitHub Actions Workflow, etc.
    • Operations within flows map to Jenkins Step, GitHub Actions Action, etc.
• Security
  • Allows for automated auditing

[johnandersen777]

• Orchestrator allows for
  • Switching execution method easily
    • Run on my local machine to test
    • Run in kubernetes cluster
    • Run in Intel DevCloud for access to machines with ML hardware
  • Local development
    • Rapid iteration on CI jobs
    • No need to push repo to validate CI is working
    • Finally we can support running all tests locally
• Operation abstraction layer allows for
  • Overrides
    • Use implementation X when running in k8s, do Y when running in DevCloud, do Z when running locally
  • Overlays
  • Add extend flows when in different environments or for different purposes or different deployment models
  • https://intel.github.io/dffml/examples/dataflows.html

[johnandersen777]

• Zero-th phase parser
  • Every format has a parser
  • Formats with multiple versions need parser per version
    • You might maintain the version 1 parser in the 1.x branch, version 2 in the 2.x branch, etc.
• Write "next phase" parsers specific to task and format name and format version
  • Generate data flow description of format (in case of a manifest)
• First line of decision making
  • shim layer acts as reverse proxy
  • validates, parses, directs to appropriate next phase
  • next phase only ever needs to worry about its job
    • Verification that we should be executing is done (security)
    • Validation that format conforms to schema is done (complete and correct)

[johnandersen777]

• What does this allow us to do?
  • downstream validation of all DFFML plugins
  • throw all data from every execution into a data lake

[johnandersen777]

Generic flow (data,work,program) executor

[johnandersen777]

Why: unikernels
Can build smallest possibls attack surface
Could even build scilicon / RTL to optimize for sepfic data flow

[johnandersen777]

• Show how we convert from abitrary manifests into dataflow format
  • Will likely need to flush out config / object loading.
  • Probably need to support full object paths and then validate that they are BaseConfigurables
• Need to support secret "unlock" stuff, this is probably similar to dataflow for config
  • This also amounts to: my input goes through this flow to become a different value

[johnandersen777]

Manifest Schema

Manifests allow us to focus less on code and more on data.
By focusing on the data going into and out of systems. We can achieve standard
documentation of processes via a standard interface (manifests).

Our manifests can be thought of as ways to provide a config class with it's
parameters or ways to provide an operation with it's inputs.

References:

• JSON Schema

  • https://json-schema.org/learn/getting-started-step-by-step

  • https://schema.org/

  • https://www.schemastore.org/json/

• Manifest Shim (parser and validator for manifests)

  • util: testing: manifest: shim: Initial commit #1273

Validating

Install jsonschema, and pyyaml python modules

• https://python-jsonschema.readthedocs.io/en/latest/
• https://pyyaml.org/wiki/PyYAMLDocumentation

pip install pyyaml jsonschema

This is how you convert from yaml to json

$ python -c "import sys, pathlib, json, yaml; pathlib.Path(sys.argv[-1]).write_text(json.dumps(yaml.safe_load(pathlib.Path(sys.argv[-2]).read_text()), indent=4) + '\n')" manifest.yaml manifest.json

Example below validates, checking status code we see exit code 0 which means
success, the document conforms to the schema.

$ jsonschema --instance manifest.json manifest-format-name.0.0.2.schema.json
$ echo $?
0

Writing

Suggested process (in flux)

• Make sure you can run the jsonschema validator

  • TODO Validation micro service

• Look at existing problem space

  • What data is needed? This likely will becomes the inputs of a dataflow,
    or an operation, or config.

  • Write first draft of what a valid manifest would be

• Write schema based off initial manifest

  • Do not include fields for future use. Only include what you currently intend
    to use for each version

    • Instead, create a new format name and new schema. If we stick to the rule
      of if you have the data you have to act on it, there is never any if A then
      B situations. If you want a different outcome, you create different manifest.
      This helps keep architectures loosely coupled
      https://medium.com/@marciosete/loosely-coupled-architecture-6a2b06082316

    • We also decided that we could potentially combine manifests. This allows for
      you to use the data you wanted, but just keep it sperate and make the decision
      to combine the equivalent of adding variables purely as conditional on use of data.
      This way if the data is present, it is always used!

    • By ensuring that data present is always used, we can begin to map manifests to
      dataflows, in this way, we can check the validity of a dataflow simply by ensuring
      all manifest data is used as an input or config.

      • As such, a passing validity check ensures we have a complete description of a
        problem. We know all the inputs and system constraints (manifests), and we are
        sure that they will be taken into account on execution (dataflow run).

  • Each field with a type MUST have a description

• Write ADR describing context around creation and usage of manifest

  • The ADR should describe how the author intends the manifest to be used

  • Treat the ADR + manifest like a contract. If something
    accepts the manifest (valid format and version, see shim)
    it is obligated to fulfil the intent of the ADR. The consumer
    MUST return an error response when given a manifest if it
    cannot use each piece of data in the manifest as directed by
    the ADR and descriptions of fields within the manifest schema.

  • The Intent section of the ADR should describe how you want manifest
    consumers to use each field.

ADR Template

my-format-name
##############

Version: 0.0.1
Date: 2022-01-22

Status
******

Proposed|Evolving|Final

Description
***********

ADR for a declaration of assets (manifest) involved in the process
of greeting an entity.

Context
*******

- We need a way to describe the data involved in a greeting

Intent
******

- Ensure valid communication path to ``entity``

- Send ``entity`` message containing ``greeting``

[johnandersen777]

State transition, issue filing, estimating time to close issue, all have to do with having the complete mapping of inputs to problem (data flow). If we have an accurate mapping then we have a valid flow, we can create an estimate that we understand how we created the estimate because we have a complete description of the problem. See also: estimation of GSoC project time, estimation of time to complete best practices badging program activities, time to complete any issue, helps with prioritization of who in an org should work on what, when, to unblock others in the org. Related to builtree discussion.