port OCI module back from python_gardenlinux_cli

Author: yeoldegrove

README.md

@@ -4,33 +4,75 @@
 ![security check](https://github.com/gardenlinux/parse_features_lib/actions/workflows/bandit.yml/badge.svg)
 
 # Parse features lib
-This library helps you to work with the gardenlinux/features folder. It parses all info.yamls and builds a tree.
 
-Features (planned):
-* validate CNAMEs
-* validate info.yamls
-* Deduct dependencies from cname_base
+This library includes tooling to build and distribute [Garden Linux](https://github.com/gardenlinux/gardenlinux).
+
+Features:
+
+-   compare APT repositories
+-   parse features
+-   parse flavors
+-   push OCI artifacts to a registry
 
 ## Quickstart
+
+### Example: get a list of features for a given cname
+
 **Inclusion via poetry**:
 
-`parse_features_lib = { git  = "https://github.com/gardenlinux/parse_features_lib", rev="main" }`
+`gardenlinux = { git  = "https://github.com/gardenlinux/python_gardenlinux_lib", rev="0.6.0" }`
+
 ```python
-import parse_features_lib
+import gardenlinux.features as features
 
 if __name__ == "__main__":
-    # Step 1: parse the "features directory" and get the full graph containing all features
-    all_features = parse_features_lib.read_feature_files("features")
+    # Step 1: parse the "features directory" and get a list of features
+    features_list = features.filter_as_list("aws-gardener_prod")
+    print(features_list)
+```
+
+## Developer Documentation
+
+The library is documented with docstrings, which are used to generate the developer documentation available [here](https://gardenlinux.github.io/python-gardenlinux-lib/).
+
+## Push OCI artifacts to a registry
+
+this tool helps you to push oci artifacts.
 
-    # Step 2: supply desired features and get all their dependencies
-    dependencies = parse_features_lib.filter_graph(all_features, {"gardener", "_prod", "server", "ociExample"})
+### Installation
 
-    # Step 3: play with the retrieved data.
-    for feature, info in dependencies.nodes(data="content"):
-        if "oci_artifacts" in info:
-            print(feature, info["oci_artifacts"])
+```bash
+git clone https://github.com/gardenlinux/python-gardenlinux-lib.git
+mkdir venv
+python -m venv venv
+source venv/bin/activate.sh
+poetry install
+gl-oci --help
 ```
 
-## Developer Documentation
-The library is documented with docstrings, which are used to generate the developer documentation available [here](https://gardenlinux.github.io/python-gardenlinux-lib/). 
+### Usage
+
+The process to push a Gardenlinux build-output folder to an OCI registry is split into two steps: In the first step all files are pushed to the registry and a manifest that includes all those pushed files (layers) is created and pushed as well. An index entry that links to this manifest is created offline and written to a local file but not pushed to any index. This push to an index can be done in the second step where the local file containing the index entry is read and pushed to an index. The seperation into two steps was done because pushing of manifests takes long and writes to dedicated resources (possible to run in parallel). Updating the index on the other hand is quick but writes to a share resource (not possible to run in parallel). By splitting the process up into two steps it is possible to run the slow part in parallel and the quick part sequentially.
+
+#### 1. Push layers + manifest
+
+To push layers you have to supply the directory with the build outputs `--dir`. Also you have to supply cname (`--cname`), architecture `--arch` and version `--version` of the build. This information will be included in the manifest. You have to supply an endpoint where the artifacts shall be pushed to `--container`, for example `ghcr.io/gardenlinux/gardenlinux`. You can disable enforced HTTPS connections to your registry with `--insecure True`. You can supply `--cosign_file <filename>` if you want to have the hash saved in `<filename>`. This can be handy to read the hash later to sign the manifest with cosign. With `--manifest_file <filename>` you tell the program in which file to store the manifests index entry. This is the file that can be used in the next step to update the index. You can use the environment variable GL_CLI_REGISTRY_TOKEN to authenticate against the registry. Below is an example of a full program call of `push-manifest`
+
+```bash
+GL_CLI_REGISTRY_TOKEN=asdf123 gl-oci push-manifest --dir build-metal-gardener_prod --container ghcr.io/gardenlinux/gl-oci --arch amd64 --version 1592.1 --cname metal-gardener_prod --cosign_file digest --manifest_file oci_manifest_entry_metal.json
+```
 
+#### 2. Update index with manifest entry
+
+Parameters that are the same as for `push-manifest`:
+
+-   env-var `GL_CLI_REGISTRY_TOKEN`
+-   `--version`
+-   `--container`
+-   `--manifest-file` this time this parameter adjusts the manifest entry file to be read from instead of being written to
+
+A full example looks like this:
+
+```bash
+GL_CLI_REGISTRY_TOKEN=asdf123 gl-oci update-index --container ghcr.io/gardenlinux/gl-oci --version 1592.1 --manifest_file oci_manifest_entry_metal.json
+```

pyproject.toml

@@ -23,6 +23,7 @@ boto3 = "*"
 [tool.poetry.group.dev.dependencies]
 bandit = "^1.8.3"
 black = "^24.8.0"
+opencontainers = "^0.0.14"
 
 [tool.poetry.group.docs.dependencies]
 sphinx-rtd-theme = "^2.0.0"
@@ -31,6 +32,7 @@ sphinx-rtd-theme = "^2.0.0"
 gl-cname = "gardenlinux.features.cname_main:main"
 gl-features-parse = "gardenlinux.features.__main__:main"
 gl-flavors-parse = "gardenlinux.flavors.__main__:main"
+gl-oci = "gardenlinux.oci.__main__:main"
 flavors-parse = "gardenlinux.flavors.__main__:main"
 
 [tool.pytest.ini_options]

src/gardenlinux/constants.py

@@ -55,8 +55,19 @@
 
 # It is important that this list is sorted in descending length of the entries
 GL_MEDIA_TYPES = [
+    "secureboot.aws-efivars",
+    "secureboot.kek.auth",
     "gcpimage.tar.gz.log",
+    "secureboot.pk.auth",
+    "secureboot.kek.crt",
+    "secureboot.kek.der",
+    "secureboot.db.auth",
     "firecracker.tar.gz",
+    "secureboot.pk.crt",
+    "secureboot.pk.der",
+    "secureboot.db.crt",
+    "secureboot.db.der",
+    "secureboot.db.arn",
     "platform.test.log",
     "platform.test.xml",
     "gcpimage.tar.gz",
@@ -65,11 +76,15 @@
     "pxe.tar.gz.log",
     "root.squashfs",
     "manifest.log",
+    "squashfs.log",
     "release.log",
+    "vmlinuz.log",
+    "initrd.log",
     "pxe.tar.gz",
     "qcow2.log",
     "test-log",
     "boot.efi",
+    "squashfs",
     "manifest",
     "vmdk.log",
     "tar.log",
@@ -122,12 +137,30 @@
     "vhd.log": "application/io.gardenlinux.log",
     "ova.log": "application/io.gardenlinux.log",
     "vmlinuz": "application/io.gardenlinux.kernel",
+    "vmlinuz.log": "application/io.gardenlinux.log",
     "initrd": "application/io.gardenlinux.initrd",
+    "initrd.log": "application/io.gardenlinux.log",
     "root.squashfs": "application/io.gardenlinux.squashfs",
+    "squashfs": "application/io.gardenlinux.squashfs",
+    "squashfs.log": "application/io.gardenlinux.log",
     "boot.efi": "application/io.gardenlinux.efi",
     "platform.test.log": "application/io.gardenlinux.io.platform.test.log",
     "platform.test.xml": "application/io.gardenlinux.io.platform.test.xml",
     "chroot.test.log": "application/io.gardenlinux.io.chroot.test.log",
     "chroot.test.xml": "application/io.gardenlinux.io.chroot.test.xml",
     "oci.log": "application/io.gardenlinux.log",
+    "secureboot.pk.crt": "application/io.gardenlinux.cert.secureboot.pk.crt",
+    "secureboot.pk.der": "application/io.gardenlinux.cert.secureboot.pk.der",
+    "secureboot.pk.auth": "application/io.gardenlinux.cert.secureboot.pk.auth",
+    "secureboot.kek.crt": "application/io.gardenlinux.cert.secureboot.kek.crt",
+    "secureboot.kek.der": "application/io.gardenlinux.cert.secureboot.kek.der",
+    "secureboot.kek.auth": "application/io.gardenlinux.cert.secureboot.kek.auth",
+    "secureboot.db.crt": "application/io.gardenlinux.cert.secureboot.db.crt",
+    "secureboot.db.der": "application/io.gardenlinux.cert.secureboot.db.der",
+    "secureboot.db.auth": "application/io.gardenlinux.cert.secureboot.db.auth",
+    "secureboot.db.arn": "application/io.gardenlinux.cert.secureboot.db.arn",
+    "secureboot.aws-efivars": "application/io.gardenlinux.cert.secureboot.aws-efivars",
 }
+
+OCI_ANNOTATION_SIGNATURE_KEY = "io.gardenlinux.oci.signature"
+OCI_ANNOTATION_SIGNED_STRING_KEY = "io.gardenlinux.oci.signed-string"

src/gardenlinux/features/__main__.py

@@ -212,5 +212,29 @@ def sort_subset(input_set, order_list):
     return [item for item in order_list if item in input_set]
 
 
+def get_flavor_from_cname(cname: str, get_arch: bool = True) -> str:
+    """
+    Extracts the flavor from a canonical name.
+
+    :param str cname: Canonical name of an image
+    :param bool get_arch: Whether to include the architecture in the flavor
+    :return: Flavor string
+    """
+
+    # cname:
+    # azure-gardener_prod_tpm2_trustedboot-amd64-1312.2-80ffcc87
+    # transform to flavor:
+    # azure-gardener_prod_tpm2_trustedboot-amd64
+
+    platform = cname.split("-")[0]
+    features = cname.split("-")[1:-1]
+    arch = cname.split("-")[-1]
+
+    if get_arch:
+        return f"{platform}-{features}-{arch}"
+    else:
+        return f"{platform}-{features}"
+
+
 if __name__ == "__main__":
     main()

src/gardenlinux/oci/__init__.py

@@ -0,0 +1,5 @@
+# -*- coding: utf-8 -*-
+
+# from .__main__ import Cli
+
+# __all__ = ["Cli"]

src/gardenlinux/oci/__main__.py

@@ -0,0 +1,128 @@
+#!/usr/bin/env python3
+
+import os
+import click
+
+from pygments.lexer import default
+
+from .registry import GlociRegistry
+
+
+@click.group()
+def cli():
+    pass
+
+
+@cli.command()
+@click.option(
+    "--container",
+    required=True,
+    type=click.Path(),
+    help="Container Name",
+)
+@click.option(
+    "--version",
+    required=True,
+    type=click.Path(),
+    help="Version of image",
+)
+@click.option(
+    "--commit",
+    required=False,
+    type=click.Path(),
+    default=None,
+    help="Commit of image",
+)
+@click.option(
+    "--arch",
+    required=True,
+    type=click.Path(),
+    help="Target Image CPU Architecture",
+)
+@click.option(
+    "--cname", required=True, type=click.Path(), help="Canonical Name of Image"
+)
+@click.option("--dir", "directory", required=True, help="path to the build artifacts")
+@click.option(
+    "--cosign_file",
+    required=False,
+    help="A file where the pushed manifests digests is written to. The content can be used by an external tool (e.g. cosign) to sign the manifests contents",
+)
+@click.option(
+    "--manifest_file",
+    default="manifests/manifest.json",
+    help="A file where the index entry for the pushed manifest is written to.",
+)
+@click.option(
+    "--insecure",
+    default=False,
+    help="Use HTTP to communicate with the registry",
+)
+def push_manifest(
+    container,
+    version,
+    commit,
+    arch,
+    cname,
+    directory,
+    cosign_file,
+    manifest_file,
+    insecure,
+):
+    """push artifacts from a dir to a registry, get the index-entry for the manifest in return"""
+    container_name = f"{container}:{version}"
+    registry = GlociRegistry(
+        container_name=container_name,
+        token=os.getenv("GL_CLI_REGISTRY_TOKEN"),
+        insecure=insecure,
+    )
+    digest = registry.push_from_dir(
+        arch, version, cname, directory, manifest_file, commit=commit
+    )
+    if cosign_file:
+        print(digest, file=open(cosign_file, "w"))
+
+
+@cli.command()
+@click.option(
+    "--container",
+    "container",
+    required=True,
+    type=click.Path(),
+    help="Container Name",
+)
+@click.option(
+    "--version",
+    "version",
+    required=True,
+    type=click.Path(),
+    help="Version of image",
+)
+@click.option(
+    "--manifest_folder",
+    default="manifests",
+    help="A folder where the index entries are read from.",
+)
+@click.option(
+    "--insecure",
+    default=False,
+    help="Use HTTP to communicate with the registry",
+)
+def update_index(container, version, manifest_folder, insecure):
+    """push a index entry from a list of files to an index"""
+    container_name = f"{container}:{version}"
+    registry = GlociRegistry(
+        container_name=container_name,
+        token=os.getenv("GL_CLI_REGISTRY_TOKEN"),
+        insecure=insecure,
+    )
+    registry.update_index(manifest_folder)
+
+
+def main():
+    """Entry point for the gl-oci command."""
+    cli()
+
+
+if __name__ == "__main__":
+    cli()

src/gardenlinux/oci/crypto.py

@@ -0,0 +1,16 @@
+import hashlib
+
+
+def verify_sha256(checksum: str, data: bytes):
+    data_checksum = f"sha256:{hashlib.sha256(data).hexdigest()}"
+    if checksum != data_checksum:
+        raise ValueError(f"Invalid checksum. {checksum} != {data_checksum}")
+
+
+def calculate_sha256(file_path: str) -> str:
+    """Calculate the SHA256 checksum of a file."""
+    sha256_hash = hashlib.sha256()
+    with open(file_path, "rb") as f:
+        for byte_block in iter(lambda: f.read(4096), b""):
+            sha256_hash.update(byte_block)
+    return sha256_hash.hexdigest()

src/gardenlinux/oci/defaults.py

@@ -0,0 +1,2 @@
+annotation_signature_key = "io.gardenlinux.oci.signature"
+annotation_signed_string_key = "io.gardenlinux.oci.signed-string"

src/gardenlinux/oci/helper.py

@@ -0,0 +1,24 @@
+import json
+import os
+import re
+
+
+def write_dict_to_json_file(input, output_path):
+    if os.path.exists(output_path):
+        raise ValueError(f"{output_path} already exists")
+    with open(output_path, "w") as fp:
+        json.dump(input, fp)
+
+
+def get_uri_for_digest(uri, digest):
+    """
+    Given a URI for an image, return a URI for the related digest.
+
+    URI may be in any of the following forms:
+
+        ghcr.io/homebrew/core/hello
+        ghcr.io/homebrew/core/hello:2.10
+        ghcr.io/homebrew/core/hello@sha256:ff81...47a
+    """
+    base_uri = re.split(r"[@:]", uri, maxsplit=1)[0]
+    return f"{base_uri}@{digest}"