diff --git a/METADATA.md b/METADATA.md new file mode 100644 index 0000000000..9323b48df9 --- /dev/null +++ b/METADATA.md @@ -0,0 +1,86 @@ +# Metadata + +Metadata files provide information that clients can use to make update decisions. Different metadata files provide different information. The various metadata files are signed by different roles. The concept of roles allows TUF to only trust information that a role is trusted to provide. + +The signed metadata files always include the time they were created and their expiration dates. This ensures that outdated metadata will be detected and that clients can refuse to accept metadata older than that which they've already seen. + +All TUF metadata uses a subset of the JSON object format. When calculating the digest of an object, we use the [Canonical JSON](http://wiki.laptop.org/go/Canonical_JSON) format. Implementation-level detail about the metadata can be found in the [spec](docs/tuf-spec.txt). + +There are four required top-level roles and one optional top-level role, each with their own metadata file. + +Required: + +* Root +* Targets +* Snapshot +* Timestamp + +Optional: + +* Mirrors + +There may also be any number of delegated target roles. + +## Root Metadata (root.json) + +Signed by: Root role. + +Specifies the other top-level roles. When specifying these roles, the trusted keys for each role are listed along with the minimum number of those keys which are required to sign the role's metadata. We call this number the signature threshold. + +Note: Metadata content and name out-of-date. +See [example](http://mirror1.poly.edu/test-pypi/metadata/root.txt). + +## Targets Metadata (targets.json) + +Signed by: Targets role. + +The targets.json metadata file lists hashes and sizes of target files. Target files are the actual files that clients are intending to download (for example, the software updates they are trying to obtain). + +This file can optionally define other roles to which it delegates trust. Delegating trust means that the delegated role is trusted for some or all of the target files available from the repository. When delegated roles are specified, they are specified in a similar way to how the Root role specifies the top-level roles: the trusted keys and signature threshold for each role is given. Additionally, one or more patterns are specified which indicate the target file paths for which clients should trust each delegated role. + +Note: Metadata content and name out-of-date. +See [example](http://mirror1.poly.edu/test-pypi/metadata/targets.txt). + +## Delegated Targets Metadata (targets/foo.json) + +Signed by: A delegated targets role. + +The metadata files provided by delegated targets roles follow exactly the same format as the metadata file provided by the top-level Targets role. + +The location of the metadata file for each delegated target role is based on the delegation ancestry of the role. If the top-level Targets role defines a role named foo, then the delegated target role's full name would be targets/foo and its metadata file will be available on the repository at the path targets/foo.json (this is relative to the base directory from which all metadata is available). This path is just the full name of the role followed by a file extension. + +If this delegated role foo further delegates to a role bar, then the result is a role whose full name is targets/foo/bar and whose signed metadata file is made available on the repository at targets/foo/bar.json. + +Note: Metadata content and name out-of-date. +See [example](http://mirror1.poly.edu/test-pypi/metadata/targets/unclaimed.txt). + +## snapshot Metadata (snapshot.json) + +Signed by: Snapshot role. + +The snapshot.json metadata file lists hashes and sizes of all metadata files other than timestamp.json. This file ensures that clients will see a consistent view of the files on the repository. That is, metadata files (and thus target file) that existed on the repository at different times cannot be combined and presented to clients by an attacker. + +Note: Metadata content and name out-of-date. +​See [example](http://mirror1.poly.edu/test-pypi/metadata/release.txt). + +## Timestamp Metadata (timestamp.json) + +Signed by: Timestamp role. + +The timestamp.json metadata file lists the hashes and size of the snapshot.json file. This is the first and potentially only file that needs to be downloaded when clients poll for the existence of updates. This file is frequently resigned and has a short expiration date, thus allowing clients to quickly detect if they are being prevented from obtaining the most recent metadata. An online key is generally used to automatically resign this file at regular intervals. + +There are two primary reasons why the timestamp.json file doesn't contain all of the information that the snapshot.json file does. + +* The timestamp.json file is downloaded very frequently and so should be kept as small as possible, especially considering that the snapshot.json file grows in size in proportion to the number of delegated target roles. +* As the Timestamp role's key is an online key and thus at high risk, separate keys should be used for signing the snapshot.json metadata file so that the Snapshot role's keys can be kept offline and thus more secure. + +Note: Metadata content and name out-of-date. +See [example](http://mirror1.poly.edu/test-pypi/metadata/timestamp.txt). + +## Mirrors Metadata (mirrors.json) + +Optionally signed by: Mirrors role. + +The mirrors.json file provides an optional way to provide mirror list updates to TUF clients. Mirror lists can alternatively be provided directly by the software update system and obtained in any way the system sees fit, including being hard coded if that is what an applications wants to do. + +No example available. At the time of writing, this hasn't been implemented in TUF. Currently mirrors are specified by the client code. diff --git a/README.md b/README.md index 410070307e..5f57b33034 100644 --- a/README.md +++ b/README.md @@ -15,9 +15,8 @@ completely new software. Three major classes of software update systems are: -* **Application updaters** which are used by applications use to update -themselves. For example, Firefox updates itself through its own application -updater. +* **Application updaters** which are used by applications to update themselves. +For example, Firefox updates itself through its own application updater. * **Library package managers** such as those offered by many programming languages for installing additional libraries. These are systems such as @@ -30,8 +29,63 @@ YaST are examples of these. ## Our Approach There are literally thousands of different software update systems in common -use today. (In fact the average Windows user has about two dozen different +use today. (In fact the average Windows user has about [two dozen](http://secunia.com/gfx/pdf/Secunia_RSA_Software_Portfolio_Security_Exposure.pdf) different software updaters on their machine!) We are building a library that can be universally (and in most cases transparently) used to secure software update systems. + +## Overview + +At the highest level, TUF simply provides applications with a secure method of obtaining files and knowing when new versions of files are available. We call these files, the ones that are supposed to be downloaded, "target files". The most common need for these abilities is in software update systems and that's what we had in mind when creating TUF. + +On the surface, this all sounds simple. Securely obtaining updates just means: + +* Knowing when an update exists. +* Downloading the updated file. + +The problem is that this is only simple when there are no malicious parties involved. If an attacker is trying to interfere with these seemingly simple steps, there is plenty they can do. + +## Background + +Let's assume you take the approach that most systems do (at least, the ones that even try to be secure). You download both the file you want and a cryptographic signature of the file. You already know which key you trust to make the signature. You check that the signature is correct and was made by this trusted key. All seems well, right? Wrong. You are still at risk in many ways, including: + +* An attacker keeps giving you the same file, so you never realize there is an update. +* An attacker gives you an older, insecure version of a file that you already have, so you download that one and blindly use it thinking it's newer. +* An attacker gives you a newer version of a file you have but it's not the newest one. It's newer to you, but it may be insecure and exploitable by the attacker. +* An attacker compromises the key used to sign these files and now you download a malicious file that is properly signed. + +These are just some of the attacks software update systems are vulnerable to when only using signed files. +See [Security](SECURITY.md) for a full list of attacks and updater weaknesses TUF is designed to prevent. + +The following papers provide detailed information on securing software updater systems, TUF's design and implementation details, attacks on package managers, and package management security: + +* [Survivable Key Compromise in Software Update Systems](docs/papers/survivable-key-compromise-ccs2010.pdf?raw=true) + +* [A Look In the Mirror: Attacks on Package Managers](docs/papers/package-management-security-tr08-02.pdf?raw=true) + +* [Package Management Security](docs/papers/attacks-on-package-managers-ccs2008.pdf?raw=true) + + +##What TUF Does + +In order to securely download and verify target files, TUF requires a few extra files to exist on a repository. These are called metadata files. TUF metadata files contain additional information, including information about which keys are trusted, the cryptographic hashes of files, signatures on the metadata, metadata version numbers, and the date after which the metadata should be considered expired. + +When a software update system using TUF wants to check for updates, it asks TUF to do the work. That is, your software update system never has to deal with this additional metadata or understand what's going on underneath. If TUF reports back that there are updates available, your software update system can then ask TUF to download these files. TUF downloads them and checks them against the TUF metadata that it also downloads from the repository. If the downloaded target files are trustworthy, TUF hands them over to your software update system. +See [Metadata](METADATA.md) for more information and examples. + +TUF specification document is also available: + +* [The Update Framework Specification](docs/tuf-spec.txt?raw=true) + + + +##Using TUF + +TUF has four major classes of users: clients, for whom TUF is largely transparent; mirrors, who will (in most cases) have nothing at all to do with TUF; upstream servers, who will largely be responsible for care and feeding of repositories; and integrators, who do the work of putting TUF into existing projects. + +* [Creating a Repository](tuf/README.md) + +* [Low-level Integration](tuf/client/README.md) + +* [High-level Integration](tuf/interposition/README.md) diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 0000000000..be58249d2e --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,69 @@ +#Security + +Generally, a software update system is secure if it can be sure that it knows about the latest available updates in a timely manner, any files it downloads are the correct files, and no harm results from checking or downloading files. The details of making this happen are complicated by various attacks that can be carried out against software update systems. + +## Attacks and Weaknesses + +The following are some of the known attacks on software update systems, including weaknesses that make attacks possible. In order to design a secure software update framework, these need to be understood and protected against. Some of these issues are or can be related depending on the design and implementation of a software update system. + +* **Arbitrary software installation**. An attacker installs anything they want on the client system. That is, an attacker can provide arbitrary files in response to download requests and the files will not be detected as illegitimate. + +* **Rollback attacks**. An attacker presents a software update system with older files than those the client has already seen, causing the client to use files older than those the client knows about. + +* **Indefinite freeze attacks**. An attacker continues to present a software update system with the same files the client has already seen. The result is that the client does not know that new files are available. + +* **Endless data attacks**. An attacker responds to a file download request with an endless stream of data, causing harm to clients (e.g. a disk partition filling up or memory exhaustion). + +* **Slow retrieval attacks**. An attacker responds to clients with a very slow stream of data that essentially results in the client never continuing the update process. + +* **Extraneous dependencies attacks**. An attacker indicates to clients that in order to install the software they wanted, they also need to install unrelated software. This unrelated software can be from a trusted source but may have known vulnerabilities that are exploitable by the attacker. + +* **Mix-and-match attacks**. An attacker presents clients with a view of a repository that includes files that never existed together on the repository at the same time. This can result in, for example, outdated versions of dependencies being installed. + +* **Wrong software installation**. An attacker provides a client with a trusted file that is not the one the client wanted. + +* **Malicious mirrors preventing updates**. An attacker in control of one repository mirror is able to prevent users from obtaining updates from other, good mirrors. + +* **Vulnerability to key compromises**. At attacker who is able to compromise a single key or less than a given threshold of keys can compromise clients. This includes relying on a single online key (such as only being protected by SSL) or a single offline key (such as most software update systems use to sign files). + +##Design Concepts + +The design and implementation of TUF aims to be secure against all of the above attacks. A few general ideas drive much of the security of TUF. + +For the details of how TUF conveys the information discussed below, see the [Metadata documentation](METADATA.md). + +## Trust + +Trusting downloaded files really means trusting that the files were provided by some trusted party. Two frequently overlooked aspects of trust in a secure software update system are: + +* Trust should not be granted forever. Trust should expire if it is not renewed. +* Compartmentalized trust. A trusted party should only be trusted for files that it is supposed to provide. + +## Mitigated Key Risk + +Cryptographic signatures are a necessary component in securing a software update system. The safety of the keys that are used to create these signatures affects the security of clients. Rather than incorrectly assume that private keys are always safe from compromise, a secure software update system must strive to keep clients as safe as possible even when compromises happen. + +Keeping clients safe despite dangers to keys involves: + +* Fast and secure key replacement and revocation. +* Minimally trusting keys that are at high risk. Keys that are kept online or used in an automated fashion shouldn't pose immediate risk to clients if compromised. +* Supporting the use of multiple keys with threshold/quorum signatures trust. + +## Integrity + +File integrity is important both with respect to single files as well as collections of files. It's fairly obvious that clients must verify that individual downloaded files are correct. Not as obvious but still very important is the need for clients to be certain that their entire view of a repository is correct. For example, if a trusted party is providing two files, a software update system should see the latest versions of both of those files, not just one of the files and not versions of the two files that were never provided together. + +## Freshness + +As software updates often fix security bugs, it is important that software update systems be able to obtain the latest versions of files that are available. An attacker may want to trick a client into installing outdated versions of software or even just convince a client that no updates are available. + +Ensuring freshness means to: + +* Never accept files older than those that have been seen previously. +* Recognize when there may be a problem obtaining updates. + +Note that it won't always be possible for a client to successfully update if an attacker is responding to their requests. However, a client should be able to recognize that updates may exist that they haven't been able to obtain. + +## Implementation Safety + +In addition to a secure design, TUF also works to be secure against implementation vulnerabilities including those common to software update systems. In some cases this is assisted by the inclusion of additional information in metadata. For example, knowing the expected size of a target file that is to be downloaded allows TUF to limit the amount of data it will download when retrieving the file. As a result, TUF is secure against endless data attacks (discussed above). diff --git a/docs/images/repository_tool-diagram.png b/docs/images/repository_tool-diagram.png new file mode 100644 index 0000000000..36d444f944 Binary files /dev/null and b/docs/images/repository_tool-diagram.png differ diff --git a/docs/latex/tuf-client-spec.pdf.old b/docs/latex/tuf-client-spec.pdf.old new file mode 100644 index 0000000000..4228832d12 Binary files /dev/null and b/docs/latex/tuf-client-spec.pdf.old differ diff --git a/docs/latex/tuf-client-spec.tex b/docs/latex/tuf-client-spec.tex index a8148e1edf..7acc33f0a4 100644 --- a/docs/latex/tuf-client-spec.tex +++ b/docs/latex/tuf-client-spec.tex @@ -1,8 +1,37 @@ % tuf_client_spec.tex -\documentclass{letter} -\usepackage{listings} +% This document has been deprecated. We may later update and include it in +% the supported documentation. +\documentclass{article} \setlength\parindent{0pt} -\lstset{frameshape={ynn}{nny}{nnn}{yyn}, showstringspaces=false, basicstyle=\ttfamily} +\usepackage{listings} +\usepackage{hyperref} +\usepackage{color} +\usepackage{textcomp} +\definecolor{listinggray}{gray}{0.9} +\definecolor{lbcolor}{rgb}{0.9,0.9,0.9} +\lstset{ + backgroundcolor=\color{lbcolor}, + tabsize=4, + rulecolor=, + language=matlab, + basicstyle=\scriptsize, + upquote=true, + aboveskip={1.5\baselineskip}, + columns=fixed, + showstringspaces=false, + extendedchars=true, + breaklines=true, + prebreak = \raisebox{0ex}[0ex][0ex]{\ensuremath{\hookleftarrow}}, + frame=single, + showtabs=false, + showspaces=false, + showstringspaces=false, + identifierstyle=\ttfamily, + keywordstyle=\color[rgb]{0,0,1}, + commentstyle=\color[rgb]{0.133,0.545,0.133}, + stringstyle=\color[rgb]{0.627,0.126,0.941}, +} + \begin{document} %--------------------------------- Header -------------------------------------- @@ -44,7 +73,23 @@ \section{Example} \subsection{Setting up the Repository} You'll need to run the following steps to set the stage: -\lstinputlisting[language=csh]{tufsetup.sh} + +\begin{lstlisting} +#! /bin/sh + +# create the relevant directories +mkdir tufdemo +cd tufdemo +mkdir demorepo +mkdir demoproject + +# add a file to the project +echo "#! /usr/bin/env python" > demoproject/helloworld.py +echo "print 'hello, world!'" >> demoproject/helloworld.py + +# run the quickstart script +quickstart.py -t 1 -k keystore -l demorepo -r demoproject +\end{lstlisting} This will prompt you for a password for your keystore and an expiration date. Choosing your expiration date is something of a balancing act: on the one hand, @@ -72,7 +117,14 @@ \subsection{Setting up the Client} a mechanism with which to perform the initial installation of our demo project's metadata. To do that, open up another terminal and run the following: -\lstinputlisting[language=csh]{tufclientsetup.sh} +\begin{lstlisting} +#! /bin/sh + +mkdir democlient +cp -r demorepo/meta democlient/cur +cp -r democlient/cur democlient/prev +\end{lstlisting} + Once we've installed our metadata, getting the software is a simple matter of running the demonstration client, found with TUF's source at diff --git a/docs/tuf-server-spec.pdf b/docs/latex/tuf-server-spec.pdf.old similarity index 59% rename from docs/tuf-server-spec.pdf rename to docs/latex/tuf-server-spec.pdf.old index e1633ba6f6..bdb4893857 100644 Binary files a/docs/tuf-server-spec.pdf and b/docs/latex/tuf-server-spec.pdf.old differ diff --git a/docs/latex/tuf-server-spec.tex b/docs/latex/tuf-server-spec.tex index 104ec8a263..6bf1717d7d 100644 --- a/docs/latex/tuf-server-spec.tex +++ b/docs/latex/tuf-server-spec.tex @@ -1,8 +1,37 @@ -% tuf_repo_spec.tex +% tuf_server_spec.tex +% This document has been deprecated. We may later update and include it in +% the supported documentation. \documentclass{article} \setlength\parindent{0pt} \usepackage{listings} -\lstset{frameshape={ynn}{nny}{nnn}{yyn}, showstringspaces=false, basicstyle=\ttfamily} +\usepackage{hyperref} +\usepackage{color} +\usepackage{textcomp} +\definecolor{listinggray}{gray}{0.9} +\definecolor{lbcolor}{rgb}{0.9,0.9,0.9} +\lstset{ + backgroundcolor=\color{lbcolor}, + tabsize=4, + rulecolor=, + language=matlab, + basicstyle=\scriptsize, + upquote=true, + aboveskip={1.5\baselineskip}, + columns=fixed, + showstringspaces=false, + extendedchars=true, + breaklines=true, + prebreak = \raisebox{0ex}[0ex][0ex]{\ensuremath{\hookleftarrow}}, + frame=single, + showtabs=false, + showspaces=false, + showstringspaces=false, + identifierstyle=\ttfamily, + keywordstyle=\color[rgb]{0,0,1}, + commentstyle=\color[rgb]{0.133,0.545,0.133}, + stringstyle=\color[rgb]{0.627,0.126,0.941}, +} + \begin{document} %--------------------------------- Header -------------------------------------- diff --git a/docs/tuf-client-spec.pdf b/docs/tuf-client-spec.pdf deleted file mode 100644 index 2ba1659989..0000000000 Binary files a/docs/tuf-client-spec.pdf and /dev/null differ diff --git a/docs/tuf-spec.txt b/docs/tuf-spec.txt index 7437ade5e7..e54069a4f2 100644 --- a/docs/tuf-spec.txt +++ b/docs/tuf-spec.txt @@ -1,4 +1,4 @@ - TUF: The Update Framework + The Update Framework Specification 1. Introduction @@ -220,7 +220,7 @@ There are four fundamental top-level roles in the framework: - Root role - Targets role - - Release role + - Snapshot role - Timestamp role There is also one optional top-level role: @@ -262,9 +262,9 @@ Delegated trust can be revoked at any time by the delegating role signing new metadata that indicates the delegated role is no longer trusted. -2.1.3 Release role +2.1.3 Snapshot role - The release role signs a metadata file that provides information about the + The snapshot role signs a metadata file that provides information about the latest version of all of the other metadata on the repository (excluding the timestamp file, discussed below). This information allows clients to know which metadata files have been updated and also prevents mix-and-match @@ -274,7 +274,7 @@ To prevent an adversary from replaying an out-of-date signed metadata file whose signature has not yet expired, an automated process periodically signs - a timestamped statement containing the the hash of the release file. Even + a timestamped statement containing the the hash of the snapshot file. Even though this timestamp key must be kept online, the risk posed to clients by compromise of this key is minimal. @@ -348,28 +348,28 @@ defined. The following are the metadata files of top-level roles relative to the base URL of metadata available from a given repository mirror. - /root.txt + /root.json Signed by the root keys; specifies trusted keys for the other top-level roles. - /release.txt + /snapshot.json - Signed by the release role's keys. Lists hashes and sizes of all - metadata files other than timestamp.txt. + Signed by the snapshot role's keys. Lists hashes and sizes of all + metadata files other than timestamp.json. - /targets.txt + /targets.json Signed by the target role's keys. Lists hashes and sizes of target files. - /timestamp.txt + /timestamp.json Signed by the timestamp role's keys. Lists hashes and size of the - release file. This is the first and potentially only file that needs + snapshot file. This is the first and potentially only file that needs to be downloaded when clients poll for the existence of updates. - /mirrors.txt (optional) + /mirrors.json (optional) Signed by the mirrors role's keys. Lists information about available mirrors and the content available from each mirror. @@ -378,7 +378,7 @@ any metadata files in compressed (e.g. gzip'd) format. In doing so, the filename of the compressed file should be the same as the original with the addition of the file name extension for the compression type (e.g. - release.txt.gz). The original (uncompressed) file should always be made + snapshot.json.gz). The original (uncompressed) file should always be made available, as well. 3.1.2.1 Metadata files for targets delegation @@ -386,13 +386,13 @@ When the targets role delegates trust to other roles, each delegated role provides one signed metadata file. This file is located at: - /targets/DELEGATED_ROLE.txt + /targets/DELEGATED_ROLE.json where DELEGATED_ROLE is the name of the delegated role that has been - specified in targets.txt. If this role further delegates trust to a role + specified in targets.json. If this role further delegates trust to a role named ANOTHER_ROLE, that role's signed metadata file is made available at: - /targets/DELEGATED_ROLE/ANOTHER_ROLE.txt + /targets/DELEGATED_ROLE/ANOTHER_ROLE.json 4. Document formats @@ -410,7 +410,8 @@ 4.2. File formats: general principles - All signed files are of the format: + All signed metadata files have the format: + { "signed" : ROLE, "signatures" : [ { "keyid" : KEYID, @@ -424,12 +425,20 @@ METHOD is the key signing method used to generate the signature. SIGNATURE is a signature of the canonical JSON form of ROLE. - We define one signing method at present: - "evp" : An interface to OpenSSL's EVP functions. - - All times are given as strings of the format "YYYY-MM-DD HH:MM:SS UTC". - - All keys are of the format: + The current Python implementation of TUF defines two signing methods, + although TUF is not restricted to any particular key signing method, + key type, or cryptographic library: + + "RSASSA-PSS" : RSA Probabilistic signature scheme with appendix. + + "ed25519" : Elliptic curve digital signature algorithm based on Twisted + Edwards curves. + + RSASSA-PSS: http://tools.ietf.org/html/rfc3447#page-29 + ed25519: http://ed25519.cr.yp.to/ + + All keys have the format: + { "keytype" : KEYTYPE, "keyval" : KEYVAL } @@ -437,27 +446,48 @@ used to sign documents. The type determines the interpretation of KEYVAL. - The KEYID of a key is the hexdigest of the SHA-256 hash of the - canonical JSON form of the key. - - We define one keytype at present: 'rsa'. Its format is: + We define two keytypes at present: 'rsa' and 'ed25519'. + + The 'rsa' format is: + { "keytype" : "rsa", "keyval" : { "public" : PUBLIC, "private" : PRIVATE } } where PUBLIC and PRIVATE are in PEM format and are strings. All RSA keys - must be at least 2048 bits long. + must be at least 2048 bits. + + The 'ed25519' format is: + + { "keytype" : "ed25519", + "keyval" : { "public" : PUBLIC, + "private" : PRIVATE } + } + + where PUBLIC and PRIVATE are both 32-byte strings. + + Metadata does not include the private portion of the key object: + + { "keytype" : "rsa", + "keyval" : { "public" : PUBLIC} + } + + The KEYID of a key is the hexdigest of the SHA-256 hash of the + canonical JSON form of the key, where the "private" object key is excluded. + + All times are given as strings of the format "YYYY-MM-DD HH:MM:SS UTC". + -4.3. File formats: root.txt +4.3. File formats: root.json - The root.txt file is signed by the root role's keys. It indicates + The root.json file is signed by the root role's keys. It indicates which keys are authorized for all top-level roles, including the root role itself. Revocation and replacement of top-level role keys, including for the root role, is done by changing the keys listed for the roles in this file. - The format of root.txt is as follows: + The format of root.json is as follows: { "_type" : "Root", "version" : VERSION, @@ -478,8 +508,8 @@ EXPIRES determines when metadata should be considered expired and no longer trusted by clients. Clients MUST NOT trust an expired file. - A ROLE is one of "root", "release", "targets", "timestamp", or "mirrors". - A role for each of "root", "release", "timestamp", and "targets" MUST be + A ROLE is one of "root", "snapshot", "targets", "timestamp", or "mirrors". + A role for each of "root", "snapshot", "timestamp", and "targets" MUST be specified in the key list. The role of "mirror" is optional. If not specified, the mirror list will not need to be signed if mirror lists are being used. @@ -493,15 +523,15 @@ whose signatures are required in order to consider a file as being properly signed by that role. -4.4. File formats: release.txt +4.4. File formats: snapshot.json - The release.txt file is signed by the release role. It lists hashes and - sizes of all metadata on the repository, excluding timestamp.txt and - mirrors.txt. + The snapshot.json file is signed by the snapshot role. It lists hashes and + sizes of all metadata on the repository, excluding timestamp.json and + mirrors.json. - The format of release.txt is as follows: + The format of snapshot.json is as follows: - { "_type" : "Release", + { "_type" : "Snapshot", "version" : VERSION, "expires" : EXPIRES, "meta" : METAFILES @@ -519,9 +549,9 @@ METAPATH is the the metadata file's path on the repository relative to the metadata base URL. -4.5. File formats: targets.txt and delegated target roles +4.5. File formats: targets.json and delegated target roles - The format of targets.txt is as follows: + The format of targets.json is as follows: { "_type" : "Targets", "version" : VERSION, @@ -593,8 +623,8 @@ A path to a directory is used to indicate all possible targets sharing that directory as a prefix; e.g. if the directory is "targets/A", then targets - which match that directory include "targets/A/B.txt" and - "targets/A/B/C.txt". + which match that directory include "targets/A/B.json" and + "targets/A/B/C.json". We are currently investigating a few "priority tag" schemes to resolve conflicts between delegated roles that share responsibility for overlapping @@ -614,9 +644,9 @@ 08-04)[https://isis.poly.edu/~jcappos/papers/cappos_stork_dissertation_08.pdf]. The metadata files for delegated target roles has the same format as the - top-level targets.txt metadata file. + top-level targets.json metadata file. -4.6. File formats: timestamp.txt +4.6. File formats: timestamp.json The timestamp file is signed by a timestamp key. It indicates the latest versions of other files and is frequently resigned to limit the @@ -634,17 +664,17 @@ "meta" : METAFILES } - METAFILES is the same is described for the release.txt file. In the case of - the timestamp.txt file, this will commonly only include a description of the - release.txt file. + METAFILES is the same is described for the snapshot.json file. In the case of + the timestamp.json file, this will commonly only include a description of the + snapshot.json file. -4.7. File formats: mirrors.txt +4.7. File formats: mirrors.json - The mirrors.txt file is signed by the mirrors role. It indicates which + The mirrors.json file is signed by the mirrors role. It indicates which mirrors are active and believed to be mirroring specific parts of the repository. - The format of mirrors.txt is as follows: + The format of mirrors.json is as follows: { "_type" : "Mirrorlist", "version" : VERSION, @@ -680,24 +710,25 @@ 5.1. The client application - Note: At any point in the following process there is a problem (e.g. only - expired metadata can be retrieved), the software update system using the - framework must decide how to proceed. + Note: If at any point in the following process there is a problem (e.g., only + expired metadata can be retrieved), the Root file is downloaded and the process + starts over. Optionally, the software update system using the framework can + decide how to proceed rather than automatically downloading a new Root file. The client code instructs the framework to check for updates. The framework - downloads the timestamp.txt file from a mirror and checks that the file is + downloads the timestamp.json file from a mirror and checks that the file is properly signed by the timestamp role, is not expired, and is not older than - the last timestamp.txt file retrieved. If the timestamp file lists the same - release.txt file as was previously seen, the client code is informed that no + the last timestamp.json file retrieved. If the timestamp file lists the same + snapshot.json file as was previously seen, the client code is informed that no updates are available and the update checking process stops. - If the release.txt file has changed, the framework downloads the file and - verifies that it is properly signed by the release role, is not expired, has - a newer timestamp than the last release.txt file seen, and matches the - description (hashes and size) in the timestamp.txt file. The framework then - checks which metadata files listed in release.txt differ from those - described in the last release.txt file the framework had seen. If the - root.txt file has changed, the framework updates this (following the same + If the snapshot.json file has changed, the framework downloads the file and + verifies that it is properly signed by the snapshot role, is not expired, has + a newer timestamp than the last snapshot.json file seen, and matches the + description (hashes and size) in the timestamp.json file. The framework then + checks which metadata files listed in snapshot.json differ from those + described in the last snapshot.json file the framework had seen. If the + root.json file has changed, the framework updates this (following the same security measures as with the other files) and starts the process over. If any other metadata files have changed, the framework downloads and checks those. @@ -715,8 +746,8 @@ 6. Usage - See https://www.updateframework.com/ for discussion of recommended usage in - various situations. + See http://www.theupdateframework.com/ for discussion of recommended usage + in various situations. 6.1. Key management and migration @@ -725,12 +756,12 @@ machine, in special-purpose hardware, etc.). To replace a compromised root key or any other top-level role key, the root - role signs a new root.txt file that lists the updated trusted keys for the - role. When replacing root keys, an application will sign the new root.txt + role signs a new root.json file that lists the updated trusted keys for the + role. When replacing root keys, an application will sign the new root.json file with both the new and old root keys until all clients are known to have - obtained the new root.txt file (a safe assumption is that this will be a + obtained the new root.json file (a safe assumption is that this will be a very long time or never). There is no risk posed by continuing to sign the - root.txt file with revoked keys as once clients have updated they no longer + root.json file with revoked keys as once clients have updated they no longer trust the revoked key. This is only to ensure outdated clients remain able to update. @@ -748,7 +779,7 @@ targets, then should clients read metadata while the same metadata is being written to, they would effectively see denial-of-service attacks. Therefore, the repository needs to be careful about how it writes metadata - and targets. The high-level idea of the solution is that each release will + and targets. The high-level idea of the solution is that each snapshot will be contained in a so-called consistent snapshot. If a client is reading from one consistent snapshot, then the repository is free to write another consistent snapshot without interrupting that client. For more reasons on @@ -772,12 +803,12 @@ from the selection of a digest (which includes the name of the cryptographic function) from all digests in the referred file. - Additionally, the timestamp metadata (timestamp.txt) should also be written + Additionally, the timestamp metadata (timestamp.json) should also be written to disk whenever it is updated. It is optional for an implementation to - write identical copies at timestamp.digest.txt for record-keeping purposes, + write identical copies at digest.timestamp.json for record-keeping purposes, because a cryptographic hash of the timestamp metadata is usually not - known in advance. The same step applies to the root metadata (root.txt), - although an implementation must write both root.txt and root.digest.txt + known in advance. The same step applies to the root metadata (root.json), + although an implementation must write both root.json and digest.root.json because it is possible to download root metadata both with and without known hashes. These steps are required because these are the only metadata files that may be requested without known hashes. @@ -791,7 +822,7 @@ it is updated. In the next subsection, we will see how clients will reproduce the name of the intended file. - Finally, the root metadata should write the Boolean "consistent_snapshots" + Finally, the root metadata should write the Boolean "consistent_snapshot" attribute at the root level of its keys of attributes. If consistent snapshots are not written by the repository, then the attribute may either be left unspecified or be set to the False value. Otherwise, it must be @@ -805,16 +836,16 @@ We now explain how a client should read a self-contained consistent snapshot. - If the root metadata (root.txt) is either missing the Boolean - "consistent_snapshots" attribute or the attribute is set to False, then the + If the root metadata (root.json) is either missing the Boolean + "consistent_snapshot" attribute or the attribute is set to False, then the client should do nothing different from the workflow in Section 5.1. Otherwise, the client must perform as follows: - 1. It must first retrieve the timestamp metadata (timestamp.txt) from the + 1. It must first retrieve the timestamp metadata (timestamp.json) from the repository. - 2. If a threshold number of signatures of the timestamp or release + 2. If a threshold number of signatures of the timestamp or snapshot metadata are not valid, then the client must download the root metadata - (root.txt) from the repository and return to step 1. + (root.json) from the repository and return to step 1. 3. Otherwise, the client must download every subsequent metadata or target file as follows: if the metadata or target file has the name filename.ext, then the client must actually retrieve the file with the diff --git a/ed25519/ed25519.py b/ed25519/ed25519.py deleted file mode 100755 index 7f8613b8d8..0000000000 --- a/ed25519/ed25519.py +++ /dev/null @@ -1,104 +0,0 @@ -import hashlib - -b = 256 -q = 2**255 - 19 -l = 2**252 + 27742317777372353535851937790883648493 - -def H(m): - return hashlib.sha512(m).digest() - -def expmod(b,e,m): - if e == 0: return 1 - t = expmod(b,e/2,m)**2 % m - if e & 1: t = (t*b) % m - return t - -def inv(x): - return expmod(x,q-2,q) - -d = -121665 * inv(121666) -I = expmod(2,(q-1)/4,q) - -def xrecover(y): - xx = (y*y-1) * inv(d*y*y+1) - x = expmod(xx,(q+3)/8,q) - if (x*x - xx) % q != 0: x = (x*I) % q - if x % 2 != 0: x = q-x - return x - -By = 4 * inv(5) -Bx = xrecover(By) -B = [Bx % q,By % q] - -def edwards(P,Q): - x1 = P[0] - y1 = P[1] - x2 = Q[0] - y2 = Q[1] - x3 = (x1*y2+x2*y1) * inv(1+d*x1*x2*y1*y2) - y3 = (y1*y2+x1*x2) * inv(1-d*x1*x2*y1*y2) - return [x3 % q,y3 % q] - -def scalarmult(P,e): - if e == 0: return [0,1] - Q = scalarmult(P,e/2) - Q = edwards(Q,Q) - if e & 1: Q = edwards(Q,P) - return Q - -def encodeint(y): - bits = [(y >> i) & 1 for i in range(b)] - return ''.join([chr(sum([bits[i * 8 + j] << j for j in range(8)])) for i in range(b/8)]) - -def encodepoint(P): - x = P[0] - y = P[1] - bits = [(y >> i) & 1 for i in range(b - 1)] + [x & 1] - return ''.join([chr(sum([bits[i * 8 + j] << j for j in range(8)])) for i in range(b/8)]) - -def bit(h,i): - return (ord(h[i/8]) >> (i%8)) & 1 - -def publickey(sk): - h = H(sk) - a = 2**(b-2) + sum(2**i * bit(h,i) for i in range(3,b-2)) - A = scalarmult(B,a) - return encodepoint(A) - -def Hint(m): - h = H(m) - return sum(2**i * bit(h,i) for i in range(2*b)) - -def signature(m,sk,pk): - h = H(sk) - a = 2**(b-2) + sum(2**i * bit(h,i) for i in range(3,b-2)) - r = Hint(''.join([h[i] for i in range(b/8,b/4)]) + m) - R = scalarmult(B,r) - S = (r + Hint(encodepoint(R) + pk + m) * a) % l - return encodepoint(R) + encodeint(S) - -def isoncurve(P): - x = P[0] - y = P[1] - return (-x*x + y*y - 1 - d*x*x*y*y) % q == 0 - -def decodeint(s): - return sum(2**i * bit(s,i) for i in range(0,b)) - -def decodepoint(s): - y = sum(2**i * bit(s,i) for i in range(0,b-1)) - x = xrecover(y) - if x & 1 != bit(s,b-1): x = q-x - P = [x,y] - if not isoncurve(P): raise Exception("decoding point that is not on curve") - return P - -def checkvalid(s,m,pk): - if len(s) != b/4: raise Exception("signature length is wrong") - if len(pk) != b/8: raise Exception("public-key length is wrong") - R = decodepoint(s[0:b/8]) - A = decodepoint(pk) - S = decodeint(s[b/8:b/4]) - h = Hint(encodepoint(R) + pk + m) - if scalarmult(B,S) != edwards(R,scalarmult(A,h)): - raise Exception("signature does not pass verification") diff --git a/setup.py b/setup.py index d035942fbd..37203b3b22 100755 --- a/setup.py +++ b/setup.py @@ -55,7 +55,6 @@ $ quickstart.py --project ./project-files $ signercli.py --genrsakey ./keystore - """ from setuptools import setup @@ -69,6 +68,7 @@ url='https://www.updateframework.com', install_requires=['pycrypto>=2.6'], packages=[ + 'ed25519', 'tuf', 'tuf.client', 'tuf.compatibility', @@ -79,9 +79,8 @@ 'tuf.tests' ], scripts=[ - 'tuf/repo/quickstart.py', 'tuf/pushtools/push.py', 'tuf/pushtools/receivetools/receive.py', - 'tuf/repo/signercli.py' + 'tuf/client/basic_client.py' ] ) diff --git a/tests/integration/test_arbitrary_package_attack.py b/tests/integration/test_arbitrary_package_attack.py index f54c5d0815..3412c761b0 100755 --- a/tests/integration/test_arbitrary_package_attack.py +++ b/tests/integration/test_arbitrary_package_attack.py @@ -168,8 +168,10 @@ def test_arbitrary_package_attack(using_tuf=False, modify_metadata=False): print('Attempting arbitrary package attack without TUF:') try: test_arbitrary_package_attack(using_tuf=False) + except ArbitraryPackageAlert, error: print(error) + else: print('Extraneous dependency attack failed.') print() @@ -180,6 +182,7 @@ def test_arbitrary_package_attack(using_tuf=False, modify_metadata=False): test_arbitrary_package_attack(using_tuf=True, modify_metadata=False) except ArbitraryPackageAlert, error: print(error) + else: print('Extraneous dependency attack failed.') print() @@ -189,8 +192,10 @@ def test_arbitrary_package_attack(using_tuf=False, modify_metadata=False): ' (and tampering with metadata):') try: test_arbitrary_package_attack(using_tuf=True, modify_metadata=True) + except ArbitraryPackageAlert, error: print(error) + else: print('Extraneous dependency attack failed.') print() diff --git a/tests/integration/test_extraneous_dependencies_attack.py b/tests/integration/test_extraneous_dependencies_attack.py index 9d07b6a455..a97b549446 100755 --- a/tests/integration/test_extraneous_dependencies_attack.py +++ b/tests/integration/test_extraneous_dependencies_attack.py @@ -218,7 +218,6 @@ def test_extraneous_dependency_attack(using_tuf=False, modify_metadata=False): print('Extraneous dependency attack failed.') print() - print('Attempting extraneous dependency attack with TUF:') try: test_extraneous_dependency_attack(using_tuf=True, modify_metadata=False) diff --git a/tests/unit/aggregate_tests.py b/tests/unit/aggregate_tests.py index 620c54de5c..0713e217b5 100755 --- a/tests/unit/aggregate_tests.py +++ b/tests/unit/aggregate_tests.py @@ -21,10 +21,8 @@ Run all the unit tests from every .py file beginning with "test_" in 'tuf/tests'. Use --random to run the tests in random order. - """ - import sys import unittest import glob diff --git a/tests/unit/test_keystore.py b/tests/unit/deprecated/test_keystore.py similarity index 99% rename from tests/unit/test_keystore.py rename to tests/unit/deprecated/test_keystore.py index 5eed223a42..e1a8c7f253 100755 --- a/tests/unit/test_keystore.py +++ b/tests/unit/deprecated/test_keystore.py @@ -13,7 +13,6 @@ Unit test for keystore.py. - """ import unittest @@ -25,7 +24,7 @@ import tuf import tuf.repo.keystore -import tuf.rsa_key +import tuf.keys import tuf.formats import tuf.util import tuf.log @@ -56,7 +55,7 @@ for i in range(3): # Populating the original 'RSAKEYS' and 'PASSWDS' lists. - RSAKEYS.append(tuf.rsa_key.generate()) + RSAKEYS.append(tuf.keys.generate_rsa_key()) PASSWDS.append('passwd_'+str(i)) # Saving original copies of 'RSAKEYS' and 'PASSWDS' to temp variables @@ -350,6 +349,7 @@ def tearDownModule(): tuf.repo.keystore.clear_keystore() + # Run the unit tests. if __name__ == '__main__': unittest.main() diff --git a/tests/unit/test_push.py b/tests/unit/deprecated/test_push.py similarity index 100% rename from tests/unit/test_push.py rename to tests/unit/deprecated/test_push.py diff --git a/tests/unit/test_pushtoolslib.py b/tests/unit/deprecated/test_pushtoolslib.py similarity index 100% rename from tests/unit/test_pushtoolslib.py rename to tests/unit/deprecated/test_pushtoolslib.py diff --git a/tests/unit/test_quickstart.py b/tests/unit/deprecated/test_quickstart.py similarity index 99% rename from tests/unit/test_quickstart.py rename to tests/unit/deprecated/test_quickstart.py index 00ddf44ede..e1bca8c47e 100755 --- a/tests/unit/test_quickstart.py +++ b/tests/unit/deprecated/test_quickstart.py @@ -18,7 +18,6 @@ Given that all message prompts don't change - this will work pretty well for running quickstart without having to manually enter input to prompts every time you want to run quickstart. - """ import os diff --git a/tests/unit/test_signercli.py b/tests/unit/deprecated/test_signercli.py similarity index 100% rename from tests/unit/test_signercli.py rename to tests/unit/deprecated/test_signercli.py diff --git a/tests/unit/test_signerlib.py b/tests/unit/deprecated/test_signerlib.py similarity index 100% rename from tests/unit/test_signerlib.py rename to tests/unit/deprecated/test_signerlib.py diff --git a/tests/unit/test_ed25519_keys.py b/tests/unit/test_ed25519_keys.py new file mode 100755 index 0000000000..a65a85dc0f --- /dev/null +++ b/tests/unit/test_ed25519_keys.py @@ -0,0 +1,123 @@ +""" + + test_ed25519_keys.py + + + Vladimir Diaz + + + October 11, 2013. + + + See LICENSE for licensing information. + + + Test cases for test_ed25519_keys.py. +""" + +import unittest +import logging + +import tuf +import tuf.log +import tuf.formats +import tuf.ed25519_keys as ed25519 + +logger = logging.getLogger('tuf.test_ed25519_keys') + +public, private = ed25519.generate_public_and_private() +FORMAT_ERROR_MSG = 'tuf.FormatError raised. Check object\'s format.' + + +class TestEd25519_keys(unittest.TestCase): + def setUp(self): + pass + + + def test_generate_public_and_private(self): + pub, priv = ed25519.generate_public_and_private() + + # Check format of 'pub' and 'priv'. + self.assertEqual(True, tuf.formats.ED25519PUBLIC_SCHEMA.matches(pub)) + self.assertEqual(True, tuf.formats.ED25519SEED_SCHEMA.matches(priv)) + + # Check for invalid argument. + self.assertRaises(tuf.FormatError, + ed25519.generate_public_and_private, 'True') + + self.assertRaises(tuf.FormatError, + ed25519.generate_public_and_private, 2048) + + + def test_create_signature(self): + global public + global private + data = 'The quick brown fox jumps over the lazy dog' + signature, method = ed25519.create_signature(public, private, data) + + # Verify format of returned values. + self.assertEqual(True, + tuf.formats.ED25519SIGNATURE_SCHEMA.matches(signature)) + + self.assertEqual(True, tuf.formats.NAME_SCHEMA.matches(method)) + self.assertEqual('ed25519', method) + + # Check for improperly formatted argument. + self.assertRaises(tuf.FormatError, + ed25519.create_signature, 123, private, data) + + self.assertRaises(tuf.FormatError, + ed25519.create_signature, public, 123, data) + + # Check for invalid 'data'. + self.assertRaises(tuf.CryptoError, + ed25519.create_signature, public, private, 123) + + + def test_verify_signature(self): + global public + global private + data = 'The quick brown fox jumps over the lazy dog' + signature, method = ed25519.create_signature(public, private, data) + + valid_signature = ed25519.verify_signature(public, method, signature, data) + self.assertEqual(True, valid_signature) + + # Check for improperly formatted arguments. + self.assertRaises(tuf.FormatError, ed25519.verify_signature, 123, method, + signature, data) + + # Signature method improperly formatted. + self.assertRaises(tuf.FormatError, ed25519.verify_signature, public, 123, + signature, data) + + # Signature not a string. + self.assertRaises(tuf.FormatError, ed25519.verify_signature, public, method, + 123, data) + + # Invalid signature length, which must be exactly 64 bytes.. + self.assertRaises(tuf.FormatError, ed25519.verify_signature, public, method, + 'bad_signature', data) + + # Check for invalid signature and data. + # Mismatched data. + self.assertEqual(False, ed25519.verify_signature(public, method, + signature, '123')) + + # Mismatched signature. + bad_signature = 'a'*64 + self.assertEqual(False, ed25519.verify_signature(public, method, + bad_signature, data)) + + # Generated signature created with different data. + new_signature, method = ed25519.create_signature(public, private, + 'mismatched data') + + self.assertEqual(False, ed25519.verify_signature(public, method, + new_signature, data)) + + + +# Run the unit tests. +if __name__ == '__main__': + unittest.main() diff --git a/tests/unit/test_formats.py b/tests/unit/test_formats.py index 1e375ca3ec..387c1a255f 100755 --- a/tests/unit/test_formats.py +++ b/tests/unit/test_formats.py @@ -15,7 +15,6 @@ Unit test for 'formats.py' - """ import unittest @@ -76,7 +75,7 @@ def test_schemas(self): 'NAME_SCHEMA': (tuf.formats.NAME_SCHEMA, 'Marty McFly'), - 'TOGGLE_SCHEMA': (tuf.formats.TOGGLE_SCHEMA, True), + 'BOOLEAN_SCHEMA': (tuf.formats.BOOLEAN_SCHEMA, True), 'THRESHOLD_SCHEMA': (tuf.formats.THRESHOLD_SCHEMA, 1), @@ -108,7 +107,7 @@ def test_schemas(self): 'custom': {'type': 'paintjob'}}), 'FILEDICT_SCHEMA': (tuf.formats.FILEDICT_SCHEMA, - {'metadata/root.txt': {'length': 1024, + {'metadata/root.json': {'length': 1024, 'hashes': {'sha256': 'ABCD123'}, 'custom': {'type': 'metadata'}}}), @@ -156,7 +155,7 @@ def test_schemas(self): 'SCPCONFIG_SCHEMA': (tuf.formats.SCPCONFIG_SCHEMA, {'general': {'transfer_module': 'scp', - 'metadata_path': '/path/meta.txt', + 'metadata_path': '/path/meta.json', 'targets_directory': '/targets'}, 'scp': {'host': 'http://localhost:8001', 'user': 'McFly', @@ -184,6 +183,7 @@ def test_schemas(self): 'ROOT_SCHEMA': (tuf.formats.ROOT_SCHEMA, {'_type': 'Root', 'version': 8, + 'consistent_snapshot': False, 'expires': '2012-10-16 06:42:12 UTC', 'keys': {'123abc': {'keytype': 'rsa', 'keyval': {'public': 'pubkey', @@ -196,7 +196,7 @@ def test_schemas(self): {'_type': 'Targets', 'version': 8, 'expires': '2012-10-16 06:42:12 UTC', - 'targets': {'metadata/targets.txt': {'length': 1024, + 'targets': {'metadata/targets.json': {'length': 1024, 'hashes': {'sha256': 'ABCD123'}, 'custom': {'type': 'metadata'}}}, 'delegations': {'keys': {'123abc': {'keytype':'rsa', @@ -206,11 +206,11 @@ def test_schemas(self): 'threshold': 1, 'paths': ['path1/', 'path2']}]}}), - 'RELEASE_SCHEMA': (tuf.formats.RELEASE_SCHEMA, - {'_type': 'Release', + 'SNAPSHOT_SCHEMA': (tuf.formats.SNAPSHOT_SCHEMA, + {'_type': 'Snapshot', 'version': 8, 'expires': '2012-10-16 06:42:12 UTC', - 'meta': {'metadata/release.txt': {'length': 1024, + 'meta': {'metadata/snapshot.json': {'length': 1024, 'hashes': {'sha256': 'ABCD123'}, 'custom': {'type': 'metadata'}}}}), @@ -218,7 +218,7 @@ def test_schemas(self): {'_type': 'Timestamp', 'version': 8, 'expires': '2012-10-16 06:42:12 UTC', - 'meta': {'metadata/timestamp.txt': {'length': 1024, + 'meta': {'metadata/timestamp.json': {'length': 1024, 'hashes': {'sha256': 'ABCD123'}, 'custom': {'type': 'metadata'}}}}), @@ -246,8 +246,8 @@ def test_schemas(self): 'confined_target_dirs': ['path1/', 'path2/'], 'custom': {'type': 'mirror'}}]})} - # Iterate through 'valid_schemas', ensuring each 'valid_schema' correctly - # matches its respective 'schema_type'. + # Iterate 'valid_schemas', ensuring each 'valid_schema' correctly matches + # its respective 'schema_type'. for schema_name, (schema_type, valid_schema) in valid_schemas.items(): self.assertEqual(True, schema_type.matches(valid_schema)) @@ -290,7 +290,7 @@ def test_TimestampFile(self): # Test conditions for valid instances of 'tuf.formats.TimestampFile'. version = 8 expires = '2012-10-16 06:42:12 UTC' - filedict = {'metadata/timestamp.txt': {'length': 1024, + filedict = {'metadata/timestamp.json': {'length': 1024, 'hashes': {'sha256': 'ABCD123'}, 'custom': {'type': 'metadata'}}} @@ -322,7 +322,8 @@ def test_TimestampFile(self): def test_RootFile(self): # Test conditions for valid instances of 'tuf.formats.RootFile'. version = 8 - expiration_seconds = 691200 + consistent_snapshot = False + expires = '2018-10-16 06:42:12 UTC' keydict = {'123abc': {'keytype': 'rsa', 'keyval': {'public': 'pubkey', 'private': 'privkey'}}} @@ -335,50 +336,56 @@ def test_RootFile(self): from_metadata = tuf.formats.RootFile.from_metadata ROOT_SCHEMA = tuf.formats.ROOT_SCHEMA - self.assertTrue(ROOT_SCHEMA.matches(make_metadata(version, expiration_seconds, - keydict, roledict))) - metadata = make_metadata(version, expiration_seconds, keydict, roledict,) + self.assertTrue(ROOT_SCHEMA.matches(make_metadata(version, expires, + keydict, roledict, + consistent_snapshot))) + metadata = make_metadata(version, expires, keydict, roledict, + consistent_snapshot) self.assertTrue(isinstance(from_metadata(metadata), tuf.formats.RootFile)) # Test conditions for invalid arguments. bad_version = '8' - bad_expiration_seconds = 'eight' + bad_expires = 'eight' bad_keydict = 123 bad_roledict = 123 self.assertRaises(tuf.FormatError, make_metadata, bad_version, - expiration_seconds, - keydict, roledict) + expires, + keydict, roledict, + consistent_snapshot) self.assertRaises(tuf.FormatError, make_metadata, version, - bad_expiration_seconds, - keydict, roledict) + bad_expires, + keydict, roledict, + consistent_snapshot) self.assertRaises(tuf.FormatError, make_metadata, version, - expiration_seconds, - bad_keydict, roledict) + expires, + bad_keydict, roledict, + consistent_snapshot) self.assertRaises(tuf.FormatError, make_metadata, version, - expiration_seconds, - keydict, bad_roledict) + expires, + keydict, bad_roledict, + consistent_snapshot) self.assertRaises(tuf.FormatError, from_metadata, 'bad') - def test_ReleaseFile(self): - # Test conditions for valid instances of 'tuf.formats.ReleaseFile'. + def test_SnapshotFile(self): + # Test conditions for valid instances of 'tuf.formats.SnapshotFile'. version = 8 expires = '2012-10-16 06:42:12 UTC' - filedict = {'metadata/release.txt': {'length': 1024, + filedict = {'metadata/snapshot.json': {'length': 1024, 'hashes': {'sha256': 'ABCD123'}, 'custom': {'type': 'metadata'}}} - make_metadata = tuf.formats.ReleaseFile.make_metadata - from_metadata = tuf.formats.ReleaseFile.from_metadata - RELEASE_SCHEMA = tuf.formats.RELEASE_SCHEMA + make_metadata = tuf.formats.SnapshotFile.make_metadata + from_metadata = tuf.formats.SnapshotFile.from_metadata + SNAPSHOT_SCHEMA = tuf.formats.SNAPSHOT_SCHEMA - self.assertTrue(RELEASE_SCHEMA.matches(make_metadata(version, expires, + self.assertTrue(SNAPSHOT_SCHEMA.matches(make_metadata(version, expires, filedict))) metadata = make_metadata(version, expires, filedict) - self.assertTrue(isinstance(from_metadata(metadata), tuf.formats.ReleaseFile)) + self.assertTrue(isinstance(from_metadata(metadata), tuf.formats.SnapshotFile)) # Test conditions for invalid arguments. bad_version = '8' @@ -399,7 +406,7 @@ def test_TargetsFile(self): # Test conditions for valid instances of 'tuf.formats.TargetsFile'. version = 8 expires = '2012-10-16 06:42:12 UTC' - filedict = {'metadata/targets.txt': {'length': 1024, + filedict = {'metadata/targets.json': {'length': 1024, 'hashes': {'sha256': 'ABCD123'}, 'custom': {'type': 'metadata'}}} @@ -490,6 +497,7 @@ def test_make_signable(self): # Test conditions for expected make_signable() behavior. root = {'_type': 'Root', 'version': 8, + 'consistent_snapshot': False, 'expires': '2012-10-16 06:42:12 UTC', 'keys': {'123abc': {'keytype': 'rsa', 'keyval': {'public': 'pubkey', @@ -580,7 +588,7 @@ def test_get_role_class(self): self.assertEqual(tuf.formats.RootFile, get_role_class('Root')) self.assertEqual(tuf.formats.TargetsFile, get_role_class('Targets')) - self.assertEqual(tuf.formats.ReleaseFile, get_role_class('Release')) + self.assertEqual(tuf.formats.SnapshotFile, get_role_class('Snapshot')) self.assertEqual(tuf.formats.TimestampFile, get_role_class('Timestamp')) self.assertEqual(tuf.formats.MirrorsFile, get_role_class('Mirrors')) @@ -599,7 +607,7 @@ def test_expected_meta_rolename(self): self.assertEqual('Root', expected_rolename('root')) self.assertEqual('Targets', expected_rolename('targets')) - self.assertEqual('Release', expected_rolename('release')) + self.assertEqual('Snapshot', expected_rolename('snapshot')) self.assertEqual('Timestamp', expected_rolename('timestamp')) self.assertEqual('Mirrors', expected_rolename('mirrors')) self.assertEqual('Targets Role', expected_rolename('targets role')) @@ -616,6 +624,7 @@ def test_check_signable_object_format(self): # Test condition for a valid argument. root = {'_type': 'Root', 'version': 8, + 'consistent_snapshot': False, 'expires': '2012-10-16 06:42:12 UTC', 'keys': {'123abc': {'keytype': 'rsa', 'keyval': {'public': 'pubkey', diff --git a/tests/unit/test_keydb.py b/tests/unit/test_keydb.py index c7330875fc..7fd0d293ec 100755 --- a/tests/unit/test_keydb.py +++ b/tests/unit/test_keydb.py @@ -13,7 +13,6 @@ Unit test for 'keydb.py'. - """ import unittest @@ -21,7 +20,7 @@ import tuf import tuf.formats -import tuf.rsa_key +import tuf.keys import tuf.keydb import tuf.log @@ -31,7 +30,7 @@ # Generate the three keys to use in our test cases. KEYS = [] for junk in range(3): - KEYS.append(tuf.rsa_key.generate(2048)) + KEYS.append(tuf.keys.generate_rsa_key(2048)) @@ -89,7 +88,7 @@ def test_get_key(self): - def test_add_rsakey(self): + def test_add_key(self): # Test conditions using valid 'keyid' arguments. rsakey = KEYS[0] keyid = KEYS[0]['keyid'] @@ -97,9 +96,9 @@ def test_add_rsakey(self): keyid2 = KEYS[1]['keyid'] rsakey3 = KEYS[2] keyid3 = KEYS[2]['keyid'] - self.assertEqual(None, tuf.keydb.add_rsakey(rsakey, keyid)) - self.assertEqual(None, tuf.keydb.add_rsakey(rsakey2, keyid2)) - self.assertEqual(None, tuf.keydb.add_rsakey(rsakey3)) + self.assertEqual(None, tuf.keydb.add_key(rsakey, keyid)) + self.assertEqual(None, tuf.keydb.add_key(rsakey2, keyid2)) + self.assertEqual(None, tuf.keydb.add_key(rsakey3)) self.assertEqual(rsakey, tuf.keydb.get_key(keyid)) self.assertEqual(rsakey2, tuf.keydb.get_key(keyid2)) @@ -109,26 +108,26 @@ def test_add_rsakey(self): tuf.keydb.clear_keydb() rsakey3['keytype'] = 'bad_keytype' - self.assertRaises(tuf.FormatError, tuf.keydb.add_rsakey, None, keyid) - self.assertRaises(tuf.FormatError, tuf.keydb.add_rsakey, '', keyid) - self.assertRaises(tuf.FormatError, tuf.keydb.add_rsakey, ['123'], keyid) - self.assertRaises(tuf.FormatError, tuf.keydb.add_rsakey, {'a': 'b'}, keyid) - self.assertRaises(tuf.FormatError, tuf.keydb.add_rsakey, rsakey, {'keyid': ''}) - self.assertRaises(tuf.FormatError, tuf.keydb.add_rsakey, rsakey, 123) - self.assertRaises(tuf.FormatError, tuf.keydb.add_rsakey, rsakey, False) - self.assertRaises(tuf.FormatError, tuf.keydb.add_rsakey, rsakey, ['keyid']) - self.assertRaises(tuf.FormatError, tuf.keydb.add_rsakey, rsakey3, keyid3) + self.assertRaises(tuf.FormatError, tuf.keydb.add_key, None, keyid) + self.assertRaises(tuf.FormatError, tuf.keydb.add_key, '', keyid) + self.assertRaises(tuf.FormatError, tuf.keydb.add_key, ['123'], keyid) + self.assertRaises(tuf.FormatError, tuf.keydb.add_key, {'a': 'b'}, keyid) + self.assertRaises(tuf.FormatError, tuf.keydb.add_key, rsakey, {'keyid': ''}) + self.assertRaises(tuf.FormatError, tuf.keydb.add_key, rsakey, 123) + self.assertRaises(tuf.FormatError, tuf.keydb.add_key, rsakey, False) + self.assertRaises(tuf.FormatError, tuf.keydb.add_key, rsakey, ['keyid']) + self.assertRaises(tuf.FormatError, tuf.keydb.add_key, rsakey3, keyid3) rsakey3['keytype'] = 'rsa' # Test conditions where keyid does not match the rsakey. - self.assertRaises(tuf.Error, tuf.keydb.add_rsakey, rsakey, keyid2) - self.assertRaises(tuf.Error, tuf.keydb.add_rsakey, rsakey2, keyid) + self.assertRaises(tuf.Error, tuf.keydb.add_key, rsakey, keyid2) + self.assertRaises(tuf.Error, tuf.keydb.add_key, rsakey2, keyid) # Test conditions using keyids that have already been added. - tuf.keydb.add_rsakey(rsakey, keyid) - tuf.keydb.add_rsakey(rsakey2, keyid2) - self.assertRaises(tuf.KeyAlreadyExistsError, tuf.keydb.add_rsakey, rsakey) - self.assertRaises(tuf.KeyAlreadyExistsError, tuf.keydb.add_rsakey, rsakey2) + tuf.keydb.add_key(rsakey, keyid) + tuf.keydb.add_key(rsakey2, keyid2) + self.assertRaises(tuf.KeyAlreadyExistsError, tuf.keydb.add_key, rsakey) + self.assertRaises(tuf.KeyAlreadyExistsError, tuf.keydb.add_key, rsakey2) @@ -140,12 +139,13 @@ def test_remove_key(self): keyid2 = KEYS[1]['keyid'] rsakey3 = KEYS[2] keyid3 = KEYS[2]['keyid'] - tuf.keydb.add_rsakey(rsakey, keyid) - tuf.keydb.add_rsakey(rsakey2, keyid2) - tuf.keydb.add_rsakey(rsakey3, keyid3) + tuf.keydb.add_key(rsakey, keyid) + tuf.keydb.add_key(rsakey2, keyid2) + tuf.keydb.add_key(rsakey3, keyid3) self.assertEqual(None, tuf.keydb.remove_key(keyid)) self.assertEqual(None, tuf.keydb.remove_key(keyid2)) + # Ensure the keys were actually removed. self.assertRaises(tuf.UnknownKeyError, tuf.keydb.get_key, keyid) self.assertRaises(tuf.UnknownKeyError, tuf.keydb.get_key, keyid2) @@ -170,11 +170,13 @@ def test_create_keydb_from_root_metadata(self): roledict = {'Root': {'keyids': [keyid], 'threshold': 1}, 'Targets': {'keyids': [keyid2], 'threshold': 1}} version = 8 - expiration_seconds = 200 + consistent_snapshot = False + expires = '2012-10-16 06:42:12 UTC' root_metadata = tuf.formats.RootFile.make_metadata(version, - expiration_seconds, - keydict, roledict) + expires, + keydict, roledict, + consistent_snapshot) self.assertEqual(None, tuf.keydb.create_keydb_from_root_metadata(root_metadata)) # Ensure 'keyid' and 'keyid2' were added to the keydb database. self.assertEqual(rsakey, tuf.keydb.get_key(keyid)) @@ -206,11 +208,12 @@ def test_create_keydb_from_root_metadata(self): rsakey3['keytype'] = 'bad_keytype' keydict[keyid3] = rsakey3 version = 8 - expiration_seconds = 200 + expires = '2012-10-16 06:42:12 UTC' root_metadata = tuf.formats.RootFile.make_metadata(version, - expiration_seconds, - keydict, roledict) + expires, + keydict, roledict, + consistent_snapshot) self.assertEqual(None, tuf.keydb.create_keydb_from_root_metadata(root_metadata)) # Ensure only 'keyid2' was added to the keydb database. 'keyid' and diff --git a/tests/unit/test_keys.py b/tests/unit/test_keys.py new file mode 100755 index 0000000000..758c100906 --- /dev/null +++ b/tests/unit/test_keys.py @@ -0,0 +1,193 @@ +""" + + test_keys.py + + + Vladimir Diaz + + + October 10, 2013. + + + See LICENSE for licensing information. + + + Test cases for test_keys.py. + TODO: test case for ed25519 key generation and refactor. +""" + +import unittest +import logging + +import tuf +import tuf.log +import tuf.formats +import tuf.keys + +logger = logging.getLogger('tuf.test_keys') + +KEYS = tuf.keys +FORMAT_ERROR_MSG = 'tuf.FormatError was raised! Check object\'s format.' +DATA = 'SOME DATA REQUIRING AUTHENTICITY.' + + +rsakey_dict = KEYS.generate_rsa_key() +temp_key_info_vals = rsakey_dict.values() +temp_key_vals = rsakey_dict['keyval'].values() + + +class TestKeys(unittest.TestCase): + def setUp(self): + rsakey_dict['keytype']=temp_key_info_vals[0] + rsakey_dict['keyid']=temp_key_info_vals[1] + rsakey_dict['keyval']=temp_key_info_vals[2] + rsakey_dict['keyval']['public']=temp_key_vals[0] + rsakey_dict['keyval']['private']=temp_key_vals[1] + + + def test_generate_rsa_key(self): + _rsakey_dict = KEYS.generate_rsa_key() + + # Check if the format of the object returned by generate() corresponds + # to RSAKEY_SCHEMA format. + self.assertEqual(None, tuf.formats.RSAKEY_SCHEMA.check_match(_rsakey_dict), + FORMAT_ERROR_MSG) + + # Passing a bit value that is <2048 to generate() - should raise + # 'tuf.FormatError'. + self.assertRaises(tuf.FormatError, KEYS.generate_rsa_key, 555) + + # Passing a string instead of integer for a bit value. + self.assertRaises(tuf.FormatError, KEYS.generate_rsa_key, 'bits') + + # NOTE if random bit value >=2048 (not 4096) is passed generate(bits) + # does not raise any errors and returns a valid key. + self.assertTrue(tuf.formats.RSAKEY_SCHEMA.matches(KEYS.generate_rsa_key(2048))) + self.assertTrue(tuf.formats.RSAKEY_SCHEMA.matches(KEYS.generate_rsa_key(4096))) + + + def test_format_keyval_to_metadata(self): + keyvalue = rsakey_dict['keyval'] + keytype = rsakey_dict['keytype'] + key_meta = KEYS.format_keyval_to_metadata(keytype, keyvalue) + + # Check if the format of the object returned by this function corresponds + # to KEY_SCHEMA format. + self.assertEqual(None, + tuf.formats.KEY_SCHEMA.check_match(key_meta), + FORMAT_ERROR_MSG) + key_meta = KEYS.format_keyval_to_metadata(keytype, keyvalue, private=True) + + # Check if the format of the object returned by this function corresponds + # to KEY_SCHEMA format. + self.assertEqual(None, tuf.formats.KEY_SCHEMA.check_match(key_meta), + FORMAT_ERROR_MSG) + + # Supplying a 'bad' keyvalue. + self.assertRaises(tuf.FormatError, KEYS.format_keyval_to_metadata, + 'bad_keytype', keyvalue) + + del keyvalue['public'] + self.assertRaises(tuf.FormatError, KEYS.format_keyval_to_metadata, + keytype, keyvalue) + + + def test_format_metadata_to_key(self): + # Reconfiguring rsakey_dict to conform to KEY_SCHEMA + # i.e. {keytype: 'rsa', keyval: {public: pub_key, private: priv_key}} + #keyid = rsakey_dict['keyid'] + del rsakey_dict['keyid'] + + rsakey_dict_from_meta = KEYS.format_metadata_to_key(rsakey_dict) + + # Check if the format of the object returned by this function corresponds + # to RSAKEY_SCHEMA format. + self.assertEqual(None, + tuf.formats.RSAKEY_SCHEMA.check_match(rsakey_dict_from_meta), + FORMAT_ERROR_MSG) + + # Supplying a wrong number of arguments. + self.assertRaises(TypeError, KEYS.format_metadata_to_key) + args = (rsakey_dict, rsakey_dict) + self.assertRaises(TypeError, KEYS.format_metadata_to_key, *args) + + # Supplying a malformed argument to the function - should get FormatError + del rsakey_dict['keyval'] + self.assertRaises(tuf.FormatError, KEYS.format_metadata_to_key, + rsakey_dict) + + + def test_helper_get_keyid(self): + keytype = rsakey_dict['keytype'] + keyvalue = rsakey_dict['keyval'] + + # Check format of 'keytype'. + self.assertEqual(None, tuf.formats.KEYTYPE_SCHEMA.check_match(keytype), + FORMAT_ERROR_MSG) + + # Check format of 'keyvalue'. + self.assertEqual(None, tuf.formats.KEYVAL_SCHEMA.check_match(keyvalue), + FORMAT_ERROR_MSG) + + keyid = KEYS._get_keyid(keytype, keyvalue) + + # Check format of 'keyid' - the output of '_get_keyid()' function. + self.assertEqual(None, tuf.formats.KEYID_SCHEMA.check_match(keyid), + FORMAT_ERROR_MSG) + + + def test_create_signature(self): + # Creating a signature for 'DATA'. + signature = KEYS.create_signature(rsakey_dict, DATA) + + # Check format of output. + self.assertEqual(None, + tuf.formats.SIGNATURE_SCHEMA.check_match(signature), + FORMAT_ERROR_MSG) + + # Removing private key from 'rsakey_dict' - should raise a TypeError. + rsakey_dict['keyval']['private'] = '' + + args = (rsakey_dict, DATA) + self.assertRaises(TypeError, KEYS.create_signature, *args) + + # Supplying an incorrect number of arguments. + self.assertRaises(TypeError, KEYS.create_signature) + + + def test_verify_signature(self): + # Creating a signature 'signature' of 'DATA' to be verified. + signature = KEYS.create_signature(rsakey_dict, DATA) + + # Verifying the 'signature' of 'DATA'. + verified = KEYS.verify_signature(rsakey_dict, signature, DATA) + self.assertTrue(verified, "Incorrect signature.") + + # Testing an invalid 'signature'. Same 'signature' is passed, with + # 'DATA' different than the original 'DATA' that was used + # in creating the 'signature'. Function should return 'False'. + + # Modifying 'DATA'. + _DATA = '1111'+DATA+'1111' + + # Verifying the 'signature' of modified '_DATA'. + verified = KEYS.verify_signature(rsakey_dict, signature, _DATA) + self.assertFalse(verified, + 'Returned \'True\' on an incorrect signature.') + + # Modifying 'signature' to pass an incorrect method since only + # 'PyCrypto-PKCS#1 PSS' + # is accepted. + signature['method'] = 'Biff' + + args = (rsakey_dict, signature, DATA) + self.assertRaises(tuf.UnknownMethodError, KEYS.verify_signature, *args) + + # Passing incorrect number of arguments. + self.assertRaises(TypeError, KEYS.verify_signature) + + + +# Run the unit tests. +if __name__ == '__main__': + unittest.main() diff --git a/tests/unit/test_pycrypto_keys.py b/tests/unit/test_pycrypto_keys.py new file mode 100755 index 0000000000..f6adf7d063 --- /dev/null +++ b/tests/unit/test_pycrypto_keys.py @@ -0,0 +1,198 @@ +""" + + test_pycrypto_keys.py + + + Vladimir Diaz + + + October 10, 2013. + + + See LICENSE for licensing information. + + + Test cases for test_pycrypto_keys.py. +""" + +import unittest +import logging + +import tuf +import tuf.log +import tuf.formats +import tuf.pycrypto_keys as pycrypto + +logger = logging.getLogger('tuf.test_pycrypto_keys') + +public_rsa, private_rsa = pycrypto.generate_rsa_public_and_private() +FORMAT_ERROR_MSG = 'tuf.FormatError raised. Check object\'s format.' + + +class TestPycrypto_keys(unittest.TestCase): + def setUp(self): + pass + + + def test_generate_rsa_public_and_private(self): + pub, priv = pycrypto.generate_rsa_public_and_private() + + # Check format of 'pub' and 'priv'. + self.assertEqual(None, tuf.formats.PEMRSA_SCHEMA.check_match(pub), + FORMAT_ERROR_MSG) + self.assertEqual(None, tuf.formats.PEMRSA_SCHEMA.check_match(priv), + FORMAT_ERROR_MSG) + + # Check for invalid bits argument. bit >= 2048 and a multiple of 256. + self.assertRaises(tuf.FormatError, + pycrypto.generate_rsa_public_and_private, 1024) + + self.assertRaises(ValueError, + pycrypto.generate_rsa_public_and_private, 2049) + + self.assertRaises(tuf.FormatError, + pycrypto.generate_rsa_public_and_private, '2048') + + + def test_create_rsa_signature(self): + global private_rsa + data = 'The quick brown fox jumps over the lazy dog' + signature, method = pycrypto.create_rsa_signature(private_rsa, data) + + # Verify format of returned values. + self.assertNotEqual(None, signature) + self.assertEqual(None, tuf.formats.NAME_SCHEMA.check_match(method), + FORMAT_ERROR_MSG) + self.assertEqual('RSASSA-PSS', method) + + # Check for improperly formatted argument. + self.assertRaises(tuf.FormatError, + pycrypto.create_rsa_signature, 123, data) + + # Check for invalid 'data'. + self.assertRaises(tuf.CryptoError, + pycrypto.create_rsa_signature, private_rsa, 123) + + + def test_verify_rsa_signature(self): + global public_rsa + global private_rsa + data = 'The quick brown fox jumps over the lazy dog' + signature, method = pycrypto.create_rsa_signature(private_rsa, data) + + valid_signature = pycrypto.verify_rsa_signature(signature, method, public_rsa, + data) + self.assertEqual(True, valid_signature) + + # Check for improperly formatted arguments. + self.assertRaises(tuf.FormatError, pycrypto.verify_rsa_signature, signature, + 123, public_rsa, data) + + self.assertRaises(tuf.FormatError, pycrypto.verify_rsa_signature, signature, + method, 123, data) + + self.assertRaises(tuf.FormatError, pycrypto.verify_rsa_signature, 123, method, + public_rsa, data) + + # Check for invalid signature and data. + self.assertRaises(tuf.CryptoError, pycrypto.verify_rsa_signature, signature, + method, public_rsa, 123) + + self.assertEqual(False, pycrypto.verify_rsa_signature(signature, method, + public_rsa, 'mismatched data')) + + mismatched_signature, method = pycrypto.create_rsa_signature(private_rsa, + 'mismatched data') + + self.assertEqual(False, pycrypto.verify_rsa_signature(mismatched_signature, + method, public_rsa, data)) + + + + def test_create_rsa_encrypted_pem(self): + global public_rsa + global private_rsa + passphrase = 'pw' + + # Check format of 'public_rsa'. + self.assertEqual(None, tuf.formats.PEMRSA_SCHEMA.check_match(public_rsa), + FORMAT_ERROR_MSG) + + # Check format of 'passphrase'. + self.assertEqual(None, tuf.formats.PASSWORD_SCHEMA.check_match(passphrase), + FORMAT_ERROR_MSG) + + # Generate the encrypted PEM string of 'public_rsa'. + pem_rsakey = pycrypto.create_rsa_encrypted_pem(private_rsa, passphrase) + + # Check format of 'pem_rsakey'. + self.assertEqual(None, tuf.formats.PEMRSA_SCHEMA.check_match(pem_rsakey), + FORMAT_ERROR_MSG) + + # Check for invalid arguments. + self.assertRaises(tuf.FormatError, + pycrypto.create_rsa_encrypted_pem, 1, passphrase) + self.assertRaises(tuf.FormatError, + pycrypto.create_rsa_encrypted_pem, private_rsa, ['pw']) + + + def test_create_rsa_public_and_private_from_encrypted_pem(self): + global private_rsa + passphrase = 'pw' + + # Generate the encrypted PEM string of 'private_rsa'. + pem_rsakey = pycrypto.create_rsa_encrypted_pem(private_rsa, passphrase) + + # Check format of 'passphrase'. + self.assertEqual(None, tuf.formats.PASSWORD_SCHEMA.check_match(passphrase), + FORMAT_ERROR_MSG) + + # Decrypt 'pem_rsakey' and verify the decrypted object is properly + # formatted. + public_decrypted, private_decrypted = \ + pycrypto.create_rsa_public_and_private_from_encrypted_pem(pem_rsakey, + passphrase) + self.assertEqual(None, + tuf.formats.PEMRSA_SCHEMA.check_match(public_decrypted), + FORMAT_ERROR_MSG) + + self.assertEqual(None, + tuf.formats.PEMRSA_SCHEMA.check_match(private_decrypted), + FORMAT_ERROR_MSG) + + # Does 'public_decrypted' and 'private_decrypted' match the originals? + self.assertEqual(public_rsa, public_decrypted) + self.assertEqual(private_rsa, private_decrypted) + + # Attempt decryption of 'pem_rsakey' using an incorrect passphrase. + self.assertRaises(tuf.CryptoError, + pycrypto.create_rsa_public_and_private_from_encrypted_pem, + pem_rsakey, 'bad_pw') + + # Check for non-encrypted PEM strings. + # create_rsa_public_and_private_from_encrypted_pem() + # returns a tuple of tuf.formats.PEMRSA_SCHEMA objects if the PEM formatted + # string is not actually encrypted but still a valid PEM string. + pub, priv = pycrypto.create_rsa_public_and_private_from_encrypted_pem( + private_rsa, passphrase) + self.assertEqual(None, tuf.formats.PEMRSA_SCHEMA.check_match(pub), + FORMAT_ERROR_MSG) + self.assertEqual(None, tuf.formats.PEMRSA_SCHEMA.check_match(priv), + FORMAT_ERROR_MSG) + + # Check for invalid arguments. + self.assertRaises(tuf.FormatError, + pycrypto.create_rsa_public_and_private_from_encrypted_pem, + 123, passphrase) + self.assertRaises(tuf.FormatError, + pycrypto.create_rsa_public_and_private_from_encrypted_pem, + pem_rsakey, ['pw']) + self.assertRaises(tuf.CryptoError, + pycrypto.create_rsa_public_and_private_from_encrypted_pem, + 'invalid_pem', passphrase) + + + +# Run the unit tests. +if __name__ == '__main__': + unittest.main() diff --git a/tests/unit/test_roledb.py b/tests/unit/test_roledb.py index 0b0259b595..630ccaef78 100755 --- a/tests/unit/test_roledb.py +++ b/tests/unit/test_roledb.py @@ -13,7 +13,6 @@ Unit test for 'roledb.py'. - """ @@ -22,7 +21,7 @@ import tuf import tuf.formats -import tuf.rsa_key +import tuf.keys import tuf.roledb import tuf.log @@ -32,7 +31,7 @@ # Generate the three keys to use in our test cases. KEYS = [] for junk in range(3): - KEYS.append(tuf.rsa_key.generate(2048)) + KEYS.append(tuf.keys.generate_rsa_key(2048)) @@ -320,11 +319,13 @@ def test_create_roledb_from_root_metadata(self): roledict = {'root': {'keyids': [keyid], 'threshold': 1}, 'targets': {'keyids': [keyid2], 'threshold': 1}} version = 8 - expiration_seconds = 200 + consistent_snapshot = False + expires = '2012-10-16 06:42:12 UTC' root_metadata = tuf.formats.RootFile.make_metadata(version, - expiration_seconds, - keydict, roledict) + expires, + keydict, roledict, + consistent_snapshot) self.assertEqual(None, tuf.roledb.create_roledb_from_root_metadata(root_metadata)) # Ensure 'Root' and 'Targets' were added to the role database. @@ -355,22 +356,23 @@ def test_create_roledb_from_root_metadata(self): 'targets/role1': {'keyids': [keyid2], 'threshold': 1}, 'release': {'keyids': [keyid3], 'threshold': 1}} version = 8 - expiration_seconds = 200 # Add a third key for 'release'. keydict[keyid3] = rsakey3 root_metadata = tuf.formats.RootFile.make_metadata(version, - expiration_seconds, - keydict, roledict) + expires, + keydict, roledict, + consistent_snapshot) self.assertRaises(tuf.Error, tuf.roledb.create_roledb_from_root_metadata, root_metadata) # Remove the invalid role and re-generate 'root_metadata' to test for the # other two roles. del roledict['targets/role1'] root_metadata = tuf.formats.RootFile.make_metadata(version, - expiration_seconds, - keydict, roledict) + expires, + keydict, roledict, + consistent_snapshot) self.assertEqual(None, tuf.roledb.create_roledb_from_root_metadata(root_metadata)) diff --git a/tests/unit/test_rsa_key.py b/tests/unit/test_rsa_key.py deleted file mode 100755 index 881ee1d7b1..0000000000 --- a/tests/unit/test_rsa_key.py +++ /dev/null @@ -1,258 +0,0 @@ -""" - - test_rsa_key.py - - - Konstantin Andrianov - - - April 24, 2012. - - - See LICENSE for licensing information. - - - Test cases for rsa_key.py. - - - I'm using 'global rsakey_dict' - there is no harm in doing so since - in order to modify the global variable in any method, python requires - explicit indication to modify i.e. declaring 'global' in each method - that modifies the global variable 'rsakey_dict'. - -""" - -import unittest -import logging - -import tuf -import tuf.log -import tuf.formats -import tuf.rsa_key - -logger = logging.getLogger('tuf.test_rsa_key') - -RSA_KEY = tuf.rsa_key -FORMAT_ERROR_MSG = 'tuf.FormatError was raised! Check object\'s format.' -DATA = 'SOME DATA REQUIRING AUTHENTICITY.' - - -rsakey_dict = RSA_KEY.generate() -temp_key_info_vals = rsakey_dict.values() -temp_key_vals = rsakey_dict['keyval'].values() - - -class TestRsa_key(unittest.TestCase): - def setUp(self): - rsakey_dict['keytype']=temp_key_info_vals[0] - rsakey_dict['keyid']=temp_key_info_vals[1] - rsakey_dict['keyval']=temp_key_info_vals[2] - rsakey_dict['keyval']['public']=temp_key_vals[0] - rsakey_dict['keyval']['private']=temp_key_vals[1] - - - def test_generate(self): - _rsakey_dict = RSA_KEY.generate() - - # Check if the format of the object returned by generate() corresponds - # to RSAKEY_SCHEMA format. - self.assertEqual(None, tuf.formats.RSAKEY_SCHEMA.check_match(_rsakey_dict), - FORMAT_ERROR_MSG) - - # Passing a bit value that is <2048 to generate() - should raise - # 'tuf.FormatError'. - self.assertRaises(tuf.FormatError, RSA_KEY.generate, 555) - - # Passing a string instead of integer for a bit value. - self.assertRaises(tuf.FormatError, RSA_KEY.generate, 'bits') - - # NOTE if random bit value >=2048 (not 4096) is passed generate(bits) - # does not raise any errors and returns a valid key. - self.assertTrue(tuf.formats.RSAKEY_SCHEMA.matches(RSA_KEY.generate(2048))) - self.assertTrue(tuf.formats.RSAKEY_SCHEMA.matches(RSA_KEY.generate(4096))) - - def test_create_in_metadata_format(self): - key_value = rsakey_dict['keyval'] - key_meta = RSA_KEY.create_in_metadata_format(key_value) - - # Check if the format of the object returned by this function corresponds - # to KEY_SCHEMA format. - self.assertEqual(None, - tuf.formats.KEY_SCHEMA.check_match(key_meta), - FORMAT_ERROR_MSG) - key_meta = RSA_KEY.create_in_metadata_format(key_value, private=True) - - # Check if the format of the object returned by this function corresponds - # to KEY_SCHEMA format. - self.assertEqual(None, tuf.formats.KEY_SCHEMA.check_match(key_meta), - FORMAT_ERROR_MSG) - - # Supplying a 'bad' key_value. - del key_value['public'] - self.assertRaises(tuf.FormatError, RSA_KEY.create_in_metadata_format, - key_value) - - - def test_create_from_metadata_format(self): - # Reconfiguring rsakey_dict to conform to KEY_SCHEMA - # i.e. {keytype: 'rsa', keyval: {public: pub_key, private: priv_key}} - #keyid = rsakey_dict['keyid'] - del rsakey_dict['keyid'] - - rsakey_dict_from_meta = RSA_KEY.create_from_metadata_format(rsakey_dict) - - # Check if the format of the object returned by this function corresponds - # to RSAKEY_SCHEMA format. - self.assertEqual(None, - tuf.formats.RSAKEY_SCHEMA.check_match(rsakey_dict_from_meta), - FORMAT_ERROR_MSG) - - # Supplying a wrong number of arguments. - self.assertRaises(TypeError, RSA_KEY.create_from_metadata_format) - args = (rsakey_dict, rsakey_dict) - self.assertRaises(TypeError, RSA_KEY.create_from_metadata_format, *args) - - # Supplying a malformed argument to the function - should get FormatError - del rsakey_dict['keyval'] - self.assertRaises(tuf.FormatError, RSA_KEY.create_from_metadata_format, - rsakey_dict) - - - def test_helper_get_keyid(self): - key_value = rsakey_dict['keyval'] - - # Check format of 'key_value'. - self.assertEqual(None, tuf.formats.KEYVAL_SCHEMA.check_match(key_value), - FORMAT_ERROR_MSG) - - keyid = RSA_KEY._get_keyid(key_value) - - # Check format of 'keyid' - the output of '_get_keyid()' function. - self.assertEqual(None, tuf.formats.KEYID_SCHEMA.check_match(keyid), - FORMAT_ERROR_MSG) - - - def test_createsignature(self): - # Creating a signature for 'DATA'. - signature = RSA_KEY.create_signature(rsakey_dict, DATA) - - # Check format of output. - self.assertEqual(None, - tuf.formats.SIGNATURE_SCHEMA.check_match(signature), - FORMAT_ERROR_MSG) - - # Removing private key from 'rsakey_dict' - should raise a TypeError. - rsakey_dict['keyval']['private'] = '' - - args = (rsakey_dict, DATA) - self.assertRaises(TypeError, RSA_KEY.create_signature, *args) - - # Supplying an incorrect number of arguments. - self.assertRaises(TypeError, RSA_KEY.create_signature) - - - def test_verify_signature(self): - # Creating a signature 'signature' of 'DATA' to be verified. - signature = RSA_KEY.create_signature(rsakey_dict, DATA) - - # Verifying the 'signature' of 'DATA'. - verified = RSA_KEY.verify_signature(rsakey_dict, signature, DATA) - self.assertTrue(verified, "Incorrect signature.") - - # Testing an invalid 'signature'. Same 'signature' is passed, with - # 'DATA' different than the original 'DATA' that was used - # in creating the 'signature'. Function should return 'False'. - - # Modifying 'DATA'. - _DATA = '1111'+DATA+'1111' - - # Verifying the 'signature' of modified '_DATA'. - verified = RSA_KEY.verify_signature(rsakey_dict, signature, _DATA) - self.assertFalse(verified, - 'Returned \'True\' on an incorrect signature.') - - # Modifying 'signature' to pass an incorrect method since only - # 'PyCrypto-PKCS#1 PSS' - # is accepted. - signature['method'] = 'Biff' - - args = (rsakey_dict, signature, DATA) - self.assertRaises(tuf.UnknownMethodError, RSA_KEY.verify_signature, *args) - - # Passing incorrect number of arguments. - self.assertRaises(TypeError,RSA_KEY.verify_signature) - - - def test_create_encrypted_pem(self): - passphrase = 'pw' - - # Check format of 'rsakey_dict'. - self.assertEqual(None, tuf.formats.RSAKEY_SCHEMA.check_match(rsakey_dict), - FORMAT_ERROR_MSG) - - # Check format of 'passphrase'. - self.assertEqual(None, tuf.formats.PASSWORD_SCHEMA.check_match(passphrase), - FORMAT_ERROR_MSG) - - # Generate the encrypted PEM string of 'rsakey_dict'. - pem_rsakey = tuf.rsa_key.create_encrypted_pem(rsakey_dict, passphrase) - - # Check for invalid arguments. - self.assertRaises(tuf.FormatError, - tuf.rsa_key.create_encrypted_pem, 'Biff', passphrase) - self.assertRaises(tuf.FormatError, - tuf.rsa_key.create_encrypted_pem, rsakey_dict, ['pw']) - - - - def test_create_from_encrypted_pem(self): - passphrase = 'pw' - - # Check format of 'rsakey_dict'. - self.assertEqual(None, tuf.formats.RSAKEY_SCHEMA.check_match(rsakey_dict), - FORMAT_ERROR_MSG) - - # Check format of 'passphrase'. - self.assertEqual(None, tuf.formats.PASSWORD_SCHEMA.check_match(passphrase), - FORMAT_ERROR_MSG) - - # Generate the encrypted PEM string of 'rsakey_dict'. - pem_rsakey = tuf.rsa_key.create_encrypted_pem(rsakey_dict, passphrase) - - # Decrypt 'pem_rsakey' and verify the decrypted object is properly - # formatted. - decrypted_rsakey = tuf.rsa_key.create_from_encrypted_pem(pem_rsakey, - passphrase) - self.assertEqual(None, tuf.formats.RSAKEY_SCHEMA.check_match(decrypted_rsakey), - FORMAT_ERROR_MSG) - - # Does 'decrypted_rsakey' match the original 'rsakey_dict'. - self.assertEqual(rsakey_dict, decrypted_rsakey) - - # Attempt decryption of 'pem_rsakey' using an incorrect passphrase. - self.assertRaises(tuf.CryptoError, - tuf.rsa_key.create_from_encrypted_pem, pem_rsakey, - 'bad_pw') - # Check for non-encrypted PEM string. create_from_encrypted_pem()/PyCrypto - # returns a tuf.formats.RSAKEY_SCHEMA object if PEM formatted string is - # not actually encrypted but still a valid PEM string. - non_encrypted_private_key = rsakey_dict['keyval']['private'] - decrypted_non_encrypted = tuf.rsa_key.create_from_encrypted_pem( - non_encrypted_private_key, passphrase) - self.assertEqual(None, tuf.formats.RSAKEY_SCHEMA.check_match( - decrypted_non_encrypted), FORMAT_ERROR_MSG) - - # Check for invalid arguments. - self.assertRaises(tuf.FormatError, - tuf.rsa_key.create_from_encrypted_pem, 123, passphrase) - self.assertRaises(tuf.FormatError, - tuf.rsa_key.create_from_encrypted_pem, pem_rsakey, ['pw']) - self.assertRaises(tuf.CryptoError, - tuf.rsa_key.create_from_encrypted_pem, 'invalid_pem', - passphrase) - - - -# Run the unit tests. -if __name__ == '__main__': - unittest.main() diff --git a/tests/unit/test_sig.py b/tests/unit/test_sig.py index efdbe5162b..b8205a0721 100755 --- a/tests/unit/test_sig.py +++ b/tests/unit/test_sig.py @@ -14,10 +14,8 @@ Test cases for for sig.py. - """ - import unittest import logging @@ -26,7 +24,7 @@ import tuf.formats import tuf.keydb import tuf.roledb -import tuf.rsa_key +import tuf.keys import tuf.sig logger = logging.getLogger('tuf.test_sig') @@ -34,7 +32,7 @@ # Setup the keys to use in our test cases. KEYS = [] for _ in range(3): - KEYS.append(tuf.rsa_key.generate(2048)) + KEYS.append(tuf.keys.generate_rsa_key(2048)) @@ -52,10 +50,10 @@ def test_get_signature_status_no_role(self): # Should verify we are not adding a duplicate signature # when doing the following action. Here we know 'signable' # has only one signature so it's okay. - signable['signatures'].append(tuf.sig.generate_rsa_signature( - signable['signed'], KEYS[0])) + signable['signatures'].append(tuf.keys.create_signature( + KEYS[0], signable['signed'])) - tuf.keydb.add_rsakey(KEYS[0]) + tuf.keydb.add_key(KEYS[0]) # No specific role we're considering. sig_status = tuf.sig.get_signature_status(signable, None) @@ -78,11 +76,11 @@ def test_get_signature_status_no_role(self): def test_get_signature_status_bad_sig(self): signable = {'signed' : 'test', 'signatures' : []} - signable['signatures'].append(tuf.sig.generate_rsa_signature( - signable['signed'], KEYS[0])) + signable['signatures'].append(tuf.keys.create_signature( + KEYS[0], signable['signed'])) signable['signed'] += 'signature no longer matches signed data' - tuf.keydb.add_rsakey(KEYS[0]) + tuf.keydb.add_key(KEYS[0]) threshold = 1 roleinfo = tuf.formats.make_role_metadata( [KEYS[0]['keyid']], threshold) @@ -108,11 +106,11 @@ def test_get_signature_status_bad_sig(self): def test_get_signature_status_unknown_method(self): signable = {'signed' : 'test', 'signatures' : []} - signable['signatures'].append(tuf.sig.generate_rsa_signature( - signable['signed'], KEYS[0])) + signable['signatures'].append(tuf.keys.create_signature( + KEYS[0], signable['signed'])) signable['signatures'][0]['method'] = 'fake-sig-method' - tuf.keydb.add_rsakey(KEYS[0]) + tuf.keydb.add_key(KEYS[0]) threshold = 1 roleinfo = tuf.formats.make_role_metadata( [KEYS[0]['keyid']], threshold) @@ -139,10 +137,10 @@ def test_get_signature_status_unknown_method(self): def test_get_signature_status_single_key(self): signable = {'signed' : 'test', 'signatures' : []} - signable['signatures'].append(tuf.sig.generate_rsa_signature( - signable['signed'], KEYS[0])) + signable['signatures'].append(tuf.keys.create_signature( + KEYS[0], signable['signed'])) - tuf.keydb.add_rsakey(KEYS[0]) + tuf.keydb.add_key(KEYS[0]) threshold = 1 roleinfo = tuf.formats.make_role_metadata( [KEYS[0]['keyid']], threshold) @@ -168,10 +166,10 @@ def test_get_signature_status_single_key(self): def test_get_signature_status_below_threshold(self): signable = {'signed' : 'test', 'signatures' : []} - signable['signatures'].append(tuf.sig.generate_rsa_signature( - signable['signed'], KEYS[0])) + signable['signatures'].append(tuf.keys.create_signature( + KEYS[0], signable['signed'])) - tuf.keydb.add_rsakey(KEYS[0]) + tuf.keydb.add_key(KEYS[0]) threshold = 2 roleinfo = tuf.formats.make_role_metadata( [KEYS[0]['keyid'], @@ -200,13 +198,13 @@ def test_get_signature_status_below_threshold_unrecognized_sigs(self): signable = {'signed' : 'test', 'signatures' : []} # Two keys sign it, but only one of them will be trusted. - signable['signatures'].append(tuf.sig.generate_rsa_signature( - signable['signed'], KEYS[0])) - signable['signatures'].append(tuf.sig.generate_rsa_signature( - signable['signed'], KEYS[2])) + signable['signatures'].append(tuf.keys.create_signature( + KEYS[0], signable['signed'])) + signable['signatures'].append(tuf.keys.create_signature( + KEYS[2], signable['signed'])) - tuf.keydb.add_rsakey(KEYS[0]) - tuf.keydb.add_rsakey(KEYS[1]) + tuf.keydb.add_key(KEYS[0]) + tuf.keydb.add_key(KEYS[1]) threshold = 2 roleinfo = tuf.formats.make_role_metadata( [KEYS[0]['keyid'], @@ -237,13 +235,13 @@ def test_get_signature_status_below_threshold_unauthorized_sigs(self): # Two keys sign it, but one of them is only trusted for a different # role. - signable['signatures'].append(tuf.sig.generate_rsa_signature( - signable['signed'], KEYS[0])) - signable['signatures'].append(tuf.sig.generate_rsa_signature( - signable['signed'], KEYS[1])) + signable['signatures'].append(tuf.keys.create_signature( + KEYS[0], signable['signed'])) + signable['signatures'].append(tuf.keys.create_signature( + KEYS[1], signable['signed'])) - tuf.keydb.add_rsakey(KEYS[0]) - tuf.keydb.add_rsakey(KEYS[1]) + tuf.keydb.add_key(KEYS[0]) + tuf.keydb.add_key(KEYS[1]) threshold = 2 roleinfo = tuf.formats.make_role_metadata( [KEYS[0]['keyid'], KEYS[2]['keyid']], threshold) @@ -275,10 +273,10 @@ def test_get_signature_status_below_threshold_unauthorized_sigs(self): def test_check_signatures_no_role(self): signable = {'signed' : 'test', 'signatures' : []} - signable['signatures'].append(tuf.sig.generate_rsa_signature( - signable['signed'], KEYS[0])) + signable['signatures'].append(tuf.keys.create_signature( + KEYS[0], signable['signed'])) - tuf.keydb.add_rsakey(KEYS[0]) + tuf.keydb.add_key(KEYS[0]) # No specific role we're considering. It's invalid to use the # function tuf.sig.verify() without a role specified because @@ -292,10 +290,10 @@ def test_check_signatures_no_role(self): def test_verify_single_key(self): signable = {'signed' : 'test', 'signatures' : []} - signable['signatures'].append(tuf.sig.generate_rsa_signature( - signable['signed'], KEYS[0])) + signable['signatures'].append(tuf.keys.create_signature( + KEYS[0], signable['signed'])) - tuf.keydb.add_rsakey(KEYS[0]) + tuf.keydb.add_key(KEYS[0]) threshold = 1 roleinfo = tuf.formats.make_role_metadata( [KEYS[0]['keyid']], threshold) @@ -316,13 +314,13 @@ def test_verify_unrecognized_sig(self): signable = {'signed' : 'test', 'signatures' : []} # Two keys sign it, but only one of them will be trusted. - signable['signatures'].append(tuf.sig.generate_rsa_signature( - signable['signed'], KEYS[0])) - signable['signatures'].append(tuf.sig.generate_rsa_signature( - signable['signed'], KEYS[2])) + signable['signatures'].append(tuf.keys.create_signature( + KEYS[0], signable['signed'])) + signable['signatures'].append(tuf.keys.create_signature( + KEYS[2], signable['signed'])) - tuf.keydb.add_rsakey(KEYS[0]) - tuf.keydb.add_rsakey(KEYS[1]) + tuf.keydb.add_key(KEYS[0]) + tuf.keydb.add_key(KEYS[1]) threshold = 2 roleinfo = tuf.formats.make_role_metadata( [KEYS[0]['keyid'], KEYS[1]['keyid']], threshold) @@ -341,15 +339,15 @@ def test_verify_unrecognized_sig(self): def test_generate_rsa_signature(self): signable = {'signed' : 'test', 'signatures' : []} - signable['signatures'].append(tuf.sig.generate_rsa_signature( - signable['signed'], KEYS[0])) + signable['signatures'].append(tuf.keys.create_signature( + KEYS[0], signable['signed'])) self.assertEqual(1, len(signable['signatures'])) signature = signable['signatures'][0] self.assertEqual(KEYS[0]['keyid'], signature['keyid']) - signable['signatures'].append(tuf.sig.generate_rsa_signature( - signable['signed'], KEYS[1])) + signable['signatures'].append(tuf.keys.create_signature( + KEYS[1], signable['signed'])) self.assertEqual(2, len(signable['signatures'])) signature = signable['signatures'][1] @@ -360,10 +358,10 @@ def test_may_need_new_keys(self): # One untrusted key in 'signable'. signable = {'signed' : 'test', 'signatures' : []} - signable['signatures'].append(tuf.sig.generate_rsa_signature( - signable['signed'], KEYS[0])) + signable['signatures'].append(tuf.keys.create_signature( + KEYS[0], signable['signed'])) - tuf.keydb.add_rsakey(KEYS[1]) + tuf.keydb.add_key(KEYS[1]) threshold = 1 roleinfo = tuf.formats.make_role_metadata( [KEYS[1]['keyid']], threshold) diff --git a/tests/unit/test_updater.py b/tests/unit/test_updater.py index 0b8385ab56..cecba6feac 100755 --- a/tests/unit/test_updater.py +++ b/tests/unit/test_updater.py @@ -20,7 +20,6 @@ unittest_toolbox module was created to provide additional testing tools for tuf's modules. For more info see unittest_toolbox.py. - Unit tests must follow a specific structure i.e. independent methods should be tested prior to dependent methods. More accurately: least dependent @@ -32,7 +31,6 @@ class guarantees the order of unit tests. So that, 'test_something_A' a number will be placed after 'test' and before methods name like so: 'test_1_check_directory'. The number is a measure of dependence, where 1 is less dependent than 2. - """ import os @@ -51,7 +49,7 @@ class guarantees the order of unit tests. So that, 'test_something_A' import tuf.formats import tuf.keydb import tuf.repo.keystore as keystore -import tuf.repo.signerlib as signerlib +import tuf.repository_tool as repo_tool import tuf.roledb import tuf.tests.repository_setup as setup import tuf.tests.unittest_toolbox as unittest_toolbox @@ -108,7 +106,7 @@ def test__init__exceptions(self): for directory, junk, role_list in os.walk(client_current_dir): for role_filepath in role_list: role_filepath = os.path.join(directory, role_filepath) - if role_filepath.endswith('root.txt'): + if role_filepath.endswith('root.json'): continue os.remove(role_filepath) updater.Updater('Repo_Name', self.mirrors) @@ -134,18 +132,18 @@ def setUpClass(cls): # Server side references. cls.server_repo_dir = cls.repositories['server_repository'] cls.server_meta_dir = os.path.join(cls.server_repo_dir, 'metadata') - cls.root_filepath = os.path.join(cls.server_meta_dir, 'root.txt') - cls.timestamp_filepath = os.path.join(cls.server_meta_dir, 'timestamp.txt') - cls.targets_filepath = os.path.join(cls.server_meta_dir, 'targets.txt') - cls.release_filepath = os.path.join(cls.server_meta_dir, 'release.txt') + cls.root_filepath = os.path.join(cls.server_meta_dir, 'root.json') + cls.timestamp_filepath = os.path.join(cls.server_meta_dir, 'timestamp.json') + cls.targets_filepath = os.path.join(cls.server_meta_dir, 'targets.json') + cls.snapshot_filepath = os.path.join(cls.server_meta_dir, 'snapshot.json') # References to delegated metadata paths and directories. cls.delegated_dir1 = os.path.join(cls.server_meta_dir, 'targets') cls.delegated_filepath1 = os.path.join(cls.delegated_dir1, - 'delegated_role1.txt') + 'delegated_role1.json') cls.delegated_dir2 = os.path.join(cls.delegated_dir1, 'delegated_role1') cls.delegated_filepath2 = os.path.join(cls.delegated_dir2, - 'delegated_role2.txt') + 'delegated_role2.json') cls.targets_dir = os.path.join(cls.server_repo_dir, 'targets') # Client side references. @@ -172,7 +170,7 @@ def setUp(self): # used as an optional argument to 'download_url_to_tempfileobj' patch # function. self.all_role_paths = [self.timestamp_filepath, - self.release_filepath, + self.snapshot_filepath, self.root_filepath, self.targets_filepath, self.delegated_filepath1, @@ -248,7 +246,7 @@ def _remove_filepath(self, filepath): def _add_target_to_targets_dir(self, targets_keyids): """ Adds a file to server's 'targets' directory and rebuilds - targets metadata (targets.txt). + targets metadata (targets.json). """ targets_sub_dir = os.path.join(self.targets_dir, 'targets_sub_dir') @@ -274,7 +272,7 @@ def _add_target_to_targets_dir(self, targets_keyids): def _remove_target_from_targets_dir(self, target_filename, remove_all=True): """ Remove a target 'target_filename' from server's targets directory and - rebuild 'targets', 'release', 'timestamp' metadata files. + rebuild 'targets', 'snapshot', 'timestamp' metadata files. 'target_filename' is relative to targets directory. Example of 'target_filename': 'targets_sub_dir/somefile.txt'. @@ -345,8 +343,8 @@ def _update_top_level_roles(self): # Reference self.Repository._update_metadata_if_changed(). update_if_changed = self.Repository._update_metadata_if_changed - self._mock_download_url_to_tempfileobj(self.release_filepath) - update_if_changed('release', referenced_metadata = 'timestamp') + self._mock_download_url_to_tempfileobj(self.snapshot_filepath) + update_if_changed('snapshot', referenced_metadata = 'timestamp') self._mock_download_url_to_tempfileobj(self.root_filepath) update_if_changed('root') @@ -363,19 +361,25 @@ def _update_top_level_roles(self): def test_1__load_metadata_from_file(self): # Setup - # Get root.txt file path. Extract root metadata, + # Get root.json file path. Extract root metadata, # it will be compared with content of loaded root metadata. - root_filepath = os.path.join(self.client_current_dir, 'root.txt') + root_filepath = os.path.join(self.client_current_dir, 'root.json') root_meta = tuf.util.load_json_file(root_filepath) - # Test: normal case. - for role in self.role_list: - self.Repository._load_metadata_from_file('current', role) + #for role in self.role_list: + # self.Repository._load_metadata_from_file('current', role) + self.Repository.refresh() - # Verify that the correct number of metadata objects has been loaded. + root_filepath = os.path.join(self.client_current_dir, 'root.json') + root_meta = tuf.util.load_json_file(root_filepath) + + # Verify that the correct number of metadata objects has been loaded. self.assertEqual(len(self.Repository.metadata['current']), 4) + print('root_metadata: '+repr(root_meta)+'\n\n') + print('self.Repository: '+repr(self.Repository.metadata['current']['root'])) + # Verify that the content of root metadata is valid. self.assertEqual(self.Repository.metadata['current']['root'], root_meta['signed']) @@ -414,16 +418,16 @@ def test_1__update_fileinfo(self): # Load file info for top level roles. This populates the fileinfo # dictionary. for role in self.role_list: - self.Repository._update_fileinfo(role+'.txt') + self.Repository._update_fileinfo(role+'.json') # Verify that fileinfo has been populated and contains appropriate data. self.assertTrue(self.Repository.fileinfo) for role in self.role_list: - role_filepath = os.path.join(self.client_current_dir, role+'.txt') + role_filepath = os.path.join(self.client_current_dir, role+'.json') role_info = tuf.util.get_file_details(role_filepath) role_info_dict = {'length':role_info[0], 'hashes':role_info[1]} - self.assertTrue(role+'.txt' in self.Repository.fileinfo.keys()) - self.assertEqual(self.Repository.fileinfo[role+'.txt'], role_info_dict) + self.assertTrue(role+'.json' in self.Repository.fileinfo.keys()) + self.assertEqual(self.Repository.fileinfo[role+'.json'], role_info_dict) @@ -465,7 +469,7 @@ def test_2__import_delegations(self): - + """ def test_2__ensure_all_targets_allowed(self): # Setup # Reference to self.Repository._ensure_all_targets_allowed() @@ -477,7 +481,7 @@ def test_2__ensure_all_targets_allowed(self): targets_meta_dir = os.path.join(self.server_meta_dir, 'targets') role1_meta_dir = os.path.join(targets_meta_dir, 'delegated_role1') - role1_path = os.path.join(targets_meta_dir, 'delegated_role1.txt') + role1_path = os.path.join(targets_meta_dir, 'delegated_role1.json') role1_metadata_signable = tuf.util.load_json_file(role1_path) role1_metadata = role1_metadata_signable['signed'] @@ -495,15 +499,14 @@ def test_2__ensure_all_targets_allowed(self): # delegated role's metadata are not indicated in the metadata of the # delegated role's parent, we need to modify delegated role's 'targets' # field. - target = self.random_string()+'.txt' + target = self.random_string()+'.json' deleg_target_path = os.path.join('delegated_level', target) role1_metadata['targets'][deleg_target_path] = self.random_string() # Test: targets not included in the parent's metadata. self.assertRaises(tuf.RepositoryError, ensure_all_targets_allowed, - 'targets/delegated_role1', - role1_metadata) - + 'targets/delegated_role1', role1_metadata) + """ @@ -511,26 +514,26 @@ def test_2__ensure_all_targets_allowed(self): def test_2__fileinfo_has_changed(self): # Verify that the method returns 'False' if file info was not changed. for role in self.role_list: - role_filepath = os.path.join(self.client_current_dir, role+'.txt') + role_filepath = os.path.join(self.client_current_dir, role+'.json') role_info = tuf.util.get_file_details(role_filepath) role_info_dict = {'length':role_info[0], 'hashes':role_info[1]} - self.assertFalse(self.Repository._fileinfo_has_changed(role+'.txt', + self.assertFalse(self.Repository._fileinfo_has_changed(role+'.json', role_info_dict)) # Verify that the method returns 'True' if length or hashes were changed. for role in self.role_list: - role_filepath = os.path.join(self.client_current_dir, role+'.txt') + role_filepath = os.path.join(self.client_current_dir, role+'.json') role_info = tuf.util.get_file_details(role_filepath) role_info_dict = {'length':8, 'hashes':role_info[1]} - self.assertTrue(self.Repository._fileinfo_has_changed(role+'.txt', + self.assertTrue(self.Repository._fileinfo_has_changed(role+'.json', role_info_dict)) for role in self.role_list: - role_filepath = os.path.join(self.client_current_dir, role+'.txt') + role_filepath = os.path.join(self.client_current_dir, role+'.json') role_info = tuf.util.get_file_details(role_filepath) role_info_dict = {'length':role_info[0], 'hashes':{'sha256':self.random_string()}} - self.assertTrue(self.Repository._fileinfo_has_changed(role+'.txt', + self.assertTrue(self.Repository._fileinfo_has_changed(role+'.json', role_info_dict)) @@ -540,13 +543,13 @@ def test_2__move_current_to_previous(self): # The test will consist of removing a metadata file from client's # {client_repository}/metadata/previous directory, executing the method # and then verifying that the 'previous' directory contains - # the release file. - release_meta_path = os.path.join(self.client_previous_dir, 'release.txt') - os.remove(release_meta_path) - self.assertFalse(os.path.exists(release_meta_path)) - self.Repository._move_current_to_previous('release') - self.assertTrue(os.path.exists(release_meta_path)) - shutil.copy(release_meta_path, self.client_current_dir) + # the snapshot file. + snapshot_meta_path = os.path.join(self.client_previous_dir, 'snapshot.json') + os.remove(snapshot_meta_path) + self.assertFalse(os.path.exists(snapshot_meta_path)) + self.Repository._move_current_to_previous('snapshot') + self.assertTrue(os.path.exists(snapshot_meta_path)) + shutil.copy(snapshot_meta_path, self.client_current_dir) @@ -561,7 +564,7 @@ def test_2__delete_metadata(self): self.Repository._delete_metadata('timestamp') self.assertFalse('timestamp' in self.Repository.metadata['current']) timestamp_meta_path = os.path.join(self.client_previous_dir, - 'timestamp.txt') + 'timestamp.json') shutil.copy(timestamp_meta_path, self.client_current_dir) @@ -614,7 +617,7 @@ def test_3__update_metadata(self): # Test: Invalid file downloaded. # Patch 'download.download_url_to_tempfileobj' function. - self._mock_download_url_to_tempfileobj(self.release_filepath) + self._mock_download_url_to_tempfileobj(self.snapshot_filepath) # TODO: Is this the original intent of this test? self.assertRaises(TypeError, _update_metadata, 'targets', None) @@ -624,7 +627,7 @@ def test_3__update_metadata(self): # Patch 'download.download_url_to_tempfileobj' function. self._mock_download_url_to_tempfileobj(self.targets_filepath) uncompressed_fileinfo = \ - signerlib.get_metadata_file_info(self.targets_filepath) + repo_tool.get_metadata_file_info(self.targets_filepath) _update_metadata('targets', uncompressed_fileinfo) list_of_targets = self.Repository.metadata['current']['targets']['targets'] @@ -637,23 +640,25 @@ def test_3__update_metadata(self): # Add a file to targets directory and rebuild targets metadata. added_target_2 = self._add_target_to_targets_dir(targets_keyids) uncompressed_fileinfo = \ - signerlib.get_metadata_file_info(self.targets_filepath) + repo_tool.get_metadata_file_info(self.targets_filepath) # To test compressed file handling, compress targets metadata file. targets_filepath_compressed = self._compress_file(self.targets_filepath) compressed_fileinfo = \ - signerlib.get_metadata_file_info(targets_filepath_compressed) + repo_tool.get_metadata_file_info(targets_filepath_compressed) # Re-patch 'download.download_url_to_tempfileobj' function. self._mock_download_url_to_tempfileobj(targets_filepath_compressed) + # The length (but not the hash) passed to this function is incorrect. The # length must be that of the compressed file, whereas the hash must be that # of the uncompressed file. mixed_fileinfo = { 'length': compressed_fileinfo['length'], - 'hashes': uncompressed_fileinfo['hashes'] - } - _update_metadata('targets', mixed_fileinfo, compression='gzip') + 'hashes': uncompressed_fileinfo['hashes']} + + _update_metadata('targets', mixed_fileinfo, compression='gzip', + compressed_fileinfo=compressed_fileinfo) list_of_targets = self.Repository.metadata['current']['targets']['targets'] # Verify that the added target's path is listed in target's metadata. @@ -663,7 +668,7 @@ def test_3__update_metadata(self): # Restoring server's repository to the initial state. os.remove(targets_filepath_compressed) - os.remove(os.path.join(self.client_current_dir,'targets.txt')) + os.remove(os.path.join(self.client_current_dir,'targets.json')) self._remove_target_from_targets_dir(added_target_1) @@ -688,24 +693,24 @@ def test_3__update_metadata_if_changed(self): update_if_changed = self.Repository._update_metadata_if_changed - # Test: normal case. Update 'release' metadata. + # Test: normal case. Update 'snapshot' metadata. # Patch download_file. self._mock_download_url_to_tempfileobj(self.timestamp_filepath) - # Update timestamp metadata, it will indicate change in release metadata. + # Update timestamp metadata, it will indicate change in snapshot metadata. self.Repository._update_metadata('timestamp', DEFAULT_TIMESTAMP_FILEINFO) - # Save current release metadata before updating. It will be used to + # Save current snapshot metadata before updating. It will be used to # verify the update. - old_release_meta = self.Repository.metadata['current']['release'] - self._mock_download_url_to_tempfileobj(self.release_filepath) + old_snapshot_meta = self.Repository.metadata['current']['snapshot'] + self._mock_download_url_to_tempfileobj(self.snapshot_filepath) - # Update release metadata, it will indicate change in targets metadata. - update_if_changed(metadata_role='release', referenced_metadata='timestamp') - current_release_meta = self.Repository.metadata['current']['release'] - previous_release_meta = self.Repository.metadata['previous']['release'] - self.assertEqual(old_release_meta, previous_release_meta) - self.assertNotEqual(old_release_meta, current_release_meta) + # Update snapshot metadata, it will indicate change in targets metadata. + update_if_changed(metadata_role='snapshot', referenced_metadata='timestamp') + current_snapshot_meta = self.Repository.metadata['current']['snapshot'] + previous_snapshot_meta = self.Repository.metadata['previous']['snapshot'] + self.assertEqual(old_snapshot_meta, previous_snapshot_meta) + self.assertNotEqual(old_snapshot_meta, current_snapshot_meta) # Test: normal case. Update 'targets' metadata. @@ -719,37 +724,37 @@ def test_3__update_metadata_if_changed(self): self.fail('\nFailed to update targets metadata.') - # Test: normal case. Update compressed release file. - release_filepath_compressed = self._compress_file(self.release_filepath) + # Test: normal case. Update compressed snapshot file. + snapshot_filepath_compressed = self._compress_file(self.snapshot_filepath) # Since client's '.../metadata/current' will need to have separate # gzipped metadata file in order to test compressed file handling, # we need to copy it there. - shutil.copy(release_filepath_compressed, self.client_current_dir) + shutil.copy(snapshot_filepath_compressed, self.client_current_dir) # Add a target file and rebuild metadata files at the server side. added_target_2 = self._add_target_to_targets_dir(targets_keyids) - # Since release file was updated, update compressed release file. - release_filepath_compressed = self._compress_file(self.release_filepath) + # Since snapshot file was updated, update compressed snapshot file. + snapshot_filepath_compressed = self._compress_file(self.snapshot_filepath) # Patch download_file. self._mock_download_url_to_tempfileobj(self.timestamp_filepath) - # Update timestamp metadata, it will indicate change in release metadata. + # Update timestamp metadata, it will indicate change in snapshot metadata. self.Repository._update_metadata('timestamp', DEFAULT_TIMESTAMP_FILEINFO) - # Save current release metadata before updating. It will be used to + # Save current snapshot metadata before updating. It will be used to # verify the update. - old_release_meta = self.Repository.metadata['current']['release'] - self._mock_download_url_to_tempfileobj(self.release_filepath) - - # Update release metadata, and verify the change. - update_if_changed(metadata_role='release', referenced_metadata='timestamp') - current_release_meta = self.Repository.metadata['current']['release'] - previous_release_meta = self.Repository.metadata['previous']['release'] - self.assertEqual(old_release_meta, previous_release_meta) - self.assertNotEqual(old_release_meta, current_release_meta) + old_snapshot_meta = self.Repository.metadata['current']['snapshot'] + self._mock_download_url_to_tempfileobj(self.snapshot_filepath) + + # Update snapshot metadata, and verify the change. + update_if_changed(metadata_role='snapshot', referenced_metadata='timestamp') + current_snapshot_meta = self.Repository.metadata['current']['snapshot'] + previous_snapshot_meta = self.Repository.metadata['previous']['snapshot'] + self.assertEqual(old_snapshot_meta, previous_snapshot_meta) + self.assertNotEqual(old_snapshot_meta, current_snapshot_meta) # Test: Invalid targets metadata file downloaded. @@ -764,8 +769,8 @@ def test_3__update_metadata_if_changed(self): assert isinstance(mirror_error, tuf.DownloadLengthMismatchError) # Restoring repositories to the initial state. - os.remove(release_filepath_compressed) - os.remove(os.path.join(self.client_current_dir, 'release.txt.gz')) + os.remove(snapshot_filepath_compressed) + os.remove(os.path.join(self.client_current_dir, 'snapshot.json.gz')) self._remove_target_from_targets_dir(added_target_1) @@ -786,7 +791,7 @@ def test_3__targets_of_role(self): for target in range(len(targets_list)): targets_filepaths.append(targets_list[target]['filepath']) for dir_target in targets_dir_content: - if dir_target.endswith('.txt'): + if dir_target.endswith('.json'): self.assertTrue(dir_target in targets_filepaths) @@ -843,9 +848,9 @@ def test_4__refresh_targets_metadata(self): # Delegated roles paths. role1_dir = os.path.join(self.server_meta_dir, 'targets') - role1_filepath = os.path.join(role1_dir, 'delegated_role1.txt') + role1_filepath = os.path.join(role1_dir, 'delegated_role1.json') role2_dir = os.path.join(role1_dir, 'delegated_role1') - role2_filepath = os.path.join(role2_dir, 'delegated_role2.txt') + role2_filepath = os.path.join(role2_dir, 'delegated_role2.json') # Create a file in the delegated targets directory. deleg_target_filepath2 = self._add_file_to_directory(targets_deleg_dir2) @@ -914,7 +919,7 @@ def test_5_all_targets(self): # Verify that there is a correct number of records in 'all_targets' list. # On the repository there are 4 target files, 2 of which are delegated. # The targets role lists all targets, for a total of 4. The two delegated - # roles each list 1 of the already listed targets in 'targets.txt', for a + # roles each list 1 of the already listed targets in 'targets.json', for a # total of 2 (the delegated targets are listed twice). The total number of # targets in 'all_targets' should then be 6. self.assertTrue(len(all_targets) is 6) @@ -937,7 +942,7 @@ def test_5_targets_of_role(self): for target in range(len(targets_list)): targets_filepaths.append(targets_list[target]['filepath']) for dir_target in targets_dir_content: - if dir_target.endswith('.txt'): + if dir_target.endswith('.json'): self.assertTrue(dir_target in targets_filepaths) @@ -960,7 +965,7 @@ def test_6_target(self): # Test: normal case. for _target in targets_dir_content: - if _target.endswith('.txt'): + if _target.endswith('.json'): target_info = target(_target) # Verify that 'target_info' corresponds to 'TARGETFILE_SCHEMA'. self.assertTrue(tuf.formats.TARGETFILE_SCHEMA.matches(target_info)) diff --git a/tests/unit/test_util_test_tools.py b/tests/unit/test_util_test_tools.py index 0acf513176..bfd86d7f74 100755 --- a/tests/unit/test_util_test_tools.py +++ b/tests/unit/test_util_test_tools.py @@ -93,14 +93,14 @@ def test_correct_directory_structure(self): metadata_dir = os.path.join(tuf_repo, 'metadata') current_dir = os.path.join(tuf_client, 'metadata', 'current') - # Verify '{root_repo}/tuf_repo/metadata/role.txt' paths exists. - for role in ['root', 'targets', 'release', 'timestamp']: + # Verify '{root_repo}/tuf_repo/metadata/role.json' paths exists. + for role in ['root', 'targets', 'snapshot', 'timestamp']: # Repository side. - role_file = os.path.join(metadata_dir, role+'.txt') + role_file = os.path.join(metadata_dir, role+'.json') self.assertTrue(os.path.isfile(role_file)) # Client side. - role_file = os.path.join(current_dir, role+'.txt') + role_file = os.path.join(current_dir, role+'.json') self.assertTrue(os.path.isfile(role_file)) # Verify '{root_repo}/tuf_repo/keystore/keyid.key' exists. @@ -126,8 +126,8 @@ def test_methods(self): Note: here file at the 'filepath' and the 'target' file at tuf-targets directory are identical files. - Ex: filepath = '{root_repo}/reg_repo/file.txt' - target = '{root_repo}/tuf_repo/targets/file.txt' + Ex: filepath = '{root_repo}/reg_repo/file.json' + target = '{root_repo}/tuf_repo/targets/file.json' """ reg_repo = os.path.join(self.root_repo, 'reg_repo') diff --git a/tuf/README.md b/tuf/README.md new file mode 100644 index 0000000000..d0c9609ae0 --- /dev/null +++ b/tuf/README.md @@ -0,0 +1,428 @@ +# Repository Management # + +## Table of Contents ## +- [Repository Tool Diagram](#repository-tool-diagram) +- [Create TUF Repository](#create-tuf-repository) + - [Purpose](#purpose) + - [Keys](#keys) + - [Create RSA Keys](#create-rsa-keys) + - [Import RSA Keys](#import-rsa-keys) + - [Create and Import ED25519 Keys](#create-and-import-ed25519-keys) + - [Create a New Repository](#create-a-new-repository) + - [Create Root](#create-root) + - [Create Timestamp, Snapshot, Targets](#create-timestamp-snapshot-targets) + - [Targets](#targets) + - [Add Target Files](#add-target-files) + - [Remove Target Files](#remove-target-files) + - [Delegations](#delegations) + - [Revoke Delegated Role](#revoke-delegated-role) + - [Delegate to Hashed Bins](#delegate-to-hashed-bins) + - [Consistent Snapshots](#consistent-snapshots) +- [Client Setup and Repository Trial](#client-setup-and-repository-trial) + - [Using TUF Within an Example Client Updater](#using-tuf-within-an-example-client-updater) + - [Test TUF Locally](#test-tuf-locally) + + +## Repository Tool Diagram ## +![Repo Tools Diagram 1](https://raw.github.com/theupdateframework/tuf/repository-tools/docs/images/repository_tool-diagram.png) + + +## Create TUF Repository ## + +### Purpose ### + +The **tuf.repository_tool** module can be used to create a TUF repository. +It may either be imported into a Python module or used with the Python +interpreter in interactive mode. + +```Bash +$ python +Python 2.7.3 (default, Sep 26 2013, 20:08:41) +[GCC 4.6.3] on linux2 +Type "help", "copyright", "credits" or "license" for more information. +>>> from tuf.repository_tool import * +>>> repository = load_repository("path/to/repository") +``` +Note that *tuf.repository_tool.py* is not used in TUF integrations. The +**tuf.interposition** package and **tuf.client.updater** module assist in +integrating TUF with a software updater. + + +### Keys ### + +#### Create RSA Keys #### +```python +from tuf.repository_tool import * + +# Generate and write the first of two root keys for the TUF repository. +# The following function creates an RSA key pair, where the private key is saved to +# "path/to/root_key" and the public key to "path/to/root_key.pub". +generate_and_write_rsa_keypair("path/to/root_key", bits=2048, password="password") + +# If the key length is unspecified, it defaults to 3072 bits. A length of less +# than 2048 bits raises an exception. A password may be supplied as an +# argument, otherwise a user prompt is presented. +generate_and_write_rsa_keypair("path/to/root_key2") +Enter a password for the RSA key: +Confirm: +``` +The following four key files should now exist: + +1. root_key +2. root_key.pub +3. root_key2 +4. root_key2.pub + +### Import RSA Keys ### +```python +from tuf.repository_tool import * + +# Import an existing public key. +public_root_key = import_rsa_publickey_from_file("path/to/root_key.pub") + +# Import an existing private key. Importing a private key requires a password, whereas +# importing a public key does not. +private_root_key = import_rsa_privatekey_from_file("path/to/root_key") +Enter a password for the encrypted RSA key: +``` +import_rsa_privatekey_from_file() raises a "tuf.CryptoError" exception if the key/password +is invalid. + +### Create and Import ED25519 Keys ### +```Python +from tuf.repository_tool import * + +# Generate and write an ed25519 key pair. The private key is saved encrypted. +# A 'password' argument may be supplied, otherwise a prompt is presented. +generate_and_write_ed25519_keypair('path/to/ed25519_key') +Enter a password for the ED25519 key: +Confirm: + +# Import the ed25519 public key just created . . . +public_ed25519_key = import_ed25519_publickey_from_file('path/to/ed25519_key.pub') + +# and its corresponding private key. +private_ed25519_key = import_ed25519_privatekey_from_file('path/to/ed25519_key') +Enter a password for the encrypted ED25519 key: +``` + +### Create a New Repository ### + +#### Create Root #### +```python +# Continuing from the previous section . . . + +# Create a new Repository object that holds the file path to the repository and the four +# top-level role objects (Root, Targets, Snapshot, Timestamp). Metadata files are created when +# repository.write() is called. The repository directory is created if it does not exist. +repository = create_new_repository("path/to/repository/") + +# The Repository instance, 'repository', initially contains top-level Metadata objects. +# Add one of the public keys, created in the previous section, to the root role. Metadata is +# considered valid if it is signed by the public key's corresponding private key. +repository.root.add_verification_key(public_root_key) + +# Role keys (i.e., the key's keyid) may be queried. Other attributes include: signing_keys, version, +# signatures, expiration, threshold, delegations (Targets role), and compressions. +repository.root.keys +['b23514431a53676595922e955c2d547293da4a7917e3ca243a175e72bbf718df'] + +# Add a second public key to the root role. Although previously generated and saved to a file, +# the second public key must be imported before it can added to a role. +public_root_key2 = import_rsa_publickey_from_file("path/to/root_key2.pub") +repository.root.add_verification_key(public_root_key2) + +# Threshold of each role defaults to 1. Users may change the threshold value, but repository_tool.py +# validates thresholds and warns users. Set the threshold of the root role to 2, +# which means the root metadata file is considered valid if it contains at least two valid +# signatures. +repository.root.threshold = 2 +private_root_key2 = import_rsa_privatekey_from_file("path/to/root_key2", password="password") + +# Load the root signing keys to the repository, which write() uses to sign the root metadata. +# The load_signing_key() method SHOULD warn when the key is NOT explicitly allowed to +# sign for it. +repository.root.load_signing_key(private_root_key) +repository.root.load_signing_key(private_root_key2) + +# Print the number of valid signatures and public/private keys of the repository's metadata. +repository.status() +'root' role contains 2 / 2 signatures. +'targets' role contains 0 / 1 public keys. + +try: + repository.write() + +# An exception is raised here by write() because the other top-level roles (targets, snapshot, +# and timestamp) have not been configured with keys. Another option is to call +# repository.write_partial() and generate metadata that may contain an invalid threshold of +# signatures, required public keys, etc. write_partial() allows multiple repository maintainers to +# independently sign metadata and generate them separately. load_repository() can load partially +# written metadata. +except tuf.UnsignedMetadataError, e: + print e +Not enough signatures for 'path/to/repository/metadata.staged/targets.json' + +# In the next section, update the other top-level roles and create a repository with valid metadata. +``` + +#### Create Timestamp, Snapshot, Targets + +```python +# Continuing from the previous section . . . + +# Generate keys for the remaining top-level roles. The root keys have been set above. +# The password argument may be omitted if a password prompt is needed. +generate_and_write_rsa_keypair("path/to/targets_key", password="password") +generate_and_write_rsa_keypair("path/to/snapshot_key", password="password") +generate_and_write_rsa_keypair("path/to/timestamp_key", password="password") + +# Add the public keys of the remaining top-level roles. +repository.targets.add_verification_key(import_rsa_publickey_from_file("path/to/targets_key.pub")) +repository.snapshot.add_verification_key(import_rsa_publickey_from_file("path/to/snapshot_key.pub")) +repository.timestamp.add_verification_key(import_rsa_publickey_from_file("path/to/timestamp_key.pub")) + +# Import the signing keys of the remaining top-level roles. Prompt for passwords. +private_targets_key = import_rsa_privatekey_from_file("path/to/targets_key") +Enter a password for the encrypted RSA key: + +private_snapshot_key = import_rsa_privatekey_from_file("path/to/snapshot_key") +Enter a password for the encrypted RSA key: + +private_timestamp_key = import_rsa_privatekey_from_file("path/to/timestamp_key") +Enter a password for the encrypted RSA key: + +# Load the signing keys of the remaining roles so that valid signatures are generated when +# repository.write() is called. +repository.targets.load_signing_key(private_targets_key) +repository.snapshot.load_signing_key(private_snapshot_key) +repository.timestamp.load_signing_key(private_timestamp_key) + +# Optionally set the expiration date of the timestamp role. By default, roles are set to expire +# as follows: root(1 year), targets(3 months), snapshot(1 week), timestamp(1 day). +repository.timestamp.expiration = "2014-10-28 12:08:00" + +# Metadata files may also be compressed. Only "gz" is currently supported. +repository.targets.compressions = ["gz"] +repository.snapshot.compressions = ["gz"] + +# Write all metadata to "path/to/repository/metadata.staged/". The common case is to crawl the +# filesystem for all delegated roles in "path/to/repository/metadata.staged/targets/". +repository.write() +``` + +### Targets ### + +#### Add Target Files #### +```Bash +# Create and save target files to the targets directory of the repository. +$ cd path/to/repository/targets/ +$ echo 'file1' > file1.txt +$ echo 'file2' > file2.txt +$ echo 'file3' > file3.txt +$ mkdir django; echo 'file4' > django/file4.txt +``` + +```python +from tuf.repository_tool import * + +# Load the repository created in the previous section. This repository so far contains metadata for +# the top-level roles, but no targets. +repository = load_repository("path/to/repository/") + +# get_filepaths_in_directory() returns a list of file paths in a directory. It can also return +# files in sub-directories if 'recursive_walk' is True. +list_of_targets = repository.get_filepaths_in_directory("path/to/repository/targets/", + recursive_walk=False, followlinks=True) + +# Add the list of target paths to the metadata of the Targets role. Any target file paths +# that may already exist are NOT replaced. add_targets() does not create or move target files. +# Any target paths added to a role must be relative to the targets directory, otherwise an +# exception is raised. +repository.targets.add_targets(list_of_targets) + +# Individual target files may also be added. +repository.targets.add_target("path/to/repository/targets/file3.txt") + +# The private key of the updated targets metadata must be loaded before it can be signed and +# written (Note the load_repository() call above). +private_targets_key = import_rsa_privatekey_from_file("path/to/targets_key") +Enter a password for the encrypted RSA key: + +repository.targets.load_signing_key(private_targets_key) + +# Due to the load_repository(), we must also load the private keys of the other top-level roles +# to generate a valid set of metadata. +private_root_key = import_rsa_privatekey_from_file("path/to/root_key") +Enter a password for the encrypted RSA key: + +private_root_key2 = import_rsa_privatekey_from_file("path/to/root_key2") +Enter a password for the encrypted RSA key: + +private_snapshot_key = import_rsa_privatekey_from_file("path/to/snapshot_key") +Enter a password for the encrypted RSA key: + +private_timestamp_key = import_rsa_privatekey_from_file("path/to/timestamp_key") +Enter a password for the encrypted RSA key: + +repository.root.load_signing_key(private_root_key) +repository.root.load_signing_key(private_root_key2) +repository.snapshot.load_signing_key(private_snapshot_key) +repository.timestamp.load_signing_key(private_timestamp_key) + +# Generate new versions of all the top-level metadata. +repository.write() +``` + +#### Remove Target Files #### +```python +# Continuing from the previous section . . . + +# Remove a target file listed in the "targets" metadata. The target file is not actually deleted +# from the file system. +repository.targets.remove_target("path/to/repository/targets/file3.txt") + +# repository.write() creates any new metadata files, updates those that have changed, and any that +# need updating to make a new "snapshot" (new snapshot.json and timestamp.json). +repository.write() +``` + +### Delegations ### +```python +# Continuing from the previous section . . . + +# Generate a key for a new delegated role named "unclaimed". +generate_and_write_rsa_keypair("path/to/unclaimed_key", bits=2048, password="password") +public_unclaimed_key = import_rsa_publickey_from_file("path/to/unclaimed_key.pub") + +# Make a delegation from "targets" to "targets/unclaimed", initially containing zero targets. +# The delegated role’s full name is not expected. +# delegate(rolename, list_of_public_keys, list_of_file_paths, threshold, +# restricted_paths, path_hash_prefixes) +repository.targets.delegate("unclaimed", [public_unclaimed_key], []) + +# Load the private key of "targets/unclaimed" so that signatures are later added and valid +# metadata is created. +private_unclaimed_key = import_rsa_privatekey_from_file("path/to/unclaimed_key") +Enter a password for the encrypted RSA key: + +repository.targets(unclaimed).load_signing_key(private_unclaimed_key) + +# Update an attribute of the unclaimed role. +repository.targets('unclaimed').version = 2 + +# Delegations may also be nested. Create the delegated role "targets/unclaimed/django", +# where it initially contains zero targets and future targets are restricted to a +# particular directory. +repository.targets('unclaimed').delegate("django", [public_unclaimed_key], [], + restricted_paths=["path/to/repository/targets/django/"]) +repository.targets('unclaimed')('django').load_signing_key(private_unclaimed_key) +repository.targets('unclaimed')('django').add_target("path/to/repository/targets/django/file4.txt") +repository.targets('unclaimed')('django').compressions = ["gz"] + +# Write the metadata of "targets/unclaimed", "targets/unclaimed/django", root, targets, snapshot, +# and timestamp. +repository.write() +``` + +#### Revoke Delegated Role #### +```python +# Continuing from the previous section . . . + +# Create a delegated role that will be revoked in the next step. +repository.targets('unclaimed').delegate("flask", [public_unclaimed_key], []) + +# Revoke "targets/unclaimed/flask" and write the metadata of all remaining roles. +repository.targets('unclaimed').revoke("flask") +repository.write() +``` + +```Bash +# Copy the staged metadata directory changes to the live repository. +$ cp -r "path/to/repository/metadata.staged/" "path/to/repository/metadata/" +``` + +#### Delegate to Hashed Bins #### + +A large number of target files may also be distributed to multiple hashed bins +(delegated roles). The metadata files of delegated roles will be nearly equal in size +(i.e., target file paths are uniformly distributed by calculating the target filepath's +digest and determining which bin it should reside in. The updater client will use +"lazy bin walk" to find a target file's hashed bin destination. This method is intended +for repositories with a large number of target files, a way of easily distributing and +managing the metadata that lists the targets, and minimizing the number of metadata files +(and size) downloaded by the client. + +Method that handles hashed bin delegations and example: +```Python +delegate_hashed_bins(list_of_targets, keys_of_hashed_bins, number_of_bins) +``` + +```Python +# Get a list of target paths for the hashed bins. +targets = \ + repository.get_filepaths_in_directory('path/to/repository/targets/django', recursive_walk=True) +repository.targets('unclaimed')('django').delegate_hashed_bins(targets, [public_unclaimed_key], 32) + +# delegated_hashed_bins() only assigns the public key(s) of the hashed bins, so the private keys may +# be manually loaded as follows: +for delegation in repository.targets('unclaimed')('django').delegations: + delegation.load_signing_key(private_unclaimed_key) + +# Delegated roles can be restricted to particular paths with add_restricted_paths(). +repository.targets('unclaimed').add_restricted_paths('path/to/repository/targets/django', 'django') +``` + +#### Consistent Snapshots #### +A repository may optionally support multiple versions of `snapshot.json` simultaneously, where +a client with version 1 of `snapshot.json` can download `target_file.zip` and another client with +version 2 of `snapshot.json` can also download a different `target_file.zip` (same file +name, but different file digest). If the `consistent_snapshot` parameter of write() is True, +metadata and target file names on the file system have their digests prepended (note: target file +names specified in metadata do not have digests included in their names). The repository +maintainer is responsible for the duration of multiple versions of metadata and target files +available on a repository. +```Python +repository.write(consistent_snapshot=True) +``` + + +## Client Setup and Repository Trial ## + +### Using TUF Within an Example Client Updater ### +```python +from tuf.repository_tool import * + +# The following function creates a directory structure that a client +# downloading new software using TUF (via tuf/client/updater.py) will expect. +# The root.json metadata file must exist, and also the directories that hold the metadata files +# downloaded from a repository. Software updaters integrating with TUF may use this +# directory to store TUF updates saved on the client side. create_tuf_client_directory() +# moves metadata from "path/to/repository/metadata" to "path/to/client/". The repository +# in "path/to/repository/" is the repository created in the "Create TUF Repository" section. +create_tuf_client_directory("path/to/repository/", "path/to/client/") +``` + +### Test TUF Locally ### +```Bash +# Run the local TUF repository server. +$ cd "path/to/repository/"; python -m SimpleHTTPServer 8001 + +# Retrieve targets from the TUF repository and save them to "path/to/client/". The +# basic_client.py module is available in "tuf/client/". +# In a different command-line prompt . . . +$ cd "path/to/client/" +$ ls +metadata/ + +$ basic_client.py --repo http://localhost:8001 +$ ls . targets/ targets/django/ +.: +metadata targets tuf.log + +targets/: +django file1.txt file2.txt + +targets/django/: +file4.txt +``` diff --git a/tuf/__init__.py b/tuf/__init__.py index 7fa5e4af3f..14a61f6c18 100755 --- a/tuf/__init__.py +++ b/tuf/__init__.py @@ -18,7 +18,6 @@ The names chosen for TUF Exception classes should end in 'Error' except where there is a good reason not to, and provide that reason in those cases. - """ import urlparse @@ -117,6 +116,14 @@ class RepositoryError(Error): +class InsufficientKeysError(Error): + """Indicate that metadata role lacks a threshold of pubic or private keys.""" + pass + + + + + class ForbiddenTargetError(RepositoryError): """Indicate that a role signed for a target that it was not delegated to.""" pass @@ -165,7 +172,7 @@ class CryptoError(Error): class BadSignatureError(CryptoError): - """Indicate that some metadata file had a bad signature.""" + """Indicate that some metadata file has a bad signature.""" def __init__(self, metadata_role_name): self.metadata_role_name = metadata_role_name @@ -285,6 +292,13 @@ class InvalidNameError(Error): +class UnsignedMetadataError(Error): + """Indicate metadata object with insufficient threshold of signatures.""" + + + + + class NoWorkingMirrorError(Error): """An updater will throw this exception in case it could not download a metadata or target file. diff --git a/ed25519/__init__.py b/tuf/_vendor/__init__.py old mode 100755 new mode 100644 similarity index 100% rename from ed25519/__init__.py rename to tuf/_vendor/__init__.py diff --git a/tuf/_vendor/ed25519/.gitignore b/tuf/_vendor/ed25519/.gitignore new file mode 100644 index 0000000000..0d20b6487c --- /dev/null +++ b/tuf/_vendor/ed25519/.gitignore @@ -0,0 +1 @@ +*.pyc diff --git a/tuf/_vendor/ed25519/.travis.yml b/tuf/_vendor/ed25519/.travis.yml new file mode 100644 index 0000000000..8518ca28be --- /dev/null +++ b/tuf/_vendor/ed25519/.travis.yml @@ -0,0 +1,28 @@ +language: python +python: 2.7 +env: + - TOXENV=py26 + - TOXENV=py27 + #- TOXENV=py32 + #- TOXENV=py33 + - TOXENV=pypy + +install: + # Add the PyPy repository + - "if [[ $TOXENV == 'pypy' ]]; then sudo add-apt-repository -y ppa:pypy/ppa; fi" + # Upgrade PyPy + - "if [[ $TOXENV == 'pypy' ]]; then sudo apt-get -y install pypy; fi" + # This is required because we need to get rid of the Travis installed PyPy + # or it'll take precedence over the PPA installed one. + - "if [[ $TOXENV == 'pypy' ]]; then sudo rm -rf /usr/local/pypy/bin; fi" + - pip install tox + +script: + - tox + +notifications: + irc: + channels: + - "irc.freenode.org#cryptography-dev" + use_notice: true + skip_join: true diff --git a/tuf/_vendor/ed25519/__init__.py b/tuf/_vendor/ed25519/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/ed25519/checkparams.py b/tuf/_vendor/ed25519/checkparams.py similarity index 100% rename from ed25519/checkparams.py rename to tuf/_vendor/ed25519/checkparams.py diff --git a/tuf/_vendor/ed25519/ed25519.py b/tuf/_vendor/ed25519/ed25519.py new file mode 100644 index 0000000000..b7e9ff046e --- /dev/null +++ b/tuf/_vendor/ed25519/ed25519.py @@ -0,0 +1,161 @@ +import hashlib + + +b = 256 +q = 2 ** 255 - 19 +l = 2 ** 252 + 27742317777372353535851937790883648493 + + +def H(m): + return hashlib.sha512(m).digest() + + +def pow2(x, p): + """== pow(x, 2**p, q)""" + while p > 0: + x = x * x % q + p -= 1 + return x + +def inv(z): + """$= z^{-1} \mod q$, for z != 0""" + # Adapted from curve25519_athlon.c in djb's Curve25519. + z2 = z * z % q # 2 + z9 = pow2(z2, 2) * z % q # 9 + z11 = z9 * z2 % q # 11 + z2_5_0 = (z11*z11)%q * z9 % q # 31 == 2^5 - 2^0 + z2_10_0 = pow2(z2_5_0, 5) * z2_5_0 % q # 2^10 - 2^0 + z2_20_0 = pow2(z2_10_0, 10) * z2_10_0 % q # ... + z2_40_0 = pow2(z2_20_0, 20) * z2_20_0 % q + z2_50_0 = pow2(z2_40_0, 10) * z2_10_0 % q + z2_100_0 = pow2(z2_50_0, 50) * z2_50_0 % q + z2_200_0 = pow2(z2_100_0, 100) * z2_100_0 % q + z2_250_0 = pow2(z2_200_0, 50) * z2_50_0 % q # 2^250 - 2^0 + return pow2(z2_250_0, 5) * z11 % q # 2^255 - 2^5 + 11 = q - 2 + + +d = -121665 * inv(121666) +I = pow(2, (q - 1) / 4, q) + + +def xrecover(y): + xx = (y * y - 1) * inv(d * y * y + 1) + x = pow(xx, (q + 3) / 8, q) + + if (x * x - xx) % q != 0: + x = (x * I) % q + + if x % 2 != 0: + x = q-x + + return x + + +By = 4 * inv(5) +Bx = xrecover(By) +B = (Bx % q, By % q) + + +def edwards(P, Q): + x1, y1 = P + x2, y2 = Q + x3 = (x1 * y2 + x2 * y1) * inv(1 + d * x1 * x2 * y1 * y2) + y3 = (y1 * y2 + x1 * x2) * inv(1 - d * x1 * x2 * y1 * y2) + + return (x3 % q, y3 % q) + + +def scalarmult(P, e): + if e == 0: + return (0, 1) + + Q = scalarmult(P, e / 2) + Q = edwards(Q, Q) + + if e & 1: + Q = edwards(Q, P) + + return Q + + +def encodeint(y): + bits = [(y >> i) & 1 for i in range(b)] + return ''.join([ + chr(sum([bits[i * 8 + j] << j for j in range(8)])) + for i in range(b/8) + ]) + + +def encodepoint(P): + x = P[0] + y = P[1] + bits = [(y >> i) & 1 for i in range(b - 1)] + [x & 1] + return ''.join([ + chr(sum([bits[i * 8 + j] << j for j in range(8)])) + for i in range(b/8) + ]) + + +def bit(h, i): + return (ord(h[i / 8]) >> (i % 8)) & 1 + + +def publickey(sk): + h = H(sk) + a = 2 ** (b - 2) + sum(2 ** i * bit(h, i) for i in range(3, b - 2)) + A = scalarmult(B, a) + return encodepoint(A) + + +def Hint(m): + h = H(m) + return sum(2 ** i * bit(h, i) for i in range(2 * b)) + + +def signature(m, sk, pk): + h = H(sk) + a = 2 ** (b - 2) + sum(2 ** i * bit(h, i) for i in range(3, b - 2)) + r = Hint(''.join([h[j] for j in range(b / 8, b / 4)]) + m) + R = scalarmult(B, r) + S = (r + Hint(encodepoint(R) + pk + m) * a) % l + return encodepoint(R) + encodeint(S) + + +def isoncurve(P): + x, y = P + return (-x * x + y * y - 1 - d * x * x * y * y) % q == 0 + + +def decodeint(s): + return sum(2 ** i * bit(s, i) for i in range(0, b)) + + +def decodepoint(s): + y = sum(2 ** i * bit(s, i) for i in range(0, b - 1)) + x = xrecover(y) + + if x & 1 != bit(s, b-1): + x = q-x + + P = (x, y) + + if not isoncurve(P): + raise Exception("decoding point that is not on curve") + + return P + + +def checkvalid(s, m, pk): + if len(s) != b / 4: + raise Exception("signature length is wrong") + + if len(pk) != b / 8: + raise Exception("public-key length is wrong") + + R = decodepoint(s[:b / 8]) + A = decodepoint(pk) + S = decodeint(s[b / 8:b / 4]) + h = Hint(encodepoint(R) + pk + m) + + if scalarmult(B, S) != edwards(R, scalarmult(A, h)): + raise Exception("signature does not pass verification") diff --git a/tuf/_vendor/ed25519/runtests.sh b/tuf/_vendor/ed25519/runtests.sh new file mode 100755 index 0000000000..4dec6743ab --- /dev/null +++ b/tuf/_vendor/ed25519/runtests.sh @@ -0,0 +1,8 @@ +#!/bin/sh +set -e + +python -u signfast.py < sign.input + +if [[ $TEST == 'slow' ]]; then + python -u sign.py < sign.input +fi diff --git a/tuf/_vendor/ed25519/science.py b/tuf/_vendor/ed25519/science.py new file mode 100644 index 0000000000..18a2fcc2c1 --- /dev/null +++ b/tuf/_vendor/ed25519/science.py @@ -0,0 +1,32 @@ +import os +import timeit + +import ed25519 + + +seed = os.urandom(32) + +data = "The quick brown fox jumps over the lazy dog" +private_key = seed +public_key = ed25519.publickey(seed) +signature = ed25519.signature(data, private_key, public_key) + + +print('Time generate') +print(timeit.timeit("ed25519.publickey(seed)", + setup="from __main__ import ed25519, seed", + number=10, +)) + +print('\nTime create signature') +print(timeit.timeit("ed25519.signature(data, private_key, public_key)", + setup="from __main__ import ed25519, data, private_key, public_key", + number=10, +)) + + +print('\nTime verify signature') +print(timeit.timeit("ed25519.checkvalid(signature, data, public_key)", + setup="from __main__ import ed25519, signature, data, public_key", + number=10, +)) diff --git a/tuf/_vendor/ed25519/setup.py b/tuf/_vendor/ed25519/setup.py new file mode 100644 index 0000000000..d65e3d2a50 --- /dev/null +++ b/tuf/_vendor/ed25519/setup.py @@ -0,0 +1,11 @@ +from setuptools import setup + + +setup( + name="ed25519", + version="1.0", + + py_modules="ed25519", + + zip_safe=False, +) diff --git a/ed25519/sign.input b/tuf/_vendor/ed25519/sign.input similarity index 100% rename from ed25519/sign.input rename to tuf/_vendor/ed25519/sign.input diff --git a/ed25519/sign.py b/tuf/_vendor/ed25519/sign.py similarity index 94% rename from ed25519/sign.py rename to tuf/_vendor/ed25519/sign.py index be099ad2ac..18eea684e6 100644 --- a/ed25519/sign.py +++ b/tuf/_vendor/ed25519/sign.py @@ -1,3 +1,5 @@ +from __future__ import print_function + import sys import binascii import ed25519 @@ -16,6 +18,7 @@ while 1: line = sys.stdin.readline() if not line: break + print(".", end="") x = line.split(':') sk = binascii.unhexlify(x[0][0:64]) pk = ed25519.publickey(sk) diff --git a/tuf/_vendor/ed25519/signfast.py b/tuf/_vendor/ed25519/signfast.py new file mode 100644 index 0000000000..630c92cac4 --- /dev/null +++ b/tuf/_vendor/ed25519/signfast.py @@ -0,0 +1,48 @@ +from __future__ import print_function + +import sys +import binascii +import ed25519 + +# examples of inputs: see sign.input +# should produce no output: python sign.py < sign.input + +# warning: currently 37 seconds/line on a fast machine + +# fields on each input line: sk, pk, m, sm +# each field hex +# each field colon-terminated +# sk includes pk at end +# sm includes m at end + +MAX = 10 + +i = 0 +while 1: + if i >= MAX: + break + i += 1 + line = sys.stdin.readline() + if not line: break + print(".", end="") + x = line.split(':') + sk = binascii.unhexlify(x[0][0:64]) + pk = ed25519.publickey(sk) + m = binascii.unhexlify(x[2]) + s = ed25519.signature(m,sk,pk) + ed25519.checkvalid(s,m,pk) + forgedsuccess = 0 + try: + if len(m) == 0: + forgedm = "x" + else: + forgedmlen = len(m) + forgedm = ''.join([chr(ord(m[i])+(i==forgedmlen-1)) for i in range(forgedmlen)]) + ed25519.checkvalid(s,forgedm,pk) + forgedsuccess = 1 + except: + pass + assert not forgedsuccess + assert x[0] == binascii.hexlify(sk + pk) + assert x[1] == binascii.hexlify(pk) + assert x[3] == binascii.hexlify(s + m) diff --git a/tuf/_vendor/ed25519/tox.ini b/tuf/_vendor/ed25519/tox.ini new file mode 100644 index 0000000000..1fdba16678 --- /dev/null +++ b/tuf/_vendor/ed25519/tox.ini @@ -0,0 +1,5 @@ +[tox] +envlist = py26,py27,pypy,py32,py33 + +[testenv] +commands = ./runtests.sh diff --git a/tuf/client/README.md b/tuf/client/README.md new file mode 100644 index 0000000000..633e151657 --- /dev/null +++ b/tuf/client/README.md @@ -0,0 +1,172 @@ +#updater.py +**updater.py** is intended as the only TUF module that software update +systems need to utilize for a low-level integration. It provides a single +class representing an updater that includes methods to download, install, and +verify metadata or target files in a secure manner. Importing +**tuf.client.updater.py** and instantiating its main class is all that is +required by the client prior to a TUF update request. The importation and +instantiation steps allow TUF to load all of the required metadata files +and set the repository mirror information. + +The **tuf.repository_tool** module can be used to create a TUF repository. See +[tuf/README](../README.md) for more information on creating TUF repositories. + +The **tuf.interposition** package can also assist in integrating TUF with a +software updater. See [tuf/interposition/README](../interposition/README.md) +for more information on interposing Python urllib calls with TUF. + + +## Overview of the Update Process +1. The software update system instructs TUF to check for updates. + +2. TUF downloads and verifies timestamp.json. + +3. If timestamp.json indicates that snapshot.json has changed, TUF downloads and +verifies snapshot.json. + +4. TUF determines which metadata files listed in snapshot.json differ from those +described in the last snapshot.json that TUF has seen. If root.json has changed, +the update process starts over using the new root.json. + +5. TUF provides the software update system with a list of available files +according to targets.json. + +6. The software update system instructs TUF to download a specific target +file. + +7. TUF downloads and verifies the file and then makes the file available to +the software update system. + + +## Example Client +### Refresh TUF Metadata and Download Target Files +```Python +# The client first imports the 'updater.py' module, the only module the +# client is required to import. The client will utilize a single class +# from this module. +import tuf.client.updater + +# The only other module the client interacts with is 'tuf.conf'. The +# client accesses this module solely to set the repository directory. +# This directory will hold the files downloaded from a remote repository. +tuf.conf.repository_directory = 'path/to/local_repository' + +# Next, the client creates a dictionary object containing the repository +# mirrors. The client may download content from any one of these mirrors. +# In the example below, a single mirror named 'mirror1' is defined. The +# mirror is located at 'http://localhost:8001', and all of the metadata +# and targets files can be found in the 'metadata' and 'targets' directory, +# respectively. If the client wishes to only download target files from +# specific directories on the mirror, the 'confined_target_dirs' field +# should be set. In the example, the client has chosen '', which is +# interpreted as no confinement. In other words, the client can download +# targets from any directory or subdirectories. If the client had chosen +# 'targets1/', they would have been confined to the '/targets/targets1/' +# directory on the 'http://localhost:8001' mirror. +repository_mirrors = {'mirror1': {'url_prefix': 'http://localhost:8001', + 'metadata_path': 'metadata', + 'targets_path': 'targets', + 'confined_target_dirs': ['']}} + +# The updater may now be instantiated. The Updater class of 'updater.py' +# is called with two arguments. The first argument assigns a name to this +# particular updater and the second argument the repository mirrors defined +# above. +updater = tuf.client.updater.Updater('updater', repository_mirrors) + +# The client calls the refresh() method to ensure it has the latest +# copies of the top-level metadata files (i.e., Root, Targets, Snapshot, +# Timestamp). +updater.refresh() + +# The target file information of all the repository targets is determined next. +# Since all_targets() downloads the target files of every role, all role +# metadata is updated. +targets = updater.all_targets() + +# Among these targets, determine the ones that have changed since the client's +# last refresh(). A target is considered updated if it does not exist in +# 'destination_directory' (current directory) or the target located there has +# changed. +destination_directory = '.' +updated_targets = updater.updated_targets(targets, destination_directory) + +# Lastly, attempt to download each target among those that have changed. +# The updated target files are saved locally to 'destination_directory'. +for target in updated_targets: + updater.download_target(target, destination_directory) + +# Remove any files from the destination directory that are no longer being +# tracked. For example, a target file from a previous snapshot that has since +# been removed on the remote repository. +updater.remove_obsolete_targets(destination_directory) +``` + +### Download Target Files of a Role +```Python +# Example demonstrating an update that only downloads the targets of +# a specific role (i.e., 'targets/django'). + +# Refresh the metadata of the top-level roles (i.e., Root, Targets, Snapshot, Timestamp). +updater.refresh() + +# Update the 'targets/django' role, and determine the target files that have changed. +# targets_of_role() refreshes the minimum metadata needed to download the target files +# of the specified role (e.g., R1->R4->R5, where R2 and R3 are excluded). +targets_of_django = updater.targets_of_role('targets/django') +updated_targets = updater.updated_targets(targets_of_django, destination_directory) + +for target in updated_targets: + updater.download_target(target, destination_directory) +``` + +### Download Specific Target File +```Python +# Example demonstrating an update that downloads a specific target. + +# Refresh the metadata of the top-level roles (i.e., Root, Targets, Snapshot, Timestamp). +updater.refresh() + +# target() updates role metadata when required. +target = updater.target('LICENSE.txt') +updated_target = updater.updated_targets([target], destination_directory) + +for target in updated_target: + updater.download_target(target, destination_directory) +``` + +###A Simple Integration Example with basic_client.py +```Bash +# Assume a simple TUF repository has been setup with 'tuf.repository_tool.py'. +$ basic_client.py --repo http://localhost:8001 + +# Metadata and target files are silently updated. An exception is only raised if an error, +# or attack, is detected. Inspect 'tuf.log' for the outcome of the update process. + +$ cat tuf.log +[2013-12-16 16:17:05,267 UTC] [tuf.download] [INFO][_download_file:726@download.py] +Downloading: http://localhost:8001/metadata/timestamp.json + +[2013-12-16 16:17:05,269 UTC] [tuf.download] [WARNING][_check_content_length:589@download.py] +reported_length (545) < required_length (2048) + +[2013-12-16 16:17:05,269 UTC] [tuf.download] [WARNING][_check_downloaded_length:656@download.py] +Downloaded 545 bytes, but expected 2048 bytes. There is a difference of 1503 bytes! + +[2013-12-16 16:17:05,611 UTC] [tuf.download] [INFO][_download_file:726@download.py] +Downloading: http://localhost:8001/metadata/snapshot.json + +[2013-12-16 16:17:05,612 UTC] [tuf.client.updater] [INFO][_check_hashes:636@updater.py] +The file\'s sha256 hash is correct: 782675fadd650eeb2926d33c401b5896caacf4fd6766498baf2bce2f3b739db4 + +[2013-12-16 16:17:05,951 UTC] [tuf.download] [INFO][_download_file:726@download.py] +Downloading: http://localhost:8001/metadata/targets.json + +[2013-12-16 16:17:05,952 UTC] [tuf.client.updater] [INFO][_check_hashes:636@updater.py] +The file\'s sha256 hash is correct: a5019c28a1595c43a14cad2b6252c4d1db472dd6412a9204181ad6d61b1dd69a + +[2013-12-16 16:17:06,299 UTC] [tuf.download] [INFO][_download_file:726@download.py] +Downloading: http://localhost:8001/targets/file1.txt + +[2013-12-16 16:17:06,303 UTC] [tuf.client.updater] [INFO][_check_hashes:636@updater.py] +The file's sha256 hash is correct: ecdc5536f73bdae8816f0ea40726ef5e9b810d914493075903bb90623d97b1d8 diff --git a/tuf/client/basic_client.py b/tuf/client/basic_client.py index 1ba36dbc32..c57852caf3 100755 --- a/tuf/client/basic_client.py +++ b/tuf/client/basic_client.py @@ -72,7 +72,7 @@ def update_client(repository_mirror): in the current working directory. The current directory must already include a 'metadata' directory, which in turn must contain the 'current' and 'previous' directories. At a minimum, these two directories require - the 'root.txt' metadata file. + the 'root.json' metadata file. repository_mirror: @@ -211,7 +211,7 @@ def parse_options(): # the current directory. try: update_client(repository_mirror) - except (tuf.RepositoryError, tuf.ExpiredMetadataError), e: + except (tuf.NoWorkingMirrorError, tuf.RepositoryError), e: sys.stderr.write('Error: '+str(e)+'\n') sys.exit(1) diff --git a/tuf/client/updater.py b/tuf/client/updater.py index 3b875fb359..9de3002920 100755 --- a/tuf/client/updater.py +++ b/tuf/client/updater.py @@ -26,23 +26,23 @@ 1. The software update system instructs TUF to check for updates. - 2. TUF downloads and verifies timestamp.txt. + 2. TUF downloads and verifies timestamp.json. - 3. If timestamp.txt indicates that release.txt has changed, TUF downloads and - verifies release.txt. + 3. If timestamp.json indicates that snapshot.json has changed, TUF downloads + and verifies snapshot.json. - 4. TUF determines which metadata files listed in release.txt differ from those - described in the last release.txt that TUF has seen. If root.txt has changed, - the update process starts over using the new root.txt. + 4. TUF determines which metadata files listed in snapshot.json differ from + those described in the last snapshot.json that TUF has seen. If root.json + has changed, the update process starts over using the new root.json. 5. TUF provides the software update system with a list of available files - according to targets.txt. + according to targets.json. 6. The software update system instructs TUF to download a specific target - file. + file. 7. TUF downloads and verifies the file and then makes the file available to - the software update system. + the software update system. @@ -97,7 +97,6 @@ # The updated target files are saved locally to 'destination_directory'. for target in updated_targets: updater.download_target(target, destination_directory) - """ import errno @@ -105,12 +104,15 @@ import os import shutil import time +import urllib +import random import tuf import tuf.conf import tuf.download import tuf.formats import tuf.hash +import tuf.keys import tuf.keydb import tuf.log import tuf.mirrors @@ -144,7 +146,7 @@ class Updater(object): self.fileinfo: A cache of lengths and hashes of stored metadata files. - Example: {'root.txt': {'length': 13323, + Example: {'root.json': {'length': 13323, 'hashes': {'sha256': dbfac345..}}, ...} @@ -158,7 +160,7 @@ class Updater(object): refresh(): This method downloads, verifies, and loads metadata for the top-level - roles in a specific order (i.e., timestamp -> release -> root -> targets) + roles in a specific order (i.e., timestamp -> snapshot -> root -> targets) The expiration time for downloaded metadata is also verified. The metadata for delegated roles are not refreshed by this method, but by @@ -195,6 +197,12 @@ class Updater(object): served by the repository but have since been removed, can be deleted from disk by the client by calling this method. + Note: The methods listed above are public and intended for the software + updater integrating TUF with this module. All other methods that may begin + with a single leading underscore are non-public and only used internally. + updater.py is not subclassed in TUF, nor is it designed to be subclassed, + so double leading underscores is not used. + http://www.python.org/dev/peps/pep-0008/#method-names-and-instance-variables """ def __init__(self, updater_name, repository_mirrors): @@ -219,7 +227,7 @@ def __init__(self, updater_name, repository_mirrors): and, at a minimum, the root metadata file must exist: - {tuf.conf.repository_directory}/metadata/current/root.txt + {tuf.conf.repository_directory}/metadata/current/root.json updater_name: @@ -243,15 +251,14 @@ def __init__(self, updater_name, repository_mirrors): tuf.RepositoryError: If there is an error with the updater's repository files, such - as a missing 'root.txt' file. + as a missing 'root.json' file. - Th metadata files (e.g., 'root.txt', 'targets.txt') for the top- + Th metadata files (e.g., 'root.json', 'targets.json') for the top- level roles are read from disk and stored in dictionaries. None. - """ # Do the arguments have the correct format? @@ -282,6 +289,11 @@ def __init__(self, updater_name, repository_mirrors): # Store the location of the client's metadata directory. self.metadata_directory = {} + + # Store the 'consistent_snapshot' of the Root role. This setting + # determines if metadata and target files downloaded from remote + # repositories include the digest. + self.consistent_snapshot = False # Ensure the repository metadata directory has been set. if tuf.conf.repository_directory is None: @@ -312,13 +324,13 @@ def __init__(self, updater_name, repository_mirrors): # Load current and previous metadata. for metadata_set in ['current', 'previous']: - for metadata_role in ['root', 'targets', 'release', 'timestamp']: + for metadata_role in ['root', 'targets', 'snapshot', 'timestamp']: self._load_metadata_from_file(metadata_set, metadata_role) # Raise an exception if the repository is missing the required 'root' # metadata. if 'root' not in self.metadata['current']: - message = 'No root of trust! Could not find the "root.txt" file.' + message = 'No root of trust! Could not find the "root.json" file.' raise tuf.RepositoryError(message) @@ -327,7 +339,6 @@ def __init__(self, updater_name, repository_mirrors): def __str__(self): """ The string representation of an Updater object. - """ return self.name @@ -340,7 +351,7 @@ def _load_metadata_from_file(self, metadata_set, metadata_role): """ Load current or previous metadata if there is a local file. If the - expected file belonging to 'metadata_role' (e.g., 'root.txt') cannot + expected file belonging to 'metadata_role' (e.g., 'root.json') cannot be loaded, raise an exception. The extracted metadata object loaded from file is saved to the metadata store (i.e., self.metadata). @@ -351,7 +362,7 @@ def _load_metadata_from_file(self, metadata_set, metadata_role): metadata_role: The name of the metadata. This is a role name and should - not end in '.txt'. Examples: 'root', 'targets', 'targets/linux/x86'. + not end in '.json'. Examples: 'root', 'targets', 'targets/linux/x86'. tuf.FormatError: @@ -370,7 +381,6 @@ def _load_metadata_from_file(self, metadata_set, metadata_role): None. - """ # Ensure we have a valid metadata set. @@ -379,7 +389,7 @@ def _load_metadata_from_file(self, metadata_set, metadata_role): # Save and construct the full metadata path. metadata_directory = self.metadata_directory[metadata_set] - metadata_filename = metadata_role + '.txt' + metadata_filename = metadata_role + '.json' metadata_filepath = os.path.join(metadata_directory, metadata_filename) # Ensure the metadata path is valid/exists, else ignore the call. @@ -395,12 +405,15 @@ def _load_metadata_from_file(self, metadata_set, metadata_role): # Save the metadata object to the metadata store. self.metadata[metadata_set][metadata_role] = metadata_object - - # We need to rebuild the key and role databases if - # metadata object is 'root' or target metadata. + + # If 'metadata_role' is 'root' or targets metadata, the key and role + # databases must be rebuilt. If 'root', ensure self.consistent_snaptshots + # is updated. if metadata_set == 'current': if metadata_role == 'root': self._rebuild_key_and_role_db() + self.consistent_snapshot = metadata_object['consistent_snapshot'] + elif metadata_object['_type'] == 'Targets': # TODO: Should we also remove the keys of the delegated roles? tuf.roledb.remove_delegated_roles(metadata_role) @@ -414,10 +427,10 @@ def _rebuild_key_and_role_db(self): """ Rebuild the key and role databases from the currently trusted - 'root' metadata object extracted from 'root.txt'. This private - function is called when a new/updated 'root' metadata file is loaded. - This function will only store the role information for the top-level - roles (i.e., 'root', 'targets', 'release', 'timestamp'). + 'root' metadata object extracted from 'root.json'. This private + method is called when a new/updated 'root' metadata file is loaded. + This method will only store the role information for the top-level + roles (i.e., 'root', 'targets', 'snapshot', 'timestamp'). None. @@ -435,7 +448,6 @@ def _rebuild_key_and_role_db(self): None. - """ # Clobbering this means all delegated metadata files are rendered outdated @@ -475,7 +487,6 @@ def _import_delegations(self, parent_role): None. - """ current_parent_metadata = self.metadata['current'][parent_role] @@ -492,13 +503,13 @@ def _import_delegations(self, parent_role): # Iterate through the keys of the delegated roles of 'parent_role' # and load them. for keyid, keyinfo in keys_info.items(): - if keyinfo['keytype'] == 'rsa': - rsa_key = tuf.rsa_key.create_from_metadata_format(keyinfo) + if keyinfo['keytype'] in ['rsa', 'ed25519']: + key = tuf.keys.format_metadata_to_key(keyinfo) # We specify the keyid to ensure that it's the correct keyid # for the key. try: - tuf.keydb.add_rsakey(rsa_key, keyid) + tuf.keydb.add_key(key, keyid) except tuf.KeyAlreadyExistsError: pass except (tuf.FormatError, tuf.Error), e: @@ -527,7 +538,7 @@ def _import_delegations(self, parent_role): - def refresh(self): + def refresh(self, unsafely_update_root_if_necessary=True): """ Update the latest copies of the metadata for the top-level roles. @@ -537,11 +548,13 @@ def refresh(self): The client would call refresh() prior to requesting target file information. Calling refresh() ensures target methods, like all_targets() and target(), refer to the latest available content. - The latest copies for delegated metadata are downloaded and updated - by the target methods. + The latest copies, according to the currently trusted top-level metadata, + of delegated metadata are downloaded and updated by the target methods. - None. + unsafely_update_root_if_necessary: + Boolean that indicates whether to unsafely update the Root metadata + if any of the top-level metadata cannot be downloaded successfully. tuf.NoWorkingMirrorError: @@ -556,17 +569,32 @@ def refresh(self): None. - """ + + # Do the arguments have the correct format? + # This check ensures the arguments have the appropriate + # number of objects and object types, and that all dict + # keys are properly named. + # Raise 'tuf.FormatError' if the check fail. + tuf.formats.BOOLEAN_SCHEMA.check_match(unsafely_update_root_if_necessary) # The timestamp role does not have signed metadata about it; otherwise we # would need an infinite regress of metadata. Therefore, we use some # default, sane metadata about it. DEFAULT_TIMESTAMP_FILEINFO = { - 'hashes':None, + 'hashes': {}, 'length': tuf.conf.DEFAULT_TIMESTAMP_REQUIRED_LENGTH } + # The Root role may be updated without knowing its hash if top-level + # metadata cannot be safely downloaded (e.g., keys may have been revoked, + # thus requiring a new Root file that includes the updated keys) and + # 'unsafely_update_root_if_necessary' is True. + DEFAULT_ROOT_FILEINFO = { + 'hashes': {}, + 'length': tuf.conf.DEFAULT_ROOT_REQUIRED_LENGTH + } + # Update the top-level metadata. The _update_metadata_if_changed() and # _update_metadata() calls below do NOT perform an update if there # is insufficient trusted signatures for the specified metadata. @@ -574,40 +602,54 @@ def refresh(self): # Use default but sane information for timestamp metadata, and do not # require strict checks on its required length. - self._update_metadata('timestamp', DEFAULT_TIMESTAMP_FILEINFO) - - self._update_metadata_if_changed('release', referenced_metadata='timestamp') - - self._update_metadata_if_changed('root') - - self._update_metadata_if_changed('targets') + try: + self._update_metadata('timestamp', DEFAULT_TIMESTAMP_FILEINFO) + self._update_metadata_if_changed('snapshot', + referenced_metadata='timestamp') + self._update_metadata_if_changed('root') + self._update_metadata_if_changed('targets') + + except tuf.NoWorkingMirrorError, e: + if unsafely_update_root_if_necessary: + message = 'Valid top-level metadata cannot be downloaded. Unsafely '+\ + 'update the Root metadata.' + logger.info(message) + + self._update_metadata('root', DEFAULT_ROOT_FILEINFO) + self.refresh(unsafely_update_root_if_necessary=False) + + else: + raise - # Updated the top-level metadata (which all had valid signatures), however, - # have they expired? Raise 'tuf.ExpiredMetadataError' if any of the metadata - # has expired. - for metadata_role in ['timestamp', 'root', 'release', 'targets']: - self._ensure_not_expired(metadata_role) + else: + # Updated the top-level metadata (which all had valid signatures), + # however, have they expired? Raise 'tuf.ExpiredMetadataError' if any of + # the metadata has expired. + for metadata_role in ['timestamp', 'root', 'snapshot', 'targets']: + self._ensure_not_expired(metadata_role) - def __check_hashes(self, file_object, trusted_hashes): + def _check_hashes(self, file_object, trusted_hashes): """ - A helper function that verifies multiple secure hashes of the downloaded - file. If any of these fail it raises an exception. This is to conform - with the TUF specs, which support clients with different hashing - algorithms. The 'hash.py' module is used to compute the hashes of the - 'file_object'. + A private helper method that verifies multiple secure hashes of the + downloaded file 'file_object'. If any of these fail it raises an + exception. This is to conform with the TUF spec, which support clients + with different hashing algorithms. The 'hash.py' module is used to compute + the hashes of 'file_object'. file_object: - A file-like object. + A 'tuf.util.TempFile' file-like object. 'file_object' ensures that a + read() without a size argument properly reads the entire file. trusted_hashes: A dictionary with hash-algorithm names as keys and hashes as dict values. - The hashes should be in the hexdigest format. + The hashes should be in the hexdigest format. Should be Conformant to + 'tuf.formats.HASHDICT_SCHEMA'. tuf.BadHashError, if the hashes don't match. @@ -617,15 +659,16 @@ def __check_hashes(self, file_object, trusted_hashes): None. - """ - # Verify each trusted hash of 'trusted_hashes'. Raise exception if - # any of the hashes are incorrect and return if all are correct. + # Verify each trusted hash of 'trusted_hashes'. If all are valid, simply + # return. for algorithm, trusted_hash in trusted_hashes.items(): digest_object = tuf.hash.digest(algorithm) digest_object.update(file_object.read()) computed_hash = digest_object.hexdigest() + + # Raise an exception if any of the hashes are incorrect. if trusted_hash != computed_hash: raise tuf.BadHashError(trusted_hash, computed_hash) else: @@ -635,102 +678,115 @@ def __check_hashes(self, file_object, trusted_hashes): - def __hard_check_compressed_file_length(self, file_object, - compressed_file_length): + def _hard_check_file_length(self, file_object, trusted_file_length): """ - A helper function that checks the expected compressed length of a - file-like object. The length of the file must be strictly equal to the - expected length. This is a deliberately redundant implementation designed - to complement tuf.download._check_downloaded_length(). + A private helper method that ensures the length of 'file_object' is + strictly equal to 'trusted_file_length'. This is a deliberately + redundant implementation designed to complement + tuf.download._check_downloaded_length(). file_object: - A file-like object. + A 'tuf.util.TempFile' file-like object. 'file_object' ensures that a + read() without a size argument properly reads the entire file. - compressed_file_length: - A nonnegative integer that is the expected compressed length of the - file. + trusted_file_length: + A non-negative integer that is the trusted length of the file. - tuf.DownloadLengthMismatchError, if the lengths don't match. + tuf.DownloadLengthMismatchError, if the lengths do not match. - None. + Reads the contents of 'file_object' and logs a message if 'file_object' + matches the trusted length. None. - """ - observed_length = file_object.get_compressed_length() - if observed_length != compressed_file_length: - raise tuf.DownloadLengthMismatchError(compressed_file_length, + # Read the entire contents of 'file_object', a 'tuf.util.TempFile' file-like + # object that ensures the entire file is read. + observed_length = len(file_object.read()) + + # Return and log a message if the length 'file_object' is equal to + # 'trusted_file_length', otherwise raise an exception. A hard check + # ensures that a downloaded file strictly matches a known, or trusted, + # file length. + if observed_length != trusted_file_length: + raise tuf.DownloadLengthMismatchError(trusted_file_length, observed_length) else: - logger.debug('file length ('+str(observed_length)+\ - ') == trusted length ('+str(compressed_file_length)+')') + logger.debug('Observed length ('+str(observed_length)+\ + ') == trusted length ('+str(trusted_file_length)+')') - def __soft_check_compressed_file_length(self, file_object, - compressed_file_length): + def _soft_check_file_length(self, file_object, trusted_file_length): """ - A helper function that checks the expected compressed length of a - file-like object. The length of the file must be less than or equal to - the expected length. This is a deliberately redundant implementation - designed to complement tuf.download._check_downloaded_length(). + A private helper method that checks the trusted file length of a + 'tuf.util.TempFile' file-like object. The length of the file must be less + than or equal to the expected length. This is a deliberately redundant + implementation designed to complement + tuf.download._check_downloaded_length(). file_object: - A file-like object. + A 'tuf.util.TempFile' file-like object. 'file_object' ensures that a + read() without a size argument properly reads the entire file. - compressed_file_length: - A nonnegative integer that is the expected compressed length of the - file. + trusted_file_length: + A non-negative integer that is the trusted length of the file. - tuf.DownloadLengthMismatchError, if the lengths don't match. + tuf.DownloadLengthMismatchError, if the lengths do not match. - None. + Reads the contents of 'file_object' and logs a message if 'file_object' + is less than or equal to the trusted length. None. - """ - observed_length = file_object.get_compressed_length() - if observed_length > compressed_file_length: - raise tuf.DownloadLengthMismatchError(compressed_file_length, + # Read the entire contents of 'file_object', a 'tuf.util.TempFile' file-like + # object that ensures the entire file is read. + observed_length = len(file_object.read()) + + # Return and log a message if 'file_object' is less than or equal to + # 'trusted_file_length', otherwise raise an exception. A soft check + # ensures that an upper bound restricts how large a file is downloaded. + if observed_length > trusted_file_length: + raise tuf.DownloadLengthMismatchError(trusted_file_length, observed_length) else: - logger.debug('file length ('+str(observed_length)+\ - ') <= trusted length ('+str(compressed_file_length)+')') + logger.debug('Observed length ('+str(observed_length)+\ + ') <= trusted length ('+str(trusted_file_length)+')') - def get_target_file(self, target_filepath, compressed_file_length, - uncompressed_file_hashes): + def _get_target_file(self, target_filepath, file_length, file_hashes): """ - Safely download a target file up to a certain length, and check its + Safely (i.e., the file length and hash are strictly equal to the + trusted) download a target file up to a certain length, and check its hashes thereafter. target_filepath: - The relative target filepath obtained from TUF targets metadata. + The target filepath (relative to the repository targets directory) + obtained from TUF targets metadata. - compressed_file_length: + file_length: The expected compressed length of the target file. If the file is not compressed, then it will simply be its uncompressed length. - uncompressed_file_hashes: + file_hashes: The expected hashes of the target file. @@ -744,37 +800,50 @@ def get_target_file(self, target_filepath, compressed_file_length, a temporary file and returned. - A tuf.util.TempFile file-like object containing the target. - + A 'tuf.util.TempFile' file-like object containing the target. """ - def verify_uncompressed_target_file(target_file_object): + # Define a callable function that is passed as an argument to _get_file() + # and called. The 'verify_target_file' function ensures the file length + # and hashes of 'target_filepath' are strictly equal to the trusted values. + def verify_target_file(target_file_object): + # Every target file must have its length and hashes inspected. - self.__hard_check_compressed_file_length(target_file_object, - compressed_file_length) - self.__check_hashes(target_file_object, uncompressed_file_hashes) + self._hard_check_file_length(target_file_object, file_length) + self._check_hashes(target_file_object, file_hashes) + + # Target files, unlike metadata files, are not decompressed; the + # 'compression' argument to _get_file() is needed only for decompression of + # metadata. Target files may be compressed or uncompressed. + if self.consistent_snapshot: + target_digest = random.choice(file_hashes.values()) + dirname, basename = os.path.split(target_filepath) + target_filepath = os.path.join(dirname, target_digest+'.'+basename) - return self.__get_file(target_filepath, verify_uncompressed_target_file, - 'target', compressed_file_length, - download_safely=True, compression=None) + return self._get_file(target_filepath, verify_target_file, + 'target', file_length, compression=None, + verify_compressed_file_function=None, + download_safely=True) - def __verify_uncompressed_metadata_file(self, metadata_file_object, - metadata_role): + def _verify_uncompressed_metadata_file(self, metadata_file_object, + metadata_role): """ - A private helper function to verify an uncompressed downloaded metadata + A private helper function to verify an uncompressed metadata file. metadata_file_object: - A tuf.util.TempFile instance containing the metadata file. + A 'tuf.util.TempFile' instance containing the metadata file. + 'metadata_file_object' ensures the entire file is returned with read(). metadata_role: - The role name of the metadata. + The role name of the metadata (e.g., 'root', 'targets', + 'targets/linux/x86'). tuf.ForbiddenTargetError: @@ -797,20 +866,21 @@ def __verify_uncompressed_metadata_file(self, metadata_file_object, In case the metadata file does not have a valid signature. - None. + The contents of 'metadata_file_object' is read and loaded. None. - """ metadata = metadata_file_object.read() + try: metadata_signable = tuf.util.load_json_string(metadata) except Exception, exception: raise tuf.InvalidMetadataJSONError(exception) else: - # Ensure the loaded 'metadata_signable' is properly formatted. + # Ensure the loaded 'metadata_signable' is properly formatted. Raise + # 'tuf.FormatError' if not. tuf.formats.check_signable_object_format(metadata_signable) # Is 'metadata_signable' newer than the currently installed @@ -827,9 +897,16 @@ def __verify_uncompressed_metadata_file(self, metadata_file_object, current_version) # Reject the metadata if any specified targets are not allowed. + # 'tuf.ForbiddenTargetError' raised if any of the targets of 'metadata_role' + # are not allowed. if metadata_signable['signed']['_type'] == 'Targets': - self._ensure_all_targets_allowed(metadata_role, - metadata_signable['signed']) + if metadata_role != 'targets': + metadata_targets = metadata_signable['signed']['targets'].keys() + parent_rolename = tuf.roledb.get_parent_rolename(metadata_role) + parent_role_metadata = self.metadata['current'][parent_rolename] + parent_delegations = parent_role_metadata['delegations'] + tuf.util.ensure_all_targets_allowed(metadata_role, metadata_targets, + parent_delegations) # Verify the signature on the downloaded metadata object. valid = tuf.sig.verify(metadata_signable, metadata_role) @@ -840,8 +917,10 @@ def __verify_uncompressed_metadata_file(self, metadata_file_object, - def unsafely_get_metadata_file(self, metadata_role, metadata_filepath, - compressed_file_length): + def _unsafely_get_metadata_file(self, metadata_role, metadata_filepath, + uncompressed_fileinfo, + compression=None, compressed_fileinfo=None): + """ Unsafely download a metadata file up to a certain length. The actual file @@ -850,14 +929,24 @@ def unsafely_get_metadata_file(self, metadata_role, metadata_filepath, metadata_role: - The role name of the metadata. + The role name of the metadata (e.g., 'root', 'targets', + 'targets/linux/x86'). metadata_filepath: - The relative metadata filepath. + The metadata filepath (i.e., relative to the repository metadata + directory). - compressed_file_length: - The expected compressed length of the metadata file. If the file is not - compressed, then it will simply be its uncompressed length. + uncompressed_fileinfo: + The trusted file length and hashes of the uncompressed version of the + metadata file. Should be 'tuf.formats.FILEINFO_SCHEMA'. + + compression: + The name of the compression algorithm (e.g., 'gzip'), if the metadata + file is compressed. + + compressed_fileinfo: + The fileinfo of the metadata file, if it is compressed. Should be + 'tuf.formats.FILEINFO_SCHEMA'. tuf.NoWorkingMirrorError: @@ -870,28 +959,51 @@ def unsafely_get_metadata_file(self, metadata_role, metadata_filepath, in a temporary file and returned. - A tuf.util.TempFile file-like object containing the metadata. - + A 'tuf.util.TempFile' file-like object containing the metadata. """ + + # Store file length and hashes of the uncompressed version metadata. + # The uncompressed version is always verified. + uncompressed_file_length = uncompressed_fileinfo['length'] + uncompressed_file_hashes = uncompressed_fileinfo['hashes'] + download_file_length = uncompressed_file_length + compressed_file_length = None + compressed_file_hashes = None + + # Store the file length and hashes of the compressed version of the + # metadata, if compressions is set. + if compression is not None and compressed_fileinfo is not None: + compressed_file_length = compressed_fileinfo['length'] + compressed_file_hashes = compressed_fileinfo['hashes'] + download_file_length = compressed_file_length def unsafely_verify_uncompressed_metadata_file(metadata_file_object): - self.__soft_check_compressed_file_length(metadata_file_object, - compressed_file_length) - self.__verify_uncompressed_metadata_file(metadata_file_object, - metadata_role) + self._soft_check_file_length(metadata_file_object, + uncompressed_file_length) + self._check_hashes(metadata_file_object, uncompressed_file_hashes) + self._verify_uncompressed_metadata_file(metadata_file_object, + metadata_role) + + def unsafely_verify_compressed_metadata_file(metadata_file_object): + self._hard_check_file_length(metadata_file_object, compressed_file_length) + self._check_hashes(metadata_file_object, compressed_file_hashes) - return self.__get_file(metadata_filepath, - unsafely_verify_uncompressed_metadata_file, 'meta', - compressed_file_length, download_safely=False, - compression=None) + if compression is None: + unsafely_verify_compressed_metadata_file = None + return self._get_file(metadata_filepath, + unsafely_verify_uncompressed_metadata_file, 'meta', + download_file_length, compression, + unsafely_verify_compressed_metadata_file, + download_safely=False) - def safely_get_metadata_file(self, metadata_role, metadata_filepath, - compressed_file_length, - uncompressed_file_hashes, compression): + + def _safely_get_metadata_file(self, metadata_role, metadata_filepath, + uncompressed_fileinfo, + compression=None, compressed_fileinfo=None): """ Safely download a metadata file up to a certain length, and check its @@ -899,20 +1011,24 @@ def safely_get_metadata_file(self, metadata_role, metadata_filepath, metadata_role: - The role name of the metadata. + The role name of the metadata (e.g., 'root', 'targets', + 'targets/linux/x86'). metadata_filepath: - The relative metadata filepath. - - compressed_file_length: - The expected compressed length of the metadata file. If the file is not - compressed, then it will simply be its uncompressed length. - - uncompressed_file_hashes: - The expected hashes of the metadata file. + The metadata filepath (i.e., relative to the repository metadata + directory). + + uncompressed_fileinfo: + The trusted file length and hashes of the uncompressed version of the + metadata file. Should be 'tuf.formats.FILEINFO_SCHEMA'. compression: - The name of the compression algorithm used to compress the metadata. + The name of the compression algorithm (e.g., 'gzip'), if the metadata + file is compressed. + + compressed_fileinfo: + The fileinfo of the metadata file, if it is compressed. Should be + 'tuf.formats.FILEINFO_SCHEMA'. tuf.NoWorkingMirrorError: @@ -925,21 +1041,41 @@ def safely_get_metadata_file(self, metadata_role, metadata_filepath, in a temporary file and returned. - A tuf.util.TempFile file-like object containing the metadata. - + A 'tuf.util.TempFile' file-like object containing the metadata. """ - + + # Store file length and hashes of the uncompressed version metadata. + # The uncompressed version is always verified. + uncompressed_file_length = uncompressed_fileinfo['length'] + uncompressed_file_hashes = uncompressed_fileinfo['hashes'] + download_file_length = uncompressed_file_length + + # Store the file length and hashes of the compressed version of the + # metadata, if compressions is set. + if compression and compressed_fileinfo: + compressed_file_length = compressed_fileinfo['length'] + compressed_file_hashes = compressed_fileinfo['hashes'] + download_file_length = compressed_file_length + def safely_verify_uncompressed_metadata_file(metadata_file_object): - self.__hard_check_compressed_file_length(metadata_file_object, - compressed_file_length) - self.__check_hashes(metadata_file_object, uncompressed_file_hashes) - self.__verify_uncompressed_metadata_file(metadata_file_object, + self._hard_check_file_length(metadata_file_object, + uncompressed_file_length) + self._check_hashes(metadata_file_object, uncompressed_file_hashes) + self._verify_uncompressed_metadata_file(metadata_file_object, metadata_role) - return self.__get_file(metadata_filepath, - safely_verify_uncompressed_metadata_file, 'meta', - compressed_file_length, download_safely=True, - compression=compression) + def safely_verify_compressed_metadata_file(metadata_file_object): + self._hard_check_file_length(metadata_file_object, compressed_file_length) + self._check_hashes(metadata_file_object, compressed_file_hashes) + + if compression is None: + safely_verify_compressed_metadata_file = None + + return self._get_file(metadata_filepath, + safely_verify_uncompressed_metadata_file, 'meta', + download_file_length, compression, + safely_verify_compressed_metadata_file, + download_safely=True) @@ -948,8 +1084,9 @@ def safely_verify_uncompressed_metadata_file(metadata_file_object): # TODO: Instead of the more fragile 'download_safely' switch, unroll the # function into two separate ones: one for "safe" download, and the other one # for "unsafe" download? This should induce safer and more readable code. - def __get_file(self, filepath, verify_uncompressed_file, file_type, - compressed_file_length, download_safely, compression): + def _get_file(self, filepath, verify_file_function, file_type, + file_length, compression=None, + verify_compressed_file_function=None, download_safely=True): """ Try downloading, up to a certain length, a metadata or target file from a @@ -960,26 +1097,34 @@ def __get_file(self, filepath, verify_uncompressed_file, file_type, filepath: The relative metadata or target filepath. - verify_uncompressed_file: - A function which expects an uncompressed file-like object and which - will raise an exception in case the file is not valid for any reason. + verify_file_function: + A callable function that expects a 'tuf.util.TempFile' file-like object + and raises an exception if the file is invalid. Target files and + uncompressed versions of metadata may be verified with + 'verify_file_function'. file_type: Type of data needed for download, must correspond to one of the strings in the list ['meta', 'target']. 'meta' for metadata file type or - 'target' for target file type. It should correspond to NAME_SCHEMA - format. + 'target' for target file type. It should correspond to the + 'tuf.formats.NAME_SCHEMA' format. - compressed_file_length: - The expected compressed length of the target or metadata file. If the - file is not compressed, then it will simply be its uncompressed length. + file_length: + The expected length, or upper bound, of the target or metadata file to + be downloaded. + + compression: + The name of the compression algorithm (e.g., 'gzip'), if the metadata + file is compressed. + + verify_compressed_file_function: + If compression is specified, in the case of metadata files, this + callable function may be set to perform verification of the compressed + version of the metadata file. Decompressed metadata is also verified. download_safely: A boolean switch to toggle safe or unsafe download of the file. - compression: - The name of the compression algorithm used to compress the file. - tuf.NoWorkingMirrorError: The metadata could not be fetched. This is raised only when all known @@ -991,8 +1136,7 @@ def __get_file(self, filepath, verify_uncompressed_file, file_type, file and returned. - A tuf.util.TempFile file-like object containing the metadata or target. - + A 'tuf.util.TempFile' file-like object containing the metadata or target. """ file_mirrors = tuf.mirrors.get_list_of_mirrors(file_type, filepath, @@ -1005,18 +1149,23 @@ def __get_file(self, filepath, verify_uncompressed_file, file_type, try: if download_safely: file_object = tuf.download.safe_download(file_mirror, - compressed_file_length) + file_length) else: file_object = tuf.download.unsafe_download(file_mirror, - compressed_file_length) + file_length) - if compression: - logger.debug('Decompressing '+str(file_mirror)) + if compression is not None: + if verify_compressed_file_function is not None: + verify_compressed_file_function(file_object) + logger.info('Decompressing '+str(file_mirror)) file_object.decompress_temp_file_object(compression) else: - logger.debug('Not decompressing '+str(file_mirror)) - - verify_uncompressed_file(file_object) + logger.info('Not decompressing '+str(file_mirror)) + + # Verify 'file_object' according to the callable function. + # 'file_object' is also verified if decompressed above (i.e., the + # uncompressed version). + verify_file_function(file_object) except Exception, exception: # Remember the error from this mirror, and "reset" the target file. @@ -1037,11 +1186,12 @@ def __get_file(self, filepath, verify_uncompressed_file, file_type, - def _update_metadata(self, metadata_role, fileinfo, compression=None): + def _update_metadata(self, metadata_role, uncompressed_fileinfo, + compression=None, compressed_fileinfo=None): """ Download, verify, and 'install' the metadata belonging to 'metadata_role'. - Calling this function implies the metadata has been updated by the + Calling this method implies the metadata has been updated by the repository and thus needs to be re-downloaded. The current and previous metadata stores are updated if the newly downloaded metadata is successfully downloaded and verified. @@ -1049,29 +1199,28 @@ def _update_metadata(self, metadata_role, fileinfo, compression=None): metadata_role: The name of the metadata. This is a role name and should not end - in '.txt'. Examples: 'root', 'targets', 'targets/linux/x86'. + in '.json'. Examples: 'root', 'targets', 'targets/linux/x86'. - fileinfo: - A dictionary containing length and hashes of the metadata file. + uncompressed_fileinfo: + A dictionary containing length and hashes of the uncompressed metadata + file. + Ex: {"hashes": {"sha256": "3a5a6ec1f353...dedce36e0"}, "length": 1340} - The length must be that of the compressed metadata file if it is - compressed, or uncompressed metadata file if it is uncompressed. - The hashes must be that of the uncompressed metadata file. - - STRICT_REQUIRED_LENGTH: - A Boolean indicator used to signal whether we should perform strict - checking of the required length in 'fileinfo'. True by default. True - by default. We explicitly set this to False when we know that we want - to turn this off for downloading the timestamp metadata, which has no - signed required_length. - + compression: A string designating the compression type of 'metadata_role'. - The 'release' metadata file may be optionally downloaded and stored in + The 'snapshot' metadata file may be optionally downloaded and stored in compressed form. Currently, only metadata files compressed with 'gzip' are considered. Any other string is ignored. + compressed_fileinfo: + A dictionary containing length and hashes of the compressed metadata + file. + + Ex: {"hashes": {"sha256": "3a5a6ec1f353...dedce36e0"}, + "length": 1340} + tuf.NoWorkingMirrorError: The metadata could not be updated. This is not specific to a single @@ -1085,23 +1234,17 @@ def _update_metadata(self, metadata_role, fileinfo, compression=None): None. - """ # Construct the metadata filename as expected by the download/mirror modules. - metadata_filename = metadata_role + '.txt' + metadata_filename = metadata_role + '.json' uncompressed_metadata_filename = metadata_filename - # The 'release' or Targets metadata may be compressed. Add the appropriate + # The 'snapshot' or Targets metadata may be compressed. Add the appropriate # extension to 'metadata_filename'. if compression == 'gzip': metadata_filename = metadata_filename + '.gz' - # Extract file length and file hashes. They will be passed as arguments - # to 'download_file' function. - compressed_file_length = fileinfo['length'] - uncompressed_file_hashes = fileinfo['hashes'] - # Attempt a file download from each mirror until the file is downloaded and # verified. If the signature of the downloaded file is valid, proceed, # otherwise log a warning and try the next mirror. 'metadata_file_object' @@ -1121,17 +1264,36 @@ def _update_metadata(self, metadata_role, fileinfo, compression=None): # Note also that we presently support decompression of only "safe" # metadata, but this is easily extend to "unsafe" metadata as well as # "safe" targets. - + if metadata_role == 'timestamp': metadata_file_object = \ - self.unsafely_get_metadata_file(metadata_role, metadata_filename, - compressed_file_length) + self._unsafely_get_metadata_file(metadata_role, metadata_filename, + uncompressed_fileinfo, + compression, compressed_fileinfo) + + elif metadata_role == 'root' and not len(uncompressed_fileinfo['hashes']): + metadata_file_object = \ + self._unsafely_get_metadata_file(metadata_role, metadata_filename, + uncompressed_fileinfo, + compression, compressed_fileinfo) + else: + remote_filename = metadata_filename + if self.consistent_snapshot: + if compression: + filename_digest = \ + random.choice(compressed_fileinfo['hashes'].values()) + + else: + filename_digest = \ + random.choice(uncompressed_fileinfo['hashes'].values()) + dirname, basename = os.path.split(remote_filename) + remote_filename = os.path.join(dirname, filename_digest+'.'+basename) + metadata_file_object = \ - self.safely_get_metadata_file(metadata_role, metadata_filename, - compressed_file_length, - uncompressed_file_hashes, - compression=compression) + self._safely_get_metadata_file(metadata_role, remote_filename, + uncompressed_fileinfo, + compression, compressed_fileinfo) # The metadata has been verified. Move the metadata file into place. # First, move the 'current' metadata file to the 'previous' directory @@ -1144,6 +1306,7 @@ def _update_metadata(self, metadata_role, fileinfo, compression=None): previous_filepath = os.path.join(self.metadata_directory['previous'], metadata_filename) previous_filepath = os.path.abspath(previous_filepath) + if os.path.exists(current_filepath): # Previous metadata might not exist, say when delegations are added. tuf.util.ensure_parent_dir(previous_filepath) @@ -1160,6 +1323,7 @@ def _update_metadata(self, metadata_role, fileinfo, compression=None): current_uncompressed_filepath = \ os.path.abspath(current_uncompressed_filepath) metadata_file_object.move(current_uncompressed_filepath) + else: metadata_file_object.move(current_filepath) @@ -1169,57 +1333,66 @@ def _update_metadata(self, metadata_role, fileinfo, compression=None): updated_metadata_object = metadata_signable['signed'] current_metadata_object = self.metadata['current'].get(metadata_role) - # Finally, update the metadata and fileinfo stores. + # Finally, update the metadata and fileinfo stores, and rebuild the + # key and role info for the top-level roles if 'metadata_role' is root. + # Rebuilding the the key and role info is required if the newly-installed + # root metadata has revoked keys or updated any top-level role information. logger.debug('Updated '+repr(current_filepath)+'.') self.metadata['previous'][metadata_role] = current_metadata_object self.metadata['current'][metadata_role] = updated_metadata_object - self._update_fileinfo(metadata_filename) + self._update_fileinfo(metadata_filename) + + # Ensure the role and key information of the top-level roles is also updated + # according to the newly-installed Root metadata. + if metadata_role == 'root': + self._rebuild_key_and_role_db() + self.consistent_snapshot = updated_metadata_object['consistent_snapshot'] - def _update_metadata_if_changed(self, metadata_role, referenced_metadata='release'): + def _update_metadata_if_changed(self, metadata_role, referenced_metadata='snapshot'): """ Update the metadata for 'metadata_role' if it has changed. With the exception of the 'timestamp' role, all the top-level roles are updated - by this function. The 'timestamp' role is always downloaded from a mirror + by this method. The 'timestamp' role is always downloaded from a mirror without first checking if it has been updated; it is updated in refresh() - by calling _update_metadata('timestamp'). This function is also called for - delegated role metadata, which are referenced by 'release'. + by calling _update_metadata('timestamp'). This method is also called for + delegated role metadata, which are referenced by 'snapshot'. If the metadata needs to be updated but an update cannot be obtained, - this function will delete the file (with the exception of the root + this method will delete the file (with the exception of the root metadata, which never gets removed without a replacement). Due to the way in which metadata files are updated, it is expected that 'referenced_metadata' is not out of date and trusted. The refresh() - method updates the top-level roles in 'timestamp -> release -> + method updates the top-level roles in 'timestamp -> snapshot -> root -> targets' order. For delegated metadata, the parent role is updated before the delegated role. Taking into account that 'referenced_metadata' is updated and verified before 'metadata_role', - this function determines if 'metadata_role' has changed by checking + this method determines if 'metadata_role' has changed by checking the 'meta' field of the newly updated 'referenced_metadata'. metadata_role: The name of the metadata. This is a role name and should not end - in '.txt'. Examples: 'root', 'targets', 'targets/linux/x86'. + in '.json'. Examples: 'root', 'targets', 'targets/linux/x86'. referenced_metadata: This is the metadata that provides the role information for - 'metadata_role'. For the top-level roles, the 'release' role + 'metadata_role'. For the top-level roles, the 'snapshot' role is the referenced metadata for the 'root', and 'targets' roles. The 'timestamp' metadata is always downloaded regardless. In other words, it is updated by calling _update_metadata('timestamp') - and not by this function. The referenced metadata for 'release' + and not by this method. The referenced metadata for 'snapshot' is 'timestamp'. See refresh(). tuf.NoWorkingMirrorError: - If 'metadata_role' could not be downloaded after determining - that it had changed. + If 'metadata_role' could not be downloaded after determining that it had + changed. tuf.RepositoryError: If the referenced metadata is missing. @@ -1234,17 +1407,17 @@ def _update_metadata_if_changed(self, metadata_role, referenced_metadata='releas None. - """ - uncompressed_metadata_filename = metadata_role + '.txt' + uncompressed_metadata_filename = metadata_role + '.json' # Ensure the referenced metadata has been loaded. The 'root' role may be - # updated without having 'release' available. + # updated without having 'snapshot' available. if referenced_metadata not in self.metadata['current']: message = 'Cannot update '+repr(metadata_role)+' because ' \ +referenced_metadata+' is missing.' raise tuf.RepositoryError(message) + # The referenced metadata has been loaded. Extract the new # fileinfo for 'metadata_role' from it. else: @@ -1252,59 +1425,61 @@ def _update_metadata_if_changed(self, metadata_role, referenced_metadata='releas repr(referenced_metadata)+'. '+repr(metadata_role)+' may be updated.' logger.debug(message) - # There might be a compressed version of 'release.txt' or Targets + # There might be a compressed version of 'snapshot.json' or Targets # metadata available for download. Check the 'meta' field of # 'referenced_metadata' to see if it is listed when 'metadata_role' - # is 'release'. The full rolename for delegated Targets metadata - # must begin with 'targets/'. The Release role lists all the Targets + # is 'snapshot'. The full rolename for delegated Targets metadata + # must begin with 'targets/'. The snapshot role lists all the Targets # metadata available on the repository, including any that may be in # compressed form. + # + # In addition to validating the fileinfo (i.e., file lengths and hashes) + # of the uncompressed metadata, the compressed version is also verified to + # match its respective fileinfo. Verifying the compressed fileinfo ensures + # untrusted data is not decompressed prior to verifying hashes, or + # decompressing a file that may be invalid or partially intact. compression = None + compressed_fileinfo = None # Extract the fileinfo of the uncompressed version of 'metadata_role'. uncompressed_fileinfo = self.metadata['current'][referenced_metadata] \ ['meta'] \ [uncompressed_metadata_filename] - # Check for availability of compressed versions of 'release.txt', - # 'targets.txt', and delegated Targets, which also start with 'targets'. - # For 'targets.txt' and delegated metadata, 'referenced_metata' - # should always be 'release'. 'release.txt' specifies all roles - # provided by a repository, including their file sizes and hashes. - if metadata_role == 'release' or metadata_role.startswith('targets'): + # Check for the availability of compressed versions of 'snapshot.json', + # 'targets.json', and delegated Targets (that also start with 'targets'). + # For 'targets.json' and delegated metadata, 'referenced_metata' + # should always be 'snapshot'. 'snapshot.json' specifies all roles + # provided by a repository, including their file lengths and hashes. + if metadata_role == 'snapshot' or metadata_role.startswith('targets'): gzip_metadata_filename = uncompressed_metadata_filename + '.gz' if gzip_metadata_filename in self.metadata['current'] \ [referenced_metadata]['meta']: compression = 'gzip' compressed_fileinfo = self.metadata['current'][referenced_metadata] \ ['meta'][gzip_metadata_filename] - # NOTE: When we download the compressed file, we care about its - # compressed length. However, we check the hash of the uncompressed - # file; therefore we use the hashes of the uncompressed file. - fileinfo = {'length': compressed_fileinfo['length'], - 'hashes': uncompressed_fileinfo['hashes']} + logger.debug('Compressed version of '+\ repr(uncompressed_metadata_filename)+' is available at '+\ repr(gzip_metadata_filename)+'.') else: logger.debug('Compressed version of '+\ repr(uncompressed_metadata_filename)+' not available.') - fileinfo = uncompressed_fileinfo - else: - fileinfo = uncompressed_fileinfo # Simply return if the file has not changed, according to the metadata # about the uncompressed file provided by the referenced metadata. if not self._fileinfo_has_changed(uncompressed_metadata_filename, uncompressed_fileinfo): + logger.info(repr(uncompressed_metadata_filename)+' up-to-date.') + return logger.debug('Metadata '+repr(uncompressed_metadata_filename)+\ ' has changed.') try: - self._update_metadata(metadata_role, fileinfo=fileinfo, - compression=compression) + self._update_metadata(metadata_role, uncompressed_fileinfo, compression, + compressed_fileinfo) except: # The current metadata we have is not current but we couldn't # get new metadata. We shouldn't use the old metadata anymore. @@ -1315,13 +1490,15 @@ def _update_metadata_if_changed(self, metadata_role, referenced_metadata='releas # We shouldn't need to, but we need to check the trust # implications of the current implementation. self._delete_metadata(metadata_role) - logger.error('Metadata for '+str(metadata_role)+' could not be updated') + logger.error('Metadata for '+repr(metadata_role)+' cannot be updated.') raise + else: # We need to remove delegated roles because the delegated roles # may not be trusted anymore. if metadata_role == 'targets' or metadata_role.startswith('targets/'): logger.debug('Removing delegated roles of '+repr(metadata_role)+'.') + # TODO: Should we also remove the keys of the delegated roles? tuf.roledb.remove_delegated_roles(metadata_role) self._import_delegations(metadata_role) @@ -1330,198 +1507,27 @@ def _update_metadata_if_changed(self, metadata_role, referenced_metadata='releas - def _ensure_all_targets_allowed(self, metadata_role, metadata_object): - """ - - Ensure the delegated targets of 'metadata_role' are allowed; this is - determined by inspecting the 'delegations' field of the parent role - of 'metadata_role'. If a target specified by 'metadata_object' - is not found in the parent role's delegations field, raise an exception. - - Targets allowed are either exlicitly listed under the 'paths' field, or - implicitly exist under a subdirectory of a parent directory listed - under 'paths'. A parent role may delegate trust to all files under a - particular directory, including files in subdirectories, by simply - listing the directory (e.g., 'packages/source/Django/', the equivalent - of 'packages/source/Django/*'). Targets listed in hashed bins are - also validated (i.e., its calculated path hash prefix must be delegated - by the parent role. - - TODO: Should the TUF spec restrict the repository to one particular - algorithm? Should we allow the repository to specify in the role - dictionary the algorithm used for these generated hashed paths? - - - metadata_role: - The name of the metadata. This is a role name and should not end - in '.txt'. Examples: 'root', 'targets', 'targets/linux/x86'. - - metadata_object: - The metadata role object for 'metadata_role'. This is the object - saved to the metadata store and stored in the 'signed' field of a - 'signable' object (metadata roles are saved to metadata files as a - 'signable' object). - - - tuf.ForbiddenTargetError: - If the targets of 'metadata_role' are not allowed according to - the parent's metadata file. The 'paths' and 'path_hash_prefixes' - attributes are verified. - - - None. - - - None. - - """ - - # Return if 'metadata_role' is 'targets'. 'targets' is not - # a delegated role. - if metadata_role == 'targets': - return - - # The targets of delegated roles are stored in the parent's - # metadata file. Retrieve the parent role of 'metadata_role' - # to confirm 'metadata_role' contains valid targets. - parent_role = tuf.roledb.get_parent_rolename(metadata_role) - - # Iterate over the targets of 'metadata_role' and confirm they are trusted, - # or their root parent directory exists in the role delegated paths of the - # parent role. - roles = self.metadata['current'][parent_role]['delegations']['roles'] - role_index = tuf.repo.signerlib.find_delegated_role(roles, metadata_role) - - # Ensure the delegated role exists prior to extracting trusted paths from - # the parent's 'paths', or trusted path hash prefixes from the parent's - # 'path_hash_prefixes'. - if role_index is not None: - role = roles[role_index] - allowed_child_paths = role.get('paths') - allowed_child_path_hash_prefixes = role.get('path_hash_prefixes') - actual_child_targets = metadata_object['targets'].keys() - - if allowed_child_path_hash_prefixes is not None: - consistent = self._paths_are_consistent_with_hash_prefixes - if not consistent(actual_child_targets, - allowed_child_path_hash_prefixes): - raise tuf.ForbiddenTargetError('Role '+repr(metadata_role)+\ - ' specifies target which does not'+\ - ' have a path hash prefix matching'+\ - ' the prefix listed by the parent'+\ - ' role '+repr(parent_role)+'.') - - elif allowed_child_paths is not None: - - # Check that each delegated target is either explicitly listed or a parent - # directory is found under role['paths'], otherwise raise an exception. - # If the parent role explicitly lists target file paths in 'paths', - # this loop will run in O(n^2), the worst-case. The repository - # maintainer will likely delegate entire directories, and opt for - # explicit file paths if the targets in a directory are delegated to - # different roles/developers. - for child_target in actual_child_targets: - for allowed_child_path in allowed_child_paths: - prefix = os.path.commonprefix([child_target, allowed_child_path]) - if prefix == allowed_child_path: - break - else: - raise tuf.ForbiddenTargetError('Role '+repr(metadata_role)+\ - ' specifies target '+\ - repr(child_target)+' which is not'+\ - ' an allowed path according to'+\ - ' the delegations set by '+\ - repr(parent_role)+'.') - - else: - - # 'role' should have been validated when it was downloaded. - # The 'paths' or 'path_hash_prefixes' attributes should not be missing, - # so raise an error in case this clause is reached. - raise tuf.FormatError(repr(role)+' did not contain one of '+\ - 'the required fields ("paths" or '+\ - '"path_hash_prefixes").') - - # Raise an exception if the parent has not delegated to the specified - # 'metadata_role' child role. - else: - raise tuf.RepositoryError(repr(parent_role)+' has not delegated to '+\ - repr(metadata_role)+'.') - - - - - - def _paths_are_consistent_with_hash_prefixes(self, paths, - path_hash_prefixes): - """ - - Determine whether a list of paths are consistent with theirs alleged - path hash prefixes. By default, the SHA256 hash function will be used. - - - paths: - A list of paths for which their hashes will be checked. - - path_hash_prefixes: - The list of path hash prefixes with which to check the list of paths. - - - No known exceptions. - - - No known side effects. - - - A Boolean indicating whether or not the paths are consistent with the - hash prefix. - """ - - # Assume that 'paths' and 'path_hash_prefixes' are inconsistent until - # proven otherwise. - consistent = False - - if len(paths) > 0 and len(path_hash_prefixes) > 0: - for path in paths: - path_hash = self._get_target_hash(path) - # Assume that every path is inconsistent until proven otherwise. - consistent = False - - for path_hash_prefix in path_hash_prefixes: - if path_hash.startswith(path_hash_prefix): - consistent = True - break - - # This path has no matching path_hash_prefix. Stop looking further. - if not consistent: break - - return consistent - - - - - def _fileinfo_has_changed(self, metadata_filename, new_fileinfo): """ Determine whether the current fileinfo of 'metadata_filename' differs from 'new_fileinfo'. The 'new_fileinfo' argument should be extracted from the latest copy of the metadata - that references 'metadata_filename'. Example: 'root.txt' - would be referenced by 'release.txt'. + that references 'metadata_filename'. Example: 'root.json' + would be referenced by 'snapshot.json'. 'new_fileinfo' should only be 'None' if this is for updating - 'root.txt' without having 'release.txt' available. + 'root.json' without having 'snapshot.json' available. metadadata_filename: The metadata filename for the role. For the 'root' role, - 'metadata_filename' would be 'root.txt'. + 'metadata_filename' would be 'root.json'. new_fileinfo: A dict object representing the new file information for 'metadata_filename'. 'new_fileinfo' may be 'None' when - updating 'root' without having 'release' available. This + updating 'root' without having 'snapshot' available. This dict conforms to 'tuf.formats.FILEINFO_SCHEMA' and has the form: {'length': 23423 @@ -1536,7 +1542,6 @@ def _fileinfo_has_changed(self, metadata_filename, new_fileinfo): Boolean. True if the fileinfo has changed, false otherwise. - """ # If there is no fileinfo currently stored for 'metadata_filename', @@ -1564,8 +1569,9 @@ def _fileinfo_has_changed(self, metadata_filename, new_fileinfo): for algorithm, hash_value in new_fileinfo['hashes'].items(): # We're only looking for a single match. This isn't a security # check, we just want to prevent unnecessary downloads. - if hash_value == current_fileinfo['hashes'][algorithm]: - return False + if algorithm in current_fileinfo['hashes']: + if hash_value == current_fileinfo['hashes'][algorithm]: + return False return True @@ -1584,7 +1590,7 @@ def _update_fileinfo(self, metadata_filename): metadata_filename: The metadata filename for the role. For the 'root' role, - 'metadata_filename' would be 'root.txt'. + 'metadata_filename' would be 'root.json'. None. @@ -1595,7 +1601,6 @@ def _update_fileinfo(self, metadata_filename): None. - """ # In case we delayed loading the metadata and didn't do it in @@ -1629,7 +1634,7 @@ def _move_current_to_previous(self, metadata_role): metadata_role: The name of the metadata. This is a role name and should not end - in '.txt'. Examples: 'root', 'targets', 'targets/linux/x86'. + in '.json'. Examples: 'root', 'targets', 'targets/linux/x86'. None. @@ -1640,11 +1645,10 @@ def _move_current_to_previous(self, metadata_role): None. - """ # Get the 'current' and 'previous' full file paths for 'metadata_role' - metadata_filepath = metadata_role + '.txt' + metadata_filepath = metadata_role + '.json' previous_filepath = os.path.join(self.metadata_directory['previous'], metadata_filepath) current_filepath = os.path.join(self.metadata_directory['current'], @@ -1668,13 +1672,13 @@ def _delete_metadata(self, metadata_role): Remove all (current) knowledge of 'metadata_role'. The metadata belonging to 'metadata_role' is removed from the current - 'self.metadata' store and from the role database. The 'root.txt' role + 'self.metadata' store and from the role database. The 'root.json' role file is never removed. metadata_role: The name of the metadata. This is a role name and should not end - in '.txt'. Examples: 'root', 'targets', 'targets/linux/x86'. + in '.json'. Examples: 'root', 'targets', 'targets/linux/x86'. None. @@ -1685,7 +1689,6 @@ def _delete_metadata(self, metadata_role): None. - """ # The root metadata role is never deleted without a replacement. @@ -1712,7 +1715,7 @@ def _ensure_not_expired(self, metadata_role): metadata_role: The name of the metadata. This is a role name and should not end - in '.txt'. Examples: 'root', 'targets', 'targets/linux/x86'. + in '.json'. Examples: 'root', 'targets', 'targets/linux/x86'. tuf.ExpiredMetadataError: @@ -1723,13 +1726,12 @@ def _ensure_not_expired(self, metadata_role): None. - """ # Construct the full metadata filename and the location of its # current path. The current path of 'metadata_role' is needed # to log the exact filename of the expired metadata. - metadata_filename = metadata_role + '.txt' + metadata_filename = metadata_role + '.json' rolepath = os.path.join(self.metadata_directory['current'], metadata_filename) rolepath = os.path.abspath(rolepath) @@ -1756,8 +1758,11 @@ def all_targets(self): Get a list of the target information for all the trusted targets on the repository. This list also includes all the targets of - delegated roles. The list conforms to 'tuf.formats.TARGETFILES_SCHEMA' + delegated roles. Targets of the list returned are ordered according + the trusted order of the delegated roles, where parent roles come before + children. The list conforms to 'tuf.formats.TARGETFILES_SCHEMA' and has the form: + [{'filepath': 'a/b/c.txt', 'fileinfo': {'length': 13323, 'hashes': {'sha256': dbfac345..}} @@ -1769,7 +1774,7 @@ def all_targets(self): tuf.RepositoryError: If the metadata for the 'targets' role is missing from - the 'release' metadata. + the 'snapshot' metadata. tuf.UnknownRoleError: If one of the roles could not be found in the role database. @@ -1779,7 +1784,6 @@ def all_targets(self): A list of targets, conformant to 'tuf.formats.TARGETFILES_SCHEMA'. - """ # Load the most up-to-date targets of the 'targets' role and all @@ -1787,11 +1791,12 @@ def all_targets(self): self._refresh_targets_metadata(include_delegations=True) all_targets = [] + # Fetch the targets for the 'targets' role. all_targets = self._targets_of_role('targets', skip_refresh=True) - # Fetch the targets for the delegated roles. - for delegated_role in tuf.roledb.get_delegated_rolenames('targets'): + # Fetch the targets of the delegated roles. + for delegated_role in sorted(tuf.roledb.get_delegated_rolenames('targets')): all_targets = self._targets_of_role(delegated_role, all_targets, skip_refresh=True) @@ -1810,13 +1815,13 @@ def _refresh_targets_metadata(self, rolename='targets', include_delegations=Fals _update_metadata_if_changed('targets') call, not here. Delegated roles are not loaded when the repository is first initialized. They are loaded from disk, updated if they have changed, and stored to the 'self.metadata' - store by this function. This function is called by the target methods, + store by this method. This method is called by the target methods, like all_targets() and targets_of_role(). rolename: This is a delegated role name and should not end - in '.txt'. Example: 'targets/linux/x86'. + in '.json'. Example: 'targets/linux/x86'. include_delegations: Boolean indicating if the delegated roles set by 'rolename' should @@ -1825,7 +1830,7 @@ def _refresh_targets_metadata(self, rolename='targets', include_delegations=Fals tuf.RepositoryError: If the metadata file for the 'targets' role is missing - from the 'release' metadata. + from the 'snapshot' metadata. The metadata for the delegated roles are loaded and updated if they @@ -1834,7 +1839,6 @@ def _refresh_targets_metadata(self, rolename='targets', include_delegations=Fals None. - """ roles_to_update = [] @@ -1842,19 +1846,22 @@ def _refresh_targets_metadata(self, rolename='targets', include_delegations=Fals # See if this role provides metadata and, if we're including # delegations, look for metadata from delegated roles. role_prefix = rolename + '/' - for metadata_path in self.metadata['current']['release']['meta'].keys(): - if metadata_path == rolename + '.txt': - roles_to_update.append(metadata_path[:-len('.txt')]) + for metadata_path in self.metadata['current']['snapshot']['meta'].keys(): + if metadata_path == rolename + '.json': + roles_to_update.append(metadata_path[:-len('.json')]) elif include_delegations and metadata_path.startswith(role_prefix): - roles_to_update.append(metadata_path[:-len('.txt')]) + # Add delegated roles. Skip roles names containing compression + # extensions. + if metadata_path.endswith('.json'): + roles_to_update.append(metadata_path[:-len('.json')]) - # Remove the 'targets' role because it gets updated when the targets.txt + # Remove the 'targets' role because it gets updated when the targets.json # file is updated in _update_metadata_if_changed('targets'). if rolename == 'targets': try: roles_to_update.remove('targets') except ValueError: - message = 'The Release metadata file is missing the targets.txt entry.' + message = 'The snapshot metadata file is missing the targets.json entry.' raise tuf.RepositoryError(message) # If there is nothing to refresh, we are done. @@ -1885,51 +1892,99 @@ def _refresh_targets_metadata(self, rolename='targets', include_delegations=Fals def refresh_targets_metadata_chain(self, rolename): """ - Proof-of-concept. + + Refresh the minimum targets metadata of 'rolename'. If 'rolename' is + 'targets/claimed/3.3/django', refresh the metadata of the following roles: + + targets.json + targets/claimed.json + targets/claimed/3.3.json + + Note that 'targets/claimed/3.3/django.json' is not refreshed here. + + The metadata of the 'targets' role is updated in refresh() by the + _update_metadata_if_changed('targets') call, not here. Delegated roles + are not loaded when the repository is first initialized; they can be + loaded from disk, updated if they have changed, and stored to the + 'self.metadata' store by this method. This method may be called + before targets_of_role('rolename') so that the most up-to-date metadata is + available to verify the target files of 'rolename', including the metadata + of 'rolename'. + + + rolename: + This is a full delegated rolename and should not end in '.json'. + Example: 'targets/linux/x86'. + + + tuf.FormatError: + If any of the arguments are improperly formatted. + + tuf.RepositoryError: + If the metadata of any of the parent roles of 'rolename' is missing + from the 'snapshot.json' metadata file. + + + The metadata of the parent roles of 'rolename' are loaded from disk and + updated if they have changed. Metadata is removed from the role database + if it has expired. + + A list of the roles that have been updated, loaded, and are valid. """ + # Do the arguments have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fail. + tuf.formats.ROLENAME_SCHEMA.check_match(rolename) + # List of parent roles to update. parent_roles = [] + # Separate each rolename (i.e., each rolename should exclude parent and + # child rolenames). 'rolename' should be the full rolename, as in + # 'targets/linux/x86'. parts = rolename.split('/') # Append the first role to the list. parent_roles.append(parts[0]) - # The 'roles_added' string contains the roles already added. If 'a' and 'a/b' - # have been added to 'parent_roles', 'roles_added' would contain 'a/b' + # The 'roles_added' string contains the roles (full rolename) already added. + # If 'a' and 'a/b' have been added to 'parent_roles', 'roles_added' would + # contain 'a/b'. roles_added = parts[0] # Add each subsequent role to the previous string (with a '/' separator). - # This only goes to -1 because we only want to return the parents (so we + # This only goes to -1 because we only want to store the parents (so we # ignore the last element). for next_role in parts[1:-1]: parent_roles.append(roles_added+'/'+next_role) roles_added = roles_added+'/'+next_role - message = 'Minimum metadata to download to set chain of trust: '+\ + message = 'Minimum metadata to download and set the chain of trust: '+\ repr(parent_roles)+'.' logger.info(message) - # See if this role provides metadata. All the available roles - # on the repository are specified in the 'release.txt' metadata. - targets_metadata_allowed = self.metadata['current']['release']['meta'].keys() + # Check if 'snapshot.json' provides metadata for each of the roles in + # 'parent_roles'. All the available roles on the repository are specified + # in the 'snapshot.json' metadata. + targets_metadata_allowed = self.metadata['current']['snapshot']['meta'].keys() for parent_role in parent_roles: - parent_role = parent_role + '.txt' + parent_role = parent_role + '.json' if parent_role not in targets_metadata_allowed: - message = '"release.txt" does not provide all the parent roles'+\ + message = '"snapshot.json" does not provide all the parent roles '+\ 'of '+repr(rolename)+'.' - raise tuf.Repository(message) + raise tuf.RepositoryError(message) - # Remove the 'targets' role because it gets updated when the targets.txt + # Remove the 'targets' role because it gets updated when the targets.json # file is updated in _update_metadata_if_changed('targets'). if rolename == 'targets': try: parent_roles.remove('targets') except ValueError: - message = 'The Release metadata file is missing the "targets.txt" entry.' + message = 'The snapshot metadata file is missing the "targets.json" entry.' raise tuf.RepositoryError(message) # If there is nothing to refresh, we are done. @@ -1940,8 +1995,9 @@ def refresh_targets_metadata_chain(self, rolename): parent_roles.sort() logger.debug('Roles to update: '+repr(parent_roles)+'.') - # Iterate over 'roles_to_update', load its metadata - # file, and update it if it has changed. + # Iterate 'parent_roles', load each role's metadata file from disk, and + # update it if it has changed. + refreshed_chain = [] for rolename in parent_roles: self._load_metadata_from_file('previous', rolename) self._load_metadata_from_file('current', rolename) @@ -1951,9 +2007,11 @@ def refresh_targets_metadata_chain(self, rolename): # Remove the role if it has expired. try: self._ensure_not_expired(rolename) + refreshed_chain.append(rolename) except tuf.ExpiredMetadataError: tuf.roledb.remove_role(rolename) + return refreshed_chain @@ -1965,6 +2023,7 @@ def _targets_of_role(self, rolename, targets=None, skip_refresh=False): Return the target information for all the targets of 'rolename'. The returned information is a list conformant to 'tuf.formats.TARGETFILES_SCHEMA' and has the form: + [{'filepath': 'a/b/c.txt', 'fileinfo': {'length': 13323, 'hashes': {'sha256': dbfac345..}} @@ -1973,7 +2032,7 @@ def _targets_of_role(self, rolename, targets=None, skip_refresh=False): rolename: This is a role name and should not end - in '.txt'. Examples: 'targets', 'targets/linux/x86'. + in '.json'. Examples: 'targets', 'targets/linux/x86'. targets: A list of targets containing target information, conformant to @@ -1993,7 +2052,6 @@ def _targets_of_role(self, rolename, targets=None, skip_refresh=False): A list of dict objects containing the target information of all the targets of 'rolename'. Conformant to 'tuf.formats.TARGETFILES_SCHEMA'. - """ if targets is None: @@ -2011,7 +2069,7 @@ def _targets_of_role(self, rolename, targets=None, skip_refresh=False): # Do we have metadata for 'rolename'? if rolename not in self.metadata['current']: - message = 'No metadata for '+rolename+'. Unable to determine targets.' + message = 'No metadata for '+repr(rolename)+'. Unable to determine targets.' logger.debug(message) return targets @@ -2035,13 +2093,15 @@ def targets_of_role(self, rolename='targets'): Return a list of trusted targets directly specified by 'rolename'. The returned information is a list conformant to tuf.formats.TARGETFILES_SCHEMA and has the form: + [{'filepath': 'a/b/c.txt', 'fileinfo': {'length': 13323, 'hashes': {'sha256': dbfac345..}} ...] - - This may be a very slow operation if there is a large number of - delegations and many metadata files aren't already downloaded. + + The metadata of 'rolename' is updated if out of date, including the + metadata of its parent roles (i.e., the minimum roles needed to set the + chain of trust). rolename: @@ -2063,15 +2123,15 @@ def targets_of_role(self, rolename='targets'): A list of targets, conformant to 'tuf.formats.TARGETFILES_SCHEMA'. - """ # Does 'rolename' have the correct format? # Raise 'tuf.FormatError' if there is a mismatch. tuf.formats.RELPATH_SCHEMA.check_match(rolename) + self.refresh_targets_metadata_chain(rolename) self._refresh_targets_metadata(rolename) - + return self._targets_of_role(rolename, skip_refresh=True) @@ -2081,7 +2141,8 @@ def targets_of_role(self, rolename='targets'): def target(self, target_filepath): """ - Return the target file information for 'target_filepath'. + Return the target file information of 'target_filepath' and update + its corresponding metadata, if necessary. target_filepath: @@ -2104,12 +2165,18 @@ def target(self, target_filepath): The target information for 'target_filepath', conformant to 'tuf.formats.TARGETFILE_SCHEMA'. - """ # Does 'target_filepath' have the correct format? # Raise 'tuf.FormatError' if there is a mismatch. tuf.formats.RELPATH_SCHEMA.check_match(target_filepath) + + # 'target_filepath' might contain URL encoding escapes. + # http://docs.python.org/2/library/urllib.html#urllib.unquote + target_filepath = urllib.unquote(target_filepath) + + if not target_filepath.startswith('/'): + target_filepath = '/' + target_filepath # Get target by looking at roles in order of priority tags. target = self._preorder_depth_first_walk(target_filepath) @@ -2152,18 +2219,17 @@ def _preorder_depth_first_walk(self, target_filepath): The target information for 'target_filepath', conformant to 'tuf.formats.TARGETFILE_SCHEMA'. - """ target = None current_metadata = self.metadata['current'] role_names = ['targets'] - # Ensure the client has the most up-to-date version of 'targets.txt'. - # Raise 'tuf.NoWorkingMirrorError' if the changed metadata cannot be successfully - # downloaded and 'tuf.RepositoryError' if the referenced metadata is - # missing. Target methods such as this one are called after the top-level - # metadata have been refreshed (i.e., updater.refresh()). + # Ensure the client has the most up-to-date version of 'targets.json'. + # Raise 'tuf.NoWorkingMirrorError' if the changed metadata cannot be + # successfully downloaded and 'tuf.RepositoryError' if the referenced + # metadata is missing. Target methods such as this one are called after the + # top-level metadata have been refreshed (i.e., updater.refresh()). self._update_metadata_if_changed('targets') # Preorder depth-first traversal of the tree of target delegations. @@ -2175,7 +2241,7 @@ def _preorder_depth_first_walk(self, target_filepath): # The metadata for 'role_name' must be downloaded/updated before # its targets, delegations, and child roles can be inspected. # self.metadata['current'][role_name] is currently missing. - # _refresh_targets_metadata() does not refresh 'targets.txt', it + # _refresh_targets_metadata() does not refresh 'targets.json', it # expects _update_metadata_if_changed() to have already refreshed it, # which this function has checked above. self._refresh_targets_metadata(role_name, include_delegations=False) @@ -2234,7 +2300,6 @@ def _get_target_from_targets_role(self, role_name, targets, target_filepath): The target information for 'target_filepath', conformant to 'tuf.formats.TARGETFILE_SCHEMA'. - """ target = None @@ -2265,8 +2330,8 @@ def _visit_child_role(self, child_role, target_filepath): Ensure that we explore only delegated roles trusted with the target. We assume conservation of delegated paths in the complete tree of - delegations. Note that the call to _ensure_all_targets_allowed in - __verify_uncompressed_metadata_file should already ensure that all + delegations. Note that the call to tuf.util.ensure_all_targets_allowed in + _verify_uncompressed_metadata_file should already verify that all targets metadata is valid; i.e. that the targets signed by a delegatee is a proper subset of the targets delegated to it by the delegator. Nevertheless, we check it again here for performance and safety reasons. @@ -2295,7 +2360,6 @@ def _visit_child_role(self, child_role, target_filepath): 'target_filepath', then we return the role name of 'child_role'. Otherwise, we return None. - """ child_role_name = child_role['name'] @@ -2367,7 +2431,6 @@ def _get_target_hash(self, target_filepath, hash_function='sha256'): The hash of 'target_filepath'. - """ # Calculate the hash of the filepath to determine which bin to find the @@ -2417,7 +2480,6 @@ def remove_obsolete_targets(self, destination_directory): None. - """ # Does 'destination_directory' have the correct format? @@ -2459,6 +2521,7 @@ def updated_targets(self, targets, destination_directory): The returned information is a list conformant to 'tuf.formats.TARGETFILES_SCHEMA' and has the form: + [{'filepath': 'a/b/c.txt', 'fileinfo': {'length': 13323, 'hashes': {'sha256': dbfac345..}} @@ -2466,7 +2529,8 @@ def updated_targets(self, targets, destination_directory): targets: - A list of target files. + A list of target files. Targets that come earlier in the list are + chosen over duplicates that may occur later. destination_directory: The directory containing the target files. @@ -2480,7 +2544,6 @@ def updated_targets(self, targets, destination_directory): A list of targets, conformant to 'tuf.formats.TARGETFILES_SCHEMA'. - """ # Do the arguments have the correct format? @@ -2488,13 +2551,20 @@ def updated_targets(self, targets, destination_directory): tuf.formats.TARGETFILES_SCHEMA.check_match(targets) tuf.formats.PATH_SCHEMA.check_match(destination_directory) + # Keep track of the target objects and filepaths of updated targets. + # Return 'updated_targets' and use 'updated_targetpaths' to avoid + # duplicates. updated_targets = [] + updated_targetpaths = [] for target in targets: # Get the target's filepath located in 'destination_directory'. # We will compare targets against this file. target_filepath = os.path.join(destination_directory, target['filepath']) + if target_filepath in updated_targetpaths: + continue + # Try one of the algorithm/digest combos for a mismatch. We break # as soon as we find a mismatch. for algorithm, digest in target['fileinfo']['hashes'].items(): @@ -2502,13 +2572,17 @@ def updated_targets(self, targets, destination_directory): try: digest_object = tuf.hash.digest_filename(target_filepath, algorithm=algorithm) + # This exception would occur if the target does not exist locally. except IOError: updated_targets.append(target) + updated_targetpaths.append(target_filepath) break + # The file does exist locally, check if its hash differs. if digest_object.hexdigest() != digest: updated_targets.append(target) + updated_targetpaths.append(target_filepath) break return updated_targets @@ -2545,7 +2619,6 @@ def download_target(self, target, destination_directory): None. - """ # Do the arguments have the correct format? @@ -2563,12 +2636,15 @@ def download_target(self, target, destination_directory): # get_target_file checks every mirror and returns the first target # that passes verification. - target_file_object = self.get_target_file(target_filepath, trusted_length, - trusted_hashes) + target_file_object = self._get_target_file(target_filepath, trusted_length, + trusted_hashes) # We acquired a target file object from a mirror. Move the file into - # place (i.e., locally to 'destination_directory'). - destination = os.path.join(destination_directory, target_filepath) + # place (i.e., locally to 'destination_directory'). Note: join() discards + # 'destination_directory' if 'target_path' contains a leading path separator + # (i.e., is treated as an absolute path). + destination = os.path.join(destination_directory, + target_filepath.lstrip(os.sep)) destination = os.path.abspath(destination) target_dirpath = os.path.dirname(destination) if target_dirpath: @@ -2581,8 +2657,3 @@ def download_target(self, target, destination_directory): logger.warn(str(target_dirpath)+' does not exist.') target_file_object.move(destination) - - - - - diff --git a/tuf/conf.py b/tuf/conf.py index 542bc81bb1..eec68c1444 100755 --- a/tuf/conf.py +++ b/tuf/conf.py @@ -12,11 +12,12 @@ See LICENSE for licensing information. - A central location for TUF configuration settings. - + A central location for TUF configuration settings. Example options include + setting the destination of temporary files and downloaded content, the maximum + length of downloaded metadata (unknown file attributes), download behavior, + and cryptography libraries clients wish to use. """ - # Set a directory that should be used for all temporary files. If this # is None, then the system default will be used. The system default # will also be used if a directory path set here is invalid or @@ -25,11 +26,11 @@ # The directory under which metadata for all repositories will be # stored. This is not a simple cache because each repository's root of -# trust (root.txt) will need to already be stored below here and should +# trust (root.json) will need to already be stored below here and should # not be deleted. At a minimum, each key in the mirrors dictionary # below should have a directory under 'repository_directory' # which already exists and within that directory should have the file -# 'metadata/current/root.txt'. This MUST be set. +# 'metadata/current/root.json'. This MUST be set. repository_directory = None # A PEM (RFC 1422) file where you may find SSL certificate authorities @@ -41,6 +42,12 @@ # default but sane upper bound for the number of bytes required to download it. DEFAULT_TIMESTAMP_REQUIRED_LENGTH = 16384 #bytes +# The Root role may be updated without knowing its hash if top-level metadata +# cannot be safely downloaded (e.g., keys may have been revoked, thus requiring +# a new Root file that includes the updated keys). Set a default upper bound +# for the maximum total bytes that may be downloaded for Root metadata. +DEFAULT_ROOT_REQUIRED_LENGTH = 512000 #bytes + # Set a timeout value in seconds (float) for non-blocking socket operations. SOCKET_TIMEOUT = 1 #seconds @@ -62,6 +69,27 @@ # computational restrictions. A strong user password is still important. # Modifying the number of iterations will result in a new derived key+PBDKF2 # combination if the key is loaded and re-saved, overriding any previous -# iteration setting used by the old '.key'. +# iteration setting used in the old '' key file. # https://en.wikipedia.org/wiki/PBKDF2 PBKDF2_ITERATIONS = 100000 + +# The user client may set the specific cryptography library used by The Update +# Framework updater, or the software updater integrating TUF. +# Supported RSA cryptography libraries: ['pycrypto'] +RSA_CRYPTO_LIBRARY = 'pycrypto' + +# Supported ed25519 cryptography libraries: ['pynacl', 'ed25519'] +ED25519_CRYPTO_LIBRARY = 'ed25519' + +# General purpose cryptography. Algorithms and functions that fall under general +# purpose include AES, PBKDF2, cryptographically strong random number +# generators, and cryptographic hash functions. The majority of the general +# cryptography is needed by the repository and developer tools. +# RSA_CRYPTO_LIBRARY and ED25519_CRYPTO_LIBRARY are needed on the client side +# of the software updater. +GENERAL_CRYPTO_LIBRARY = 'pycrypto' + +# The algorithm(s) in REPOSITORY_HASH_ALGORITHMS are chosen by the repository tool +# to generate the digests listed in metadata and prepended to the filenames of +# consistent snapshots. +REPOSITORY_HASH_ALGORITHMS = ['sha256'] diff --git a/tuf/download.py b/tuf/download.py index 4c3d000681..cd8bf2ace7 100755 --- a/tuf/download.py +++ b/tuf/download.py @@ -19,7 +19,6 @@ file-like object that will automatically destroys itself once closed. Note that the file-like object, 'tuf.util.TempFile', is returned by the '_download_file()' function. - """ # Induce "true division" (http://www.python.org/dev/peps/pep-0238/). @@ -106,7 +105,6 @@ def __start_clock(self): None. - """ # We must have reset the clock before this. @@ -139,7 +137,6 @@ def __stop_clock_and_check_speed(self, data_length): None. - """ # We use (platform-specific) wall time, so it will be imprecise sometimes. @@ -201,7 +198,6 @@ def read(self, size): Received data up to 'size' bytes. - """ # We should never try to specify a negative size. @@ -426,7 +422,6 @@ def _open_connection(url): File-like object. - """ # urllib2.Request produces a Request object that allows for a finer control @@ -478,7 +473,6 @@ def _download_fixed_amount_of_data(connection, temp_file, required_length): total_downloaded: The total number of bytes we have downloaded for the desired file and which should be equal to 'required_length'. - """ # Keep track of total bytes downloaded. @@ -538,7 +532,6 @@ def _get_content_length(connection): reported_length: The total number of bytes reported by server. If the process fails, we return None; otherwise we would return a nonnegative integer. - """ try: @@ -559,7 +552,7 @@ def _get_content_length(connection): -def _check_content_length(reported_length, required_length): +def _check_content_length(reported_length, required_length, strict_length=True): """ A helper function that checks whether the length reported by server is @@ -572,6 +565,10 @@ def _check_content_length(reported_length, required_length): required_length: The total number of bytes obtained from (possibly default) metadata. + strict_length: + Boolean that indicates whether the required length of the file is an + exact match, or an upper limit (e.g., downloading a Timestamp file). + No known side effects. @@ -580,21 +577,33 @@ def _check_content_length(reported_length, required_length): None. - """ + logger.debug('The server reported a length of '+repr(reported_length)+' bytes.') + comparison_result = None + try: if reported_length < required_length: - logger.warn('reported_length ('+str(reported_length)+ - ') < required_length ('+str(required_length)+')') + comparison_result = 'less than' + elif reported_length > required_length: - logger.warn('reported_length ('+str(reported_length)+ - ') > required_length ('+str(required_length)+')') + comparison_result = 'greater than' + else: - logger.debug('reported_length ('+str(reported_length)+ - ') == required_length ('+str(required_length)+')') + comparison_result = 'equal to' + except: - logger.exception('Could not check reported and required lengths!') + logger.exception('Could not check reported and required lengths.') + + if strict_length: + message = 'The reported length is '+comparison_result+' the required '+\ + 'length of '+repr(required_length)+' bytes.' + logger.debug(message) + + else: + message = 'The reported length is '+comparison_result+' the upper limit '+\ + 'of '+repr(required_length)+' bytes.' + logger.debug(message) @@ -612,8 +621,11 @@ def _check_downloaded_length(total_downloaded, required_length, The total number of bytes supposedly downloaded for the file in question. required_length: - The total number of bytes expected of the file as seen from its (possibly - default) metadata. + The total number of bytes expected of the file as seen from its metadata. + The Timestamp role is always downloaded without a known file length, and + the Root role when the client cannot download any of the required + top-level roles. In both cases, 'required_length' is actually an upper + limit on the length of the downloaded file. STRICT_REQUIRED_LENGTH: A Boolean indicator used to signal whether we should perform strict @@ -630,30 +642,33 @@ def _check_downloaded_length(total_downloaded, required_length, None. - """ if total_downloaded == required_length: - logger.debug('total_downloaded == required_length == '+ - str(required_length)) + logger.info('Downloaded '+str(total_downloaded)+' bytes out of the '+\ + 'expected '+str(required_length)+ ' bytes.') else: difference_in_bytes = abs(total_downloaded-required_length) - message = 'Downloaded '+str(total_downloaded)+' bytes, but expected '+\ - str(required_length)+' bytes. There is a difference of '+\ - str(difference_in_bytes)+' bytes!' # What we downloaded is not equal to the required length, but did we ask # for strict checking of required length? - if STRICT_REQUIRED_LENGTH: + if STRICT_REQUIRED_LENGTH: + message = 'Downloaded '+str(total_downloaded)+' bytes, but expected '+\ + str(required_length)+' bytes. There is a difference of '+\ + str(difference_in_bytes)+' bytes.' + # This must be due to a programming error, and must never happen! logger.error(message) raise tuf.DownloadLengthMismatchError(required_length, total_downloaded) + else: + message = 'Downloaded '+str(total_downloaded)+' bytes out of an upper '+\ + 'limit of '+str(required_length)+' bytes.' # We specifically disabled strict checking of required length, but we # will log a warning anyway. This is useful when we wish to download the - # timestamp metadata, for which we have no signed metadata; so, we must - # guess a reasonable required_length for it. - logger.warn(message) + # Timestamp or Root metadata, for which we have no signed metadata; so, + # we must guess a reasonable required_length for it. + logger.info(message) @@ -711,7 +726,6 @@ def _download_file(url, required_length, STRICT_REQUIRED_LENGTH=True): A 'tuf.util.TempFile' file-like object which points to the contents of 'url'. - """ # Do all of the arguments have the appropriate format? @@ -748,7 +762,8 @@ def _download_file(url, required_length, STRICT_REQUIRED_LENGTH=True): reported_length = _get_content_length(connection) # Then, we check whether the required length matches the reported length. - _check_content_length(reported_length, required_length) + _check_content_length(reported_length, required_length, + STRICT_REQUIRED_LENGTH) # Download the contents of the URL, up to the required length, to a # temporary file, and get the total number of downloaded bytes. @@ -773,8 +788,3 @@ def _download_file(url, required_length, STRICT_REQUIRED_LENGTH=True): # Restore previously saved values or functions. httplib.HTTPConnection.response_class = previous_http_response_class socket.setdefaulttimeout(previous_socket_timeout) - - - - - diff --git a/tuf/ed25519_key.py b/tuf/ed25519_key.py deleted file mode 100755 index f2583f7620..0000000000 --- a/tuf/ed25519_key.py +++ /dev/null @@ -1,624 +0,0 @@ -""" - - ed25519_key.py - - - Vladimir Diaz - - - September 24, 2013. - - - See LICENSE for licensing information. - - - The goal of this module is to support ed25519 signatures. ed25519 is an - elliptic-curve public key signature scheme, its main strength being small - signatures (64 bytes) and small public keys (32 bytes). - http://ed25519.cr.yp.to/ - - 'tuf/ed25519_key.py' calls 'ed25519/ed25519.py', which is the pure Python - implementation of ed25519 provided by the author: - http://ed25519.cr.yp.to/software.html - Optionally, ed25519 cryptographic operations may be executed by PyNaCl, which - provides Python bindings to the NaCl library and is much faster than the pure - python implementation. PyNaCl relies on the C library, libsodium. - - https://github.com/dstufft/pynacl - https://github.com/jedisct1/libsodium - http://nacl.cr.yp.to/ - - The ed25519-related functions included here are generate(), create_signature() - and verify_signature(). The 'ed25519' and PyNaCl (i.e., 'nacl') modules used - by ed25519_key.py generate the actual ed25519 keys and the functions listed - above can be viewed as an easy-to-use public interface. Additional functions - contained here include format_keyval_to_metadata() and - format_metadata_to_key(). These last two functions produce or use - ed25519 keys compatible with the key structures listed in TUF Metadata files. - The generate() function returns a dictionary containing all the information - needed of ed25519 keys, such as public/private keys and a keyID identifier. - create_signature() and verify_signature() are supplemental functions used for - generating ed25519 signatures and verifying them. - - Key IDs are used as identifiers for keys (e.g., RSA key). They are the - hexadecimal representation of the hash of key object (specifically, the key - object containing only the public key). Review 'ed25519_key.py' and the - '_get_keyid()' function to see precisely how keyids are generated. One may - get the keyid of a key object by simply accessing the dictionary's 'keyid' - key (i.e., ed25519_key_dict['keyid']). - """ - -# Help with Python 3 compatability, where the print statement is a function, an -# implicit relative import is invalid, and the '/' operator performs true -# division. Example: print 'hello world' raises a 'SyntaxError' exception. -from __future__ import print_function -from __future__ import absolute_import -from __future__ import division - -# Required for hexadecimal conversions. Signatures and public/private keys are -# hexlified. -import binascii - -# Generate OS-specific randomness (os.urandom) suitable for cryptographic use. -# http://docs.python.org/2/library/os.html#miscellaneous-functions -import os - -import tuf - -# Import the python implementation of the ed25519 algorithm that is provided by -# the author. Note: This implementation is very slow and does not include -# protection against side-channel attacks according to the author. Verifying -# signatures can take approximately 9 seconds on an intel core 2 duo @ -# 2.2 ghz x 2). Optionally, the PyNaCl module may be used to speed up ed25519 -# cryptographic operations. -# http://ed25519.cr.yp.to/software.html -# Try to import PyNaCl. The functions found in this module provide the option -# of using PyNaCl over the slower implementation of ed25519. -try: - import nacl.signing - import nacl.encoding -except (ImportError, IOError): - message = 'The PyNacl library and/or its dependencies cannot be imported.' - raise tuf.UnsupportedLibraryError(message) - -# The pure Python implementation of ed25519. -import ed25519.ed25519 - -# Digest objects needed to generate hashes. -import tuf.hash - -# Perform object format-checking. -import tuf.formats - -# The default hash algorithm to use when generating KeyIDs. -_KEY_ID_HASH_ALGORITHM = 'sha256' - -# Supported ed25519 signing methods. 'ed25519-python' is the pure Python -# implementation signing method. 'ed25519-pynacl' (i.e., 'nacl' module) is the -# (libsodium+Python bindings) implementation signing method. -_SUPPORTED_ED25519_SIGNING_METHODS = ['ed25519-python', 'ed25519-pynacl'] - - -def generate(use_pynacl=False): - """ - - Generate an ed25519 seed key ('sk') and public key ('pk'). - In addition, a keyid used as an identifier for ed25519 keys is generated. - The object returned conforms to 'tuf.formats.ED25519KEY_SCHEMA' and has the - form: - {'keytype': 'ed25519', - 'keyid': keyid, - 'keyval': {'public': '876f5584a9db99b8546c0d8608d6...', - 'private': 'bf7336055c7638276efe9afe039...'}} - - The public and private keys are strings. An ed25519 seed key is a random - 32-byte value and public key 32 bytes, although both are hexlified to 64 - bytes. - - >>> ed25519_key = generate() - >>> tuf.formats.ED25519KEY_SCHEMA.matches(ed25519_key) - True - >>> len(ed25519_key['keyval']['public']) - 64 - >>> len(ed25519_key['keyval']['private']) - 64 - >>> ed25519_key_pynacl = generate(use_pynacl=True) - >>> tuf.formats.ED25519KEY_SCHEMA.matches(ed25519_key_pynacl) - True - >>> len(ed25519_key_pynacl['keyval']['public']) - 64 - >>> len(ed25519_key_pynacl['keyval']['private']) - 64 - - - use_pynacl: - True, if the ed25519 keys should be generated with PyNaCl. False, if the - keys should be generated with the pure Python implementation of ed25519 - (much slower). - - - NotImplementedError, if a randomness source is not found. - - - The ed25519 keys are generated by first creating a random 32-byte value - 'sk' with os.urandom() and then calling ed25519's ed25519.25519.publickey(sk) - or PyNaCl's nacl.signing.SigningKey(). - - - A dictionary containing the ed25519 keys and other identifying information. - Conforms to 'tuf.formats.ED25519KEY_SCHEMA'. - """ - - # Begin building the ed25519 key dictionary. - ed25519_key_dict = {} - keytype = 'ed25519' - - # Generate ed25519's seed key by calling os.urandom(). The random bytes - # returned should be suitable for cryptographic use and is OS-specific. - # Raise 'NotImplementedError' if a randomness source is not found. - # ed25519 seed keys are fixed at 32 bytes (256-bit keys). - # http://blog.mozilla.org/warner/2011/11/29/ed25519-keys/ - seed = os.urandom(32) - public = None - - if use_pynacl: - # Generate the public key. PyNaCl (i.e., 'nacl' module) performs - # the actual key generation. - nacl_key = nacl.signing.SigningKey(seed) - public = str(nacl_key.verify_key) - - # Use the pure Python implementation of ed25519. - else: - public = ed25519.ed25519.publickey(seed) - - # Generate the keyid for the ed25519 key dict. 'key_value' corresponds to the - # 'keyval' entry of the 'ED25519KEY_SCHEMA' dictionary. The seed (private) - # key information is not included in the generation of the 'keyid' identifier. - key_value = {'public': binascii.hexlify(public), - 'private': ''} - keyid = _get_keyid(key_value) - - # Build the 'ed25519_key_dict' dictionary. Update 'key_value' with the - # ed25519 seed key prior to adding 'key_value' to 'ed25519_key_dict'. - key_value['private'] = binascii.hexlify(seed) - - ed25519_key_dict['keytype'] = keytype - ed25519_key_dict['keyid'] = keyid - ed25519_key_dict['keyval'] = key_value - - return ed25519_key_dict - - - - - -def format_keyval_to_metadata(key_value, private=False): - """ - - Return a dictionary conformant to 'tuf.formats.KEY_SCHEMA'. - If 'private' is True, include the private key. The dictionary - returned has the form: - {'keytype': 'ed25519', - 'keyval': {'public': '876f5584a9db99b8546c0d8608d6...', - 'private': 'bf7336055c7638276efe9afe039...'}} - - or if 'private' is False: - - {'keytype': 'ed25519', - 'keyval': {'public': '876f5584a9db99b8546c0d8608d6...', - 'private': ''}} - - The private and public keys are 32 bytes, although hexlified to 64 bytes. - - ed25519 keys are stored in Metadata files (e.g., root.txt) in the format - returned by this function. - - >>> ed25519_key = generate() - >>> key_val = ed25519_key['keyval'] - >>> ed25519_metadata = format_keyval_to_metadata(key_val, private=True) - >>> tuf.formats.KEY_SCHEMA.matches(ed25519_metadata) - True - - - key_value: - A dictionary containing a seed and public ed25519 key. - 'key_value' is of the form: - - {'public': '876f5584a9db99b8546c0d8608d6...', - 'private': 'bf7336055c7638276efe9afe039...'} - - conformat to 'tuf.formats.KEYVAL_SCHEMA'. - - private: - Indicates if the private key should be included in the - returned dictionary. - - - tuf.FormatError, if 'key_value' does not conform to - 'tuf.formats.KEYVAL_SCHEMA'. - - - None. - - - A 'KEY_SCHEMA' dictionary. - """ - - # Does 'key_value' have the correct format? - # This check will ensure 'key_value' has the appropriate number - # of objects and object types, and that all dict keys are properly named. - # Raise 'tuf.FormatError' if the check fails. - tuf.formats.KEYVAL_SCHEMA.check_match(key_value) - - if private is True and len(key_value['private']): - return {'keytype': 'ed25519', 'keyval': key_value} - else: - public_key_value = {'public': key_value['public'], 'private': ''} - return {'keytype': 'ed25519', 'keyval': public_key_value} - - - - - -def format_metadata_to_key(key_metadata): - """ - - Construct an ed25519 key dictionary (i.e., tuf.formats.ED25519KEY_SCHEMA) - from 'key_metadata'. The dict returned by this function has the exact - format as the dict returned by generate(). It is of the form: - - {'keytype': 'ed25519', - 'keyid': keyid, - 'keyval': {'public': '876f5584a9db99b8546c0d8608d6...', - 'private': 'bf7336055c7638276efe9afe039...'}} - - The public and private keys are 32-byte strings, although hexlified to 64 - bytes. - - ed25519 key dictionaries in 'ED25519KEY_SCHEMA' format should be used by - modules storing a collection of keys, such as a keydb keystore. - ed25519 keys as stored in metadata files use a different format, so this - function should be called if an ed25519 key is extracted from one of these - metadata files and needs converting. Generate() creates an entirely - new key and returns it in the format appropriate for 'keydb.py' and - 'keystore.py'. - - >>> ed25519_key = generate() - >>> key_val = ed25519_key['keyval'] - >>> ed25519_metadata = format_keyval_to_metadata(key_val, private=True) - >>> ed25519_key_2 = format_metadata_to_key(ed25519_metadata) - >>> tuf.formats.ED25519KEY_SCHEMA.matches(ed25519_key_2) - True - >>> ed25519_key == ed25519_key_2 - True - - - key_metadata: - The ed25519 key dictionary as stored in Metadata files, conforming to - 'tuf.formats.KEY_SCHEMA'. It has the form: - - {'keytype': 'ed25519', - 'keyval': {'public': '876f5584a9db99b8546c0d8608d6...', - 'private': 'bf7336055c7638276efe9afe039...'}} - - - tuf.FormatError, if 'key_metadata' does not conform to - 'tuf.formats.KEY_SCHEMA'. - - - None. - - - A dictionary containing the ed25519 keys and other identifying information. - """ - - # Does 'key_metadata' have the correct format? - # This check will ensure 'key_metadata' has the appropriate number - # of objects and object types, and that all dict keys are properly named. - # Raise 'tuf.FormatError' if the check fails. - tuf.formats.KEY_SCHEMA.check_match(key_metadata) - - # Construct the dictionary to be returned. - ed25519_key_dict = {} - keytype = 'ed25519' - key_value = key_metadata['keyval'] - - # Convert 'key_value' to 'tuf.formats.KEY_SCHEMA' and generate its hash - # The hash is in hexdigest form. _get_keyid() ensures the private key - # information is not included. - keyid = _get_keyid(key_value) - - # We now have all the required key values. Build 'ed25519_key_dict'. - ed25519_key_dict['keytype'] = keytype - ed25519_key_dict['keyid'] = keyid - ed25519_key_dict['keyval'] = key_value - - return ed25519_key_dict - - - - - -def _get_keyid(key_value): - """Return the keyid for 'key_value'.""" - - # 'keyid' will be generated from an object conformant to 'KEY_SCHEMA', - # which is the format Metadata files (e.g., root.txt) store keys. - # 'format_keyval_to_metadata()' returns the object needed by _get_keyid(). - ed25519_key_meta = format_keyval_to_metadata(key_value, private=False) - - # Convert the ed25519 key to JSON Canonical format suitable for adding - # to digest objects. - ed25519_key_update_data = tuf.formats.encode_canonical(ed25519_key_meta) - - # Create a digest object and call update(), using the JSON - # canonical format of 'ed25519_key_meta' as the update data. - digest_object = tuf.hash.digest(_KEY_ID_HASH_ALGORITHM) - digest_object.update(ed25519_key_update_data) - - # 'keyid' becomes the hexadecimal representation of the hash. - keyid = digest_object.hexdigest() - - return keyid - - - - - -def create_signature(ed25519_key_dict, data, use_pynacl=False): - """ - - Return a signature dictionary of the form: - {'keyid': 'a0469d9491e3c0b42dd41fe3455359dbacb3306b6e8fb59...', - 'method': 'ed25519-python', - 'sig': '4b3829671b2c6b90034518a918d2447c722474c878c2431dd...'} - - Note: 'method' may also be 'ed25519-pynacl', if the signature was created - by the 'nacl' module. - - The signing process will use the public and seed key - ed25519_key_dict['keyval']['private'], - ed25519_key_dict['keyval']['public'] - - and 'data' to generate the signature. - - >>> ed25519_key_dict = generate() - >>> data = 'The quick brown fox jumps over the lazy dog.' - >>> signature = create_signature(ed25519_key_dict, data) - >>> tuf.formats.SIGNATURE_SCHEMA.matches(signature) - True - >>> len(signature['sig']) - 128 - >>> signature_pynacl = create_signature(ed25519_key_dict, data, True) - >>> tuf.formats.SIGNATURE_SCHEMA.matches(signature_pynacl) - True - >>> len(signature_pynacl['sig']) - 128 - - - ed25519_key_dict: - A dictionary containing the ed25519 keys and other identifying information. - 'ed25519_key_dict' has the form: - - {'keytype': 'ed25519', - 'keyid': keyid, - 'keyval': {'public': '876f5584a9db99b8546c0d8608d6...', - 'private': 'bf7336055c7638276efe9afe039...'}} - - The public and private keys are 32-byte strings, although hexlified to 64 - bytes. - - data: - Data object used by create_signature() to generate the signature. - - use_pynacl: - True, if the ed25519 signature should be generated with PyNaCl. False, - if the signature should be generated with the pure Python implementation - of ed25519 (much slower). - - - TypeError, if a private key is not defined for 'ed25519_key_dict'. - - tuf.FormatError, if an incorrect format is found for 'ed25519_key_dict'. - - tuf.CryptoError, if a signature cannot be created. - - - ed25519.ed25519.signature() or nacl.signing.SigningKey.sign() called to - generate the actual signature. - - - A signature dictionary conformat to 'tuf.format.SIGNATURE_SCHEMA'. - ed25519 signatures are 64 bytes, however, the hexlified signature - (128 bytes) is stored in the dictionary returned. - """ - - # Does 'ed25519_key_dict' have the correct format? - # This check will ensure 'ed25519_key_dict' has the appropriate number - # of objects and object types, and that all dict keys are properly named. - # Raise 'tuf.FormatError' if the check fails. - tuf.formats.ED25519KEY_SCHEMA.check_match(ed25519_key_dict) - - # Signing the 'data' object requires a seed and public key. - # 'ed25519.ed25519.py' generates the actual 64-byte signature in pure Python. - # nacl.signing.SigningKey.sign() generates the signature if 'use_pynacl' - # is True. - signature = {} - private_key = ed25519_key_dict['keyval']['private'] - public_key = ed25519_key_dict['keyval']['public'] - private_key = binascii.unhexlify(private_key) - public_key = binascii.unhexlify(public_key) - - keyid = ed25519_key_dict['keyid'] - method = None - sig = None - - # Verify the signature, but only if the private key has been set. The private - # key is a NULL string if unset. Although it may be clearer to explicit check - # that 'private_key' is not '', we can/should check for a value and not - # compare identities with the 'is' keyword. - if len(private_key): - if use_pynacl: - method = 'ed25519-pynacl' - try: - nacl_key = nacl.signing.SigningKey(private_key) - nacl_sig = nacl_key.sign(data) - sig = nacl_sig.signature - except (ValueError, nacl.signing.CryptoError): - message = 'An "ed25519-pynacl" signature could not be created.' - raise tuf.CryptoError(message) - - # Generate an "ed25519-python" (i.e., pure python implementation) signature. - else: - # ed25519.ed25519.signature() requires both the seed and public keys. - # It calculates the SHA512 of the seed key, which is 32 bytes. - method = 'ed25519-python' - try: - sig = ed25519.ed25519.signature(data, private_key, public_key) - except Exception, e: - message = 'An "ed25519-python" signature could not be generated.' - raise tuf.CryptoError(message) - - # Raise an exception since the private key is not defined. - else: - message = 'The required private key is not defined for "ed25519_key_dict".' - raise TypeError(message) - - # Build the signature dictionary to be returned. - # The hexadecimal representation of 'sig' is stored in the signature. - signature['keyid'] = keyid - signature['method'] = method - signature['sig'] = binascii.hexlify(sig) - - return signature - - - - - -def verify_signature(ed25519_key_dict, signature, data, use_pynacl=False): - """ - - Determine whether the seed key belonging to 'ed25519_key_dict' produced - 'signature'. verify_signature() will use the public key found in - 'ed25519_key_dict', the 'method' and 'sig' objects contained in 'signature', - and 'data' to complete the verification. Type-checking performed on both - 'ed25519_key_dict' and 'signature'. - - >>> ed25519_key_dict = generate() - >>> data = 'The quick brown fox jumps over the lazy dog.' - >>> signature = create_signature(ed25519_key_dict, data) - >>> verify_signature(ed25519_key_dict, signature, data) - True - >>> verify_signature(ed25519_key_dict, signature, data, True) - True - >>> bad_data = 'The sly brown fox jumps over the lazy dog.' - >>> bad_signature = create_signature(ed25519_key_dict, bad_data) - >>> verify_signature(ed25519_key_dict, bad_signature, data, True) - False - - - ed25519_key_dict: - A dictionary containing the ed25519 keys and other identifying - information. 'ed25519_key_dict' has the form: - - {'keytype': 'ed25519', - 'keyid': 'a0469d9491e3c0b42dd41fe3455359dbacb3306b6e8fb59...', - 'keyval': {'public': '876f5584a9db99b8546c0d8608d6...', - 'private': 'bf7336055c7638276efe9afe039...'}} - - The public and private keys are 32-byte strings, although hexlified to - 64 bytes. - - signature: - The signature dictionary produced by tuf.ed25519_key.create_signature(). - 'signature' has the form: - - {'keyid': 'a0469d9491e3c0b42dd41fe3455359dbacb3306b6e8fb59...', - 'method': 'ed25519-python', - 'sig': '4b3829671b2c6b90034518a918d2447c722474c878c2431dd...'} - - Conformant to 'tuf.formats.SIGNATURE_SCHEMA'. - - data: - Data object used by tuf.ed25519_key.create_signature() to generate - 'signature'. 'data' is needed here to verify the signature. - - use_pynacl: - True, if the ed25519 signature should be verified with PyNaCl. False, - if the signature should be verified with the pure Python implementation - of ed25519 (much slower). - - - tuf.UnknownMethodError. Raised if the signing method used by - 'signature' is not one supported by tuf.ed25519_key.create_signature(). - - tuf.FormatError. Raised if either 'ed25519_key_dict' - or 'signature' do not match their respective tuf.formats schema. - 'ed25519_key_dict' must conform to 'tuf.formats.ED25519KEY_SCHEMA'. - 'signature' must conform to 'tuf.formats.SIGNATURE_SCHEMA'. - - - ed25519.ed25519.checkvalid() called to do the actual verification. - nacl.signing.VerifyKey.verify() called if 'use_pynacl' is True. - - - Boolean. True if the signature is valid, False otherwise. - """ - - # Does 'ed25519_key_dict' have the correct format? - # This check will ensure 'ed25519_key_dict' has the appropriate number - # of objects and object types, and that all dict keys are properly named. - # Raise 'tuf.FormatError' if the check fails. - tuf.formats.ED25519KEY_SCHEMA.check_match(ed25519_key_dict) - - # Does 'signature' have the correct format? - tuf.formats.SIGNATURE_SCHEMA.check_match(signature) - - # Using the public key belonging to 'ed25519_key_dict' - # (i.e., ed25519_key_dict['keyval']['public']), verify whether 'signature' - # was produced by ed25519_key_dict's corresponding seed key - # ed25519_key_dict['keyval']['private']. Before returning the Boolean result, - # ensure 'ed25519-python' or 'ed25519-pynacl' was used as the signing method. - method = signature['method'] - sig = signature['sig'] - sig = binascii.unhexlify(sig) - public = ed25519_key_dict['keyval']['public'] - public = binascii.unhexlify(public) - valid_signature = False - - if method in _SUPPORTED_ED25519_SIGNING_METHODS: - if use_pynacl: - try: - nacl_verify_key = nacl.signing.VerifyKey(public) - nacl_message = nacl_verify_key.verify(data, sig) - if nacl_message == data: - valid_signature = True - except nacl.signing.BadSignatureError: - pass - - # Verify signature with 'ed25519-python' (i.e., pure Python implementation). - else: - try: - ed25519.ed25519.checkvalid(sig, data, public) - valid_signature = True - - # The pure Python implementation raises 'Exception' if 'signature' is - # invalid. - except Exception, e: - pass - else: - message = 'Unsupported ed25519 signing method: '+repr(method)+'.\n'+ \ - 'Supported methods: '+repr(_SUPPORTED_ED25519_SIGNING_METHODS)+'.' - raise tuf.UnknownMethodError(message) - - return valid_signature - - - -if __name__ == '__main__': - # The interactive sessions of the documentation strings can - # be tested by running 'ed25519_key.py' as a standalone module. - # python -B ed25519_key.py - import doctest - doctest.testmod() diff --git a/tuf/ed25519_keys.py b/tuf/ed25519_keys.py new file mode 100755 index 0000000000..5a8303ec23 --- /dev/null +++ b/tuf/ed25519_keys.py @@ -0,0 +1,411 @@ +""" + + ed25519_keys.py + + + Vladimir Diaz + + + September 24, 2013. + + + See LICENSE for licensing information. + + + The goal of this module is to support ed25519 signatures. ed25519 is an + elliptic-curve public key signature scheme, its main strength being small + signatures (64 bytes) and small public keys (32 bytes). + http://ed25519.cr.yp.to/ + + 'tuf/ed25519_keys.py' calls 'tuf._vendor.ed25519/ed25519.py', which is the + pure Python implementation of ed25519 optimized for a faster runtime. + The Python reference implementation is concise, but very slow (verifying + signatures takes ~9 seconds on an Intel core 2 duo @ 2.2 ghz x 2). The + optimized version can verify signatures in ~2 seconds. + + http://ed25519.cr.yp.to/software.html + https://github.com/pyca/ed25519 + + Optionally, ed25519 cryptographic operations may be executed by PyNaCl, which + is a Python binding to the NaCl library and is faster than the pure python + implementation. Verifying signatures can take approximately 0.0009 seconds. + PyNaCl relies on the libsodium C library. + + https://github.com/pyca/pynacl + https://github.com/jedisct1/libsodium + http://nacl.cr.yp.to/ + + The ed25519-related functions included here are generate(), create_signature() + and verify_signature(). The 'ed25519' and PyNaCl (i.e., 'nacl') modules used + by ed25519_keys.py generate the actual ed25519 keys and the functions listed + above can be viewed as an easy-to-use public interface. + """ + +# Help with Python 3 compatibility, where the print statement is a function, an +# implicit relative import is invalid, and the '/' operator performs true +# division. Example: print 'hello world' raises a 'SyntaxError' exception. +from __future__ import print_function +from __future__ import absolute_import +from __future__ import division + +# 'binascii' required for hexadecimal conversions. Signatures and +# public/private keys are hexlified. +import binascii + +# 'os' required to generate OS-specific randomness (os.urandom) suitable for +# cryptographic use. +# http://docs.python.org/2/library/os.html#miscellaneous-functions +import os + +# Import the python implementation of the ed25519 algorithm provided by pyca, +# which is an optimized version of the one provided by ed25519's authors. +# Note: The pure Python version do not include protection against side-channel +# attacks. Verifying signatures can take approximately 2 seconds on a intel +# core 2 duo @ 2.2 ghz x 2). Optionally, the PyNaCl module may be used to +# speed up ed25519 cryptographic operations. +# http://ed25519.cr.yp.to/software.html +# https://github.com/pyca/ed25519 +# https://github.com/pyca/pynacl +# +# PyNaCl's 'cffi' dependency may thrown an 'IOError' exception when +# importing 'nacl.signing'. +try: + import nacl.signing + import nacl.encoding +except (ImportError, IOError): + pass + +# The optimized pure Python implementation of ed25519 provided by TUF. If +# PyNaCl cannot be imported and an attempt to use is made in this module, a +# 'tuf.UnsupportedLibraryError' exception is raised. +import tuf._vendor.ed25519.ed25519 + +import tuf + +# Digest objects needed to generate hashes. +import tuf.hash + +# Perform object format-checking. +import tuf.formats + +# Supported ed25519 signing method: 'ed25519'. The pure Python +# implementation (i.e., 'tuf._vendor.ed25519.ed25519') and PyNaCl +# (i.e., 'nacl', libsodium+Python bindgs) modules are currently supported in +# the creationg of 'ed25519' signatures. Previously, a distinction was made +# between signatures made by the pure Python implementation and PyNaCl. +_SUPPORTED_ED25519_SIGNING_METHODS = ['ed25519',] + + +def generate_public_and_private(use_pynacl=False): + """ + + Generate a pair of ed25519 public and private keys. + The public and private keys returned conform to + 'tuf.formats.ED25519PULIC_SCHEMA' and 'tuf.formats.ED25519SEED_SCHEMA', + respectively, and have the form: + + '\xa2F\x99\xe0\x86\x80%\xc8\xee\x11\xb95T\xd9\...' + + An ed25519 seed key is a random 32-byte string. Public keys are also 32 + bytes. + + >>> public, private = generate_public_and_private(use_pynacl=False) + >>> tuf.formats.ED25519PUBLIC_SCHEMA.matches(public) + True + >>> tuf.formats.ED25519SEED_SCHEMA.matches(private) + True + >>> public, private = generate_public_and_private(use_pynacl=True) + >>> tuf.formats.ED25519PUBLIC_SCHEMA.matches(public) + True + >>> tuf.formats.ED25519SEED_SCHEMA.matches(private) + True + + + use_pynacl: + True, if the ed25519 keys should be generated with PyNaCl. False, if the + keys should be generated with the pure Python implementation of ed25519 + (slower). + + + tuf.FormatError, if 'use_pynacl' is not a Boolean. + + tuf.UnsupportedLibraryError, if the PyNaCl ('nacl') module is unavailable + and 'use_pynacl' is True. + + NotImplementedError, if a randomness source is not found by 'os.urandom'. + + + The ed25519 keys are generated by first creating a random 32-byte seed + with os.urandom() and then calling ed25519's + ed25519.25519.publickey(seed) or PyNaCl's nacl.signing.SigningKey(). + + + A (public, private) tuple that conform to 'tuf.formats.ED25519PUBLIC_SCHEMA' + and 'tuf.formats.ED25519SEED_SCHEMA', respectively. + """ + + # Does 'use_pynacl' have the correct format? + # This check will ensure 'use_pynacl' conforms to 'tuf.formats.BOOLEAN_SCHEMA'. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.BOOLEAN_SCHEMA.check_match(use_pynacl) + + # Generate ed25519's seed key by calling os.urandom(). The random bytes + # returned should be suitable for cryptographic use and is OS-specific. + # Raise 'NotImplementedError' if a randomness source is not found. + # ed25519 seed keys are fixed at 32 bytes (256-bit keys). + # http://blog.mozilla.org/warner/2011/11/29/ed25519-keys/ + seed = os.urandom(32) + public = None + + if use_pynacl: + # Generate the public key. PyNaCl (i.e., 'nacl' module) performs + # the actual key generation. + try: + nacl_key = nacl.signing.SigningKey(seed) + public = str(nacl_key.verify_key) + except NameError: + message = 'The PyNaCl library and/or its dependencies unavailable.' + raise tuf.UnsupportedLibraryError(message) + + # Use the pure Python implementation of ed25519. + else: + public = tuf._vendor.ed25519.ed25519.publickey(seed) + + return public, seed + + + + + +def create_signature(public_key, private_key, data, use_pynacl=False): + """ + + Return a (signature, method) tuple, where the method is 'ed25519' and + generated by either the pure python implemenation, or by PyNaCl + (i.e., 'nacl'). The signature returns conforms to + 'tuf.formats.ED25519SIGNATURE_SCHEMA', and has the form: + + '\xae\xd7\x9f\xaf\x95{bP\x9e\xa8YO Z\x86\x9d...' + + A signature is a 64-byte string. + + >>> public, private = generate_public_and_private(use_pynacl=False) + >>> data = 'The quick brown fox jumps over the lazy dog' + >>> signature, method = \ + create_signature(public, private, data, use_pynacl=False) + >>> tuf.formats.ED25519SIGNATURE_SCHEMA.matches(signature) + True + >>> method == 'ed25519' + True + >>> signature, method = \ + create_signature(public, private, data, use_pynacl=True) + >>> tuf.formats.ED25519SIGNATURE_SCHEMA.matches(signature) + True + >>> method == 'ed25519' + True + + + public: + The ed25519 public key, which is a 32-byte string. + + private: + The ed25519 private key, which is a 32-byte string. + + data: + Data object used by create_signature() to generate the signature. + + use_pynacl: + True, if the ed25519 signature should be generated with PyNaCl. False, + if the signature should be generated with the pure Python implementation + of ed25519 (much slower). + + + tuf.FormatError, if the arguments are improperly formatted. + + tuf.CryptoError, if a signature cannot be created. + + + tuf._vendor.ed25519.ed25519.signature() or nacl.signing.SigningKey.sign() + called to generate the actual signature. + + + A signature dictionary conformat to 'tuf.format.SIGNATURE_SCHEMA'. + ed25519 signatures are 64 bytes, however, the hexlified signature is + stored in the dictionary returned. + """ + + # Does 'public_key' have the correct format? + # This check will ensure 'public_key' conforms to + # 'tuf.formats.ED25519PUBLIC_SCHEMA', which must have length 32 bytes. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.ED25519PUBLIC_SCHEMA.check_match(public_key) + + # Is 'private_key' properly formatted? + tuf.formats.ED25519SEED_SCHEMA.check_match(private_key) + + # Is 'use_pynacl' properly formatted? + tuf.formats.BOOLEAN_SCHEMA.check_match(use_pynacl) + + # Signing the 'data' object requires a seed and public key. + # 'tuf._vendor.ed25519.ed25519.py' generates the actual 64-byte signature in + # pure Python. nacl.signing.SigningKey.sign() generates the signature if + # 'use_pynacl' is True. + public = public_key + private = private_key + + method = None + signature = None + + # The private and public keys have been validated above by 'tuf.formats' and + # should be 32-byte strings. + if use_pynacl: + method = 'ed25519' + try: + nacl_key = nacl.signing.SigningKey(private) + nacl_sig = nacl_key.sign(data) + signature = nacl_sig.signature + + except NameError: + message = 'The PyNaCl library and/or its dependencies unavailable.' + raise tuf.UnsupportedLibraryError(message) + + except (ValueError, nacl.signing.CryptoError): + message = 'An "ed25519" signature could not be created with PyNaCl.' + raise tuf.CryptoError(message) + + # Generate an "ed25519" signature with the pure python implementation. + else: + # tuf._vendor.ed25519.ed25519.signature() requires both the seed and + # public keys. It calculates the SHA512 of the seed key, which is 32 bytes. + method = 'ed25519' + try: + signature = tuf._vendor.ed25519.ed25519.signature(data, private, public) + + # 'Exception' raised by ed25519.py for any exception that may occur. + except Exception, e: + message = 'An "ed25519" signature could not be generated in pure Python.' + raise tuf.CryptoError(message) + + return signature, method + + + + + +def verify_signature(public_key, method, signature, data, use_pynacl=False): + """ + + Determine whether the private key corresponding to 'public_key' produced + 'signature'. verify_signature() will use the public key, the 'method' and + 'sig', and 'data' arguments to complete the verification. + + >>> public, private = generate_public_and_private(use_pynacl=False) + >>> data = 'The quick brown fox jumps over the lazy dog' + >>> signature, method = \ + create_signature(public, private, data, use_pynacl=False) + >>> verify_signature(public, method, signature, data, use_pynacl=False) + True + >>> verify_signature(public, method, signature, data, use_pynacl=True) + True + >>> bad_data = 'The sly brown fox jumps over the lazy dog' + >>> bad_signature, method = \ + create_signature(public, private, bad_data, use_pynacl=False) + >>> verify_signature(public, method, bad_signature, data, use_pynacl=False) + False + + + public_key: + The public key is a 32-byte string. + + method: + 'ed25519' signature method generated by either the pure python + implementation (i.e., 'tuf._vendor.ed25519.ed25519.py') or PyNacl + (i.e., 'nacl'). + + signature: + The signature is a 64-byte string. + + data: + Data object used by tuf.ed25519_keys.create_signature() to generate + 'signature'. 'data' is needed here to verify the signature. + + use_pynacl: + True, if the ed25519 signature should be verified by PyNaCl. False, + if the signature should be verified with the pure Python implementation + of ed25519 (slower). + + + tuf.UnknownMethodError. Raised if the signing method used by + 'signature' is not one supported by tuf.ed25519_keys.create_signature(). + + tuf.FormatError. Raised if the arguments are improperly formatted. + + + tuf._vendor.ed25519.ed25519.checkvalid() called to do the actual + verification. nacl.signing.VerifyKey.verify() called if 'use_pynacl' is + True. + + + Boolean. True if the signature is valid, False otherwise. + """ + + # Does 'public_key' have the correct format? + # This check will ensure 'public_key' conforms to + # 'tuf.formats.ED25519PUBLIC_SCHEMA', which must have length 32 bytes. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.ED25519PUBLIC_SCHEMA.check_match(public_key) + + # Is 'method' properly formatted? + tuf.formats.NAME_SCHEMA.check_match(method) + + # Is 'signature' properly formatted? + tuf.formats.ED25519SIGNATURE_SCHEMA.check_match(signature) + + # Is 'use_pynacl' properly formatted? + tuf.formats.BOOLEAN_SCHEMA.check_match(use_pynacl) + + # Verify 'signature'. Before returning the Boolean result, + # ensure 'ed25519' was used as the signing method. + # Raise 'tuf.UnsupportedLibraryError' if 'use_pynacl' is True but 'nacl' is + # unavailable. + public = public_key + valid_signature = False + + if method in _SUPPORTED_ED25519_SIGNING_METHODS: + if use_pynacl: + try: + nacl_verify_key = nacl.signing.VerifyKey(public) + nacl_message = nacl_verify_key.verify(data, signature) + if nacl_message == data: + valid_signature = True + except NameError: + message = 'The PyNaCl library and/or its dependencies unavailable.' + raise tuf.UnsupportedLibraryError(message) + except nacl.signing.BadSignatureError: + pass + + # Verify 'ed25519' signature with pure Python implementation. + else: + try: + tuf._vendor.ed25519.ed25519.checkvalid(signature, data, public) + valid_signature = True + + # The pure Python implementation raises 'Exception' if 'signature' is + # invalid. + except Exception, e: + pass + else: + message = 'Unsupported ed25519 signing method: '+repr(method)+'.\n'+ \ + 'Supported methods: '+repr(_SUPPORTED_ED25519_SIGNING_METHODS)+'.' + raise tuf.UnknownMethodError(message) + + return valid_signature + + + +if __name__ == '__main__': + # The interactive sessions of the documentation strings can + # be tested by running 'ed25519_keys.py' as a standalone module. + # python -B ed25519_keys.py + import doctest + doctest.testmod() diff --git a/tuf/evp.py b/tuf/evp.py new file mode 100755 index 0000000000..60895bbd89 --- /dev/null +++ b/tuf/evp.py @@ -0,0 +1,425 @@ +""" + + evp.py + + + Vladimir Diaz + + + October 2013. + + + See LICENSE for licensing information. + + + The goal of this module is to support public-key cryptography using + the RSA algorithm. The RSA-related functions provided include + generate(), create_signature(), and verify_signature(). The 'evpy' package + used by 'rsa_key.py' generates the actual RSA keys and the functions listed + above can be viewed as an easy-to-use public interface. Additional functions + contained here include create_in_metadata_format() and + create_from_metadata_format(). These last two functions produce or use RSA + keys compatible with the key structures listed in TUF Metadata files. + The generate() function returns a dictionary containing all the information + needed of RSA keys, such as public and private keys, keyIDs, and an iden- + fier. create_signature() and verify_signature() are supplemental functions + used for generating RSA signatures and verifying them. + + Key IDs are used as identifiers for keys (e.g., RSA key). They are the + hexadecimal representation of the hash of key object (specifically, the key + object containing only the public key). See 'rsa_key.py' and the + '_get_keyid()' function to see precisely how keyids are generated. One may + get the keyid of a key object by simply accessing the dictionary's 'keyid' + key (i.e., rsakey['keyid']). + +""" + + +# Required for hexadecimal conversions. +import binascii + +# Needed to generate, sign, and verify RSA keys. +import evpy.signature +import evpy.envelope + +# Digest objects needed to generate hashes. +import tuf.hash + +# Perform object format-checking. +import tuf.formats + + +_KEY_ID_HASH_ALGORITHM = 'sha256' + +# Recommended RSA key sizes: http://www.rsa.com/rsalabs/node.asp?id=2004 +# According to the document above, revised May 6, 2003, RSA keys of +# size 3072 provide security through 2031 and beyond. +_DEFAULT_RSA_KEY_BITS = 3072 + + +def generate(bits=_DEFAULT_RSA_KEY_BITS): + """ + + Generate public and private RSA keys, with modulus length 'bits'. + In addition, a keyid used as an identifier for RSA keys is generated. + The object returned conforms to tuf.formats.RSAKEY_SCHEMA and as the form: + {'keytype': 'rsa', + 'keyid': keyid, + 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', + 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} + + The public and private keys are in PEM format and stored as strings. + + + bits: + The key size, or key length, of the RSA key. + + + tuf.CryptoError, if an exception occurs after calling evpy.envelope.keygen(). + + tuf.FormatError, if 'bits' does not contain the correct format. + + + The RSA keys are generated by calling evpy.envelope.keygen(). + + + A dictionary containing the RSA keys and other identifying information. + + """ + + + # Does 'bits' have the correct format? + # This check will ensure 'bits' conforms to 'tuf.formats.RSAKEYBITS_SCHEMA'. + # 'bits' must be an integer object, with a minimum value of 2048. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.RSAKEYBITS_SCHEMA.check_match(bits) + + # Begin building the RSA key dictionary. + rsakey_dict = {} + keytype = 'rsa' + + # Generate the public and private keys. 'public_key' and 'private_key' + # will both be strings containing RSA keys in PEM format. + # The evpy.envelope module performs the actual key generation. The + # evpy.envelope.keygen() function returns a (public, private) tuple. + + try: + public_key, private_key = evpy.envelope.keygen(bits, pem=True) + except (evpy.envelope.EnvelopeError, evpy.envelope.KeygenError, MemoryError), e: + raise tuf.CryptoError(e) + + # Generate the keyid for the RSA key. 'key_value' corresponds to the + # 'keyval' entry of the RSAKEY_SCHEMA dictionary. + key_value = {'public': public_key, + 'private': ''} + + keyid = _get_keyid(key_value) + + # Build the 'rsakey_dict' dictionary. + # Update 'key_value' with the RSA private key prior to adding + # 'key_value' to 'rsakey_dict'. + key_value['private'] = private_key + + rsakey_dict['keytype'] = keytype + rsakey_dict['keyid'] = keyid + rsakey_dict['keyval'] = key_value + + return rsakey_dict + + + + + +def create_in_metadata_format(key_value, private=False): + """ + + Return a dictionary conformant to tuf.formats.KEY_SCHEMA. + If 'private' is True, include the private key. The dictionary + returned has the form: + {'keytype': 'rsa', + 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', + 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} + or + + {'keytype': 'rsa', + 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', + 'private': ''}} if 'private' is False. + + The private and public keys are in PEM format. + + RSA keys are stored in Metadata files (e.g., root.txt) in the format + returned by this function. + + + key_value: + A dictionary containing a private and public RSA key. + 'key_value' is of the form: + + {'public': '-----BEGIN RSA PUBLIC KEY----- ...', + 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}}, + conformat to tuf.formats.KEYVAL_SCHEMA. + + private: + Indicates if the private key should be included in the + returned dictionary. + + + tuf.FormatError, if 'key_value' does not conform to + tuf.formats.KEYVAL_SCHEMA. + + + None. + + + An KEY_SCHEMA dictionary. + + """ + + + # Does 'key_value' have the correct format? + # This check will ensure 'key_value' has the appropriate number + # of objects and object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.KEYVAL_SCHEMA.check_match(key_value) + + if private and key_value['private']: + return {'keytype': 'rsa', 'keyval': key_value} + else: + public_key_value = {'public': key_value['public'], 'private': ''} + return {'keytype': 'rsa', 'keyval': public_key_value} + + + + + +def create_from_metadata_format(key_metadata): + """ + + Construct an RSA key dictionary (i.e., tuf.formats.RSAKEY_SCHEMA) + from 'key_metadata'. The dict returned by this function has the exact + format as the dict returned by generate(). It is of the form: + + {'keytype': 'rsa', + 'keyid': keyid, + 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', + 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} + + The public and private keys are in PEM format and stored as strings. + + RSA key dictionaries in RSAKEY_SCHEMA format should be used by + modules storing a collection of keys, such as a keydb and keystore. + RSA keys as stored in metadata files use a different format, so this + function should be called if an RSA key is extracted from one of these + metadata files and needs converting. Generate() creates an entirely + new key and returns it in the format appropriate for keydb and keystore. + + + key_metadata: + The RSA key dictionary as stored in Metadata files, conforming to + tuf.formats.KEY_SCHEMA. + + + tuf.FormatError, if 'key_metadata' does not conform to + tuf.formats.KEY_SCHEMA. + + + None. + + + A dictionary containing the RSA keys and other identifying information. + + """ + + + # Does 'key_metadata' have the correct format? + # This check will ensure 'key_metadata' has the appropriate number + # of objects and object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.KEY_SCHEMA.check_match(key_metadata) + + # Construct the dictionary to be returned. + rsakey_dict = {} + keytype = 'rsa' + key_value = key_metadata['keyval'] + + keyid = _get_keyid(key_value) + + # We now have all the required key values. + # Build 'rsakey_dict'. + rsakey_dict['keytype'] = keytype + rsakey_dict['keyid'] = keyid + rsakey_dict['keyval'] = key_value + + return rsakey_dict + + + + + +def _get_keyid(key_value): + """Return the keyid for 'key_value'.""" + + # 'keyid' will be generated from an object conformant to KEY_SCHEMA, + # which is the format Metadata files (e.g., root.txt) store keys. + # 'create_in_metadata_format()' returns the object needed by _get_keyid(). + rsakey_meta = create_in_metadata_format(key_value, private=False) + + # Convert the RSA key to JSON Canonical format suitable for adding + # to digest objects. + rsakey_update_data = tuf.formats.encode_canonical(rsakey_meta) + + # Create a digest object and call update(), using the JSON + # canonical format of 'rskey_meta' as the update data. + digest_object = tuf.hash.digest(_KEY_ID_HASH_ALGORITHM) + digest_object.update(rsakey_update_data) + + # 'keyid' becomes the hexadecimal representation of the hash. + keyid = digest_object.hexdigest() + + return keyid + + + + + +def create_signature(rsakey_dict, data): + """ + + Return a signature dictionary of the form: + {'keyid': keyid, 'method': 'evp', 'sig': sig}. + + The signing process will use the private key + rsakey_dict['keyval']['private'] and 'data' to generate the signature. + + + rsakey_dict: + A dictionary containing the RSA keys and other identifying information. + 'rsakey_dict' has the form: + + {'keytype': 'rsa', + 'keyid': keyid, + 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', + 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} + + The public and private keys are in PEM format and stored as strings. + + data: + Data object used by create_signature() to generate the signature. + + + TypeError, if a private key is not defined for 'rsakey_dict'. + + tuf.FormatError, if an incorrect format is found for the + 'rsakey_dict' object. + + + evpy.signature.sign() called to perform the actual signing. + + + A signature dictionary conformat to tuf.format.SIGNATURE_SCHEMA. + + """ + + + # Does 'rsakey_dict' have the correct format? + # This check will ensure 'rsakey_dict' has the appropriate number + # of objects and object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.RSAKEY_SCHEMA.check_match(rsakey_dict) + + # Signing the 'data' object requires a private key. + # The 'evp' (i.e., evpy) signing method is the only method + # currently supported. + signature = {} + private_key = rsakey_dict['keyval']['private'] + keyid = rsakey_dict['keyid'] + method = 'evp' + + if private_key: + sig = evpy.signature.sign(data, key=private_key) + else: + raise TypeError('The required private key is not defined for rsakey_dict.') + + # Build the signature dictionary to be returned. + # The hexadecimal representation of 'sig' is stored in the signature. + signature['keyid'] = keyid + signature['method'] = method + signature['sig'] = binascii.hexlify(sig) + + return signature + + + + + +def verify_signature(rsakey_dict, signature, data): + """ + + Determine whether the private key belonging to 'rsakey_dict' produced + 'signature'. verify_signature() will use the public key found in + 'rsakey_dict', the 'method' and 'sig' objects contained in 'signature', + and 'data' to complete the verification. Type-checking performed on both + 'rsakey_dict' and 'signature'. + + + rsakey_dict: + A dictionary containing the RSA keys and other identifying information. + 'rsakey_dict' has the form: + + {'keytype': 'rsa', + 'keyid': keyid, + 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', + 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} + + The public and private keys are in PEM format and stored as strings. + + signature: + The signature dictionary produced by tuf.rsa_key.create_signature(). + 'signature' has the form: + {'keyid': keyid, 'method': 'method', 'sig': sig}. Conformant to + tuf.formats.SIGNATURE_SCHEMA. + + data: + Data object used by tuf.rsa_key.create_signature() to generate + 'signature'. 'data' is needed here to verify the signature. + + + tuf.UnknownMethodError. Raised if the signing method used by + 'signature' is not one supported by tuf.rsa_key.create_signature(). + + tuf.FormatError. Raised if either 'rsakey_dict' + or 'signature' do not match their respective tuf.formats schema. + 'rsakey_dict' must conform to tuf.formats.RSAKEY_SCHEMA. + 'signature' must conform to tuf.formats.SIGNATURE_SCHEMA. + + + evpy.signature_verify() called to do the actual verification. + + + Boolean. + + """ + + + # Does 'rsakey_dict' have the correct format? + # This check will ensure 'rsakey_dict' has the appropriate number + # of objects and object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.RSAKEY_SCHEMA.check_match(rsakey_dict) + + # Does 'signature' have the correct format? + tuf.formats.SIGNATURE_SCHEMA.check_match(signature) + + # Using the public key belonging to 'rsakey_dict' + # (i.e., rsakey_dict['keyval']['public']), verify whether 'signature' + # was produced by rsakey_dict's corresponding private key + # rsakey_dict['keyval']['private']. Before returning the Boolean result, + # ensure 'evp' was used as the signing method. + + method = signature['method'] + sig = signature['sig'] + public_key = rsakey_dict['keyval']['public'] + + if method != 'evp': + raise tuf.UnknownMethodError(method) + return evpy.signature.verify(data, binascii.unhexlify(sig), key=public_key) diff --git a/tuf/formats.py b/tuf/formats.py index 85fc37c2f9..bf01392814 100755 --- a/tuf/formats.py +++ b/tuf/formats.py @@ -57,10 +57,8 @@ Example: signable_object = make_signable(unsigned_object) - """ - import binascii import calendar import re @@ -76,15 +74,19 @@ # easily backwards compatible with clients that are already deployed. # A date in 'YYYY-MM-DD HH:MM:SS UTC' format. +# TODO: Support timestamps according to the ISO 8601 standard. TIME_SCHEMA = SCHEMA.RegularExpression(r'\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} UTC') +# A date in 'YYYY-MM-DD HH:MM:SS UTC' format. +DATETIME_SCHEMA = SCHEMA.RegularExpression(r'\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}') + # A hexadecimal value in '23432df87ab..' format. HASH_SCHEMA = SCHEMA.RegularExpression(r'[a-fA-F0-9]+') # A dict in {'sha256': '23432df87ab..', 'sha512': '34324abc34df..', ...} format. HASHDICT_SCHEMA = SCHEMA.DictOf( - key_schema=SCHEMA.AnyString(), - value_schema=HASH_SCHEMA) + key_schema = SCHEMA.AnyString(), + value_schema = HASH_SCHEMA) # A hexadecimal value in '23432df87ab..' format. HEX_SCHEMA = SCHEMA.RegularExpression(r'[a-fA-F0-9]+') @@ -93,7 +95,7 @@ KEYID_SCHEMA = HASH_SCHEMA KEYIDS_SCHEMA = SCHEMA.ListOf(KEYID_SCHEMA) -# The method used for a generated signature (e.g., 'evp'). +# The method used for a generated signature (e.g., 'RSASSA-PSS'). SIG_METHOD_SCHEMA = SCHEMA.AnyString() # A relative file path (e.g., 'metadata/root/'). @@ -109,14 +111,14 @@ # A dictionary holding version information. VERSION_SCHEMA = SCHEMA.Object( - object_name='version', - major=SCHEMA.Integer(lo=0), - minor=SCHEMA.Integer(lo=0), - fix=SCHEMA.Integer(lo=0)) + object_name = 'VERSION_SCHEMA', + major = SCHEMA.Integer(lo=0), + minor = SCHEMA.Integer(lo=0), + fix = SCHEMA.Integer(lo=0)) # An integer representing the numbered version of a metadata file. # Must be 1, or greater. -METADATAVERSION_SCHEMA = SCHEMA.Integer(lo=1) +METADATAVERSION_SCHEMA = SCHEMA.Integer(lo=0) # An integer representing length. Must be 0, or greater. LENGTH_SCHEMA = SCHEMA.Integer(lo=0) @@ -127,9 +129,20 @@ # A string representing a named object. NAME_SCHEMA = SCHEMA.AnyString() +NAMES_SCHEMA = SCHEMA.ListOf(NAME_SCHEMA) + +# Supported hash algorithms. +HASHALGORITHMS_SCHEMA = SCHEMA.ListOf(SCHEMA.OneOf( + [SCHEMA.String('md5'), SCHEMA.String('sha1'), + SCHEMA.String('sha224'), SCHEMA.String('sha256'), + SCHEMA.String('sha384'), SCHEMA.String('sha512')])) + +# The contents of an encrypted TUF key. Encrypted TUF keys are saved to files +# in this format. +ENCRYPTEDKEY_SCHEMA = SCHEMA.AnyString() # A value that is either True or False, on or off, etc. -TOGGLE_SCHEMA = SCHEMA.Boolean() +BOOLEAN_SCHEMA = SCHEMA.Boolean() # A role's threshold value (i.e., the minimum number # of signatures required to sign a metadata file). @@ -139,9 +152,17 @@ # A string representing a role's name. ROLENAME_SCHEMA = SCHEMA.AnyString() -# The minimum number of bits for an RSA key. Must be 2048 bits and greater. +# The minimum number of bits for an RSA key recommended by TUF. Must be 2048 +# bits, or greater. Recommended RSA key sizes: +# http://www.emc.com/emc-plus/rsa-labs/historical/twirl-and-rsa-key-size.htm#table1 RSAKEYBITS_SCHEMA = SCHEMA.Integer(lo=2048) +# The number of bins used to delegate to hashed roles. +NUMBINS_SCHEMA = SCHEMA.Integer(lo=16) + +# A PyCrypto signature. +PYCRYPTOSIGNATURE_SCHEMA = SCHEMA.AnyString() + # An RSA key in PEM format. PEMRSA_SCHEMA = SCHEMA.AnyString() @@ -155,50 +176,76 @@ # key identifier ('rsa', 233df889cb). For RSA keys, the key value is a pair of # public and private keys in PEM Format stored as strings. KEYVAL_SCHEMA = SCHEMA.Object( - object_name='keyval', - public=SCHEMA.AnyString(), - private=SCHEMA.AnyString()) + object_name = 'KEYVAL_SCHEMA', + public = SCHEMA.AnyString(), + private = SCHEMA.Optional(SCHEMA.AnyString())) -# A generic key. All TUF keys should be saved to metadata files in this format. -KEY_SCHEMA = SCHEMA.Object( - object_name='key', - keytype=SCHEMA.AnyString(), - keyval=KEYVAL_SCHEMA) +# Supported TUF key types. +KEYTYPE_SCHEMA = SCHEMA.OneOf( + [SCHEMA.String('rsa'), SCHEMA.String('ed25519')]) -# An RSA key. +# A generic TUF key. All TUF keys should be saved to metadata files in this +# format. +KEY_SCHEMA = SCHEMA.Object( + object_name = 'KEY_SCHEMA', + keytype = SCHEMA.AnyString(), + keyval = KEYVAL_SCHEMA) + +# A TUF key object. This schema simplifies validation of keys that may be +# one of the supported key types. +# Supported key types: 'rsa', 'ed25519'. +ANYKEY_SCHEMA = SCHEMA.Object( + object_name = 'ANYKEY_SCHEMA', + keytype = KEYTYPE_SCHEMA, + keyid = KEYID_SCHEMA, + keyval = KEYVAL_SCHEMA) + +# A list of TUF key objects. +ANYKEYLIST_SCHEMA = SCHEMA.ListOf(ANYKEY_SCHEMA) + +# An RSA TUF key. RSAKEY_SCHEMA = SCHEMA.Object( - object_name='rsakey', - keytype=SCHEMA.String('rsa'), - keyid=KEYID_SCHEMA, - keyval=KEYVAL_SCHEMA) + object_name = 'RSAKEY_SCHEMA', + keytype = SCHEMA.String('rsa'), + keyid = KEYID_SCHEMA, + keyval = KEYVAL_SCHEMA) + +# An ED25519 raw public key, which must be 32 bytes. +ED25519PUBLIC_SCHEMA = SCHEMA.LengthString(32) -# An ed25519 key. +# An ED25519 raw seed key, which must be 32 bytes. +ED25519SEED_SCHEMA = SCHEMA.LengthString(32) + +# An ED25519 raw signature, which must be 64 bytes. +ED25519SIGNATURE_SCHEMA = SCHEMA.LengthString(64) + +# An ed25519 TUF key. ED25519KEY_SCHEMA = SCHEMA.Object( - object_name='ed25519key', - keytype=SCHEMA.String('ed25519'), - keyid=KEYID_SCHEMA, - keyval=KEYVAL_SCHEMA) + object_name = 'ED25519KEY_SCHEMA', + keytype = SCHEMA.String('ed25519'), + keyid = KEYID_SCHEMA, + keyval = KEYVAL_SCHEMA) # Info that describes both metadata and target files. # This schema allows the storage of multiple hashes for the same file # (e.g., sha256 and sha512 may be computed for the same file and stored). FILEINFO_SCHEMA = SCHEMA.Object( - object_name='fileinfo', - length=LENGTH_SCHEMA, - hashes=HASHDICT_SCHEMA, - custom=SCHEMA.Optional(SCHEMA.Object())) + object_name = 'FILEINFO_SCHEMA', + length = LENGTH_SCHEMA, + hashes = HASHDICT_SCHEMA, + custom = SCHEMA.Optional(SCHEMA.Object())) # A dict holding the information for a particular file. The keys hold the # relative file path and the values the relevant file information. FILEDICT_SCHEMA = SCHEMA.DictOf( - key_schema=RELPATH_SCHEMA, - value_schema=FILEINFO_SCHEMA) + key_schema = RELPATH_SCHEMA, + value_schema = FILEINFO_SCHEMA) # A dict holding a target file. TARGETFILE_SCHEMA = SCHEMA.Object( - object_name='targetfile', - filepath=RELPATH_SCHEMA, - fileinfo=FILEINFO_SCHEMA) + object_name = 'TARGETFILE_SCHEMA', + filepath = RELPATH_SCHEMA, + fileinfo = FILEINFO_SCHEMA) TARGETFILES_SCHEMA = SCHEMA.ListOf(TARGETFILE_SCHEMA) # A single signature of an object. Indicates the signature, the id of the @@ -210,10 +257,13 @@ # one can imagine that maybe a key wants to sign multiple times with different # signature methods. SIGNATURE_SCHEMA = SCHEMA.Object( - object_name='signature', - keyid=KEYID_SCHEMA, - method=SIG_METHOD_SCHEMA, - sig=HEX_SCHEMA) + object_name = 'SIGNATURE_SCHEMA', + keyid = KEYID_SCHEMA, + method = SIG_METHOD_SCHEMA, + sig = HEX_SCHEMA) + +# List of SIGNATURE_SCHEMA. +SIGNATURES_SCHEMA = SCHEMA.ListOf(SIGNATURE_SCHEMA) # A schema holding the result of checking the signatures of a particular # 'SIGNABLE_SCHEMA' role. @@ -221,31 +271,31 @@ # valid? This SCHEMA holds this information. See 'sig.py' for # more information. SIGNATURESTATUS_SCHEMA = SCHEMA.Object( - object_name='signaturestatus', - threshold=SCHEMA.Integer(), - good_sigs=SCHEMA.ListOf(KEYID_SCHEMA), - bad_sigs=SCHEMA.ListOf(KEYID_SCHEMA), - unknown_sigs=SCHEMA.ListOf(KEYID_SCHEMA), - untrusted_sigs=SCHEMA.ListOf(KEYID_SCHEMA), - unknown_method_sigs=SCHEMA.ListOf(KEYID_SCHEMA)) + object_name = 'SIGNATURESTATUS_SCHEMA', + threshold = SCHEMA.Integer(), + good_sigs = KEYIDS_SCHEMA, + bad_sigs = KEYIDS_SCHEMA, + unknown_sigs = KEYIDS_SCHEMA, + untrusted_sigs = KEYIDS_SCHEMA, + unknown_method_sigs = KEYIDS_SCHEMA) # A signable object. Holds the signing role and its associated signatures. SIGNABLE_SCHEMA = SCHEMA.Object( - object_name='signable', - signed=SCHEMA.Any(), - signatures=SCHEMA.ListOf(SIGNATURE_SCHEMA)) + object_name = 'SIGNABLE_SCHEMA', + signed = SCHEMA.Any(), + signatures = SCHEMA.ListOf(SIGNATURE_SCHEMA)) # A dict where the dict keys hold a keyid and the dict values a key object. KEYDICT_SCHEMA = SCHEMA.DictOf( - key_schema=KEYID_SCHEMA, - value_schema=KEY_SCHEMA) + key_schema = KEYID_SCHEMA, + value_schema = KEY_SCHEMA) # The format used by the key database to store keys. The dict keys hold a key # identifier and the dict values any object. The key database should store # key objects in the values (e.g., 'RSAKEY_SCHEMA', 'DSAKEY_SCHEMA'). KEYDB_SCHEMA = SCHEMA.DictOf( - key_schema=KEYID_SCHEMA, - value_schema=SCHEMA.Any()) + key_schema = KEYID_SCHEMA, + value_schema = SCHEMA.Any()) # The format of the resulting "scp config dict" after extraction from the # push configuration file (i.e., push.cfg). In the case of a config file @@ -255,18 +305,18 @@ # 'remote_directory' entries. See 'tuf/pushtools/pushtoolslib.py' and # 'tuf/pushtools/push.py'. SCPCONFIG_SCHEMA = SCHEMA.Object( - object_name='scp_config', - general=SCHEMA.Object( - object_name='[general]', - transfer_module=SCHEMA.String('scp'), - metadata_path=PATH_SCHEMA, - targets_directory=PATH_SCHEMA), + object_name = 'SCPCONFIG_SCHEMA', + general = SCHEMA.Object( + object_name = '[general]', + transfer_module = SCHEMA.String('scp'), + metadata_path = PATH_SCHEMA, + targets_directory = PATH_SCHEMA), scp=SCHEMA.Object( - object_name='[scp]', - host=URL_SCHEMA, - user=NAME_SCHEMA, - identity_file=PATH_SCHEMA, - remote_directory=PATH_SCHEMA)) + object_name = '[scp]', + host = URL_SCHEMA, + user = NAME_SCHEMA, + identity_file = PATH_SCHEMA, + remote_directory = PATH_SCHEMA)) # The format of the resulting "receive config dict" after extraction from the # receive configuration file (i.e., receive.cfg). The receive config file @@ -275,101 +325,136 @@ # 'backup_directory' entries. # see 'tuf/pushtools/pushtoolslib.py' and 'tuf/pushtools/receive/receive.py' RECEIVECONFIG_SCHEMA = SCHEMA.Object( - object_name='receive_config', - general=SCHEMA.Object( - object_name='[general]', - pushroots=SCHEMA.ListOf(PATH_SCHEMA), - repository_directory=PATH_SCHEMA, - metadata_directory=PATH_SCHEMA, - targets_directory=PATH_SCHEMA, - backup_directory=PATH_SCHEMA)) + object_name = 'RECEIVECONFIG_SCHEMA', general=SCHEMA.Object( + object_name = '[general]', + pushroots = SCHEMA.ListOf(PATH_SCHEMA), + repository_directory = PATH_SCHEMA, + metadata_directory = PATH_SCHEMA, + targets_directory = PATH_SCHEMA, + backup_directory = PATH_SCHEMA)) # A path hash prefix is a hexadecimal string. PATH_HASH_PREFIX_SCHEMA = HEX_SCHEMA + # A list of path hash prefixes. PATH_HASH_PREFIXES_SCHEMA = SCHEMA.ListOf(PATH_HASH_PREFIX_SCHEMA) # Role object in {'keyids': [keydids..], 'name': 'ABC', 'threshold': 1, -# 'paths':[filepaths..]} # format. +# 'paths':[filepaths..]} format. ROLE_SCHEMA = SCHEMA.Object( - object_name='role', - keyids=SCHEMA.ListOf(KEYID_SCHEMA), - name=SCHEMA.Optional(ROLENAME_SCHEMA), - threshold=THRESHOLD_SCHEMA, - paths=SCHEMA.Optional(RELPATHS_SCHEMA), - path_hash_prefixes=SCHEMA.Optional(PATH_HASH_PREFIXES_SCHEMA)) + object_name = 'ROLE_SCHEMA', + name = SCHEMA.Optional(ROLENAME_SCHEMA), + keyids = KEYIDS_SCHEMA, + threshold = THRESHOLD_SCHEMA, + paths = SCHEMA.Optional(RELPATHS_SCHEMA), + path_hash_prefixes = SCHEMA.Optional(PATH_HASH_PREFIXES_SCHEMA)) # A dict of roles where the dict keys are role names and the dict values holding # the role data/information. ROLEDICT_SCHEMA = SCHEMA.DictOf( - key_schema=ROLENAME_SCHEMA, - value_schema=ROLE_SCHEMA) + key_schema = ROLENAME_SCHEMA, + value_schema = ROLE_SCHEMA) # Like ROLEDICT_SCHEMA, except that ROLE_SCHEMA instances are stored in order. ROLELIST_SCHEMA = SCHEMA.ListOf(ROLE_SCHEMA) -# The root: indicates root keys and top-level roles. +# The delegated roles of a Targets role (a parent). +DELEGATIONS_SCHEMA = SCHEMA.Object( + keys = KEYDICT_SCHEMA, + roles = ROLELIST_SCHEMA) + +# The number of seconds before metadata expires. The minimum is 86400 seconds +# (= 1 day). This schema is used for the initial expiration date. Repository +# maintainers may later modify this value (TIME_SCHEMA). +EXPIRATION_SCHEMA = SCHEMA.Integer(lo=86400) + +# Supported compression extension (e.g., 'gz'). +COMPRESSION_SCHEMA = SCHEMA.OneOf([SCHEMA.String(''), SCHEMA.String('gz')]) + +# List of supported compression extensions. +COMPRESSIONS_SCHEMA = SCHEMA.ListOf( + SCHEMA.OneOf([SCHEMA.String(''), SCHEMA.String('gz')])) + +# tuf.roledb +ROLEDB_SCHEMA = SCHEMA.Object( + object_name = 'ROLEDB_SCHEMA', + keyids = KEYIDS_SCHEMA, + signing_keyids = SCHEMA.Optional(KEYIDS_SCHEMA), + threshold = THRESHOLD_SCHEMA, + version = SCHEMA.Optional(METADATAVERSION_SCHEMA), + expires = SCHEMA.Optional(SCHEMA.OneOf([EXPIRATION_SCHEMA, TIME_SCHEMA])), + signatures = SCHEMA.Optional(SIGNATURES_SCHEMA), + compressions = SCHEMA.Optional(COMPRESSIONS_SCHEMA), + paths = SCHEMA.Optional(RELPATHS_SCHEMA), + path_hash_prefixes = SCHEMA.Optional(PATH_HASH_PREFIXES_SCHEMA), + delegations = SCHEMA.Optional(DELEGATIONS_SCHEMA), + partial_loaded = SCHEMA.Optional(BOOLEAN_SCHEMA)) + +# Root role: indicates root keys and top-level roles. ROOT_SCHEMA = SCHEMA.Object( - object_name='root', - _type=SCHEMA.String('Root'), - version=METADATAVERSION_SCHEMA, - expires=TIME_SCHEMA, - keys=KEYDICT_SCHEMA, - roles=ROLEDICT_SCHEMA) - -# Targets. Indicates targets and delegates target paths to other roles. + object_name = 'ROOT_SCHEMA', + _type = SCHEMA.String('Root'), + version = METADATAVERSION_SCHEMA, + consistent_snapshot = BOOLEAN_SCHEMA, + expires = TIME_SCHEMA, + keys = KEYDICT_SCHEMA, + roles = ROLEDICT_SCHEMA) + +# Targets role: Indicates targets and delegates target paths to other roles. TARGETS_SCHEMA = SCHEMA.Object( - object_name='targets', - _type=SCHEMA.String('Targets'), - version=METADATAVERSION_SCHEMA, - expires=TIME_SCHEMA, - targets=FILEDICT_SCHEMA, - delegations=SCHEMA.Optional(SCHEMA.Object( - keys=KEYDICT_SCHEMA, - roles=ROLELIST_SCHEMA))) - -# A Release: indicates the latest versions of all metadata (except timestamp). -RELEASE_SCHEMA = SCHEMA.Object( - object_name='release', - _type=SCHEMA.String('Release'), - version=METADATAVERSION_SCHEMA, - expires=TIME_SCHEMA, - meta=FILEDICT_SCHEMA) - -# A Timestamp: indicates the latest version of the release file. + object_name = 'TARGETS_SCHEMA', + _type = SCHEMA.String('Targets'), + version = METADATAVERSION_SCHEMA, + expires = TIME_SCHEMA, + targets = FILEDICT_SCHEMA, + delegations = SCHEMA.Optional(DELEGATIONS_SCHEMA)) + +# Snapshot role: indicates the latest versions of all metadata (except timestamp). +SNAPSHOT_SCHEMA = SCHEMA.Object( + object_name = 'SNAPSHOT_SCHEMA', + _type = SCHEMA.String('Snapshot'), + version = METADATAVERSION_SCHEMA, + expires = TIME_SCHEMA, + meta = FILEDICT_SCHEMA) + +# Timestamp role: indicates the latest version of the snapshot file. TIMESTAMP_SCHEMA = SCHEMA.Object( - object_name='timestamp', - _type=SCHEMA.String('Timestamp'), - version=METADATAVERSION_SCHEMA, - expires=TIME_SCHEMA, - meta=FILEDICT_SCHEMA) + object_name = 'TIMESTAMP_SCHEMA', + _type = SCHEMA.String('Timestamp'), + version = METADATAVERSION_SCHEMA, + expires = TIME_SCHEMA, + meta = FILEDICT_SCHEMA) # A schema containing information a repository mirror may require, # such as a url, the path of the directory metadata files, etc. MIRROR_SCHEMA = SCHEMA.Object( - object_name='mirror', - url_prefix=URL_SCHEMA, - metadata_path=RELPATH_SCHEMA, - targets_path=RELPATH_SCHEMA, - confined_target_dirs=RELPATHS_SCHEMA, - custom=SCHEMA.Optional(SCHEMA.Object())) + object_name = 'MIRROR_SCHEMA', + url_prefix = URL_SCHEMA, + metadata_path = RELPATH_SCHEMA, + targets_path = RELPATH_SCHEMA, + confined_target_dirs = RELPATHS_SCHEMA, + custom = SCHEMA.Optional(SCHEMA.Object())) # A dictionary of mirrors where the dict keys hold the mirror's name and # and the dict values the mirror's data (i.e., 'MIRROR_SCHEMA'). # The repository class of 'updater.py' accepts dictionaries # of this type provided by the TUF client. MIRRORDICT_SCHEMA = SCHEMA.DictOf( - key_schema=SCHEMA.AnyString(), - value_schema=MIRROR_SCHEMA) + key_schema = SCHEMA.AnyString(), + value_schema = MIRROR_SCHEMA) # A Mirrorlist: indicates all the live mirrors, and what documents they # serve. MIRRORLIST_SCHEMA = SCHEMA.Object( - object_name='mirrorlist', - _type=SCHEMA.String('Mirrors'), - version=METADATAVERSION_SCHEMA, - expires=TIME_SCHEMA, - mirrors=SCHEMA.ListOf(MIRROR_SCHEMA)) + object_name = 'MIRRORLIST_SCHEMA', + _type = SCHEMA.String('Mirrors'), + version = METADATAVERSION_SCHEMA, + expires = TIME_SCHEMA, + mirrors = SCHEMA.ListOf(MIRROR_SCHEMA)) + +# Any of the role schemas (e.g., TIMESTAMP_SCHEMA, SNAPSHOT_SCHEMA, etc.) +ANYROLE_SCHEMA = SCHEMA.OneOf([ROOT_SCHEMA, TARGETS_SCHEMA, SNAPSHOT_SCHEMA, + TIMESTAMP_SCHEMA, MIRROR_SCHEMA]) @@ -380,10 +465,9 @@ class MetaFile(object): Base class for all metadata file classes. Classes representing metadata files such as RootFile - and ReleaseFile all inherit from MetaFile. The + and SnapshotFile all inherit from MetaFile. The __eq__, __ne__, perform 'equal' and 'not equal' comparisons between Metadata File objects. - """ info = None @@ -401,7 +485,6 @@ def __getattr__(self, name): Allow all metafile objects to have their interesting attributes referred to directly without the info dict. The info dict is just to be able to do the __eq__ comparison generically. - """ if name in self.info: @@ -450,12 +533,13 @@ def make_metadata(version, expiration_date, filedict): class RootFile(MetaFile): - def __init__(self, version, expires, keys, roles): + def __init__(self, version, expires, keys, roles, consistent_snapshot): self.info = {} self.info['version'] = version self.info['expires'] = expires self.info['keys'] = keys self.info['roles'] = roles + self.info['consistent_snapshot'] = consistent_snapshot @staticmethod @@ -468,21 +552,20 @@ def from_metadata(object): expires = parse_time(object['expires']) keys = object['keys'] roles = object['roles'] + consistent_snapshot = object['consistent_snapshot'] - return RootFile(version, expires, keys, roles) + return RootFile(version, expires, keys, roles, consistent_snapshot) @staticmethod - def make_metadata(version, expiration_seconds, keydict, roledict): - # Is 'expiration_seconds' properly formatted? - # Raise 'tuf.FormatError' if not. - LENGTH_SCHEMA.check_match(expiration_seconds) - + def make_metadata(version, expiration_date, keydict, roledict, + consistent_snapshot): result = {'_type' : 'Root'} result['version'] = version - result['expires'] = format_time(time.time() + expiration_seconds) + result['expires'] = expiration_date result['keys'] = keydict result['roles'] = roledict + result['consistent_snapshot'] = consistent_snapshot # Is 'result' a Root metadata file? # Raise 'tuf.FormatError' if not. @@ -494,7 +577,7 @@ def make_metadata(version, expiration_seconds, keydict, roledict): -class ReleaseFile(MetaFile): +class SnapshotFile(MetaFile): def __init__(self, version, expires, filedict): self.info = {} self.info['version'] = version @@ -504,27 +587,27 @@ def __init__(self, version, expires, filedict): @staticmethod def from_metadata(object): - # Is 'object' a Release metadata file? + # Is 'object' a Snapshot metadata file? # Raise 'tuf.FormatError' if not. - RELEASE_SCHEMA.check_match(object) + SNAPSHOT_SCHEMA.check_match(object) version = object['version'] expires = parse_time(object['expires']) filedict = object['meta'] - return ReleaseFile(version, expires, filedict) + return SnapshotFile(version, expires, filedict) @staticmethod def make_metadata(version, expiration_date, filedict): - result = {'_type' : 'Release'} + result = {'_type' : 'Snapshot'} result['version'] = version result['expires'] = expiration_date result['meta'] = filedict - # Is 'result' a Release metadata file? + # Is 'result' a Snapshot metadata file? # Raise 'tuf.FormatError' if not. - RELEASE_SCHEMA.check_match(result) + SNAPSHOT_SCHEMA.check_match(result) return result @@ -606,7 +689,7 @@ def make_metadata(): SCHEMAS_BY_TYPE = { 'Root' : ROOT_SCHEMA, 'Targets' : TARGETS_SCHEMA, - 'Release' : RELEASE_SCHEMA, + 'Snapshot' : SNAPSHOT_SCHEMA, 'Timestamp' : TIMESTAMP_SCHEMA, 'Mirrors' : MIRRORLIST_SCHEMA} @@ -615,7 +698,7 @@ def make_metadata(): ROLE_CLASSES_BY_TYPE = { 'Root' : RootFile, 'Targets' : TargetsFile, - 'Release' : ReleaseFile, + 'Snapshot' : SnapshotFile, 'Timestamp' : TimestampFile, 'Mirrors' : MirrorsFile} @@ -645,7 +728,6 @@ def format_time(timestamp): A string in 'YYYY-MM-DD HH:MM:SS UTC' format. - """ try: @@ -654,6 +736,7 @@ def format_time(timestamp): # Attach 'UTC' to the formatted time string prior to returning. return formatted_time+' UTC' + except (ValueError, TypeError): raise tuf.FormatError('Invalid argument value') @@ -677,7 +760,6 @@ def parse_time(string): A timestamp (e.g., 499137660). - """ # Is 'string' properly formatted? @@ -689,6 +771,7 @@ def parse_time(string): string = string[0:string.rfind(' UTC')] try: return calendar.timegm(time.strptime(string, '%Y-%m-%d %H:%M:%S')) + except ValueError: raise tuf.FormatError('Malformed time: '+repr(string)) @@ -715,11 +798,11 @@ def format_base64(data): A base64-encoded string. - """ try: return binascii.b2a_base64(data).rstrip('=\n ') + except (TypeError, binascii.Error), e: raise tuf.FormatError('Invalid base64 encoding: '+str(e)) @@ -746,7 +829,6 @@ def parse_base64(base64_string): A byte string representing the parsed based64 encoding of 'base64_string'. - """ if not isinstance(base64_string, basestring): @@ -760,6 +842,7 @@ def parse_base64(base64_string): try: return binascii.a2b_base64(base64_string) + except (TypeError, binascii.Error), e: raise tuf.FormatError('Invalid base64 encoding: '+str(e)) @@ -781,7 +864,7 @@ def make_signable(object): object: - A role schema dict (e.g., 'ROOT_SCHEMA', 'RELEASE_SCHEMA'). + A role schema dict (e.g., 'ROOT_SCHEMA', 'SNAPSHOT_SCHEMA'). None. @@ -791,7 +874,6 @@ def make_signable(object): A dict in 'SIGNABLE_SCHEMA' format. - """ if not isinstance(object, dict) or 'signed' not in object: @@ -832,7 +914,6 @@ def make_fileinfo(length, hashes, custom=None): A dictionary conformant to 'FILEINFO_SCHEMA', representing the file information of a metadata or target file. - """ fileinfo = {'length' : length, 'hashes' : hashes} @@ -889,7 +970,6 @@ def make_role_metadata(keyids, threshold, name=None, paths=None, A properly formatted role meta dict, conforming to 'ROLE_SCHEMA'. - """ role_meta = {} @@ -948,9 +1028,8 @@ def get_role_class(expected_rolename): The class corresponding to 'expected_rolename'. - E.g., 'Release' as an argument to this function causes - 'ReleaseFile' to be returned. - + E.g., 'Snapshot' as an argument to this function causes + SnapshotFile' to be returned. """ # Does 'expected_rolename' have the correct type? @@ -961,6 +1040,7 @@ def get_role_class(expected_rolename): try: role_class = ROLE_CLASSES_BY_TYPE[expected_rolename] + except KeyError: raise tuf.FormatError(repr(expected_rolename)+' not supported.') else: @@ -993,7 +1073,6 @@ def expected_meta_rolename(meta_rolename): A string (e.g., 'Root', 'Targets'). - """ # Does 'meta_rolename' have the correct type? @@ -1033,7 +1112,6 @@ def check_signable_object_format(object): A string representing the signing role (e.g., 'root', 'targets'). The role string is returned with characters all lower case. - """ # Does 'object' have the correct type? @@ -1043,10 +1121,13 @@ def check_signable_object_format(object): try: role_type = object['signed']['_type'] + except (KeyError, TypeError): raise tuf.FormatError('Untyped object') + try: schema = SCHEMAS_BY_TYPE[role_type] + except KeyError: raise tuf.FormatError('Unrecognized type '+repr(role_type)) @@ -1077,7 +1158,6 @@ def _canonical_string_encoder(string): A string with the canonical-encoded 'string' embedded. - """ string = '"%s"' % re.sub(r'(["\\])', r'\\\1', string) @@ -1182,7 +1262,6 @@ def encode_canonical(object, output_function=None): A string representing the 'object' encoded in canonical JSON form. - """ result = None @@ -1194,6 +1273,7 @@ def encode_canonical(object, output_function=None): try: _encode_canonical(object, output_function) + except TypeError, e: message = 'Could not encode '+repr(object)+': '+str(e) raise tuf.FormatError(message) diff --git a/tuf/interposition/__init__.py b/tuf/interposition/__init__.py index 00c53e7c6f..4360016eca 100644 --- a/tuf/interposition/__init__.py +++ b/tuf/interposition/__init__.py @@ -285,6 +285,24 @@ def configure(filename="tuf.interposition.json", +def refresh(configurations): + """Refresh the top-level metadata for previously read configurations.""" + + # Get the updater and refresh its top-level metadata. In the majority of + # integrations, a software updater integrating TUF with interposition will + # usually only require an initial refresh() (i.e., when configure() is + # called). A series of target file requests may then occur, which are all + # referenced by the latest top-level metadata updated by configure(). + # Although interposition was designed to remain transparent, for software + # updaters that require an explicit refresh of top-level metadata, this + # method is provided. + for configuration in configurations.itervalues(): + __updater_controller.refresh(configuration) + + + + + def deconfigure(configurations): """Remove TUF interposition for previously read configurations.""" diff --git a/tuf/interposition/configuration.py b/tuf/interposition/configuration.py index 295ccac61d..e165a073c0 100644 --- a/tuf/interposition/configuration.py +++ b/tuf/interposition/configuration.py @@ -45,7 +45,7 @@ def __init__(self, hostname, port, repository_directory, repository_mirrors, def __repr__(self): - MESSAGE = "Configuration(netloc={network_location})" + MESSAGE = "network location: {network_location}" return MESSAGE.format(network_location=self.network_location) diff --git a/tuf/interposition/updater.py b/tuf/interposition/updater.py index 8c065479d5..c2d266e273 100644 --- a/tuf/interposition/updater.py +++ b/tuf/interposition/updater.py @@ -50,6 +50,18 @@ def __init__(self, configuration): self.switch_context() self.updater = tuf.client.updater.Updater(self.configuration.hostname, self.configuration.repository_mirrors) + + # Update the client's top-level metadata. The download_target() method does + # not automatically refresh top-level prior to retrieving target files and + # their associated Targets metadata, so update the top-level + # metadata here. + Logger.info('Refreshing top-level metadata for interposed '+repr(configuration)) + self.updater.refresh() + + + def refresh(self): + """Refresh top-level metadata""" + self.updater.refresh() def cleanup(self): @@ -65,12 +77,18 @@ def download_target(self, target_filepath): # download file into a temporary directory shared over runtime destination_directory = self.tempdir - filename = os.path.join(destination_directory, target_filepath) - - self.switch_context() # switch TUF context - self.updater.refresh() # update TUF client repository metadata - - # then, update target at filepath + + # Note: join() discards 'destination_directory' if 'target_filepath' + # contains a leading path separator (i.e., is treated as an absolute path). + filename = os.path.join(destination_directory, target_filepath.lstrip(os.sep)) + + # Switch TUF context. + self.switch_context() + + # Locate the fileinfo of 'target_filepath'. updater.target() searches + # Targets metadata in order of trust, according to the currently trusted + # snapshot. To prevent consecutive target file requests from referring to + # different snapshots, top-level metadata is not automatically refreshed. targets = [self.updater.target(target_filepath)] # TODO: targets are always updated if destination directory is new, right? @@ -123,8 +141,6 @@ def get_target_filepath(self, source_url): raise else: - # TUF assumes that target_filepath does not begin with a '/'. - target_filepath = target_filepath.lstrip('/') return target_filepath @@ -254,15 +270,37 @@ def __check_configuration_on_add(self, configuration): def add(self, configuration): """Add an Updater based on the given Configuration.""" - UPDATER_ADDED_MESSAGE = "Updater added for {configuration}." - repository_mirror_hostnames = self.__check_configuration_on_add(configuration) - + # If all is well, build and store an Updater, and remember hostnames. + Logger.info('Adding updater for interposed '+repr(configuration)) self.__updaters[configuration.hostname] = Updater(configuration) self.__repository_mirror_hostnames.update(repository_mirror_hostnames) + + + + def refresh(self, configuration): + """Refresh the top-level metadata of the given Configuration.""" + + assert isinstance(configuration, Configuration) + + repository_mirror_hostnames = configuration.get_repository_mirror_hostnames() + + assert configuration.hostname in self.__updaters + assert repository_mirror_hostnames.issubset(self.__repository_mirror_hostnames) + + # Get the updater and refresh its top-level metadata. In the majority of + # integrations, a software updater integrating TUF with interposition will + # usually only require an initial refresh() (i.e., when configure() is + # called). A series of target file requests may then occur, which are all + # referenced by the latest top-level metadata updated by configure(). + # Although interposition was designed to remain transparent, for software + # updaters that require an explicit refresh of top-level metadata, this + # method is provided. + Logger.info('Refreshing top-level metadata for '+ repr(configuration)) + updater = self.__updaters.get(configuration.hostname) + updater.refresh() - Logger.info(UPDATER_ADDED_MESSAGE.format(configuration=configuration)) def get(self, url): @@ -273,7 +311,7 @@ def get(self, url): GENERIC_WARNING_MESSAGE = "No updater or interposition for url={url}" DIFFERENT_NETLOC_MESSAGE = "We have an updater for netloc={netloc1} but not for netlocs={netloc2}" - HOSTNAME_FOUND_MESSAGE = "Found updater for hostname={hostname}" + HOSTNAME_FOUND_MESSAGE = "Found updater for interposed network location: {netloc}" HOSTNAME_NOT_FOUND_MESSAGE = "No updater for hostname={hostname}" updater = None @@ -298,7 +336,7 @@ def get(self, url): # Ensure that the updater is meant for this (hostname, port). if updater.configuration.network_location in network_locations: - Logger.info(HOSTNAME_FOUND_MESSAGE.format(hostname=hostname)) + Logger.info(HOSTNAME_FOUND_MESSAGE.format(netloc=network_location)) # Raises an exception in case we do not recognize how to # transform this URL for TUF. In that case, there will be no # updater for this URL. @@ -324,7 +362,7 @@ def get(self, url): def remove(self, configuration): """Remove an Updater matching the given Configuration.""" - UPDATER_REMOVED_MESSAGE = "Updater removed for {configuration}." + UPDATER_REMOVED_MESSAGE = "Updater removed for interposed {configuration}." assert isinstance(configuration, Configuration) diff --git a/tuf/keydb.py b/tuf/keydb.py index 601993b3f3..60cb4d8931 100755 --- a/tuf/keydb.py +++ b/tuf/keydb.py @@ -25,15 +25,17 @@ and the '_get_keyid()' function to learn precisely how keyids are generated. One may get the keyid of a key object by simply accessing the dictionary's 'keyid' key (i.e., rsakey['keyid']). - """ - import logging +import copy import tuf import tuf.formats -import tuf.rsa_key +import tuf.keys + +# List of strings representing the key types supported by TUF. +_SUPPORTED_KEY_TYPES = ['rsa', 'ed25519'] # See 'log.py' to learn how logging is handled in TUF. logger = logging.getLogger('tuf.keydb') @@ -62,13 +64,12 @@ def create_keydb_from_root_metadata(root_metadata): A function to add the key to the database is called. In the case of RSA - keys, this function is add_rsakey(). + keys, this function is add_key(). The old keydb key database is replaced. None. - """ # Does 'root_metadata' have the correct format? @@ -84,13 +85,13 @@ def create_keydb_from_root_metadata(root_metadata): # them to 'RSAKEY_SCHEMA' if their type is 'rsa', and then # adding them the database. Duplicates are avoided. for keyid, key_metadata in root_metadata['keys'].items(): - if key_metadata['keytype'] == 'rsa': + if key_metadata['keytype'] in _SUPPORTED_KEY_TYPES: # 'key_metadata' is stored in 'KEY_SCHEMA' format. Call # create_from_metadata_format() to get the key in 'RSAKEY_SCHEMA' - # format, which is the format expected by 'add_rsakey()'. - rsakey_dict = tuf.rsa_key.create_from_metadata_format(key_metadata) + # format, which is the format expected by 'add_key()'. + key_dict = tuf.keys.format_metadata_to_key(key_metadata) try: - add_rsakey(rsakey_dict, keyid) + add_key(key_dict, keyid) # 'tuf.Error' raised if keyid does not match the keyid for 'rsakey_dict'. except tuf.Error, e: logger.error(e) @@ -105,7 +106,7 @@ def create_keydb_from_root_metadata(root_metadata): -def add_rsakey(rsakey_dict, keyid=None): +def add_key(key_dict, keyid=None): """ Add 'rsakey_dict' to the key database while avoiding duplicates. @@ -113,8 +114,8 @@ def add_rsakey(rsakey_dict, keyid=None): and raise an exception if it is not. - rsakey_dict: - A dictionary conformant to 'tuf.formats.RSAKEY_SCHEMA'. + key_dict: + A dictionary conformant to 'tuf.formats.ANYKEY_SCHEMA'. It has the form: {'keytype': 'rsa', 'keyid': keyid, @@ -138,15 +139,13 @@ def add_rsakey(rsakey_dict, keyid=None): None. - """ - # Does 'rsakey_dict' have the correct format? # This check will ensure 'rsakey_dict' has the appropriate number of objects # and object types, and that all dict keys are properly named. # Raise 'tuf.FormatError if the check fails. - tuf.formats.RSAKEY_SCHEMA.check_match(rsakey_dict) + tuf.formats.ANYKEY_SCHEMA.check_match(key_dict) # Does 'keyid' have the correct format? if keyid is not None: @@ -154,16 +153,16 @@ def add_rsakey(rsakey_dict, keyid=None): tuf.formats.KEYID_SCHEMA.check_match(keyid) # Check if the keyid found in 'rsakey_dict' matches 'keyid'. - if keyid != rsakey_dict['keyid']: - raise tuf.Error('Incorrect keyid '+rsakey_dict['keyid']+' expected '+keyid) + if keyid != key_dict['keyid']: + raise tuf.Error('Incorrect keyid '+key_dict['keyid']+' expected '+keyid) # Check if the keyid belonging to 'rsakey_dict' is not already # available in the key database before returning. - keyid = rsakey_dict['keyid'] + keyid = key_dict['keyid'] if keyid in _keydb_dict: raise tuf.KeyAlreadyExistsError('Key: '+keyid) - _keydb_dict[keyid] = rsakey_dict + _keydb_dict[keyid] = copy.deepcopy(key_dict) @@ -190,7 +189,6 @@ def get_key(keyid): The key matching 'keyid'. In the case of RSA keys, a dictionary conformant to 'tuf.formats.RSAKEY_SCHEMA' is returned. - """ # Does 'keyid' have the correct format? @@ -201,7 +199,7 @@ def get_key(keyid): # Return the key belonging to 'keyid', if found in the key database. try: - return _keydb_dict[keyid] + return copy.deepcopy(_keydb_dict[keyid]) except KeyError: raise tuf.UnknownKeyError('Key: '+keyid) @@ -229,7 +227,6 @@ def remove_key(keyid): None. - """ # Does 'keyid' have the correct format? @@ -264,7 +261,6 @@ def clear_keydb(): None. - """ _keydb_dict.clear() diff --git a/tuf/keys.py b/tuf/keys.py new file mode 100755 index 0000000000..b115221bef --- /dev/null +++ b/tuf/keys.py @@ -0,0 +1,1248 @@ +""" + + keys.py + + + Vladimir Diaz + + + October 4, 2013. + + + See LICENSE for licensing information. + + + The goal of this module is to centralize cryptographic key routines and their + supported operations (e.g., creating and verifying signatures). This module + is designed to support multiple public-key algorithms, such as RSA and + ED25519, and multiple cryptography libraries. Which cryptography library to + use is determined by the default, or user modified, values set in + 'tuf.conf.py' + + The (RSA and ED25519)-related functions provided include generate_rsa_key(), + generate_ed25519_key(), create_signature(), and verify_signature(). + The cryptography libraries called by 'tuf.keys.py' generate the actual TUF + keys and the functions listed above can be viewed as the easy-to-use public + interface. + + Additional functions contained here include format_keyval_to_metadata() and + format_metadata_to_key(). These last two functions produce or use TUF keys + compatible with the key structures listed in TUF Metadata files. The key + generation functions return a dictionary containing all the information needed + of TUF keys, such as public & private keys, and a keyID. create_signature() + and verify_signature() are supplemental functions needed for generating + signatures and verifying them. + + https://en.wikipedia.org/wiki/RSA_(algorithm) + http://ed25519.cr.yp.to/ + + Key IDs are used as identifiers for keys (e.g., RSA key). They are the + hexadecimal representation of the hash of key object (specifically, the key + object containing only the public key). Review 'keys.py' and the + '_get_keyid()' function to see precisely how keyids are generated. One may + get the keyid of a key object by simply accessing the dictionary's 'keyid' + key (i.e., rsakey['keyid']). + """ + +# Required for hexadecimal conversions. Signatures and public/private keys are +# hexlified. +import binascii + +# 'pycrypto' is the only currently supported library for the creation of RSA +# keys. +# https://github.com/dlitz/pycrypto +_SUPPORTED_RSA_CRYPTO_LIBRARIES = ['pycrypto'] + +# The currently supported libraries for the creation of ed25519 keys and +# signatures. The 'pynacl' library should be installed and used over the slower +# python implementation of ed25519. The python implementation will be used +# if 'pynacl' is unavailable. +_SUPPORTED_ED25519_CRYPTO_LIBRARIES = ['ed25519', 'pynacl'] + +# 'pycrypto' is the only currently supported library for general-purpose +# cryptography, with plans to support pyca/cryptography. +# https://github.com/dlitz/pycrypto +# https://github.com/pyca/cryptography +_SUPPORTED_GENERAL_CRYPTO_LIBRARIES = ['pycrypto'] + +# Track which libraries are imported and thus available. An optimized version +# of the ed25519 python implementation is provided by TUF and avaialable by +# default. https://github.com/pyca/ed25519 +_available_crypto_libraries = ['ed25519'] + +# Import the PyCrypto library so that RSA keys are supported. +try: + import Crypto + import tuf.pycrypto_keys + _available_crypto_libraries.append('pycrypto') +except ImportError: + pass + +# Import the PyNaCl library, if available. It is recommended this library be +# used over the pure python implementation of ed25519, due to its speedier +# routines and side-channel protections available in the libsodium library. +try: + import nacl + import nacl.signing + _available_crypto_libraries.append('pynacl') +except (ImportError, IOError): + pass + +# The optimized version of the ed25519 library provided by default is imported +# regardless of the availability of PyNaCl. +import tuf.ed25519_keys + +# Import the TUF package and TUF-defined exceptions in __init__.py. +import tuf + +# Import the cryptography library settings. +import tuf.conf + +# Digest objects needed to generate hashes. +import tuf.hash + +# Perform format checks of argument objects. +import tuf.formats + +# The hash algorithm to use in the generation of keyids. +_KEY_ID_HASH_ALGORITHM = 'sha256' + +# Recommended RSA key sizes: +# http://www.emc.com/emc-plus/rsa-labs/historical/twirl-and-rsa-key-size.htm#table1 +# According to the document above, revised May 6, 2003, RSA keys of +# size 3072 provide security through 2031 and beyond. +_DEFAULT_RSA_KEY_BITS = 3072 + +# The crypto libraries to use in 'keys.py', set by default or by the user. +# The following cryptography libraries are currently supported: +# ['pycrypto', 'pynacl', 'ed25519'] +_RSA_CRYPTO_LIBRARY = tuf.conf.RSA_CRYPTO_LIBRARY +_ED25519_CRYPTO_LIBRARY = tuf.conf.ED25519_CRYPTO_LIBRARY +_GENERAL_CRYPTO_LIBRARY = tuf.conf.GENERAL_CRYPTO_LIBRARY + + +def generate_rsa_key(bits=_DEFAULT_RSA_KEY_BITS): + """ + + Generate public and private RSA keys, with modulus length 'bits'. In + addition, a keyid identifier for the RSA key is generated. The object + returned conforms to 'tuf.formats.RSAKEY_SCHEMA' and has the + form: + + {'keytype': 'rsa', + 'keyid': keyid, + 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', + 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} + + The public and private keys are strings in PEM format. + + Although the PyCrypto crytography library called sets a 1024-bit minimum + key size, generate() enforces a minimum key size of 2048 bits. If 'bits' is + unspecified, a 3072-bit RSA key is generated, which is the key size + recommended by TUF. + + >>> rsa_key = generate_rsa_key(bits=2048) + >>> tuf.formats.RSAKEY_SCHEMA.matches(rsa_key) + True + >>> public = rsa_key['keyval']['public'] + >>> private = rsa_key['keyval']['private'] + >>> tuf.formats.PEMRSA_SCHEMA.matches(public) + True + >>> tuf.formats.PEMRSA_SCHEMA.matches(private) + True + + + bits: + The key size, or key length, of the RSA key. 'bits' must be 2048, or + greater, and a multiple of 256. + + + tuf.FormatError, if 'bits' is improperly or invalid (i.e., not an integer + and not at least 2048). + + tuf.UnsupportedLibraryError, if any of the cryptography libraries specified + in 'tuf.conf.py' are unsupported or unavailable. + + ValueError, if an exception occurs after calling the RSA key generation + routine. 'bits' must be a multiple of 256. The 'ValueError' exception is + raised by the key generation function of the cryptography library called. + + + The RSA keys are generated by calling PyCrypto's + Crypto.PublicKey.RSA.generate(). + + + A dictionary containing the RSA keys and other identifying information. + Conforms to 'tuf.formats.RSAKEY_SCHEMA'. + """ + + # Does 'bits' have the correct format? + # This check will ensure 'bits' conforms to 'tuf.formats.RSAKEYBITS_SCHEMA'. + # 'bits' must be an integer object, with a minimum value of 2048. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.RSAKEYBITS_SCHEMA.check_match(bits) + + # Raise 'tuf.UnsupportedLibraryError' if the following libraries, specified in + # 'tuf.conf', are unsupported or unavailable: + # 'tuf.conf.RSA_CRYPTO_LIBRARY' and 'tuf.conf.ED25519_CRYPTO_LIBRARY'. + _check_crypto_libraries() + + # Begin building the RSA key dictionary. + rsakey_dict = {} + keytype = 'rsa' + public = None + private = None + + # Generate the public and private RSA keys. The PyCrypto module performs + # the actual key generation. Raise 'ValueError' if 'bits' is less than 1024 + # or not a multiple of 256, although a 2048-bit minimum is enforced by + # tuf.formats.RSAKEYBITS_SCHEMA.check_match(). + if _RSA_CRYPTO_LIBRARY == 'pycrypto': + public, private = tuf.pycrypto_keys.generate_rsa_public_and_private(bits) + else: + message = 'Invalid crypto library: '+repr(_RSA_CRYPTO_LIBRARY)+'.' + raise tuf.UnsupportedLibraryError(message) + + # Generate the keyid of the RSA key. 'key_value' corresponds to the + # 'keyval' entry of the 'RSAKEY_SCHEMA' dictionary. The private key + # information is not included in the generation of the 'keyid' identifier. + key_value = {'public': public, + 'private': ''} + keyid = _get_keyid(keytype, key_value) + + # Build the 'rsakey_dict' dictionary. Update 'key_value' with the RSA + # private key prior to adding 'key_value' to 'rsakey_dict'. + key_value['private'] = private + + rsakey_dict['keytype'] = keytype + rsakey_dict['keyid'] = keyid + rsakey_dict['keyval'] = key_value + + return rsakey_dict + + + + + +def generate_ed25519_key(): + """ + + Generate public and private ED25519 keys, both of length 32-bytes, although + they are hexlified to 64 bytes. + In addition, a keyid identifier generated for the returned ED25519 object. + The object returned conforms to 'tuf.formats.ED25519KEY_SCHEMA' and has the + form: + {'keytype': 'ed25519', + 'keyid': 'f30a0870d026980100c0573bd557394f8c1bbd6...', + 'keyval': {'public': '9ccf3f02b17f82febf5dd3bab878b767d8408...', + 'private': 'ab310eae0e229a0eceee3947b6e0205dfab3...'}} + + The public and private keys are strings in PEM format and stored in the + 'keyval' field of the returned dictionary. + + >>> ed25519_key = generate_ed25519_key() + >>> tuf.formats.ED25519KEY_SCHEMA.matches(ed25519_key) + True + >>> len(ed25519_key['keyval']['public']) + 64 + >>> len(ed25519_key['keyval']['private']) + 64 + + + None. + + + tuf.UnsupportedLibraryError, if an unsupported or unavailable library is + detected. + + + The ED25519 keys are generated by calling either the optimized pure Python + implementation of ed25519, or the ed25519 routines provided by 'pynacl'. + + + A dictionary containing the ED25519 keys and other identifying information. + Conforms to 'tuf.formats.ED25519KEY_SCHEMA'. + """ + + # Raise 'tuf.UnsupportedLibraryError' if the following libraries, specified + # in 'tuf.conf', are unsupported or unavailable: + # 'tuf.conf.RSA_CRYPTO_LIBRARY' and 'tuf.conf.ED25519_CRYPTO_LIBRARY'. + _check_crypto_libraries() + + # Begin building the ED25519 key dictionary. + ed25519_key = {} + keytype = 'ed25519' + public = None + private = None + + # Generate the public and private ED25519 keys. Use the 'pynacl' library + # if available, otherwise fall back to optimized pure python implementation + # provided by pyca and available in TUF. + if 'pynacl' in _available_crypto_libraries: + public, private = \ + tuf.ed25519_keys.generate_public_and_private(use_pynacl=True) + else: + public, private = \ + tuf.ed25519_keys.generate_public_and_private(use_pynacl=False) + + # Generate the keyid of the ED25519 key. 'key_value' corresponds to the + # 'keyval' entry of the 'ED25519KEY_SCHEMA' dictionary. The private key + # information is not included in the generation of the 'keyid' identifier. + key_value = {'public': binascii.hexlify(public), + 'private': ''} + keyid = _get_keyid(keytype, key_value) + + # Build the 'ed25519_key' dictionary. Update 'key_value' with the ED25519 + # private key prior to adding 'key_value' to 'ed25519_key'. + key_value['private'] = binascii.hexlify(private) + + ed25519_key['keytype'] = keytype + ed25519_key['keyid'] = keyid + ed25519_key['keyval'] = key_value + + return ed25519_key + + + + + +def format_keyval_to_metadata(keytype, key_value, private=False): + """ + + Return a dictionary conformant to 'tuf.formats.KEY_SCHEMA'. + If 'private' is True, include the private key. The dictionary + returned has the form: + + {'keytype': keytype, + 'keyval': {'public': '...', + 'private': '...'}} + + or if 'private' is False: + + {'keytype': keytype, + 'keyval': {'public': '...', + 'private': ''}} + + TUF keys are stored in Metadata files (e.g., root.json) in the format + returned by this function. + + >>> ed25519_key = generate_ed25519_key() + >>> key_val = ed25519_key['keyval'] + >>> keytype = ed25519_key['keytype'] + >>> ed25519_metadata = \ + format_keyval_to_metadata(keytype, key_val, private=True) + >>> tuf.formats.KEY_SCHEMA.matches(ed25519_metadata) + True + + + key_type: + The 'rsa' or 'ed25519' strings. + + key_value: + A dictionary containing a private and public keys. + 'key_value' is of the form: + + {'public': '...', + 'private': '...'}}, + + conformant to 'tuf.formats.KEYVAL_SCHEMA'. + + private: + Indicates if the private key should be included in the dictionary + returned. + + + tuf.FormatError, if 'key_value' does not conform to + 'tuf.formats.KEYVAL_SCHEMA'. + + + None. + + + A 'tuf.formats.KEY_SCHEMA' dictionary. + """ + + # Does 'keytype' have the correct format? + # This check will ensure 'keytype' has the appropriate number + # of objects and object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.KEYTYPE_SCHEMA.check_match(keytype) + + # Does 'key_value' have the correct format? + tuf.formats.KEYVAL_SCHEMA.check_match(key_value) + + if private is True and 'private' in key_value: + return {'keytype': keytype, 'keyval': key_value} + + else: + public_key_value = {'public': key_value['public']} + return {'keytype': keytype, 'keyval': public_key_value} + + + + + +def format_metadata_to_key(key_metadata): + """ + + Construct a TUF key dictionary (e.g., tuf.formats.RSAKEY_SCHEMA) + according to the keytype of 'key_metadata'. The dict returned by this + function has the exact format as the dict returned by one of the key + generations functions, like generate_ed25519_key(). The dict returned + has the form: + + {'keytype': keytype, + 'keyid': 'f30a0870d026980100c0573bd557394f8c1bbd6...', + 'keyval': {'public': '...', + 'private': '...'}} + + For example, RSA key dictionaries in RSAKEY_SCHEMA format should be used by + modules storing a collection of keys, such as with keydb.py. RSA keys as + stored in metadata files use a different format, so this function should be + called if an RSA key is extracted from one of these metadata files and need + converting. The key generation functions create an entirely new key and + return it in the format appropriate for 'keydb.py'. + + >>> ed25519_key = generate_ed25519_key() + >>> key_val = ed25519_key['keyval'] + >>> keytype = ed25519_key['keytype'] + >>> ed25519_metadata = \ + format_keyval_to_metadata(keytype, key_val, private=True) + >>> ed25519_key_2 = format_metadata_to_key(ed25519_metadata) + >>> tuf.formats.ED25519KEY_SCHEMA.matches(ed25519_key_2) + True + >>> ed25519_key == ed25519_key_2 + True + + + key_metadata: + The TUF key dictionary as stored in Metadata files, conforming to + 'tuf.formats.KEY_SCHEMA'. It has the form: + + {'keytype': '...', + 'keyval': {'public': '...', + 'private': '...'}} + + + tuf.FormatError, if 'key_metadata' does not conform to + 'tuf.formats.KEY_SCHEMA'. + + + None. + + + In the case of an RSA key, a dictionary conformant to + 'tuf.formats.RSAKEY_SCHEMA'. + """ + + # Does 'key_metadata' have the correct format? + # This check will ensure 'key_metadata' has the appropriate number + # of objects and object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.KEY_SCHEMA.check_match(key_metadata) + + # Construct the dictionary to be returned. + key_dict = {} + keytype = key_metadata['keytype'] + key_value = key_metadata['keyval'] + + # Convert 'key_value' to 'tuf.formats.KEY_SCHEMA' and generate its hash + # The hash is in hexdigest form. + keyid = _get_keyid(keytype, key_value) + + # All the required key values gathered. Build 'key_dict'. + key_dict['keytype'] = keytype + key_dict['keyid'] = keyid + key_dict['keyval'] = key_value + + return key_dict + + + + + +def _get_keyid(keytype, key_value): + """Return the keyid of 'key_value'.""" + + # 'keyid' will be generated from an object conformant to KEY_SCHEMA, + # which is the format Metadata files (e.g., root.json) store keys. + # 'format_keyval_to_metadata()' returns the object needed by _get_keyid(). + key_meta = format_keyval_to_metadata(keytype, key_value, private=False) + + # Convert the TUF key to JSON Canonical format, suitable for adding + # to digest objects. + key_update_data = tuf.formats.encode_canonical(key_meta) + + # Create a digest object and call update(), using the JSON + # canonical format of 'rskey_meta' as the update data. + digest_object = tuf.hash.digest(_KEY_ID_HASH_ALGORITHM) + digest_object.update(key_update_data) + + # 'keyid' becomes the hexadecimal representation of the hash. + keyid = digest_object.hexdigest() + + return keyid + + + + + +def _check_crypto_libraries(): + """ Ensure all the crypto libraries specified in tuf.conf are available. """ + + # The checks below all raise 'tuf.UnsupportedLibraryError' if the RSA and + # ED25519 crypto libraries specified in 'tuf.conf.py' are not supported or + # unavailable. The appropriate error message is added to the exception. + # The funcions of this module that depend on user-installed crypto libraries + # should call this private function to ensure the called routine does not fail + # with unpredictable exceptions in the event of a missing library. + # The supported and available lists checked are populated when 'tuf.keys.py' + # is imported. + if _RSA_CRYPTO_LIBRARY not in _SUPPORTED_RSA_CRYPTO_LIBRARIES: + message = 'The '+repr(_RSA_CRYPTO_LIBRARY)+' crypto library specified'+ \ + ' in "tuf.conf.RSA_CRYPTO_LIBRARY" is not supported.\n'+ \ + 'Supported crypto libraries: '+repr(_SUPPORTED_RSA_CRYPTO_LIBRARIES)+'.' + raise tuf.UnsupportedLibraryError(message) + + if _ED25519_CRYPTO_LIBRARY not in _SUPPORTED_ED25519_CRYPTO_LIBRARIES: + message = 'The '+repr(_ED25519_CRYPTO_LIBRARY)+' crypto library specified'+\ + ' in "tuf.conf.ED25519_CRYPTO_LIBRARY" is not supported.\n'+ \ + 'Supported crypto libraries: '+repr(_SUPPORTED_ED25519_CRYPTO_LIBRARIES)+'.' + raise tuf.UnsupportedLibraryError(message) + + if _GENERAL_CRYPTO_LIBRARY not in _SUPPORTED_GENERAL_CRYPTO_LIBRARIES: + message = 'The '+repr(_GENERAL_CRYPTO_LIBRARY)+' crypto library specified'+\ + ' in "tuf.conf.GENERAL_CRYPTO_LIBRARY" is not supported.\n'+ \ + 'Supported crypto libraries: '+repr(_SUPPORTED_GENERAL_CRYPTO_LIBRARIES)+'.' + raise tuf.UnsupportedLibraryError(message) + + if _RSA_CRYPTO_LIBRARY not in _available_crypto_libraries: + message = 'The '+repr(_RSA_CRYPTO_LIBRARY)+' crypto library specified'+ \ + ' in "tuf.conf.RSA_CRYPTO_LIBRARY" could not be imported.' + raise tuf.UnsupportedLibraryError(message) + + if _ED25519_CRYPTO_LIBRARY not in _available_crypto_libraries: + message = 'The '+repr(_ED25519_CRYPTO_LIBRARY)+' crypto library specified'+\ + ' in "tuf.conf.ED25519_CRYPTO_LIBRARY" could not be imported.' + raise tuf.UnsupportedLibraryError(message) + + if _GENERAL_CRYPTO_LIBRARY not in _available_crypto_libraries: + message = 'The '+repr(_GENERAL_CRYPTO_LIBRARY)+' crypto library specified'+\ + ' in "tuf.conf.GENERAL_CRYPTO_LIBRARY" could not be imported.' + raise tuf.UnsupportedLibraryError(message) + + + + + +def create_signature(key_dict, data): + """ + + Return a signature dictionary of the form: + {'keyid': 'f30a0870d026980100c0573bd557394f8c1bbd6...', + 'method': '...', + 'sig': '...'}. + + The signing process will use the private key in + key_dict['keyval']['private'] and 'data' to generate the signature. + + The following signature methods are supported: + + 'RSASSA-PSS' + RFC3447 - RSASSA-PSS + http://www.ietf.org/rfc/rfc3447. + + 'ed25519' + ed25519 - high-speed high security signatures + http://ed25519.cr.yp.to/ + + Which signature to generate is determined by the key type of 'key_dict' + and the available cryptography library specified in 'tuf.conf'. + + >>> ed25519_key = generate_ed25519_key() + >>> data = 'The quick brown fox jumps over the lazy dog' + >>> signature = create_signature(ed25519_key, data) + >>> tuf.formats.SIGNATURE_SCHEMA.matches(signature) + True + >>> len(signature['sig']) + 128 + >>> rsa_key = generate_rsa_key(2048) + >>> data = 'The quick brown fox jumps over the lazy dog' + >>> signature = create_signature(rsa_key, data) + >>> tuf.formats.SIGNATURE_SCHEMA.matches(signature) + True + + + key_dict: + A dictionary containing the TUF keys. An example RSA key dict has the + form: + + {'keytype': 'rsa', + 'keyid': 'f30a0870d026980100c0573bd557394f8c1bbd6...', + 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', + 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} + + The public and private keys are strings in PEM format. + + data: + Data object used by create_signature() to generate the signature. + + + tuf.FormatError, if 'key_dict' is improperly formatted. + + tuf.UnsupportedLibraryError, if an unsupported or unavailable library is + detected. + + TypeError, if 'key_dict' contains an invalid keytype. + + + The cryptography library specified in 'tuf.conf' called to perform the + actual signing routine. + + + A signature dictionary conformat to 'tuf.format.SIGNATURE_SCHEMA'. + """ + + # Does 'key_dict' have the correct format? + # This check will ensure 'key_dict' has the appropriate number of objects + # and object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + # The key type of 'key_dict' must be either 'rsa' or 'ed25519'. + tuf.formats.ANYKEY_SCHEMA.check_match(key_dict) + + # Raise 'tuf.UnsupportedLibraryError' if the following libraries, specified + # in 'tuf.conf', are unsupported or unavailable: + # 'tuf.conf.RSA_CRYPTO_LIBRARY' and 'tuf.conf.ED25519_CRYPTO_LIBRARY'. + _check_crypto_libraries() + + # Signing the 'data' object requires a private key. + # The 'RSASSA-PSS' (i.e., PyCrypto module) and 'ed25519' (i.e., PyNaCl and the + # optimized pure Python implementation of ed25519) are the only signing + # methods currently supported. + signature = {} + keytype = key_dict['keytype'] + public = key_dict['keyval']['public'] + private = key_dict['keyval']['private'] + keyid = key_dict['keyid'] + method = None + sig = None + + # Convert 'data' to canonical JSON format so that repeatable signatures are + # generated across different platforms and Python key dictionaries. The + # resulting 'data' is a string encoded in UTF-8 and compatible with the input + # expected by the cryptography functions called below. + data = tuf.formats.encode_canonical(data) + + # Call the appropriate cryptography libraries for the supported key types, + # otherwise raise an exception. + if keytype == 'rsa': + if _RSA_CRYPTO_LIBRARY == 'pycrypto': + sig, method = tuf.pycrypto_keys.create_rsa_signature(private, data) + else: + message = 'Unsupported "tuf.conf.RSA_CRYPTO_LIBRARY": '+\ + repr(_RSA_CRYPTO_LIBRARY)+'.' + raise tuf.UnsupportedLibraryError(message) + + elif keytype == 'ed25519': + public = binascii.unhexlify(public) + private = binascii.unhexlify(private) + if _ED25519_CRYPTO_LIBRARY == 'pynacl' \ + and 'pynacl' in _available_crypto_libraries: + sig, method = tuf.ed25519_keys.create_signature(public, private, + data, use_pynacl=True) + + # Fall back to using the optimized pure python implementation of ed25519. + else: + sig, method = tuf.ed25519_keys.create_signature(public, private, + data, use_pynacl=False) + else: + raise TypeError('Invalid key type.') + + # Build the signature dictionary to be returned. + # The hexadecimal representation of 'sig' is stored in the signature. + signature['keyid'] = keyid + signature['method'] = method + signature['sig'] = binascii.hexlify(sig) + + return signature + + + + + +def verify_signature(key_dict, signature, data): + """ + + Determine whether the private key belonging to 'key_dict' produced + 'signature'. verify_signature() will use the public key found in + 'key_dict', the 'method' and 'sig' objects contained in 'signature', + and 'data' to complete the verification. + + >>> ed25519_key = generate_ed25519_key() + >>> data = 'The quick brown fox jumps over the lazy dog' + >>> signature = create_signature(ed25519_key, data) + >>> verify_signature(ed25519_key, signature, data) + True + >>> verify_signature(ed25519_key, signature, 'bad_data') + False + >>> rsa_key = generate_rsa_key() + >>> signature = create_signature(rsa_key, data) + >>> verify_signature(rsa_key, signature, data) + True + >>> verify_signature(rsa_key, signature, 'bad_data') + False + + + key_dict: + A dictionary containing the TUF keys and other identifying information. + If 'key_dict' is an RSA key, it has the form: + + {'keytype': 'rsa', + 'keyid': 'f30a0870d026980100c0573bd557394f8c1bbd6...', + 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', + 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} + + The public and private keys are strings in PEM format. + + signature: + The signature dictionary produced by one of the key generation functions. + 'signature' has the form: + + {'keyid': 'f30a0870d026980100c0573bd557394f8c1bbd6...', + 'method': 'method', + 'sig': sig}. + + Conformant to 'tuf.formats.SIGNATURE_SCHEMA'. + + data: + Data object used by tuf.rsa_key.create_signature() to generate + 'signature'. 'data' is needed here to verify the signature. + + + tuf.FormatError, raised if either 'key_dict' or 'signature' are improperly + formatted. + + tuf.UnsupportedLibraryError, if an unsupported or unavailable library is + detected. + + tuf.UnknownMethodError. Raised if the signing method used by + 'signature' is not one supported. + + + The cryptography library specified in 'tuf.conf' called to do the actual + verification. + + + Boolean. True if the signature is valid, False otherwise. + """ + + # Does 'key_dict' have the correct format? + # This check will ensure 'key_dict' has the appropriate number + # of objects and object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.ANYKEY_SCHEMA.check_match(key_dict) + + # Does 'signature' have the correct format? + tuf.formats.SIGNATURE_SCHEMA.check_match(signature) + + # Using the public key belonging to 'key_dict' + # (i.e., rsakey_dict['keyval']['public']), verify whether 'signature' + # was produced by key_dict's corresponding private key + # key_dict['keyval']['private']. + method = signature['method'] + sig = signature['sig'] + sig = binascii.unhexlify(sig) + public = key_dict['keyval']['public'] + keytype = key_dict['keytype'] + valid_signature = False + + # Convert 'data' to canonical JSON format so that repeatable signatures are + # generated across different platforms and Python key dictionaries. The + # resulting 'data' is a string encoded in UTF-8 and compatible with the input + # expected by the cryptography functions called below. + data = tuf.formats.encode_canonical(data) + + # Call the appropriate cryptography libraries for the supported key types, + # otherwise raise an exception. + if keytype == 'rsa': + if _RSA_CRYPTO_LIBRARY == 'pycrypto': + valid_signature = tuf.pycrypto_keys.verify_rsa_signature(sig, method, + public, data) + else: + message = 'Unsupported "tuf.conf.RSA_CRYPTO_LIBRARY": '+\ + repr(_RSA_CRYPTO_LIBRARY)+'.' + raise tuf.UnsupportedLibraryError(message) + + elif keytype == 'ed25519': + public = binascii.unhexlify(public) + if _ED25519_CRYPTO_LIBRARY == 'pynacl' or \ + 'pynacl' in _available_crypto_libraries: + valid_signature = tuf.ed25519_keys.verify_signature(public, + method, sig, data, + use_pynacl=True) + # Fall back to the optimized pure python implementation of ed25519. + else: + valid_signature = tuf.ed25519_keys.verify_signature(public, + method, sig, data, + use_pynacl=False) + else: + raise TypeError('Unsupported key type.') + + return valid_signature + + + + + +def import_rsakey_from_encrypted_pem(encrypted_pem, password): + """ + + Import the public and private RSA keys stored in 'encrypted_pem'. In + addition, a keyid identifier for the RSA key is generated. The object + returned conforms to 'tuf.formats.RSAKEY_SCHEMA' and has the + form: + + {'keytype': 'rsa', + 'keyid': keyid, + 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', + 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} + + The public and private keys are strings in PEM format. + + Although the PyCrypto crytography library called sets a 1024-bit minimum + key size, generate() enforces a minimum key size of 2048 bits. If 'bits' is + unspecified, a 3072-bit RSA key is generated, which is the key size + recommended by TUF. + + >>> rsa_key = generate_rsa_key() + >>> private = rsa_key['keyval']['private'] + >>> passphrase = 'secret' + >>> encrypted_pem = create_rsa_encrypted_pem(private, passphrase) + >>> rsa_key2 = import_rsakey_from_encrypted_pem(encrypted_pem, passphrase) + >>> rsa_key == rsa_key2 + True + + + encrypted_pem: + A string in PEM format. + + password: + The password, or passphrase, to decrypt the private part of the RSA + key. 'password' is not used directly as the encryption key, a stronger + encryption key is derived from it. + + + tuf.FormatError, if the arguments are improperly formatted. + + tuf.UnsupportedLibraryError, if any of the cryptography libraries specified + in 'tuf.conf.py' are unsupported or unavailable. + + ValueError, if an exception occurs after calling the RSA key generation + routine. 'bits' must be a multiple of 256. The 'ValueError' exception is + raised by the key generation function of the cryptography library called. + + + The RSA keys are generated by calling PyCrypto's + Crypto.PublicKey.RSA.generate(). + + + A dictionary containing the RSA keys and other identifying information. + Conforms to 'tuf.formats.RSAKEY_SCHEMA'. + """ + + # Does 'encrypted_pem' have the correct format? + # This check will ensure 'encrypted_pem' conforms to + # 'tuf.formats.PEMRSA_SCHEMA'. + tuf.formats.PEMRSA_SCHEMA.check_match(encrypted_pem) + + # Does 'password' have the correct format? + tuf.formats.PASSWORD_SCHEMA.check_match(password) + + # Raise 'tuf.UnsupportedLibraryError' if the following libraries, specified in + # 'tuf.conf', are unsupported or unavailable: + # 'tuf.conf.RSA_CRYPTO_LIBRARY' and 'tuf.conf.ED25519_CRYPTO_LIBRARY'. + _check_crypto_libraries() + + # Begin building the RSA key dictionary. + rsakey_dict = {} + keytype = 'rsa' + public = None + private = None + + # Generate the public and private RSA keys. The PyCrypto module performs + # the actual key generation. Raise 'ValueError' if 'bits' is less than 1024 + # or not a multiple of 256, although a 2048-bit minimum is enforced by + # tuf.formats.RSAKEYBITS_SCHEMA.check_match(). + if _RSA_CRYPTO_LIBRARY == 'pycrypto': + public, private = \ + tuf.pycrypto_keys.create_rsa_public_and_private_from_encrypted_pem(encrypted_pem, + password) + else: + message = 'Invalid crypto library: '+repr(_RSA_CRYPTO_LIBRARY)+'.' + raise tuf.UnsupportedLibraryError(message) + + # Generate the keyid of the RSA key. 'key_value' corresponds to the + # 'keyval' entry of the 'RSAKEY_SCHEMA' dictionary. The private key + # information is not included in the generation of the 'keyid' identifier. + key_value = {'public': public, + 'private': ''} + keyid = _get_keyid(keytype, key_value) + + # Build the 'rsakey_dict' dictionary. Update 'key_value' with the RSA + # private key prior to adding 'key_value' to 'rsakey_dict'. + key_value['private'] = private + + rsakey_dict['keytype'] = keytype + rsakey_dict['keyid'] = keyid + rsakey_dict['keyval'] = key_value + + return rsakey_dict + + + + + +def format_rsakey_from_pem(pem): + """ + + Generate an RSA key object from 'pem'. In addition, a keyid identifier for + the RSA key is generated. The object returned conforms to + 'tuf.formats.RSAKEY_SCHEMA' and has the form: + + {'keytype': 'rsa', + 'keyid': keyid, + 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', + 'private': ''}} + + The public portion of the RSA key is a string in PEM format. + + >>> rsa_key = generate_rsa_key() + >>> public = rsa_key['keyval']['public'] + >>> rsa_key['keyval']['private'] = '' + >>> rsa_key2 = format_rsakey_from_pem(public) + >>> rsa_key == rsa_key2 + True + >>> format_rsakey_from_pem('bad_pem') + Traceback (most recent call last): + ... + FormatError: The PEM string argument is improperly formatted. + + + pem: + A string in PEM format. + + + tuf.FormatError, if 'pem' is improperly formatted. + + + None. + + + A dictionary containing the RSA keys and other identifying information. + Conforms to 'tuf.formats.RSAKEY_SCHEMA'. + """ + + # Does 'pem' have the correct format? + # This check will ensure arguments has the appropriate number + # of objects and object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.PEMRSA_SCHEMA.check_match(pem) + + # Ensure the PEM string starts with the required number of dashes. Although + # a simple validation of 'pem' is performed here, a fully valid PEM string is + # needed to successfully verify signatures. + if not pem.startswith(b'-----'): + raise tuf.FormatError('The PEM string argument is improperly formatted.') + + # Begin building the RSA key dictionary. + rsakey_dict = {} + keytype = 'rsa' + public = pem + + # Generate the keyid of the RSA key. 'key_value' corresponds to the + # 'keyval' entry of the 'RSAKEY_SCHEMA' dictionary. The private key + # information is not included in the generation of the 'keyid' identifier. + key_value = {'public': public, + 'private': ''} + keyid = _get_keyid(keytype, key_value) + + rsakey_dict['keytype'] = keytype + rsakey_dict['keyid'] = keyid + rsakey_dict['keyval'] = key_value + + return rsakey_dict + + + + + +def encrypt_key(key_object, password): + """ + + Return a string containing 'key_object' in encrypted form. Encrypted strings + may be safely saved to a file. The corresponding decrypt_key() function can + be applied to the encrypted string to restore the original key object. + 'key_object' is a TUF key (e.g., RSAKEY_SCHEMA, ED25519KEY_SCHEMA). This + function calls the appropriate cryptography module (e.g., pycrypto_keys.py) + to perform the encryption. + + The currently supported general-purpose crypto module, 'pycrypto_keys.py', + performs the actual cryptographic operation on 'key_object'. Whereas + an encrypted PEM file uses the Triple Data Encryption Algorithm (3DES), the + Cipher-block chaining (CBC) mode of operation, and the Password-Based Key + Derivation Function 1 (PBKF1) + MD5 to strengthen 'password', encrypted + TUF keys use AES-256-CTR-Mode and passwords strengthened with + PBKDF2-HMAC-SHA256 (100K iterations by default, but may be overriden in + 'tuf.conf.PBKDF2_ITERATIONS' by the user). + + http://en.wikipedia.org/wiki/Advanced_Encryption_Standard + http://en.wikipedia.org/wiki/CTR_mode#Counter_.28CTR.29 + https://en.wikipedia.org/wiki/PBKDF2 + + >>> ed25519_key = generate_ed25519_key() + >>> password = 'secret' + >>> encrypted_key = encrypt_key(ed25519_key, password) + >>> tuf.formats.ENCRYPTEDKEY_SCHEMA.matches(encrypted_key) + True + + + key_object: + A TUF key (containing also the private key portion) of the form + 'tuf.formats.ANYKEY_SCHEMA' + + password: + The password, or passphrase, to encrypt the private part of the RSA + key. 'password' is not used directly as the encryption key, a stronger + encryption key is derived from it. + + + tuf.FormatError, if the arguments are improperly formatted. + + tuf.CryptoError, if 'key_object' cannot be encrypted. + + tuf.UnsupportedLibraryError, if the general-purpose cryptography library + specified in 'tuf.conf.GENERAL_CRYPTO_LIBRARY' is unsupported. + + + Perform crytographic operations using the library specified in + 'tuf.formats.GENERAL_CRYPTO_LIBRARY' and 'password'. + + + An encrypted string of the form: 'tuf.formats.ENCRYPTEDKEY_SCHEMA'. + """ + + # Does 'key_object' have the correct format? + # This check will ensure 'key_object' has the appropriate number + # of objects and object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.ANYKEY_SCHEMA.check_match(key_object) + + # Does 'password' have the correct format? + tuf.formats.PASSWORD_SCHEMA.check_match(password) + + # Raise 'tuf.UnsupportedLibraryError' if the following libraries, specified in + # 'tuf.conf', are unsupported or unavailable: + # 'tuf.conf.GENERAL_CRYPTO_LIBRARY'. + _check_crypto_libraries() + + # Encrypted string of 'key_object'. The encrypted string may be safely saved + # to a file and stored offline. + encrypted_key = None + + # Generate an encrypted string of 'key_object' using AES-256-CTR-Mode, where + # 'password' is strengthened with PBKDF2-HMAC-SHA256. Ensure the general- + # purpose library specified in 'tuf.conf.GENERAL_CRYPTO_LIBRARY' is supported. + if _GENERAL_CRYPTO_LIBRARY == 'pycrypto': + encrypted_key = \ + tuf.pycrypto_keys.encrypt_key(key_object, password) + else: + message = 'Invalid crypto library: '+repr(_GENERAL_CRYPTO_LIBRARY)+'.' + raise tuf.UnsupportedLibraryError(message) + + return encrypted_key + + + + + +def decrypt_key(encrypted_key, passphrase): + """ + + Return a string containing 'encrypted_key' in non-encrypted form. + The decrypt_key() function can be applied to the encrypted string to restore + the original key object, a TUF key (e.g., RSAKEY_SCHEMA, ED25519KEY_SCHEMA). + This function calls the appropriate cryptography module (e.g., + pycrypto_keys.py) to perform the decryption. + + The currently supported general-purpose crypto module, 'pycrypto_keys.py', + performs the actual cryptographic operation on 'key_object'. Whereas + an encrypted PEM file uses the Triple Data Encryption Algorithm (3DES), the + Cipher-block chaining (CBC) mode of operation, and the Password-Based Key + Derivation Function 1 (PBKF1) + MD5 to strengthen 'password', encrypted + TUF keys use AES-256-CTR-Mode and passwords strengthened with + PBKDF2-HMAC-SHA256 (100K iterations be default, but may be overriden in + 'tuf.conf.py' by the user). + + http://en.wikipedia.org/wiki/Advanced_Encryption_Standard + http://en.wikipedia.org/wiki/CTR_mode#Counter_.28CTR.29 + https://en.wikipedia.org/wiki/PBKDF2 + + >>> ed25519_key = generate_ed25519_key() + >>> password = 'secret' + >>> encrypted_key = encrypt_key(ed25519_key, password) + >>> decrypted_key = decrypt_key(encrypted_key, password) + >>> tuf.formats.ANYKEY_SCHEMA.matches(decrypted_key) + True + >>> decrypted_key == ed25519_key + True + + + encrypted_key: + An encrypted TUF key (additional data is also included, such as salt, + number of password iterations used for the derived encryption key, etc) + of the form 'tuf.formats.ENCRYPTEDKEY_SCHEMA'. 'encrypted_key' should + have been generated with encrypted_key(). + + password: + The password, or passphrase, to decrypt 'encrypted_key'. 'password' is + not used directly as the encryption key, a stronger encryption key is + derived from it. The supported general-purpose module takes care of + re-deriving the encryption key. + + + tuf.FormatError, if the arguments are improperly formatted. + + tuf.CryptoError, if 'encrypted_key' cannot be decrypted. + + tuf.UnsupportedLibraryError, if the general-purpose cryptography library + specified in 'tuf.conf.GENERAL_CRYPTO_LIBRARY' is unsupported. + + + Perform crytographic operations using the library specified in + 'tuf.formats.GENERAL_CRYPTO_LIBRARY' and 'password'. + + + A TUF key object of the form: 'tuf.formats.ANYKEY_SCHEMA' (e.g., + RSAKEY_SCHEMA, ED25519KEY_SCHEMA). + """ + + # Does 'encrypted_key' have the correct format? + # This check ensures 'encrypted_key' has the appropriate number + # of objects and object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.ENCRYPTEDKEY_SCHEMA.check_match(encrypted_key) + + # Does 'passphrase' have the correct format? + tuf.formats.PASSWORD_SCHEMA.check_match(passphrase) + + # Raise 'tuf.UnsupportedLibraryError' if the following libraries, specified in + # 'tuf.conf', are unsupported or unavailable: + # 'tuf.conf.GENERAL_CRYPTO_LIBRARY'. + _check_crypto_libraries() + + # Store and return the decrypted key object. + key_object = None + + # Decrypt 'encrypted_key' so that the original key object is restored. + # encrypt_key() generates an encrypted string of the TUF key object using + # AES-256-CTR-Mode, where 'password' is strengthened with PBKDF2-HMAC-SHA256. + # Ensure the general-purpose library specified in + # 'tuf.conf.GENERAL_CRYPTO_LIBRARY' is supported. + if _GENERAL_CRYPTO_LIBRARY == 'pycrypto': + key_object = \ + tuf.pycrypto_keys.decrypt_key(encrypted_key, passphrase) + else: + message = 'Invalid crypto library: '+repr(_GENERAL_CRYPTO_LIBRARY)+'.' + raise tuf.UnsupportedLibraryError(message) + + # The corresponding encrypt_key() encrypts and stores key objects in + # non-metadata format (i.e., original format of key object argument to + # encrypt_key()) prior to returning. + + return key_object + + + + + +def create_rsa_encrypted_pem(private_key, passphrase): + """ + + Return a string in PEM format, where the private part of the RSA key is + encrypted. The private part of the RSA key is encrypted by the Triple + Data Encryption Algorithm (3DES) and Cipher-block chaining (CBC) for the + mode of operation. Password-Based Key Derivation Function 1 (PBKF1) + MD5 + is used to strengthen 'passphrase'. + + https://en.wikipedia.org/wiki/Triple_DES + https://en.wikipedia.org/wiki/PBKDF2 + + >>> rsa_key = generate_rsa_key() + >>> private = rsa_key['keyval']['private'] + >>> passphrase = 'secret' + >>> encrypted_pem = create_rsa_encrypted_pem(private, passphrase) + >>> tuf.formats.PEMRSA_SCHEMA.matches(encrypted_pem) + True + + + private_key: + The private key string in PEM format. + + passphrase: + The passphrase, or password, to encrypt the private part of the RSA + key. 'passphrase' is not used directly as the encryption key, a stronger + encryption key is derived from it. + + + tuf.FormatError, if the arguments are improperly formatted. + + tuf.CryptoError, if an RSA key in encrypted PEM format cannot be created. + + TypeError, 'private_key' is unset. + + + PyCrypto's Crypto.PublicKey.RSA.exportKey() called to perform the actual + generation of the PEM-formatted output. + + + A string in PEM format, where the private RSA key is encrypted. + Conforms to 'tuf.formats.PEMRSA_SCHEMA'. + """ + + # Does 'private_key' have the correct format? + # This check will ensure 'private_key' has the appropriate number + # of objects and object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.PEMRSA_SCHEMA.check_match(private_key) + + # Does 'passphrase' have the correct format? + tuf.formats.PASSWORD_SCHEMA.check_match(passphrase) + + # Raise 'tuf.UnsupportedLibraryError' if the following libraries, specified in + # 'tuf.conf', are unsupported or unavailable: + # 'tuf.conf.RSA_CRYPTO_LIBRARY'. + _check_crypto_libraries() + + encrypted_pem = None + + # Generate the public and private RSA keys. The PyCrypto module performs + # the actual key generation. Raise 'ValueError' if 'bits' is less than 1024 + # or not a multiple of 256, although a 2048-bit minimum is enforced by + # tuf.formats.RSAKEYBITS_SCHEMA.check_match(). + if _RSA_CRYPTO_LIBRARY == 'pycrypto': + encrypted_pem = \ + tuf.pycrypto_keys.create_rsa_encrypted_pem(private_key, passphrase) + else: + message = 'Invalid crypto library: '+repr(_RSA_CRYPTO_LIBRARY)+'.' + raise tuf.UnsupportedLibraryError(message) + + return encrypted_pem + + + +if __name__ == '__main__': + # The interactive sessions of the documentation strings can + # be tested by running 'keys.py' as a standalone module: + # $ python keys.py + import doctest + doctest.testmod() diff --git a/tuf/log.py b/tuf/log.py index 2a0c403683..fed3a0e83f 100755 --- a/tuf/log.py +++ b/tuf/log.py @@ -54,7 +54,6 @@ processes: http://docs.python.org/2/library/logging.html#thread-safety http://docs.python.org/2/howto/logging-cookbook.html - """ @@ -76,8 +75,8 @@ # Set the format for logging messages. # Example format for '_FORMAT_STRING': # [2013-08-13 15:21:18,068 UTC] [tuf] [INFO][_update_metadata:851@updater.py] -_FORMAT_STRING = '[%(asctime)s UTC] [%(name)s] [%(levelname)s]'+\ - '[%(funcName)s:%(lineno)s@%(filename)s] %(message)s' +_FORMAT_STRING = '[%(asctime)s UTC] [%(name)s] [%(levelname)s] '+\ + '[%(funcName)s:%(lineno)s@%(filename)s]\n%(message)s\n' # Ask all Formatter instances to talk GMT. Set the 'converter' attribute of # 'logging.Formatter' so that all formatters use Greenwich Mean Time. @@ -143,7 +142,6 @@ def filter(self, record): True. - """ # If this LogRecord object has an exception, then we will replace its text. @@ -185,7 +183,6 @@ def set_log_level(log_level=_DEFAULT_LOG_LEVEL): None. - """ # Does 'log_level' have the correct format? @@ -216,7 +213,6 @@ def set_filehandler_log_level(log_level=_DEFAULT_FILE_LOG_LEVEL): None. - """ # Does 'log_level' have the correct format? @@ -248,7 +244,6 @@ def set_console_log_level(log_level=_DEFAULT_CONSOLE_LOG_LEVEL): None. - """ # Does 'log_level' have the correct format? @@ -287,16 +282,15 @@ def add_console_handler(log_level=_DEFAULT_CONSOLE_LOG_LEVEL): None. - """ - # Assign to the global console_handler object. - global console_handler - # Does 'log_level' have the correct format? # Raise 'tuf.FormatError' if there is a mismatch. tuf.formats.LOGLEVEL_SCHEMA.check_match(log_level) + # Assign to the global console_handler object. + global console_handler + if not console_handler: # Set the console handler for the logger. The built-in console handler will # log messages to 'sys.stderr' and capture 'log_level' messages. @@ -333,7 +327,6 @@ def remove_console_handler(): None. - """ # Assign to the global 'console_handler' object. diff --git a/tuf/mirrors.py b/tuf/mirrors.py index 6a3754676d..49c956aa0e 100755 --- a/tuf/mirrors.py +++ b/tuf/mirrors.py @@ -15,7 +15,6 @@ To extract a list of mirror urls corresponding to the file type and the location of the file with respect to the base url. - """ import os @@ -55,7 +54,7 @@ def get_list_of_mirrors(file_type, file_path, mirrors_dict): {'url_prefix': 'http://localhost:8001', 'metadata_path': 'metadata/', 'targets_path': 'targets/', - 'confined_target_dirs': ['targets/release1/', ...], + 'confined_target_dirs': ['targets/snapshot1/', ...], 'custom': {...}} The 'custom' field is optional. @@ -68,7 +67,6 @@ def get_list_of_mirrors(file_type, file_path, mirrors_dict): List of mirror urls corresponding to the file_type and file_path. If no match is found, empty list is returned. - """ # Checking if all the arguments have appropriate format. @@ -110,7 +108,8 @@ def get_list_of_mirrors(file_type, file_path, mirrors_dict): # the URL as UTF-8. We need a long-term solution with #61. # http://bugs.python.org/issue1712522 file_path = urllib.quote(file_path) - url = base+'/'+file_path + url = base + '/' + file_path.lstrip(os.sep) + #url = os.path.join(base, file_path.lstrip(os.sep)) list_of_mirrors.append(url) return list_of_mirrors diff --git a/tuf/pycrypto_keys.py b/tuf/pycrypto_keys.py new file mode 100755 index 0000000000..a5b59f0306 --- /dev/null +++ b/tuf/pycrypto_keys.py @@ -0,0 +1,948 @@ +""" + + pycrypto_keys.py + + + Vladimir Diaz + + + October 7, 2013. + + + See LICENSE for licensing information. + + + The goal of this module is to support public-key and general-purpose + cryptography through the PyCrypto library. The RSA-related functions provided: + generate_rsa_public_and_private() + create_rsa_signature() + verify_rsa_signature() + create_rsa_encrypted_pem() + create_rsa_public_and_private_from_encrypted_pem() + + The general-purpose functions include: + encrypt_key() + decrypt_key() + + PyCrypto (i.e., the 'Crypto' package) performs the actual cryptographic + operations and the functions listed above can be viewed as the easy-to-use + public interface. + + https://github.com/dlitz/pycrypto + https://en.wikipedia.org/wiki/RSA_(algorithm) + https://en.wikipedia.org/wiki/Advanced_Encryption_Standard + https://en.wikipedia.org/wiki/3des + https://en.wikipedia.org/wiki/PBKDF + + TUF key files are encrypted with the AES-256-CTR-Mode symmetric key + algorithm. User passwords are strengthened with PBKDF2, currently set to + 100,000 passphrase iterations. The previous evpy implementation used 1,000 + iterations. + + PEM-encrypted RSA key files use the Triple Data Encryption Algorithm (3DES) + and Cipher-block chaining (CBC) for the mode of operation. Password-Based Key + Derivation Function 1 (PBKF1) + MD5. + """ + +import os +import binascii +import json + +# Crypto.PublicKey (i.e., PyCrypto's public-key cryptography modules) supports +# algorithms like the Digital Signature Algorithm (DSA) and the ElGamal +# encryption system. 'Crypto.PublicKey.RSA' is needed here to generate, sign, +# and verify RSA keys. +import Crypto.PublicKey.RSA + +# PyCrypto requires 'Crypto.Hash' hash objects to generate PKCS#1 PSS +# signatures (i.e., Crypto.Signature.PKCS1_PSS). +import Crypto.Hash.SHA256 + +# RSA's probabilistic signature scheme with appendix (RSASSA-PSS). +# PKCS#1 v1.5 is available for compatibility with existing applications, but +# RSASSA-PSS is encouraged for newer applications. RSASSA-PSS generates +# a random salt to ensure the signature generated is probabilistic rather than +# deterministic (e.g., PKCS#1 v1.5). +# http://en.wikipedia.org/wiki/RSA-PSS#Schemes +# https://tools.ietf.org/html/rfc3447#section-8.1 +import Crypto.Signature.PKCS1_PSS + +# Import PyCrypto's Key Derivation Function (KDF) module. 'keystore.py' +# needs this module to derive a secret key according to the Password-Based +# Key Derivation Function 2 specification. The derived key is used as the +# symmetric key to encrypt TUF key information. PyCrypto's implementation: +# Crypto.Protocol.KDF.PBKDF2(). PKCS#5 v2.0 PBKDF2 specification: +# http://tools.ietf.org/html/rfc2898#section-5.2 +import Crypto.Protocol.KDF + +# PyCrypto's AES implementation. AES is a symmetric key algorithm that +# operates on fixed block sizes of 128-bits. +# https://en.wikipedia.org/wiki/Advanced_Encryption_Standard +import Crypto.Cipher.AES + +# 'Crypto.Random' is a cryptographically strong version of Python's standard +# "random" module. Random bits of data is needed for salts and +# initialization vectors suitable for the encryption algorithms used in +# 'pycrypto_keys.py'. +import Crypto.Random + +# The mode of operation is presently set to CTR (CounTeR Mode) for symmetric +# block encryption (AES-256, where the symmetric key is 256 bits). PyCrypto +# provides a callable stateful block counter that can update successive blocks +# when needed. The initial random block, or initialization vector (IV), can +# be set to begin the process of incrementing the 128-bit blocks and allowing +# the AES algorithm to perform cipher block operations on them. +import Crypto.Util.Counter + +# Import the TUF package and TUF-defined exceptions in __init__.py. +import tuf + +# Digest objects needed to generate hashes. +import tuf.hash + +# Perform object format-checking. +import tuf.formats + +# Extract the cryptography library settings. +import tuf.conf + +# Import key files containing json data. +import tuf.util + +# Recommended RSA key sizes: +# http://www.emc.com/emc-plus/rsa-labs/historical/twirl-and-rsa-key-size.htm#table1 +# According to the document above, revised May 6, 2003, RSA keys of +# size 3072 provide security through 2031 and beyond. +_DEFAULT_RSA_KEY_BITS = 3072 + +# The delimiter symbol used to separate the different sections +# of encrypted files (i.e., salt, iterations, hmac, IV, ciphertext). +# This delimiter is arbitrarily chosen and should not occur in +# the hexadecimal representations of the fields it is separating. +_ENCRYPTION_DELIMITER = '@@@@' + +# AES key size. Default key size = 32 bytes = AES-256. +_AES_KEY_SIZE = 32 + +# Default salt size, in bytes. A 128-bit salt (i.e., a random sequence of data +# to protect against attacks that use precomputed rainbow tables to crack +# password hashes) is generated for PBKDF2. +_SALT_SIZE = 16 + +# Default PBKDF2 passphrase iterations. The current "good enough" number +# of passphrase iterations. We recommend that important keys, such as root, +# be kept offline. 'tuf.conf.PBKDF2_ITERATIONS' should increase as CPU +# speeds increase, set here at 100,000 iterations by default (in 2013). +# Repository maintainers may opt to modify the default setting according to +# their security needs and computational restrictions. A strong user password +# is still important. Modifying the number of iterations will result in a new +# derived key+PBDKF2 combination if the key is loaded and re-saved, overriding +# any previous iteration setting used by the old '.key'. +# https://en.wikipedia.org/wiki/PBKDF2 +_PBKDF2_ITERATIONS = tuf.conf.PBKDF2_ITERATIONS + + +def generate_rsa_public_and_private(bits=_DEFAULT_RSA_KEY_BITS): + """ + + Generate public and private RSA keys with modulus length 'bits'. + The public and private keys returned conform to 'tuf.formats.PEMRSA_SCHEMA' + and have the form: + '-----BEGIN RSA PUBLIC KEY----- ...' + + or + + '-----BEGIN RSA PRIVATE KEY----- ...' + + The public and private keys are returned as strings in PEM format. + + Although PyCrypto sets a 1024-bit minimum key size, + generate_rsa_public_and_private() enforces a minimum key size of 2048 bits. + If 'bits' is unspecified, a 3072-bit RSA key is generated, which is the key + size recommended by TUF. + + >>> public, private = generate_rsa_public_and_private(2048) + >>> tuf.formats.PEMRSA_SCHEMA.matches(public) + True + >>> tuf.formats.PEMRSA_SCHEMA.matches(private) + True + + + bits: + The key size, or key length, of the RSA key. 'bits' must be 2048, or + greater, and a multiple of 256. + + + tuf.FormatError, if 'bits' does not contain the correct format. + + ValueError, if an exception occurs in the RSA key generation routine. + 'bits' must be a multiple of 256. The 'ValueError' exception is raised by + the PyCrypto key generation function. + + + The RSA keys are generated by PyCrypto's Crypto.PublicKey.RSA.generate(). + + + A (public, private) tuple containing the RSA keys in PEM format. + """ + + # Does 'bits' have the correct format? + # This check will ensure 'bits' conforms to 'tuf.formats.RSAKEYBITS_SCHEMA'. + # 'bits' must be an integer object, with a minimum value of 2048. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.RSAKEYBITS_SCHEMA.check_match(bits) + + # Generate the public and private RSA keys. The PyCrypto module performs + # the actual key generation. Raise 'ValueError' if 'bits' is less than 1024 + # or not a multiple of 256, although a 2048-bit minimum is enforced by + # tuf.formats.RSAKEYBITS_SCHEMA.check_match(). + rsa_key_object = Crypto.PublicKey.RSA.generate(bits) + + # Extract the public & private halves of the RSA key and generate their + # PEM-formatted representations. Return the key pair as a (public, private) + # tuple, where each RSA is a string in PEM format. + private = rsa_key_object.exportKey(format='PEM') + rsa_pubkey = rsa_key_object.publickey() + public = rsa_pubkey.exportKey(format='PEM') + + return public, private + + + + + +def create_rsa_signature(private_key, data): + """ + + Generate an RSASSA-PSS signature. The signature, and the method (signature + algorithm) used, is returned as a (signature, method) tuple. + + The signing process will use 'private_key' and 'data' to generate the + signature. + + RFC3447 - RSASSA-PSS + http://www.ietf.org/rfc/rfc3447.txt + + >>> public, private = generate_rsa_public_and_private(2048) + >>> data = 'The quick brown fox jumps over the lazy dog' + >>> signature, method = create_rsa_signature(private, data) + >>> tuf.formats.NAME_SCHEMA.matches(method) + True + >>> method == 'RSASSA-PSS' + True + >>> tuf.formats.PYCRYPTOSIGNATURE_SCHEMA.matches(method) + True + + + private_key: + The private RSA key, a string in PEM format. + + data: + Data object used by create_rsa_signature() to generate the signature. + + + tuf.FormatError, if 'private_key' is improperly formatted. + + TypeError, if 'private_key' is unset. + + tuf.CryptoError, if the signature cannot be generated. + + + PyCrypto's 'Crypto.Signature.PKCS1_PSS' called to generate the signature. + + + A (signature, method) tuple, where the signature is a string and the method + is 'RSASSA-PSS'. + """ + + # Does 'private_key' have the correct format? + # This check will ensure 'private_key' conforms to 'tuf.formats.PEMRSA_SCHEMA'. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.PEMRSA_SCHEMA.check_match(private_key) + + # Signing the 'data' object requires a private key. + # The 'RSASSA-PSS' (i.e., PyCrypto module) signing method is the + # only method currently supported. + method = 'RSASSA-PSS' + signature = None + + # Verify the signature, but only if the private key has been set. The private + # key is a NULL string if unset. Although it may be clearer to explicit check + # that 'private_key' is not '', we can/should check for a value and not + # compare identities with the 'is' keyword. Up to this point 'private_key' + # has variable size and can be an empty string. + if len(private_key): + # Calculate the SHA256 hash of 'data' and generate the hash's PKCS1-PSS + # signature. + + # PyCrypto's expected exceptions when generating RSA key object: + # "ValueError/IndexError/TypeError: When the given key cannot be parsed + # (possibly because the passphrase is wrong)." + # If the passphrase is incorrect, PyCrypto returns: "RSA key format is not + # supported". + try: + sha256_object = Crypto.Hash.SHA256.new(data) + rsa_key_object = Crypto.PublicKey.RSA.importKey(private_key) + + except (ValueError, IndexError, TypeError), e: + message = 'Invalid private key or hash data: '+str(e) + raise tuf.CryptoError(message) + + # Generate RSSA-PSS signature. Raise 'tuf.CryptoError' for the expected + # PyCrypto exceptions. + try: + pkcs1_pss_signer = Crypto.Signature.PKCS1_PSS.new(rsa_key_object) + signature = pkcs1_pss_signer.sign(sha256_object) + + except ValueError: + raise tuf.CryptoError('The RSA key too small for given hash algorithm.') + + except TypeError: + raise tuf.CryptoError('Missing required RSA private key.') + + except IndexError: + message = 'An RSA signature cannot be generated: '+str(e) + raise tuf.CryptoError(message) + + else: + raise TypeError('The required private key is unset.') + + return signature, method + + + + + +def verify_rsa_signature(signature, signature_method, public_key, data): + """ + + Determine whether the corresponding private key of 'public_key' produced + 'signature'. verify_signature() will use the public key, signature method, + and 'data' to complete the verification. + + >>> public, private = generate_rsa_public_and_private(2048) + >>> data = 'The quick brown fox jumps over the lazy dog' + >>> signature, method = create_rsa_signature(private, data) + >>> verify_rsa_signature(signature, method, public, data) + True + >>> verify_rsa_signature(signature, method, public, 'bad_data') + False + + + signature: + An RSASSA PSS signature as a string. This is the signature returned + by create_rsa_signature(). + + signature_method: + A string that indicates the signature algorithm used to generate + 'signature'. 'RSASSA-PSS' is currently supported. + + public_key: + The RSA public key, a string in PEM format. + + data: + Data object used by tuf.keys.create_signature() to generate + 'signature'. 'data' is needed here to verify the signature. + + + tuf.UnknownMethodError. Raised if the signing method used by + 'signature' is not one supported by tuf.keys.create_signature(). + + tuf.FormatError. Raised if 'signature', 'signature_method', or 'public_key' + is improperly formatted. + + + Crypto.Signature.PKCS1_PSS.verify() called to do the actual verification. + + + Boolean. True if the signature is valid, False otherwise. + """ + + # Does 'public_key' have the correct format? + # This check will ensure 'public_key' conforms to 'tuf.formats.PEMRSA_SCHEMA'. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.PEMRSA_SCHEMA.check_match(public_key) + + # Does 'signature_method' have the correct format? + tuf.formats.NAME_SCHEMA.check_match(signature_method) + + # Does 'signature' have the correct format? + tuf.formats.PYCRYPTOSIGNATURE_SCHEMA.check_match(signature) + + # Verify whether the private key of 'public_key' produced 'signature'. + # Before returning the 'valid_signature' Boolean result, ensure 'RSASSA-PSS' + # was used as the signing method. + valid_signature = False + + # Verify the signature with PyCrypto if the signature method is valid, + # otherwise raise 'tuf.UnknownMethodError'. + if signature_method == 'RSASSA-PSS': + try: + rsa_key_object = Crypto.PublicKey.RSA.importKey(public_key) + pkcs1_pss_verifier = Crypto.Signature.PKCS1_PSS.new(rsa_key_object) + sha256_object = Crypto.Hash.SHA256.new(data) + valid_signature = pkcs1_pss_verifier.verify(sha256_object, signature) + + except (ValueError, IndexError, TypeError), e: + message = 'The RSA signature could not be verified.' + raise tuf.CryptoError(message) + + else: + raise tuf.UnknownMethodError(signature_method) + + return valid_signature + + + + + +def create_rsa_encrypted_pem(private_key, passphrase): + """ + + Return a string in PEM format, where the private part of the RSA key is + encrypted. The private part of the RSA key is encrypted by the Triple + Data Encryption Algorithm (3DES) and Cipher-block chaining (CBC) for the + mode of operation. Password-Based Key Derivation Function 1 (PBKF1) + MD5 + is used to strengthen 'passphrase'. + + https://en.wikipedia.org/wiki/Triple_DES + https://en.wikipedia.org/wiki/PBKDF2 + + >>> public, private = generate_rsa_public_and_private(2048) + >>> passphrase = 'secret' + >>> encrypted_pem = create_rsa_encrypted_pem(private, passphrase) + >>> tuf.formats.PEMRSA_SCHEMA.matches(encrypted_pem) + True + + + private_key: + The private key string in PEM format. + + passphrase: + The passphrase, or password, to encrypt the private part of the RSA + key. 'passphrase' is not used directly as the encryption key, a stronger + encryption key is derived from it. + + + tuf.FormatError, if the arguments are improperly formatted. + + tuf.CryptoError, if an RSA key in encrypted PEM format cannot be created. + + TypeError, 'private_key' is unset. + + + PyCrypto's Crypto.PublicKey.RSA.exportKey() called to perform the actual + generation of the PEM-formatted output. + + + A string in PEM format, where the private RSA key is encrypted. + Conforms to 'tuf.formats.PEMRSA_SCHEMA'. + """ + + # Does 'private_key' have the correct format? + # This check will ensure 'private_key' has the appropriate number + # of objects and object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.PEMRSA_SCHEMA.check_match(private_key) + + # Does 'passphrase' have the correct format? + tuf.formats.PASSWORD_SCHEMA.check_match(passphrase) + + # 'private_key' is in PEM format and unencrypted. The extracted key will be + # imported and converted to PyCrypto's RSA key object + # (i.e., Crypto.PublicKey.RSA). Use PyCrypto's exportKey method, with a + # passphrase specified, to create the string. PyCrypto uses PBKDF1+MD5 to + # strengthen 'passphrase', and 3DES with CBC mode for encryption. + # 'private_key' may still be a NULL string after the + # 'tuf.formats.PEMRSA_SCHEMA' (i.e., 'private_key' has variable size and can + # be an empty string. + + if len(private_key): + try: + rsa_key_object = Crypto.PublicKey.RSA.importKey(private_key) + encrypted_pem = rsa_key_object.exportKey(format='PEM', + passphrase=passphrase) + + except (ValueError, IndexError, TypeError), e: + message = 'An encrypted RSA key in PEM format cannot be generated: '+str(e) + raise tuf.CryptoError(message) + + else: + raise TypeError('The required private key is unset.') + + + return encrypted_pem + + + + + +def create_rsa_public_and_private_from_encrypted_pem(encrypted_pem, passphrase): + """ + + Generate public and private RSA keys from an encrypted PEM. + The public and private keys returned conform to 'tuf.formats.PEMRSA_SCHEMA' + and have the form: + + '-----BEGIN RSA PUBLIC KEY----- ...' + + or + + '-----BEGIN RSA PRIVATE KEY----- ...' + + The public and private keys are returned as strings in PEM format. + + The private key part of 'encrypted_pem' is encrypted. PyCrypto's importKey + method is used, where a passphrase is specified. PyCrypto uses PBKDF1+MD5 + to strengthen 'passphrase', and 3DES with CBC mode for encryption/decryption. + Alternatively, key data may be encrypted with AES-CTR-Mode and the passphrase + strengthened with PBKDF2+SHA256. See 'keystore.py'. + + >>> public, private = generate_rsa_public_and_private(2048) + >>> passphrase = 'secret' + >>> encrypted_pem = create_rsa_encrypted_pem(private, passphrase) + >>> returned_public, returned_private = \ + create_rsa_public_and_private_from_encrypted_pem(encrypted_pem, passphrase) + >>> tuf.formats.PEMRSA_SCHEMA.matches(returned_public) + True + >>> tuf.formats.PEMRSA_SCHEMA.matches(returned_private) + True + >>> public == returned_public + True + >>> private == returned_private + True + + + encrypted_pem: + A byte string in PEM format, where the private key is encrypted. It has + the form: + + '-----BEGIN RSA PRIVATE KEY-----\n + Proc-Type: 4,ENCRYPTED\nDEK-Info: DES-EDE3-CBC ...' + + passphrase: + The passphrase, or password, to decrypt the private part of the RSA + key. 'passphrase' is not directly used as the encryption key, instead + it is used to derive a stronger symmetric key. + + + tuf.FormatError, if the arguments are improperly formatted. + + tuf.CryptoError, if the public and private RSA keys cannot be generated + from 'encrypted_pem', or exported in PEM format. + + + PyCrypto's 'Crypto.PublicKey.RSA.importKey()' called to perform the actual + conversion from an encrypted RSA private key. + + + A (public, private) tuple containing the RSA keys in PEM format. + """ + + # Does 'encryped_pem' have the correct format? + # This check will ensure 'encrypted_pem' has the appropriate number + # of objects and object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.PEMRSA_SCHEMA.check_match(encrypted_pem) + + # Does 'passphrase' have the correct format? + tuf.formats.PASSWORD_SCHEMA.check_match(passphrase) + + # Generate a PyCrypto key object from 'encrypted_pem'. The generated PyCrypto + # key contains the required export methods needed to generate the + # PEM-formatted representations of the public and private RSA key. + try: + rsa_key_object = Crypto.PublicKey.RSA.importKey(encrypted_pem, passphrase) + + # PyCrypto's expected exceptions: + # "ValueError/IndexError/TypeError: When the given key cannot be parsed + # (possibly because the passphrase is wrong)." + # If the passphrase is incorrect, PyCrypto returns: "RSA key format is not + # supported". + except (ValueError, IndexError, TypeError), e: + message = 'RSA (public, private) tuple cannot be generated from the'+\ + ' encrypted PEM string: '+str(e) + # Raise 'tuf.CryptoError' and PyCrypto's exception message. Avoid + # propogating PyCrypto's exception trace to avoid revealing sensitive error. + raise tuf.CryptoError(message) + + # Export the public and private halves of the PyCrypto RSA key object. The + # (public, private) tuple returned contains the public and private RSA keys + # in PEM format, as strings. + try: + private = rsa_key_object.exportKey(format='PEM') + rsa_pubkey = rsa_key_object.publickey() + public = rsa_pubkey.exportKey(format='PEM') + + # PyCrypto raises 'ValueError' if the public or private keys cannot be + # exported. See 'Crypto.PublicKey.RSA'. + except (ValueError): + message = 'The public and private keys cannot be exported in PEM format.' + raise tuf.CryptoError(message) + + return public, private + + + + + +def encrypt_key(key_object, password): + """ + + Return a string containing 'key_object' in encrypted form. Encrypted strings + may be safely saved to a file. The corresponding decrypt_key() function can + be applied to the encrypted string to restore the original key object. + 'key_object' is a TUF key (e.g., RSAKEY_SCHEMA, ED25519KEY_SCHEMA). This + function calls the PyCrypto library to perform the encryption and derive + a suitable encryption key. + + Whereas an encrypted PEM file uses the Triple Data Encryption Algorithm + (3DES), the Cipher-block chaining (CBC) mode of operation, and the Password + Based Key Derivation Function 1 (PBKF1) + MD5 to strengthen 'password', + encrypted TUF keys use AES-256-CTR-Mode and passwords strengthened with + PBKDF2-HMAC-SHA256 (100K iterations by default, but may be overriden in + 'tuf.conf.PBKDF2_ITERATIONS' by the user). + + http://en.wikipedia.org/wiki/Advanced_Encryption_Standard + http://en.wikipedia.org/wiki/CTR_mode#Counter_.28CTR.29 + https://en.wikipedia.org/wiki/PBKDF2 + + >>> ed25519_key = {'keytype': 'ed25519', \ + 'keyid': \ + 'd62247f817883f593cf6c66a5a55292488d457bcf638ae03207dbbba9dbe457d', \ + 'keyval': {'public': \ + '74addb5ad544a4306b34741bc1175a3613a8d7dc69ff64724243efdec0e301ad', \ + 'private': \ + '1f26964cc8d4f7ee5f3c5da2fbb7ab35811169573ac367b860a537e47789f8c4'}} + >>> passphrase = 'secret' + >>> encrypted_key = encrypt_key(ed25519_key, passphrase) + >>> tuf.formats.ENCRYPTEDKEY_SCHEMA.matches(encrypted_key) + True + + + key_object: + The TUF key object that should contain the private portion of the ED25519 + key. + + password: + The password, or passphrase, to encrypt the private part of the RSA + key. 'password' is not used directly as the encryption key, a stronger + encryption key is derived from it. + + + tuf.FormatError, if any of the arguments are improperly formatted or + 'key_object' does not contain the private portion of the key. + + tuf.CryptoError, if an ED25519 key in encrypted TUF format cannot be + created. + + + PyCrypto cryptographic operations called to perform the actual encryption of + 'key_object'. 'password' used to derive a suitable encryption key. + + + An encrypted string in 'tuf.formats.ENCRYPTEDKEY_SCHEMA' format. + """ + + # Do the arguments have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.ANYKEY_SCHEMA.check_match(key_object) + + # Does 'password' have the correct format? + tuf.formats.PASSWORD_SCHEMA.check_match(password) + + # Ensure the private portion of the key is included in 'key_object'. + if not key_object['keyval']['private']: + message = 'Key object does not contain a private part.' + raise tuf.FormatError(message) + + # Derive a key (i.e., an appropriate encryption key and not the + # user's password) from the given 'password'. Strengthen 'password' with + # PBKDF2-HMAC-SHA256 (100K iterations by default, but may be overriden in + # 'tuf.conf.PBKDF2_ITERATIONS' by the user). + salt, iterations, derived_key = _generate_derived_key(password) + + # Store the derived key info in a dictionary, the object expected + # by the non-public _encrypt() routine. + derived_key_information = {'salt': salt, 'iterations': iterations, + 'derived_key': derived_key} + + # Convert the key object to json string format and encrypt it with the + # derived key. + encrypted_key = _encrypt(json.dumps(key_object), derived_key_information) + + return encrypted_key + + + + + +def decrypt_key(encrypted_key, password): + """ + + + Return a string containing 'encrypted_key' in non-encrypted form. + The decrypt_key() function can be applied to the encrypted string to restore + the original key object, a TUF key (e.g., RSAKEY_SCHEMA, ED25519KEY_SCHEMA). + This function calls the appropriate cryptography module (e.g., + pycrypto_keys.py) to perform the decryption. + + Encrypted TUF keys use AES-256-CTR-Mode and passwords strengthened with + PBKDF2-HMAC-SHA256 (100K iterations be default, but may be overriden in + 'tuf.conf.py' by the user). + + http://en.wikipedia.org/wiki/Advanced_Encryption_Standard + http://en.wikipedia.org/wiki/CTR_mode#Counter_.28CTR.29 + https://en.wikipedia.org/wiki/PBKDF2 + + >>> ed25519_key = {'keytype': 'ed25519', \ + 'keyid': \ + 'd62247f817883f593cf6c66a5a55292488d457bcf638ae03207dbbba9dbe457d', \ + 'keyval': {'public': \ + '74addb5ad544a4306b34741bc1175a3613a8d7dc69ff64724243efdec0e301ad', \ + 'private': \ + '1f26964cc8d4f7ee5f3c5da2fbb7ab35811169573ac367b860a537e47789f8c4'}} + >>> passphrase = 'secret' + >>> encrypted_key = encrypt_key(ed25519_key, passphrase) + >>> decrypted_key = decrypt_key(encrypted_key, passphrase) + >>> tuf.formats.ED25519KEY_SCHEMA.matches(decrypted_key) + True + >>> decrypted_key == ed25519_key + True + + + encrypted_key: + An encrypted TUF key (additional data is also included, such as salt, + number of password iterations used for the derived encryption key, etc) + of the form 'tuf.formats.ENCRYPTEDKEY_SCHEMA'. 'encrypted_key' should + have been generated with encrypted_key(). + + password: + The password, or passphrase, to encrypt the private part of the RSA + key. 'password' is not used directly as the encryption key, a stronger + encryption key is derived from it. + + + tuf.FormatError, if the arguments are improperly formatted. + + tuf.CryptoError, if a TUF key cannot be decrypted from 'encrypted_key'. + + tuf.Error, if a valid TUF key object is not found in 'encrypted_key'. + + + The PyCrypto library called to perform the actual decryption of + 'encrypted_key'. The key derivation data stored in 'encrypted_key' is used + to re-derive the encryption/decryption key. + + + The decrypted key object in 'tuf.formats.ANYKEY_SCHEMA' format. + """ + + # Do the arguments have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.ENCRYPTEDKEY_SCHEMA.check_match(encrypted_key) + + # Does 'password' have the correct format? + tuf.formats.PASSWORD_SCHEMA.check_match(password) + + # Decrypt 'encrypted_key', using 'password' (and additional key derivation + # data like salts and password iterations) to re-derive the decryption key. + json_data = _decrypt(encrypted_key, password) + + # Raise 'tuf.Error' if 'json_data' cannot be deserialized to a valid + # 'tuf.formats.ANYKEY_SCHEMA' key object. + key_object = tuf.util.load_json_string(json_data) + + return key_object + + + + + +def _generate_derived_key(password, salt=None, iterations=None): + """ + Generate a derived key by feeding 'password' to the Password-Based Key + Derivation Function (PBKDF2). PyCrypto's PBKDF2 implementation is + currently used. 'salt' may be specified so that a previous derived key + may be regenerated. + """ + + if salt is None: + salt = Crypto.Random.new().read(_SALT_SIZE) + + if iterations is None: + iterations = _PBKDF2_ITERATIONS + + + def pseudorandom_function(password, salt): + """ + PyCrypto's PBKDF2() expects a callable function for its optional + 'prf' argument. 'prf' is set to HMAC-SHA1 (in PyCrypto's PBKDF2 function) + by default. 'pseudorandom_function' instead sets 'prf' to HMAC-SHA256. + """ + + return Crypto.Hash.HMAC.new(password, salt, Crypto.Hash.SHA256).digest() + + + # 'dkLen' is the desired key length. 'count' is the number of password + # iterations performed by PBKDF2. 'prf' is a pseudorandom function, which + # must be callable. + derived_key = Crypto.Protocol.KDF.PBKDF2(password, salt, + dkLen=_AES_KEY_SIZE, + count=iterations, + prf=pseudorandom_function) + + return salt, iterations, derived_key + + + + + +def _encrypt(key_data, derived_key_information): + """ + Encrypt 'key_data' using the Advanced Encryption Standard (AES-256) algorithm. + 'derived_key_information' should contain a key strengthened by PBKDF2. The + key size is 256 bits and AES's mode of operation is set to CTR (CounTeR Mode). + The HMAC of the ciphertext is generated to ensure the ciphertext has not been + modified. + + 'key_data' is the JSON string representation of the key. In the case + of RSA keys, this format would be 'tuf.formats.RSAKEY_SCHEMA': + {'keytype': 'rsa', + 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', + 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} + + 'derived_key_information' is a dictionary of the form: + {'salt': '...', + 'derived_key': '...', + 'iterations': '...'} + + 'tuf.CryptoError' raised if the encryption fails. + """ + + # Generate a random initialization vector (IV). The 'iv' is treated as the + # initial counter block to a stateful counter block function (i.e., + # PyCrypto's 'Crypto.Util.Counter'). The AES block cipher operates on 128-bit + # blocks, so generate a random 16-byte initialization block. PyCrypto expects + # the initial value of the stateful counter to be an integer. + # Follow the provably secure encrypt-then-MAC approach, which affords the + # ability to verify ciphertext without needing to decrypt it and preventing + # an attacker from feeding the block cipher malicious data. Modes like GCM + # provide both encryption and authentication, whereas CTR only provides + # encryption. + iv = Crypto.Random.new().read(16) + stateful_counter_128bit_blocks = Crypto.Util.Counter.new(128, + initial_value=long(iv.encode('hex'), 16)) + symmetric_key = derived_key_information['derived_key'] + aes_cipher = Crypto.Cipher.AES.new(symmetric_key, + Crypto.Cipher.AES.MODE_CTR, + counter=stateful_counter_128bit_blocks) + + # Use AES-256 to encrypt 'key_data'. The key size determines how many cycle + # repetitions are performed by AES, 14 cycles for 256-bit keys. + try: + ciphertext = aes_cipher.encrypt(key_data) + + # PyCrypto does not document the exceptions that may be raised or under + # what circumstances. PyCrypto example given is to call encrypt() without + # checking for exceptions. Avoid propogating the exception trace and only + # raise 'tuf.CryptoError', along with the cause of encryption failure. + except (ValueError, IndexError, TypeError), e: + message = 'The key data cannot be encrypted: '+str(e) + raise tuf.CryptoError(message) + + # Generate the hmac of the ciphertext to ensure it has not been modified. + # The decryption routine may verify a ciphertext without having to perform + # a decryption operation. + salt = derived_key_information['salt'] + hmac_object = Crypto.Hash.HMAC.new(symmetric_key, ciphertext, + Crypto.Hash.SHA256) + hmac = hmac_object.hexdigest() + + # Store the number of PBKDF2 iterations used to derive the symmetric key so + # that the decryption routine can regenerate the symmetric key successfully. + # The pbkdf2 iterations are allowed to vary for the keys loaded and saved. + iterations = derived_key_information['iterations'] + + # Return the salt, iterations, hmac, initialization vector, and ciphertext + # as a single string. These five values are delimited by + # '_ENCRYPTION_DELIMITER' to make extraction easier. This delimiter is + # arbitrarily chosen and should not occur in the hexadecimal representations + # of the fields it is separating. + return binascii.hexlify(salt) + _ENCRYPTION_DELIMITER + \ + binascii.hexlify(str(iterations)) + _ENCRYPTION_DELIMITER + \ + binascii.hexlify(hmac) + _ENCRYPTION_DELIMITER + \ + binascii.hexlify(iv) + _ENCRYPTION_DELIMITER + \ + binascii.hexlify(ciphertext) + + + + + +def _decrypt(file_contents, password): + """ + The corresponding decryption routine for _encrypt(). + + 'tuf.CryptoError' raised if the decryption fails. + """ + + # Extract the salt, iterations, hmac, initialization vector, and ciphertext + # from 'file_contents'. These five values are delimited by + # '_ENCRYPTION_DELIMITER'. This delimiter is arbitrarily chosen and should + # not occur in the hexadecimal representations of the fields it is separating. + salt, iterations, hmac, iv, ciphertext = \ + file_contents.split(_ENCRYPTION_DELIMITER) + + # Ensure we have the expected raw data for the delimited cryptographic data. + salt = binascii.unhexlify(salt) + iterations = int(binascii.unhexlify(iterations)) + hmac = binascii.unhexlify(hmac) + iv = binascii.unhexlify(iv) + ciphertext = binascii.unhexlify(ciphertext) + + # Generate derived key from 'password'. The salt and iterations are specified + # so that the expected derived key is regenerated correctly. Discard the old + # "salt" and "iterations" values, as we only need the old derived key. + junk_old_salt, junk_old_iterations, derived_key = \ + _generate_derived_key(password, salt, iterations) + + # Verify the hmac to ensure the ciphertext is valid and has not been altered. + # See the encryption routine for why we use the encrypt-then-MAC approach. + generated_hmac_object = Crypto.Hash.HMAC.new(derived_key, ciphertext, + Crypto.Hash.SHA256) + generated_hmac = generated_hmac_object.hexdigest() + + if generated_hmac != hmac: + raise tuf.CryptoError('Decryption failed.') + + # The following decryption routine assumes 'ciphertext' was encrypted with + # AES-256. + stateful_counter_128bit_blocks = Crypto.Util.Counter.new(128, + initial_value=long(iv.encode('hex'), 16)) + aes_cipher = Crypto.Cipher.AES.new(derived_key, + Crypto.Cipher.AES.MODE_CTR, + counter=stateful_counter_128bit_blocks) + try: + key_plaintext = aes_cipher.decrypt(ciphertext) + + # PyCrypto does not document the exceptions that may be raised or under + # what circumstances. PyCrypto example given is to call decrypt() without + # checking for exceptions. Avoid propogating the exception trace and only + # raise 'tuf.CryptoError', along with the cause of decryption failure. + except (ValueError, IndexError, TypeError), e: + raise tuf.CryptoError('Decryption failed: '+str(e)) + + return key_plaintext + + + +if __name__ == '__main__': + # The interactive sessions of the documentation strings can + # be tested by running 'pycrypto_keys.py' as a standalone module: + # $ python pycrypto_keys.py + import doctest + doctest.testmod() diff --git a/tuf/repo/keystore.py b/tuf/repo/keystore.py index 8f5492e52d..b16f7ca127 100755 --- a/tuf/repo/keystore.py +++ b/tuf/repo/keystore.py @@ -34,7 +34,6 @@ algorithm. User passwords are strengthened with PBKDF2, currently set to 100,000 passphrase iterations. The previous evpy implementation used 1,000 iterations. - """ import os @@ -68,7 +67,7 @@ # the AES algorithm to perform cipher block operations on them. import Crypto.Util.Counter -import tuf.rsa_key +import tuf.keys import tuf.util import tuf.conf @@ -104,6 +103,9 @@ # https://en.wikipedia.org/wiki/PBKDF2 _PBKDF2_ITERATIONS = tuf.conf.PBKDF2_ITERATIONS +# +_SUPPORTED_KEY_TYPES = ['rsa', 'ed25519'] + # A user password is read and a derived key generated. The derived key returned # by the key derivation function (PBKDF2) is saved in '_derived_keys', along # with the salt and iterations used, which has the form: @@ -159,7 +161,6 @@ def add_rsakey(rsakey_dict, password, keyid=None): None. - """ # Does 'rsakey_dict' have the correct format? @@ -235,7 +236,6 @@ def load_keystore_from_keyfiles(directory_name, keyids, passwords): A list containing the keyids of the loaded keys. - """ # Does 'directory_name' have the correct format? @@ -286,11 +286,11 @@ def load_keystore_from_keyfiles(directory_name, keyids, passwords): # Create the key based on its key type. RSA keys currently # supported. - if keydata['keytype'] == 'rsa': + if keydata['keytype'] in _SUPPORTED_KEY_TYPES: # 'keydata' is stored in KEY_SCHEMA format. Call - # create_from_metadata_format() to get the key in RSAKEY_SCHEMA + # format_metadata_to_key() to get the key in RSAKEY_SCHEMA # format, which is the format expected by 'add_rsakey()'. - rsa_key = tuf.rsa_key.create_from_metadata_format(keydata) + rsa_key = tuf.keys.format_metadata_to_key(keydata) # Ensure the keyid for 'rsa_key' is one of the keys specified in # 'keyids'. If not, do not load the key. @@ -343,7 +343,6 @@ def save_keystore_to_keyfiles(directory_name): None. - """ # Does 'directory_name' have the correct format? @@ -365,9 +364,11 @@ def save_keystore_to_keyfiles(directory_name): file_object = open(basefilename, 'w') # Determine the appropriate format to save the key based on its key type. - if key['keytype'] == 'rsa': + if key['keytype'] in _SUPPORTED_KEY_TYPES: + keytype = key['keytype'] + keyval = key['keyval'] key_metadata_format = \ - tuf.rsa_key.create_in_metadata_format(key['keyval'], private=True) + tuf.keys.format_keyval_to_metadata(keytype, keyval, private=True) else: logger.warn('The keystore has a key with an unrecognized key type.') continue @@ -402,7 +403,6 @@ def clear_keystore(): None. - """ _keystore.clear() @@ -442,7 +442,6 @@ def change_password(keyid, old_password, new_password): None. - """ # Does 'keyid' have the correct format? @@ -506,7 +505,6 @@ def get_key(keyid): The key belonging to 'keyid' (e.g., RSA key). - """ # Does 'keyid' have the correct format? @@ -530,7 +528,6 @@ def _generate_derived_key(password, salt=None, iterations=None): Derivation Function (PBKDF2). PyCrypto's PBKDF2 implementation is currently used. 'salt' may be specified so that a previous derived key may be regenerated. - """ if salt is None: @@ -584,7 +581,6 @@ def _encrypt(key_data, derived_key_information): 'iterations': '...'} 'tuf.CryptoError' raised if the encryption fails. - """ # Generate a random initialization vector (IV). The 'iv' is treated as the @@ -650,7 +646,6 @@ def _decrypt(file_contents, password): The corresponding decryption routine for _encrypt(). 'tuf.CryptoError' raised if the decryption fails. - """ # Extract the salt, iterations, hmac, initialization vector, and ciphertext diff --git a/tuf/repo/signercli.py b/tuf/repo/signercli.py index 5c36377b8a..70013e88fb 100755 --- a/tuf/repo/signercli.py +++ b/tuf/repo/signercli.py @@ -16,8 +16,8 @@ Provide an interactive command-line interface to create and sign metadata. This script can be used to create all of the top-level role files required - by TUF, which include 'root.txt', 'targets.txt', 'release.txt', and - 'timestamp.txt'. It also provides options to generate RSA keys, change the + by TUF, which include 'root.json', 'targets.json', 'release.json', and + 'timestamp.json'. It also provides options to generate RSA keys, change the encryption/decryption keys of encrypted key files, list the keyids of the signing keys stored in a keystore directory, create delegated roles, and dump the contents of signing keys (i.e., public and private keys, key @@ -33,7 +33,7 @@ Initially, the 'quickstart.py' script is utilized when the repository is first created. 'signercli.py' would then be executed to update the state of the repository. For example, the repository owner wants to change the - 'targets.txt' signing key. The owner would run 'signercli.py' to + 'targets.json' signing key. The owner would run 'signercli.py' to generate a new RSA key, add the new key to the configuration file created by 'quickstart.py', and then run 'signercli.py' to update the metadata files. @@ -46,7 +46,6 @@ See the parse_options() function for the full list of supported options. - """ import os @@ -92,7 +91,6 @@ def _get_password(prompt='Password: ', confirm=False): is True, the user is asked to enter the previously entered password once again. If they match, the password is returned to the caller. - """ while True: @@ -131,7 +129,6 @@ def _get_metadata_directory(): returned to the caller. 'tuf.FormatError' is raised if the directory is not properly formatted, and 'tuf.Error' if it does not exist. - """ metadata_directory = _prompt('\nEnter the metadata directory: ', str) @@ -151,10 +148,9 @@ def _list_keyids(keystore_directory, metadata_directory): It is assumed the directory arguments exist and have been validated by the caller. The keyids are listed without the '.key' extension, along with their associated roles. - """ - # Determine the 'root.txt' filename. This metadata file is needed + # Determine the 'root.json' filename. This metadata file is needed # to extract the keyids belonging to the top-level roles. filenames = tuf.repo.signerlib.get_metadata_filenames(metadata_directory) root_filename = filenames['root'] @@ -203,7 +199,7 @@ def _list_keyids(keystore_directory, metadata_directory): # Is 'keyid' listed in any of the top-level roles? for top_level_role in top_level_keyids: if keyid in top_level_keyids[top_level_role]['keyids']: - # To avoid a duplicate, ignore the 'targets.txt' role for now. + # To avoid a duplicate, ignore the 'targets.json' role for now. # 'targets_keyids' will also contain the keyids for this top-level role. if top_level_role != 'targets': keyids_dict[keyid].append(top_level_role) @@ -233,7 +229,6 @@ def _get_keyids(keystore_directory): key files are stored in encrypted form, the user is asked to enter the password that was used to encrypt the key file. - """ # The keyids list containing the keys loaded. @@ -288,7 +283,6 @@ def _get_all_config_keyids(config_filepath, keystore_directory): loaded_keyids = {'root': [1233d3d, 598djdks, ..], 'release': [sdfsd323, sdsd9090s, ..] ...} - """ # Save the 'load_keystore_from_keyfiles' function call. @@ -338,7 +332,6 @@ def _get_role_config_keyids(config_filepath, keystore_directory, role): tuf.Error, if the required keys could not be loaded. - """ # Save the 'load_keystore_from_keyfiles' function call. @@ -395,7 +388,7 @@ def _sign_and_write_metadata(metadata, keyids, filename): signable = tuf.repo.signerlib.sign_metadata(metadata, keyids, filename) # Write the 'signable' object to 'filename'. The 'filename' file is - # the final metadata file, such as 'root.txt' and 'targets.txt'. + # the final metadata file, such as 'root.json' and 'targets.json'. tuf.repo.signerlib.write_metadata_file(signable, filename) @@ -409,7 +402,6 @@ def _get_metadata_version(metadata_filename): 'metadata_filename' does not exist, return a version value of 1. Raise 'tuf.RepositoryError' if 'metadata_filename' cannot be read or validated. - """ # If 'metadata_filename' does not exist on the repository, this means @@ -442,7 +434,6 @@ def _get_metadata_expiration(): tuf.RepositoryError, if the entered expiration date is invalid. - """ message = '\nCurrent time: '+tuf.formats.format_time(time.time())+'.\n'+\ @@ -487,7 +478,6 @@ def change_password(keystore_directory): None. - """ # Save the 'load_keystore_from_keyfiles' function call. @@ -496,7 +486,7 @@ def change_password(keystore_directory): # Verify the 'keystore_directory' argument. keystore_directory = _check_directory(keystore_directory) - # Retrieve the metadata directory. The 'root.txt' and all the targets + # Retrieve the metadata directory. The 'root.json' and all the targets # metadata are needed to extract rolenames and their corresponding # keyids. try: @@ -563,7 +553,6 @@ def generate_rsa_key(keystore_directory): None. - """ # Save a reference to the generate_and_save_rsa_key() function. @@ -612,13 +601,12 @@ def list_signing_keys(keystore_directory): None. - """ # Verify the 'keystore_directory' argument. keystore_directory = _check_directory(keystore_directory) - # Retrieve the metadata directory. The 'root.txt' file and all the metadata + # Retrieve the metadata directory. The 'root.json' file and all the metadata # for the targets roles are needed to extract rolenames and their associated # keyids. try: @@ -654,7 +642,6 @@ def dump_key(keystore_directory): None. - """ # Save the 'load_keystore_from_keyfiles' function call. @@ -663,7 +650,7 @@ def dump_key(keystore_directory): # Verify the 'keystore_directory' argument. keystore_directory = _check_directory(keystore_directory) - # Retrieve the metadata directory. The 'root.txt' and all the targets + # Retrieve the metadata directory. The 'root.json' and all the targets # role metadata files are needed to extract rolenames and their corresponding # keyids. try: @@ -704,8 +691,11 @@ def dump_key(keystore_directory): # Retrieve the key metadata according to the keytype. if key['keytype'] == 'rsa': - key_metadata = tuf.rsa_key.create_in_metadata_format(key['keyval'], - private=show_private) + keytype = key['keytype'] + keyval = key['keyval'] + key_metadata = tuf.keys.format_keyval_to_metadata(keytype, keyval, + #key_metadata = tuf.keys.create_in_metadata_format(keytype, keyval, + private=show_private) else: message = 'The keystore contains an invalid key type.' raise tuf.RepositoryError(message) @@ -720,7 +710,7 @@ def dump_key(keystore_directory): def make_root_metadata(keystore_directory): """ - Create the 'root.txt' file. + Create the 'root.json' file. keystore_directory: @@ -737,7 +727,6 @@ def make_root_metadata(keystore_directory): None. - """ # Verify the 'keystore_directory' argument. @@ -770,7 +759,7 @@ def make_root_metadata(keystore_directory): raise tuf.RepositoryError(message) root_keyids = loaded_keyids['root'] - # Generate the root metadata and write it to 'root.txt'. + # Generate the root metadata and write it to 'root.json'. try: tuf.repo.signerlib.build_root_file(config_filepath, root_keyids, metadata_directory, version) @@ -784,7 +773,7 @@ def make_root_metadata(keystore_directory): def make_targets_metadata(keystore_directory): """ - Create the 'targets.txt' metadata file. The targets must exist at the + Create the 'targets.json' metadata file. The targets must exist at the same path they should on the repository. This takes a list of targets. We're not worrying about custom metadata at the moment. It's allowed to not provide keys. @@ -804,7 +793,6 @@ def make_targets_metadata(keystore_directory): None. - """ # Verify the 'keystore_directory' argument. @@ -852,7 +840,7 @@ def make_targets_metadata(keystore_directory): raise tuf.RepositoryError(message) try: - # Create, sign, and write the "targets.txt" file. + # Create, sign, and write the "targets.json" file. tuf.repo.signerlib.build_targets_file(targets, targets_keyids, metadata_directory, version, expiration_date) @@ -869,7 +857,7 @@ def make_release_metadata(keystore_directory): """ Create the release metadata file. - The minimum metadata must exist. This is root.txt and targets.txt. + The minimum metadata must exist. This is root.json and targets.json. keystore_directory: @@ -886,7 +874,6 @@ def make_release_metadata(keystore_directory): None. - """ # Verify the 'keystore_directory' argument. @@ -920,7 +907,7 @@ def make_release_metadata(keystore_directory): try: release_keyids = _get_role_config_keyids(config_filepath, keystore_directory, 'release') - # Generate the release metadata and write it to 'release.txt' + # Generate the release metadata and write it to 'release.json' tuf.repo.signerlib.build_release_file(release_keyids, metadata_directory, version, expiration_date) except (tuf.FormatError, tuf.Error), e: @@ -934,7 +921,7 @@ def make_release_metadata(keystore_directory): def make_timestamp_metadata(keystore_directory): """ - Create the timestamp metadata file. The 'release.txt' file must exist. + Create the timestamp metadata file. The 'release.json' file must exist. keystore_directory: @@ -951,7 +938,6 @@ def make_timestamp_metadata(keystore_directory): None. - """ # Verify the 'keystore_directory' argument. @@ -986,7 +972,7 @@ def make_timestamp_metadata(keystore_directory): try: timestamp_keyids = _get_role_config_keyids(config_filepath, keystore_directory, 'timestamp') - # Generate the timestamp metadata and write it to 'timestamp.txt' + # Generate the timestamp metadata and write it to 'timestamp.json' tuf.repo.signerlib.build_timestamp_file(timestamp_keyids, metadata_directory, version, expiration_date) except (tuf.FormatError, tuf.Error), e: @@ -1017,13 +1003,12 @@ def sign_metadata_file(keystore_directory): None. - """ # Verify the 'keystore_directory' argument. keystore_directory = _check_directory(keystore_directory) - # Retrieve the metadata directory. The 'root.txt' and all the targets + # Retrieve the metadata directory. The 'root.json' and all the targets # role metadata files are needed to extract rolenames and their corresponding # keyids. try: @@ -1084,7 +1069,6 @@ def make_delegation(keystore_directory): None. - """ # Verify the 'keystore_directory' argument. @@ -1154,7 +1138,6 @@ def _load_parent_role(metadata_directory, keystore_directory, targets_roles): list of known targets roles and asked to enter the parent role to load. Ensure the parent role is loaded properly and return a string containing the parent role's full rolename and a list of keyids belonging to the parent. - """ # 'load_key' is a reference to the 'load_keystore_from_keyfiles function'. @@ -1210,7 +1193,6 @@ def _get_delegated_role(keystore_directory, metadata_directory): a list of keyids available in the keystore and asked to enter the keyid belonging to the delegated role. Return a string containing the delegated role's full rolename and its keyids. - """ # Retrieve the delegated rolename from the user (e.g., 'role1'). @@ -1240,7 +1222,6 @@ def _make_delegated_metadata(metadata_directory, delegated_targets, role. Determine the target files from the paths in 'delegated_targets' and the other information needed to generate the targets metadata file for delegated_role'. Return the delegated paths to the caller. - """ repository_directory, junk = os.path.split(metadata_directory) @@ -1291,11 +1272,11 @@ def _make_delegated_metadata(metadata_directory, delegated_targets, # containing the parent role's name is created in the metadata # directory. For example, if the targets roles creates a delegated # role 'role1', the metadata directory would then contain: - # '{metadata_directory}/targets/role1.txt', where 'role1.txt' is the + # '{metadata_directory}/targets/role1.json', where 'role1.json' is the # delegated role's metadata file. # If delegated role 'role1' creates its own delegated role 'role2', the # metadata directory would then contain: - # '{metadata_directory}/targets/role1/role2.txt'. + # '{metadata_directory}/targets/role1/role2.json'. # When creating a delegated role, if the parent directory already # exists, this means a prior delegation has been perform by the parent. parent_directory = os.path.join(metadata_directory, parent_role) @@ -1314,7 +1295,7 @@ def _make_delegated_metadata(metadata_directory, delegated_targets, expiration_date = _get_metadata_expiration() # Sign and write the delegated metadata file. - delegated_role_filename = delegated_role+'.txt' + delegated_role_filename = delegated_role+'.json' metadata_filename = os.path.join(parent_directory, delegated_role_filename) repository_directory, junk = os.path.split(metadata_directory) generate_metadata = tuf.repo.signerlib.generate_targets_metadata @@ -1336,7 +1317,6 @@ def _update_parent_metadata(metadata_directory, delegated_role, metadata file is updated with the key and role information belonging to the newly added delegated role. Finally, the metadata file is signed and written to the metadata directory. - """ # According to the specification, the 'paths' and 'path_hash_prefixes' @@ -1363,7 +1343,7 @@ def _update_parent_metadata(metadata_directory, delegated_role, # Extract the metadata from the parent role's file. parent_filename = os.path.join(metadata_directory, parent_role) - parent_filename = parent_filename+'.txt' + parent_filename = parent_filename+'.json' parent_signable = tuf.repo.signerlib.read_metadata_file(parent_filename) parent_metadata = parent_signable['signed'] @@ -1376,8 +1356,10 @@ def _update_parent_metadata(metadata_directory, delegated_role, # Retrieve the key belonging to 'delegated_keyid' from the keystore. role_key = tuf.repo.keystore.get_key(delegated_keyid) if role_key['keytype'] == 'rsa': + keytype = role_key['keytype'] keyval = role_key['keyval'] - keys[delegated_keyid] = tuf.rsa_key.create_in_metadata_format(keyval) + keys[delegated_keyid] = tuf.keys.format_keyval_to_metadata(keytype, keyval) + #keys[delegated_keyid] = tuf.keys.create_in_metadata_format(keytype, keyval) else: message = 'Invalid keytype encountered: '+delegated_keyid+'\n' raise tuf.RepositoryError(message) @@ -1450,7 +1432,6 @@ def process_option(options): None. - """ # Determine which option was chosen and call its corresponding @@ -1506,7 +1487,6 @@ def parse_options(): The options object returned by the parser's parse_args() method. - """ usage = 'usage: %prog [option] ' @@ -1531,19 +1511,19 @@ def parse_options(): option_parser.add_option('--makeroot', action='store', type='string', help='Create the Root metadata file '\ - '(root.txt).') + '(root.json).') option_parser.add_option('--maketargets', action='store', type='string', help='Create the Targets metadata file '\ - '(targets.txt).') + '(targets.json).') option_parser.add_option('--makerelease', action='store', type='string', help='Create the Release metadata file '\ - '(release.txt).') + '(release.json).') option_parser.add_option('--maketimestamp', action='store', type='string', help='Create the Timestamp metadata file '\ - '(timestamp.txt).') + '(timestamp.json).') option_parser.add_option('--sign', action='store', type='string', help='Sign a metadata file.') diff --git a/tuf/repo/signerlib.py b/tuf/repo/signerlib.py index 3f44ea5e8d..4682348c7f 100755 --- a/tuf/repo/signerlib.py +++ b/tuf/repo/signerlib.py @@ -16,20 +16,20 @@ These functions contain code that can extract or create needed repository data, such as the extraction of role and keyid information from a config file, and the generation of actual metadata content. - """ import gzip import os import ConfigParser import logging +import time import tuf import tuf.formats import tuf.hash -import tuf.rsa_key +import tuf.keys import tuf.repo.keystore -import tuf.sig +import tuf.keys import tuf.util # See 'log.py' to learn how logging is handled in TUF. @@ -45,10 +45,10 @@ DEFAULT_RSA_KEY_BITS = 3072 # The metadata filenames for the top-level roles. -ROOT_FILENAME = 'root.txt' -TARGETS_FILENAME = 'targets.txt' -RELEASE_FILENAME = 'release.txt' -TIMESTAMP_FILENAME = 'timestamp.txt' +ROOT_FILENAME = 'root.json' +TARGETS_FILENAME = 'targets.json' +SNAPSHOT_FILENAME = 'snapshot.json' +TIMESTAMP_FILENAME = 'timestamp.json' # The filename for the repository configuration file. # This file holds the keyids and threshold values for @@ -81,7 +81,6 @@ def read_config_file(filename): A dictionary containing the data loaded from the configuration file. - """ # Does 'filename' have the correct format? @@ -128,7 +127,7 @@ def get_metadata_file_info(filename): Retrieve the file information for 'filename'. The object returned conforms to 'tuf.formats.FILEINFO_SCHEMA'. The information - generated for 'filename' is stored in metadata files like 'targets.txt'. + generated for 'filename' is stored in metadata files like 'targets.json'. The fileinfo object returned has the form: fileinfo = {'length': 1024, 'hashes': {'sha256': 1233dfba312, ...}, @@ -151,7 +150,6 @@ def get_metadata_file_info(filename): A dictionary conformant to 'tuf.formats.FILEINFO_SCHEMA'. This dictionary contains the length, hashes, and custom data about the 'filename' metadata file. - """ # Does 'filename' have the correct format? @@ -183,10 +181,10 @@ def get_metadata_filenames(metadata_directory=None): If 'metadata_directory' is set to 'metadata', the dictionary returned would contain: - filenames = {'root': 'metadata/root.txt', - 'targets': 'metadata/targets.txt', - 'release': 'metadata/release.txt', - 'timestamp': 'metadata/timestamp.txt'} + filenames = {'root': 'metadata/root.json', + 'targets': 'metadata/targets.json', + 'snapshot': 'metadata/snapshot.json', + 'timestamp': 'metadata/timestamp.json'} If the metadata directory is not set by the caller, the current directory is used. @@ -203,8 +201,7 @@ def get_metadata_filenames(metadata_directory=None): A dictionary containing the expected filenames of the top-level - metadata files, such as 'root.txt' and 'release.txt'. - + metadata files, such as 'root.json' and 'snapshot.json'. """ if metadata_directory is None: @@ -217,7 +214,7 @@ def get_metadata_filenames(metadata_directory=None): filenames = {} filenames['root'] = os.path.join(metadata_directory, ROOT_FILENAME) filenames['targets'] = os.path.join(metadata_directory, TARGETS_FILENAME) - filenames['release'] = os.path.join(metadata_directory, RELEASE_FILENAME) + filenames['snapshot'] = os.path.join(metadata_directory, SNAPSHOT_FILENAME) filenames['timestamp'] = os.path.join(metadata_directory, TIMESTAMP_FILENAME) return filenames @@ -255,7 +252,6 @@ def generate_root_metadata(config_filepath, version): A root 'signable' object conformant to 'tuf.formats.SIGNABLE_SCHEMA'. - """ # Does 'config_filepath' have the correct format? @@ -272,7 +268,7 @@ def generate_root_metadata(config_filepath, version): # Extract the role, threshold, and keyid information from the config. # The necessary role metadata is generated from this information. - for rolename in ['root', 'targets', 'release', 'timestamp']: + for rolename in ['root', 'targets', 'snapshot', 'timestamp']: # If a top-level role is missing from the config, raise an exception. if rolename not in config: raise tuf.Error('No '+rolename+' section found in config file.') @@ -290,8 +286,10 @@ def generate_root_metadata(config_filepath, version): keyid = key['keyid'] # This appears to be a new keyid. Let's generate the key for it. if keyid not in keydict: - if key['keytype'] == 'rsa': - keydict[keyid] = tuf.rsa_key.create_in_metadata_format(key['keyval']) + if key['keytype'] in ['rsa', 'ed25519']: + keytype = key['keytype'] + keyval = key['keyval'] + keydict[keyid] = tuf.keys.format_keyval_to_metadata(keytype, keyval) # This is not a recognized key. Raise an exception. else: raise tuf.Error('Unsupported keytype: '+keyid) @@ -312,8 +310,9 @@ def generate_root_metadata(config_filepath, version): 3600 * 24 * expiration['days']) # Generate the root metadata object. - root_metadata = tuf.formats.RootFile.make_metadata(version, expiration_seconds, - keydict, roledict) + expiration_date = tuf.formats.format_time(time.time()+expiration_seconds) + root_metadata = tuf.formats.RootFile.make_metadata(version, expiration_date, + keydict, roledict, False) # Note: make_signable() returns the following dictionary: # {'signed' : role_metadata, 'signatures' : []} @@ -334,9 +333,9 @@ def generate_targets_metadata(repository_directory, target_files, version, target_files: - The target files tracked by 'targets.txt'. 'target_files' is a list of + The target files tracked by 'targets.json'. 'target_files' is a list of paths/directories of target files that are relative to the repository - (e.g., ['targets/file1.txt', ...]). If the target files are saved in + (e.g., ['targets/file1.json', ...]). If the target files are saved in the root folder 'targets' on the repository, then 'targets' must be included in the target paths. The repository does not have to name this folder 'targets'. @@ -364,7 +363,6 @@ def generate_targets_metadata(repository_directory, target_files, version, A targets 'signable' object, conformant to 'tuf.formats.SIGNABLE_SCHEMA'. - """ # Do the arguments have the correct format. @@ -401,17 +399,17 @@ def generate_targets_metadata(repository_directory, target_files, version, -def generate_release_metadata(metadata_directory, version, expiration_date): +def generate_snapshot_metadata(metadata_directory, version, expiration_date): """ - Create the release metadata. The minimum metadata must exist - (i.e., 'root.txt' and 'targets.txt'). This will also look through + Create the snapshot metadata. The minimum metadata must exist + (i.e., 'root.json' and 'targets.json'). This will also look through the 'targets/' directory in 'metadata_directory' and the resulting - release file will list all the delegated roles. + snapshot file will list all the delegated roles. metadata_directory: - The directory containing the 'root.txt' and 'targets.txt' metadata + The directory containing the 'root.json' and 'targets.json' metadata files. version: @@ -425,15 +423,14 @@ def generate_release_metadata(metadata_directory, version, expiration_date): tuf.FormatError, if 'metadata_directory' is improperly formatted. - tuf.Error, if an error occurred trying to generate the release metadata + tuf.Error, if an error occurred trying to generate the snapshot metadata object. - The 'root.txt' and 'targets.txt' files are read. + The 'root.json' and 'targets.json' files are read. - The release 'signable' object, conformant to 'tuf.formats.SIGNABLE_SCHEMA'. - + The snapshot 'signable' object, conformant to 'tuf.formats.SIGNABLE_SCHEMA'. """ # Does 'metadata_directory' have the correct format? @@ -445,18 +442,18 @@ def generate_release_metadata(metadata_directory, version, expiration_date): metadata_directory = check_directory(metadata_directory) # Retrieve the full filepath of the root and targets metadata file. - root_filename = os.path.join(metadata_directory, 'root.txt') - targets_filename = os.path.join(metadata_directory, 'targets.txt') + root_filename = os.path.join(metadata_directory, 'root.json') + targets_filename = os.path.join(metadata_directory, 'targets.json') - # Retrieve the file info of 'root.txt' and 'targets.txt'. This file + # Retrieve the file info of 'root.json' and 'targets.json'. This file # information includes data such as file length, hashes of the file, etc. filedict = {} - filedict['root.txt'] = get_metadata_file_info(root_filename) - filedict['targets.txt'] = get_metadata_file_info(targets_filename) + filedict['root.json'] = get_metadata_file_info(root_filename) + filedict['targets.json'] = get_metadata_file_info(targets_filename) # Walk the 'targets/' directory and generate the file info for all # the files listed there. This information is stored in the 'meta' - # field of the release metadata object. + # field of the snapshot metadata object. targets_metadata = os.path.join(metadata_directory, 'targets') if os.path.exists(targets_metadata) and os.path.isdir(targets_metadata): for directory_path, junk, files in os.walk(targets_metadata): @@ -466,26 +463,26 @@ def generate_release_metadata(metadata_directory, version, expiration_date): metadata_name = metadata_path[len(metadata_directory):].lstrip(os.path.sep) filedict[metadata_name] = get_metadata_file_info(metadata_path) - # Generate the release metadata object. - release_metadata = tuf.formats.ReleaseFile.make_metadata(version, - expiration_date, - filedict) + # Generate the snapshot metadata object. + snapshot_metadata = tuf.formats.SnapshotFile.make_metadata(version, + expiration_date, + filedict) - return tuf.formats.make_signable(release_metadata) + return tuf.formats.make_signable(snapshot_metadata) -def generate_timestamp_metadata(release_filename, version, +def generate_timestamp_metadata(snapshot_filename, version, expiration_date, compressions=()): """ - Generate the timestamp metadata object. The 'release.txt' file must exist. + Generate the timestamp metadata object. The 'snapshot.json' file must exist. - release_filename: - The required filename of the release metadata file. + snapshot_filename: + The required filename of the snapshot metadata file. version: The metadata version number. Clients use the version number to @@ -496,7 +493,7 @@ def generate_timestamp_metadata(release_filename, version, Conformant to 'tuf.formats.TIME_SCHEMA'. compressions: - Compression extensions (e.g., 'gz'). If 'release.txt' is also saved in + Compression extensions (e.g., 'gz'). If 'snapshot.json' is also saved in compressed form, these compression extensions should be stored in 'compressions' so the compressed timestamp files can be added to the timestamp metadata object. @@ -510,30 +507,29 @@ def generate_timestamp_metadata(release_filename, version, A timestamp 'signable' object, conformant to 'tuf.formats.SIGNABLE_SCHEMA'. - """ # Do the arguments have the correct format? # Raise 'tuf.FormatError' if there is mismatch. - tuf.formats.PATH_SCHEMA.check_match(release_filename) + tuf.formats.PATH_SCHEMA.check_match(snapshot_filename) tuf.formats.METADATAVERSION_SCHEMA.check_match(version) tuf.formats.TIME_SCHEMA.check_match(expiration_date) - # Retrieve the file info for the release metadata file. + # Retrieve the file info for the snapshot metadata file. # This file information contains hashes, file length, custom data, etc. fileinfo = {} - fileinfo['release.txt'] = get_metadata_file_info(release_filename) + fileinfo['snapshot.json'] = get_metadata_file_info(snapshot_filename) - # Save the file info of the compressed versions of 'timestamp.txt'. + # Save the file info of the compressed versions of 'timestamp.json'. for file_extension in compressions: - compressed_filename = release_filename + '.' + file_extension + compressed_filename = snapshot_filename + '.' + file_extension try: compressed_fileinfo = get_metadata_file_info(compressed_filename) except: logger.warn('Could not get fileinfo about '+str(compressed_filename)) else: logger.info('Including fileinfo about '+str(compressed_filename)) - fileinfo['release.txt.' + file_extension] = compressed_fileinfo + fileinfo['snapshot.json.' + file_extension] = compressed_fileinfo # Generate the timestamp metadata object. timestamp_metadata = tuf.formats.TimestampFile.make_metadata(version, @@ -557,7 +553,7 @@ def write_metadata_file(metadata, filename, compression=None): filename: The filename (absolute path) of the metadata to be - written (e.g., 'root.txt'). + written (e.g., 'root.json'). compression: Specify an algorithm as a string to compress the file; otherwise, the @@ -575,7 +571,6 @@ def write_metadata_file(metadata, filename, compression=None): The path to the written metadata file. - """ # Are the arguments properly formatted? @@ -645,7 +640,6 @@ def read_metadata_file(filename): The metadata object. - """ return tuf.util.load_json_file(filename) @@ -671,7 +665,7 @@ def sign_metadata(metadata, keyids, filename): filename: The intended filename of the signed metadata object. - For example, 'root.txt' or 'targets.txt'. This function + For example, 'root.json' or 'targets.json'. This function does NOT save the signed metadata to this filename. @@ -684,7 +678,6 @@ def sign_metadata(metadata, keyids, filename): A signable object conformant to 'tuf.formats.SIGNABLE_SCHEMA'. - """ # Does 'keyids' and 'filename' have the correct format? @@ -715,7 +708,7 @@ def sign_metadata(metadata, keyids, filename): # Generate the signature using the appropriate signing method. if key['keytype'] == 'rsa': signed = signable['signed'] - signature = tuf.sig.generate_rsa_signature(signed, key) + signature = tuf.keys.create_signature(key, signed) signable['signatures'].append(signature) else: raise tuf.Error('The keystore contains a key with an invalid key type') @@ -767,7 +760,6 @@ def generate_and_save_rsa_key(keystore_directory, password, 'keyid': keyid, 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} - """ # Are the arguments correctly formatted? @@ -778,7 +770,7 @@ def generate_and_save_rsa_key(keystore_directory, password, keystore_directory = check_directory(keystore_directory) # tuf.FormatError or tuf.CryptoError raised. - rsakey = tuf.rsa_key.generate(bits) + rsakey = tuf.keys.generate_rsa_key(bits) logger.info('Generated a new key: '+rsakey['keyid']) @@ -820,7 +812,6 @@ def check_directory(directory): The normalized absolutized path of 'directory'. - """ # Does 'directory' have the correct format? @@ -843,14 +834,14 @@ def get_target_keyids(metadata_directory): """ Retrieve the role keyids for all the target roles located - in 'metadata_directory'. The target's '.txt' metadata - file is inspected and the keyids extracted. The 'targets.txt' - role, including delegated roles (e.g., 'targets/role1.txt'), + in 'metadata_directory'. The target's '.json' metadata + file is inspected and the keyids extracted. The 'targets.json' + role, including delegated roles (e.g., 'targets/role1.json'), are all read. metadata_directory: - The directory containing the 'targets.txt' metadata file and + The directory containing the 'targets.json' metadata file and the metadata for optional delegated roles. The delegated role 'role1' whose parent is 'targets', would be located in the '{metadata_directory}/targets/role1' directory. @@ -868,7 +859,6 @@ def get_target_keyids(metadata_directory): A dictionary containing the role information extracted from the metadata. Ex: {'targets':[keyid1, ...], 'targets/role1':[keyid], ...} - """ # Does 'metadata_directory' have the correct format? @@ -881,19 +871,19 @@ def get_target_keyids(metadata_directory): # This dict will be returned to the caller. role_keyids = {} - # Read the 'targets.txt' file. This file must exist. - targets_filepath = os.path.join(metadata_directory, 'targets.txt') + # Read the 'targets.json' file. This file must exist. + targets_filepath = os.path.join(metadata_directory, 'targets.json') if not os.path.exists(targets_filepath): - raise tuf.RepositoryError('"targets.txt" not found') + raise tuf.RepositoryError('"targets.json" not found') - # Read the contents of 'targets.txt' and save the signable. + # Read the contents of 'targets.json' and save the signable. targets_signable = tuf.util.load_json_file(targets_filepath) # Ensure the signable is properly formatted. try: tuf.formats.check_signable_object_format(targets_signable) except tuf.FormatError, e: - raise tuf.RepositoryError('"targets.txt" is improperly formatted') + raise tuf.RepositoryError('"targets.json" is improperly formatted') # Store the keyids of the 'targets' role. This target role is # required. @@ -903,18 +893,18 @@ def get_target_keyids(metadata_directory): # Walk the 'targets/' directory and generate the file info for all # the targets. This information is stored in the 'meta' field of - # the release metadata object. The keyids for the optional + # the snapshot metadata object. The keyids for the optional # delegated roles will now be extracted. targets_metadata = os.path.join(metadata_directory, 'targets') if os.path.exists(targets_metadata) and os.path.isdir(targets_metadata): for directory_path, junk, files in os.walk(targets_metadata): for basename in files: # Store the metadata's file path and the role's full name (without - # the '.txt'). The target role is identified by its full name. + # the '.json'). The target role is identified by its full name. # The metadata's file path is needed so it can be loaded. metadata_path = os.path.join(directory_path, basename) metadata_name = metadata_path[len(metadata_directory):].lstrip(os.path.sep) - metadata_name = metadata_name[:-len('.txt')] + metadata_name = metadata_name[:-len('.json')] # Read the contents of 'metadata_path' and save the signable. targets_signable = tuf.util.load_json_file(metadata_path) @@ -954,7 +944,7 @@ def build_config_file(config_file_directory, timeout, role_info): top-level roles. Must conform to 'tuf.formats.ROLEDICT_SCHEMA': {'rolename': {'keyids': ['34345df32093bd12...'], 'threshold': 1 - 'paths': ['path/to/role.txt']}} + 'paths': ['path/to/role.json']}} tuf.FormatError, if any of the arguments are improperly formatted. @@ -964,7 +954,6 @@ def build_config_file(config_file_directory, timeout, role_info): The normalized absolutized path of the saved configuration file. - """ # Do the arguments have the correct format? @@ -980,7 +969,7 @@ def build_config_file(config_file_directory, timeout, role_info): # Verify that only the top-level roles are presented. for role in role_info.keys(): - if role not in ['root', 'targets', 'release', 'timestamp']: + if role not in ['root', 'targets', 'snapshot', 'timestamp']: msg = ('\nCannot build configuration file: role '+repr(role)+ ' is not a top-level role.') raise tuf.Error(msg) @@ -1054,7 +1043,6 @@ def build_root_file(config_filepath, root_keyids, metadata_directory, version): The path for the written root metadata file. - """ # Do the arguments have the correct format? @@ -1090,7 +1078,7 @@ def build_targets_file(target_paths, targets_keyids, metadata_directory, target_paths: The list of directories and/or filepaths specifying the target files of the targets metadata. For example: - ['targets/2.5/', 'targets/3.0/file.txt', 'targes/3.2/'] + ['targets/2.5/', 'targets/3.0/file.json', 'targes/3.2/'] targets_keyids: The list of keyids to be used as the signing keys for the targets file. @@ -1116,7 +1104,6 @@ def build_targets_file(target_paths, targets_keyids, metadata_directory, The path for the written targets metadata file. - """ # Do the arguments have the correct format? @@ -1171,19 +1158,19 @@ def build_targets_file(target_paths, targets_keyids, metadata_directory, -def build_release_file(release_keyids, metadata_directory, +def build_snapshot_file(snapshot_keyids, metadata_directory, version, expiration_date, compress=False): """ - Build the release metadata file using the signing keys in 'release_keyids'. + Build the snapshot metadata file using the signing keys in 'snapshot_keyids'. The generated metadata file is saved in 'metadata_directory'. - release_keyids: - The list of keyids to be used as the signing keys for the release file. + snapshot_keyids: + The list of keyids to be used as the signing keys for the snapshot file. metadata_directory: - The directory (absolute path) to save the release metadata file. + The directory (absolute path) to save the snapshot metadata file. version: The metadata version number. Clients use the version number to @@ -1194,49 +1181,48 @@ def build_release_file(release_keyids, metadata_directory, Conformant to 'tuf.formats.TIME_SCHEMA'. compress: - Should we *include* a compressed version of the release file? By default, + Should we *include* a compressed version of the snapshot file? By default, the answer is no. tuf.FormatError, if any of the arguments are improperly formatted. - tuf.Error, if there was an error while building the release file. + tuf.Error, if there was an error while building the snapshot file. - The release metadata file is written to a file. + The snapshot metadata file is written to a file. - The path for the written release metadata file. - + The path for the written snapshot metadata file. """ # Do the arguments have the correct format? # Raise 'tuf.FormatError' if there is a mismatch. - tuf.formats.KEYIDS_SCHEMA.check_match(release_keyids) + tuf.formats.KEYIDS_SCHEMA.check_match(snapshot_keyids) tuf.formats.PATH_SCHEMA.check_match(metadata_directory) tuf.formats.METADATAVERSION_SCHEMA.check_match(version) tuf.formats.TIME_SCHEMA.check_match(expiration_date) metadata_directory = check_directory(metadata_directory) - # Generate the file path of the release metadata. - release_filepath = os.path.join(metadata_directory, RELEASE_FILENAME) + # Generate the file path of the snapshot metadata. + snapshot_filepath = os.path.join(metadata_directory, SNAPSHOT_FILENAME) - # Generate and sign the release metadata. - release_metadata = generate_release_metadata(metadata_directory, + # Generate and sign the snapshot metadata. + snapshot_metadata = generate_snapshot_metadata(metadata_directory, version, expiration_date) - signable = sign_metadata(release_metadata, release_keyids, release_filepath) + signable = sign_metadata(snapshot_metadata, snapshot_keyids, snapshot_filepath) - # Should we also include a compressed version of release.txt? + # Should we also include a compressed version of snapshot.json? if compress: - # If so, write a gzip version of release.txt. + # If so, write a gzip version of snapshot.json. compressed_written_filepath = \ - write_metadata_file(signable, release_filepath, compression='gz') + write_metadata_file(signable, snapshot_filepath, compression='gz') logger.info('Wrote '+str(compressed_written_filepath)) else: - logger.debug('No compressed version of release metadata will be included.') + logger.debug('No compressed version of snapshot metadata will be included.') - written_filepath = write_metadata_file(signable, release_filepath) + written_filepath = write_metadata_file(signable, snapshot_filepath) logger.info('Wrote '+str(written_filepath)) return written_filepath @@ -1247,7 +1233,7 @@ def build_release_file(release_keyids, metadata_directory, def build_timestamp_file(timestamp_keyids, metadata_directory, version, expiration_date, - include_compressed_release=True): + include_compressed_snapshot=True): """ Build the timestamp metadata file using the signing keys in 'timestamp_keyids'. @@ -1268,8 +1254,8 @@ def build_timestamp_file(timestamp_keyids, metadata_directory, The expiration date, in UTC, of the metadata file. Conformant to 'tuf.formats.TIME_SCHEMA'. - include_compressed_release: - Should the timestamp role *include* compression versions of the release + include_compressed_snapshot: + Should the timestamp role *include* compression versions of the snapshot metadata, if any? We do this by default. @@ -1282,7 +1268,6 @@ def build_timestamp_file(timestamp_keyids, metadata_directory, The path for the written timestamp metadata file. - """ # Do the arguments have the correct format? @@ -1294,23 +1279,23 @@ def build_timestamp_file(timestamp_keyids, metadata_directory, metadata_directory = check_directory(metadata_directory) - # Generate the file path of the release and timestamp metadata. - release_filepath = os.path.join(metadata_directory, RELEASE_FILENAME) + # Generate the file path of the snapshot and timestamp metadata. + snapshot_filepath = os.path.join(metadata_directory, SNAPSHOT_FILENAME) timestamp_filepath = os.path.join(metadata_directory, TIMESTAMP_FILENAME) - # Should we include compressed versions of release in timestamp? + # Should we include compressed versions of snapshot in timestamp? compressions = () - if include_compressed_release: + if include_compressed_snapshot: # Presently, we include only gzip versions by default. compressions = ('gz',) - logger.info('Including '+str(compressions)+' versions of release in '\ + logger.info('Including '+str(compressions)+' versions of snapshot in '\ 'timestamp.') else: - logger.warn('No compressed versions of release will be included in '\ + logger.warn('No compressed versions of snapshot will be included in '\ 'timestamp.') # Generate and sign the timestamp metadata. - timestamp_metadata = generate_timestamp_metadata(release_filepath, + timestamp_metadata = generate_timestamp_metadata(snapshot_filepath, version, expiration_date, compressions=compressions) @@ -1349,7 +1334,7 @@ def build_delegated_role_file(delegated_targets_directory, delegated_keyids, The location of the delegated role's metadata. delegation_role_name: - The delegated role's file name ending in '.txt'. Ex: 'role1.txt'. + The delegated role's file name ending in '.json'. Ex: 'role1.json'. version: The metadata version number. Clients use the version number to @@ -1370,7 +1355,6 @@ def build_delegated_role_file(delegated_targets_directory, delegated_keyids, The path for the written targets metadata file. - """ # Do the arguments have the correct format? @@ -1432,7 +1416,6 @@ def find_delegated_role(roles, delegated_role): None, if the role with the given name does not exist, or its unique index in the list of roles. - """ # Check argument types. @@ -1487,7 +1470,6 @@ def accept_any_file(full_target_path): True. - """ return True @@ -1524,7 +1506,6 @@ def get_targets(files_directory, recursive_walk=False, followlinks=True, A list of absolute paths to target files in the given files_directory. - """ targets = [] @@ -1546,8 +1527,3 @@ def get_targets(files_directory, recursive_walk=False, followlinks=True, del dirnames[:] return targets - - - - - diff --git a/tuf/repository_tool.py b/tuf/repository_tool.py new file mode 100755 index 0000000000..30888fd95e --- /dev/null +++ b/tuf/repository_tool.py @@ -0,0 +1,4672 @@ +""" + + repository_tool.py + + + Vladimir Diaz + + + October 19, 2013 + + + See LICENSE for licensing information. + + + Provide a tool that can create a TUF repository. It can be used with the + Python interpreter in interactive mode, or imported directly into a Python + module. See 'tuf/README' for the complete guide to using + 'tuf.repository_tool.py'. +""" + +# Help with Python 3 compatibility, where the print statement is a function, an +# implicit relative import is invalid, and the '/' operator performs true +# division. Example: print 'hello world' raises a 'SyntaxError' exception. +from __future__ import print_function +from __future__ import absolute_import +from __future__ import division + +import os +import errno +import sys +import time +import getpass +import logging +import tempfile +import shutil +import json +import gzip +import random + +import tuf +import tuf.formats +import tuf.util +import tuf.keydb +import tuf.roledb +import tuf.keys +import tuf.sig +import tuf.log +import tuf.conf + + +# See 'log.py' to learn how logging is handled in TUF. +logger = logging.getLogger('tuf.repository_tool') + +# Add a console handler so that users are aware of potentially unintended +# states, such as multiple roles that share keys. +tuf.log.add_console_handler() +tuf.log.set_console_log_level(logging.WARNING) + +# Recommended RSA key sizes: +# http://www.emc.com/emc-plus/rsa-labs/historical/twirl-and-rsa-key-size.htm#table1 +# According to the document above, revised May 6, 2003, RSA keys of +# size 3072 provide security through 2031 and beyond. 2048-bit keys +# are the recommended minimum and are good from the present through 2030. +DEFAULT_RSA_KEY_BITS = 3072 + +# The algorithm used by the repository to generate the digests of the +# target filepaths, which are included in metadata files and may be prepended +# to the filenames of consistent snapshots. +HASH_FUNCTION = 'sha256' + +# The extension of TUF metadata. +METADATA_EXTENSION = '.json' + +# The metadata filenames of the top-level roles. +ROOT_FILENAME = 'root' + METADATA_EXTENSION +TARGETS_FILENAME = 'targets' + METADATA_EXTENSION +SNAPSHOT_FILENAME = 'snapshot' + METADATA_EXTENSION +TIMESTAMP_FILENAME = 'timestamp' + METADATA_EXTENSION + +# The targets and metadata directory names. Metadata files are written +# to the staged metadata directory instead of the "live" one. +METADATA_STAGED_DIRECTORY_NAME = 'metadata.staged' +METADATA_DIRECTORY_NAME = 'metadata' +TARGETS_DIRECTORY_NAME = 'targets' + +# The full list of supported TUF metadata extensions. +METADATA_EXTENSIONS = ['.json', '.json.gz'] + +# The recognized compression extensions. +SUPPORTED_COMPRESSION_EXTENSIONS = ['.gz'] + +# Supported key types. +SUPPORTED_KEY_TYPES = ['rsa', 'ed25519'] + +# Expiration date delta, in seconds, of the top-level roles. A metadata +# expiration date is set by taking the current time and adding the expiration +# seconds listed below. + +# Initial 'root.json' expiration time of 1 year. +ROOT_EXPIRATION = 31556900 + +# Initial 'targets.json' expiration time of 3 months. +TARGETS_EXPIRATION = 7889230 + +# Initial 'snapshot.json' expiration time of 1 week. +SNAPSHOT_EXPIRATION = 604800 + +# Initial 'timestamp.json' expiration time of 1 day. +TIMESTAMP_EXPIRATION = 86400 + + +class Repository(object): + """ + + Represent a TUF repository that contains the metadata of the top-level + roles, including all those delegated from the 'targets.json' role. The + repository object returned provides access to the top-level roles, and any + delegated targets that are added as the repository is modified. For + example, a Repository object named 'repository' provides the following + access by default: + + repository.root.version = 2 + repository.timestamp.expiration = "2015-08-08 12:00:00" + repository.snapshot.add_verification_key(...) + repository.targets.delegate('unclaimed', ...) + + Delegating a role from 'targets' updates the attributes of the parent + delegation, which then provides: + + repository.targets('unclaimed').add_verification_key(...) + + + + repository_directory: + The root folder of the repository that contains the metadata and targets + sub-directories. + + metadata_directory: + The metadata sub-directory contains the files of the top-level + roles, including all roles delegated from 'targets.json'. + + targets_directory: + The targets sub-directory contains all the target files that are + downloaded by clients and are referenced in TUF Metadata. The hashes and + file lengths are listed in Metadata files so that they are securely + downloaded. Metadata files are similarly referenced in the top-level + metadata. + + + tuf.FormatError, if the arguments are improperly formatted. + + + Creates top-level role objects and assigns them as attributes. + + + A Repository object that contains default Metadata objects for the top-level + roles. + """ + + def __init__(self, repository_directory, metadata_directory, targets_directory): + + # Do the arguments have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if any are improperly formatted. + tuf.formats.PATH_SCHEMA.check_match(repository_directory) + tuf.formats.PATH_SCHEMA.check_match(metadata_directory) + tuf.formats.PATH_SCHEMA.check_match(targets_directory) + + self._repository_directory = repository_directory + self._metadata_directory = metadata_directory + self._targets_directory = targets_directory + + # Set the top-level role objects. + self.root = Root() + self.snapshot = Snapshot() + self.timestamp = Timestamp() + self.targets = Targets(self._targets_directory, 'targets') + + + + def write(self, write_partial=False, consistent_snapshot=False): + """ + + Write all the JSON Metadata objects to their corresponding files. + write() raises an exception if any of the role metadata to be written to + disk is invalid, such as an insufficient threshold of signatures, missing + private keys, etc. + + + write_partial: + A boolean indicating whether partial metadata should be written to + disk. Partial metadata may be written to allow multiple maintainters + to independently sign and update role metadata. write() raises an + exception if a metadata role cannot be written due to not having enough + signatures. + + consistent_snapshot: + A boolean indicating whether written metadata and target files should + include a digest in the filename (i.e., .root.json, + .targets.json.gz, .README.json, where is the + file's SHA256 digest. Example: + 1f4e35a60c8f96d439e27e858ce2869c770c1cdd54e1ef76657ceaaf01da18a3.root.json' + + + tuf.UnsignedMetadataError, if any of the top-level and delegated roles do + not have the minimum threshold of signatures. + + + Creates metadata files in the repository's metadata directory. + + + None. + """ + + # Does 'write_partial' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if any are improperly formatted. + tuf.formats.BOOLEAN_SCHEMA.check_match(write_partial) + tuf.formats.BOOLEAN_SCHEMA.check_match(consistent_snapshot) + + # At this point the tuf.keydb and tuf.roledb stores must be fully + # populated, otherwise write() throwns a 'tuf.Repository' exception if + # any of the top-level roles are missing signatures, keys, etc. + + # Write the metadata files of all the delegated roles. Ensure target paths + # are allowed, metadata is valid and properly signed, and required files and + # directories are created. + delegated_rolenames = tuf.roledb.get_delegated_rolenames('targets') + for delegated_rolename in delegated_rolenames: + delegated_filename = os.path.join(self._metadata_directory, + delegated_rolename + METADATA_EXTENSION) + roleinfo = tuf.roledb.get_roleinfo(delegated_rolename) + delegated_targets = roleinfo['paths'] + parent_rolename = tuf.roledb.get_parent_rolename(delegated_rolename) + parent_roleinfo = tuf.roledb.get_roleinfo(parent_rolename) + parent_delegations = parent_roleinfo['delegations'] + + # Raise exception if any of the targets of 'delegated_rolename' are not + # allowed. + tuf.util.ensure_all_targets_allowed(delegated_rolename, delegated_targets, + parent_delegations) + + # Ensure the parent directories of 'metadata_filepath' exist, otherwise an + # IO exception is raised if 'metadata_filepath' is written to a + # sub-directory. + tuf.util.ensure_parent_dir(delegated_filename) + + try: + _generate_and_write_metadata(delegated_rolename, delegated_filename, + write_partial, self._targets_directory, + self._metadata_directory, + consistent_snapshot) + + # Include only the exception message. + except tuf.UnsignedMetadataError, e: + raise tuf.UnsignedMetadataError(e[0]) + + # Generate the 'root.json' metadata file. + # _generate_and_write_metadata() raises a 'tuf.Error' exception if the + # metadata cannot be written. + root_filename = 'root' + METADATA_EXTENSION + root_filename = os.path.join(self._metadata_directory, root_filename) + try: + signable_junk, root_filename = \ + _generate_and_write_metadata('root', root_filename, write_partial, + self._targets_directory, + self._metadata_directory, + consistent_snapshot) + + # Include only the exception message. + except tuf.UnsignedMetadataError, e: + raise tuf.UnsignedMetadataError(e[0]) + + # Generate the 'targets.json' metadata file. + targets_filename = 'targets' + METADATA_EXTENSION + targets_filename = os.path.join(self._metadata_directory, targets_filename) + try: + signable_junk, targets_filename = \ + _generate_and_write_metadata('targets', targets_filename, write_partial, + self._targets_directory, + self._metadata_directory, + consistent_snapshot) + + # Include only the exception message. + except tuf.UnsignedMetadataError, e: + raise tuf.UnsignedMetadataError(e[0]) + + # Generate the 'snapshot.json' metadata file. + snapshot_filename = os.path.join(self._metadata_directory, 'snapshot') + snapshot_filename = 'snapshot' + METADATA_EXTENSION + snapshot_filename = os.path.join(self._metadata_directory, snapshot_filename) + filenames = {'root': root_filename, 'targets': targets_filename} + snapshot_signable = None + try: + snapshot_signable, snapshot_filename = \ + _generate_and_write_metadata('snapshot', snapshot_filename, write_partial, + self._targets_directory, + self._metadata_directory, + consistent_snapshot, filenames) + + # Include only the exception message. + except tuf.UnsignedMetadataError, e: + raise tuf.UnsignedMetadataError(e[0]) + + # Generate the 'timestamp.json' metadata file. + timestamp_filename = 'timestamp' + METADATA_EXTENSION + timestamp_filename = os.path.join(self._metadata_directory, timestamp_filename) + filenames = {'snapshot': snapshot_filename} + try: + _generate_and_write_metadata('timestamp', timestamp_filename, write_partial, + self._targets_directory, + self._metadata_directory, consistent_snapshot, + filenames) + + # Include only the exception message. + except tuf.UnsignedMetadataError, e: + raise tuf.UnsignedMetadataError(e[0]) + + + # Delete the metadata of roles no longer in 'tuf.roledb'. Obsolete roles + # may have been revoked and should no longer have their metadata files + # available on disk, otherwise loading a repository may unintentionally load + # them. + _delete_obsolete_metadata(self._metadata_directory, + snapshot_signable['signed'], consistent_snapshot) + + + + def write_partial(self): + """ + + Write all the JSON Metadata objects to their corresponding files, but + allow metadata files to contain an invalid threshold of signatures. + + + None. + + + None. + + + Creates metadata files in the repository's metadata directory. + + + None. + """ + + self.write(write_partial=True) + + + + def status(self): + """ + + Determine the status of the top-level roles, including those delegated by + the Targets role. status() checks if each role provides sufficient public + and private keys, signatures, and that a valid metadata file is generated + if write() were to be called. Metadata files are temporarily written so + that file hashes and lengths may be verified, determine if delegated role + trust is fully obeyed, and target paths valid according to parent roles. + status() does not do a simple check for number of threshold keys and + signatures. + + + None. + + + None. + + + Generates and writes temporary metadata files. + + + None. + """ + + temp_repository_directory = None + + # Generate and write temporary metadata so that full verification of + # metadata is possible, such as verifying signatures, digests, and file + # content. Ensure temporary files generated are removed after verification + # results are completed. + try: + temp_repository_directory = tempfile.mkdtemp() + targets_directory = self._targets_directory + metadata_directory = os.path.join(temp_repository_directory, + METADATA_STAGED_DIRECTORY_NAME) + os.mkdir(metadata_directory) + + + # Retrieve the roleinfo of the delegated roles, exluding the top-level + # targets role. + delegated_roles = tuf.roledb.get_delegated_rolenames('targets') + insufficient_keys = [] + insufficient_signatures = [] + + # Iterate the list of delegated roles and determine the list of invalid + # roles. First verify the public and private keys, and then the generated + # metadata file. + for delegated_role in delegated_roles: + filename = delegated_role + METADATA_EXTENSION + filename = os.path.join(metadata_directory, filename) + + # Ensure the parent directories of 'filename' exist, otherwise an + # IO exception is raised if 'filename' is written to a sub-directory. + tuf.util.ensure_parent_dir(filename) + + # Append any invalid roles to the 'insufficient_keys' and + # 'insufficient_signatures' lists + try: + _check_role_keys(delegated_role) + + except tuf.InsufficientKeysError, e: + insufficient_keys.append(delegated_role) + continue + + try: + _generate_and_write_metadata(delegated_role, filename, False, + targets_directory, metadata_directory) + except tuf.UnsignedMetadataError, e: + insufficient_signatures.append(delegated_role) + + # Print the verification results of the delegated roles and return + # immediately after each invalid case. + if len(insufficient_keys): + message = \ + 'Delegated roles with insufficient keys:\n'+repr(insufficient_keys) + print(message) + return + + if len(insufficient_signatures): + message = \ + 'Delegated roles with insufficient signatures:\n'+\ + repr(insufficient_signatures) + print(message) + return + + # Verify the top-level roles and print the results. + _print_status_of_top_level_roles(targets_directory, metadata_directory) + + finally: + shutil.rmtree(temp_repository_directory, ignore_errors=True) + + + + def get_filepaths_in_directory(self, files_directory, recursive_walk=False, + followlinks=True): + """ + + Walk the given 'files_directory' and build a list of target files found. + + + files_directory: + The path to a directory of target files. + + recursive_walk: + To recursively walk the directory, set recursive_walk=True. + + followlinks: + To follow symbolic links, set followlinks=True. + + + tuf.FormatError, if the arguments are improperly formatted. + + tuf.Error, if 'file_directory' is not a valid directory. + + Python IO exceptions. + + + None. + + + A list of absolute paths to target files in the given 'files_directory'. + """ + + # Do the arguments have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if any are improperly formatted. + tuf.formats.PATH_SCHEMA.check_match(files_directory) + tuf.formats.BOOLEAN_SCHEMA.check_match(recursive_walk) + tuf.formats.BOOLEAN_SCHEMA.check_match(followlinks) + + # Ensure a valid directory is given. + if not os.path.isdir(files_directory): + message = repr(files_directory)+' is not a directory.' + raise tuf.Error(message) + + # A list of the target filepaths found in 'file_directory'. + targets = [] + + # FIXME: We need a way to tell Python 2, but not Python 3, to return + # filenames in Unicode; see #61 and: + # http://docs.python.org/2/howto/unicode.html#unicode-filenames + for dirpath, dirnames, filenames in os.walk(files_directory, + followlinks=followlinks): + for filename in filenames: + full_target_path = os.path.join(dirpath, filename) + targets.append(full_target_path) + + # Prune the subdirectories to walk right now if we do not wish to + # recursively walk files_directory. + if recursive_walk is False: + del dirnames[:] + + return targets + + + + + +class Metadata(object): + """ + + Provide a base class to represent a TUF Metadata role. There are four + top-level roles: Root, Targets, Snapshot, and Timestamp. The Metadata class + provides methods that are needed by all top-level roles, such as adding + and removing public keys, private keys, and signatures. Metadata + attributes, such as rolename, version, threshold, expiration, key list, and + compressions, is also provided by the Metadata base class. + + + None. + + + None. + + + None. + + + None. + """ + + def __init__(self): + self._rolename = None + + + + def add_verification_key(self, key): + """ + + Add 'key' to the role. Adding a key, which should contain only the public + portion, signifies the corresponding private key and signatures the role + is expected to provide. A threshold of signatures is required for a role + to be considered properly signed. If a metadata file contains an + insufficient threshold of signatures, it must not be accepted. + + >>> + >>> + >>> + + + key: + The role key to be added, conformant to 'tuf.formats.ANYKEY_SCHEMA'. + Adding a public key to a role means that its corresponding private key + must generate and add its signature to the role. A threshold number of + signatures is required for a role to be fully signed. + + + tuf.FormatError, if the 'key' argument is improperly formatted. + + + The role's entries in 'tuf.keydb.py' and 'tuf.roledb.py' are updated. + + + None. + """ + + # Does 'key' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if any are improperly formatted. + tuf.formats.ANYKEY_SCHEMA.check_match(key) + + # Ensure 'key', which should contain the public portion, is added to + # 'tuf.keydb.py'. Add 'key' to the list of recognized keys. Keys may be + # shared, so do not raise an exception if 'key' has already been loaded. + try: + tuf.keydb.add_key(key) + + except tuf.KeyAlreadyExistsError, e: + message = 'Adding a verification key that has already been used.' + logger.warn(message) + + keyid = key['keyid'] + roleinfo = tuf.roledb.get_roleinfo(self.rolename) + + # Add 'key' to the role's entry in 'tuf.roledb.py' and avoid duplicates. + if keyid not in roleinfo['keyids']: + roleinfo['keyids'].append(keyid) + + tuf.roledb.update_roleinfo(self._rolename, roleinfo) + + + + def remove_verification_key(self, key): + """ + + Remove 'key' from the role's currently recognized list of role keys. + The role expects a threshold number of signatures. + + >>> + >>> + >>> + + + key: + The role's key, conformant to 'tuf.formats.ANYKEY_SCHEMA'. 'key' + should contain only the public portion, as only the public key is + needed. The 'add_verification_key()' method should have previously + added 'key'. + + + tuf.FormatError, if the 'key' argument is improperly formatted. + + tuf.Error, if the 'key' argument has not been previously added. + + + Updates the role's 'tuf.roledb.py' entry. + + + None. + """ + + # Does 'key' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if any are improperly formatted. + tuf.formats.ANYKEY_SCHEMA.check_match(key) + + keyid = key['keyid'] + roleinfo = tuf.roledb.get_roleinfo(self.rolename) + + if keyid in roleinfo['keyids']: + roleinfo['keyids'].remove(keyid) + + tuf.roledb.update_roleinfo(self._rolename, roleinfo) + + else: + raise tuf.Error('Verification key not found.') + + + + def load_signing_key(self, key): + """ + + Load the role key, which must contain the private portion, so that role + signatures may be generated when the role's metadata file is eventually + written to disk. + + >>> + >>> + >>> + + + key: + The role's key, conformant to 'tuf.formats.ANYKEY_SCHEMA'. It must + contain the private key, so that role signatures may be generated when + write() or write_partial() is eventually called to generate valid + metadata files. + + + tuf.FormatError, if 'key' is improperly formatted. + + tuf.Error, if the private key is not found in 'key'. + + + Updates the role's 'tuf.keydb.py' and 'tuf.roledb.py' entries. + + + None. + """ + + # Does 'key' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if any are improperly formatted. + tuf.formats.ANYKEY_SCHEMA.check_match(key) + + # Ensure the private portion of the key is available, otherwise signatures + # cannot be generated when the metadata file is written to disk. + if not len(key['keyval']['private']): + message = 'This is not a private key.' + raise tuf.Error(message) + + # Has the key, with the private portion included, been added to the keydb? + # The public version of the key may have been previously added. + try: + tuf.keydb.add_key(key) + + except tuf.KeyAlreadyExistsError, e: + tuf.keydb.remove_key(key['keyid']) + tuf.keydb.add_key(key) + + # Update the role's 'signing_keys' field in 'tuf.roledb.py'. + roleinfo = tuf.roledb.get_roleinfo(self.rolename) + if key['keyid'] not in roleinfo['signing_keyids']: + roleinfo['signing_keyids'].append(key['keyid']) + + tuf.roledb.update_roleinfo(self.rolename, roleinfo) + + + + def unload_signing_key(self, key): + """ + + Remove a previously loaded role private key (i.e., load_signing_key()). + The keyid of the 'key' is removed from the list of recognized signing + keys. + + >>> + >>> + >>> + + + key: + The role key to be unloaded, conformant to 'tuf.formats.ANYKEY_SCHEMA'. + + + tuf.FormatError, if the 'key' argument is improperly formatted. + + tuf.Error, if the 'key' argument has not been previously loaded. + + + Updates the signing keys of the role in 'tuf.roledb.py'. + + + None. + """ + + # Does 'key' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if any are improperly formatted. + tuf.formats.ANYKEY_SCHEMA.check_match(key) + + # Update the role's 'signing_keys' field in 'tuf.roledb.py'. + roleinfo = tuf.roledb.get_roleinfo(self.rolename) + + if key['keyid'] in roleinfo['signing_keyids']: + roleinfo['signing_keyids'].remove(key['keyid']) + + tuf.roledb.update_roleinfo(self.rolename, roleinfo) + + else: + raise tuf.Error('Signing key not found.') + + + + def add_signature(self, signature): + """ + + Add a signature to the role. A role is considered fully signed if it + contains a threshold of signatures. The 'signature' should have been + generated by the private key corresponding to one of the role's expected + keys. + + >>> + >>> + >>> + + + signature: + The signature to be added to the role, conformant to + 'tuf.formats.SIGNATURE_SCHEMA'. + + + tuf.FormatError, if the 'signature' argument is improperly formatted. + + + Adds 'signature', if not already added, to the role's 'signatures' field + in 'tuf.roledb.py'. + + + None. + """ + + # Does 'signature' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if any are improperly formatted. + tuf.formats.SIGNATURE_SCHEMA.check_match(signature) + + roleinfo = tuf.roledb.get_roleinfo(self.rolename) + + # Ensure the roleinfo contains a 'signatures' field. + if 'signatures' not in roleinfo: + roleinfo['signatures'] = [] + + # Update the role's roleinfo by adding 'signature', if it has not been + # added. + if signature not in roleinfo['signatures']: + roleinfo['signatures'].append(signature) + tuf.roledb.update_roleinfo(self.rolename, roleinfo) + + + + def remove_signature(self, signature): + """ + + Remove a previously loaded, or added, role 'signature'. A role must + contain a threshold number of signatures to be considered fully signed. + + >>> + >>> + >>> + + + signature: + The role signature to remove, conformant to + 'tuf.formats.SIGNATURE_SCHEMA'. + + + tuf.FormatError, if the 'signature' argument is improperly formatted. + + tuf.Error, if 'signature' has not been previously added to this role. + + + Updates the 'signatures' field of the role in 'tuf.roledb.py'. + + + None. + """ + + # Does 'signature' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if any are improperly formatted. + tuf.formats.SIGNATURE_SCHEMA.check_match(signature) + + roleinfo = tuf.roledb.get_roleinfo(self.rolename) + + if signature in roleinfo['signatures']: + roleinfo['signatures'].remove(signature) + + tuf.roledb.update_roleinfo(self.rolename, roleinfo) + + else: + raise tuf.Error('Signature not found.') + + + + @property + def signatures(self): + """ + + A getter method that returns the role's signatures. A role is considered + fully signed if it contains a threshold number of signatures, where each + signature must be provided by the generated by the private key. Keys + are added to a role with the add_verification_key() method. + + + None. + + + None. + + + None. + + + A list of signatures, conformant to 'tuf.formats.SIGNATURES_SCHEMA'. + """ + + roleinfo = tuf.roledb.get_roleinfo(self.rolename) + signatures = roleinfo['signatures'] + + return signatures + + + + @property + def keys(self): + """ + + A getter method that returns the role's keyids of the keys. The role + is expected to eventually contain a threshold of signatures generated + by the private keys of each of the role's keys (returned here as a keyid). + + + None. + + + None. + + + None. + + + A list of the role's keyids (i.e., keyids of the keys). + """ + + roleinfo = tuf.roledb.get_roleinfo(self.rolename) + keyids = roleinfo['keyids'] + + return keyids + + + + @property + def rolename(self): + """ + + Return the role's name. + Examples: 'root', 'timestamp', 'targets/unclaimed/django'. + + + None. + + + None. + + + None. + + + The role's name, conformant to 'tuf.formats.ROLENAME_SCHEMA'. + Examples: 'root', 'timestamp', 'targets/unclaimed/django'. + """ + + return self._rolename + + + + @property + def version(self): + """ + + A getter method that returns the role's version number, conformant to + 'tuf.formats.VERSION_SCHEMA'. + + + None. + + + None. + + + None. + + + The role's version number, conformant to 'tuf.formats.VERSION_SCHEMA'. + """ + + roleinfo = tuf.roledb.get_roleinfo(self.rolename) + version = roleinfo['version'] + + return version + + + + @version.setter + def version(self, version): + """ + + A setter method that updates the role's version number. TUF clients + download new metadata with version number greater than the version + currently trusted. New metadata start at version 1 when either write() + or write_partial() is called. Version numbers are automatically + incremented, when the write methods are called, as follows: + + 1. write_partial==True and the metadata is the first to be written. + + 2. write_partial=False (i.e., write()), the metadata was not loaded as + partially written, and a write_partial is not needed. + + >>> + >>> + >>> + + + version: + The role's version number, conformant to 'tuf.formats.VERSION_SCHEMA'. + + + tuf.FormatError, if the 'version' argument is improperly formatted. + + + Modifies the 'version' attribute of the Repository object and updates + the role's version in 'tuf.roledb.py'. + + + None. + """ + + # Does 'version' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if any are improperly formatted. + tuf.formats.METADATAVERSION_SCHEMA.check_match(version) + + roleinfo = tuf.roledb.get_roleinfo(self.rolename) + roleinfo['version'] = version + + tuf.roledb.update_roleinfo(self._rolename, roleinfo) + + + + @property + def threshold(self): + """ + + Return the role's threshold value. A role is considered fully signed if + a threshold number of signatures is available. + + + None. + + + None. + + + None. + + + The role's threshold value, conformant to 'tuf.formats.THRESHOLD_SCHEMA'. + """ + + roleinfo = tuf.roledb.get_roleinfo(self._rolename) + threshold = roleinfo['threshold'] + + return threshold + + + + @threshold.setter + def threshold(self, threshold): + """ + + A setter method that modified the threshold value of the role. Metadata + is considered fully signed if a 'threshold' number of signatures is + available. + + >>> + >>> + >>> + + + threshold: + An integer value that sets the role's threshold value, or the miminum + number of signatures needed for metadata to be considered fully + signed. Conformant to 'tuf.formats.THRESHOLD_SCHEMA'. + + + tuf.FormatError, if the 'threshold' argument is improperly formatted. + + + Modifies the threshold attribute of the Repository object and updates + the roles threshold in 'tuf.roledb.py'. + + + None. + """ + + # Does 'threshold' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if any are improperly formatted. + tuf.formats.THRESHOLD_SCHEMA.check_match(threshold) + + roleinfo = tuf.roledb.get_roleinfo(self._rolename) + roleinfo['threshold'] = threshold + + tuf.roledb.update_roleinfo(self._rolename, roleinfo) + + + @property + def expiration(self): + """ + + A getter method that returns the role's expiration datetime. + + >>> + >>> + >>> + + + None. + + + None. + + + None. + + + The role's expiration datetime, conformant to + 'tuf.formats.DATETIME_SCHEMA'. + """ + + roleinfo = tuf.roledb.get_roleinfo(self.rolename) + + return roleinfo['expires'] + + + + @expiration.setter + def expiration(self, expiration_datetime_utc): + """ + + A setter method for the role's expiration datetime. The top-level + roles have a default expiration (e.g., ROOT_EXPIRATION), but may later + be modified by this setter method. + + TODO: expiration_datetime_utc in ISO 8601 format. + + >>> + >>> + >>> + + + expiration_datetime_utc: + The datetime expiration of the role, conformant to + 'tuf.formats.DATETIME_SCHEMA'. + + + tuf.FormatError, if 'expiration_datetime_utc' is improperly formatted. + + + Modifies the expiration attribute of the Repository object. + + + None. + """ + + # Does 'expiration_datetime_utc' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if any are improperly formatted. + tuf.formats.DATETIME_SCHEMA.check_match(expiration_datetime_utc) + + # Further validate the datetime, such as a correct date, time, expiration. + # Convert 'expiration_datetime_utc' to a unix timestamp so that it can be + # compared with time.time(). + expiration_datetime_utc = expiration_datetime_utc+' UTC' + try: + unix_timestamp = tuf.formats.parse_time(expiration_datetime_utc) + + except (tuf.FormatError, ValueError), e: + message = 'Invalid datetime argument: '+repr(expiration_datetime_utc) + raise tuf.FormatError(message) + + # Ensure the expiration has not already passed. + if unix_timestamp < time.time(): + message = 'The expiration date must occur after the current date.' + raise tuf.FormatError(message) + + # Update the role's 'expires' entry in 'tuf.roledb.py'. + roleinfo = tuf.roledb.get_roleinfo(self.rolename) + roleinfo['expires'] = expiration_datetime_utc + tuf.roledb.update_roleinfo(self.rolename, roleinfo) + + + + @property + def signing_keys(self): + """ + + A getter method that returns a list of the role's signing keys. + + >>> + >>> + >>> + + + None. + + + None. + + + None. + + + A list of keyids of the role's signing keys, conformant to + 'tuf.formats.KEYIDS_SCHEMA'. + """ + + roleinfo = tuf.roledb.get_roleinfo(self.rolename) + signing_keyids = roleinfo['signing_keyids'] + + return signing_keyids + + + + @property + def compressions(self): + """ + + A getter method that returns a list of the file compression algorithms + used when the metadata is written to disk. If ['gz'] is set for the + 'targets.json' role, the metadata files 'targets.json' and 'targets.json.gz' + are written. + + >>> + >>> + >>> + + + None. + + + None. + + + None. + + + A list of compression algorithms, conformant to + 'tuf.formats.COMPRESSIONS_SCHEMA'. + """ + + tuf.roledb.get_roleinfo(self.rolename) + compressions = roleinfo['compressions'] + + return compressions + + + + @compressions.setter + def compressions(self, compression_list): + """ + + A setter method for the file compression algorithms used when the + metadata is written to disk. If ['gz'] is set for the 'targets.json' role + the metadata files 'targets.json' and 'targets.json.gz' are written. + + >>> + >>> + >>> + + + compression_list: + A list of file compression algorithms, conformant to + 'tuf.formats.COMPRESSIONS_SCHEMA'. + + + tuf.FormatError, if 'compression_list' is improperly formatted. + + + Updates the role's compression algorithms listed in 'tuf.roledb.py'. + + + None. + """ + + # Does 'compression_name' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if any are improperly formatted. + tuf.formats.COMPRESSIONS_SCHEMA.check_match(compression_list) + + roleinfo = tuf.roledb.get_roleinfo(self.rolename) + + # Add the compression algorithms of 'compression_list' to the role's + # entry in 'tuf.roledb.py'. + for compression in compression_list: + if compression not in roleinfo['compressions']: + roleinfo['compressions'].append(compression) + + tuf.roledb.update_roleinfo(self.rolename, roleinfo) + + + + + +class Root(Metadata): + """ + + Represent a Root role object. The root role is responsible for + listing the public keys and threshold of all the top-level roles, including + itself. Top-level metadata is rejected if it does not comply with what is + specified by the Root role. + + This Root object sub-classes Metadata, so the expected Metadata + operations like adding/removing public keys, signatures, private keys, and + updating metadata attributes (e.g., version and expiration) is supported. + Since Root is a top-level role and must exist, a default Root object + is instantiated when a new Repository object is created. + + >>> + >>> + >>> + + + None. + + + None. + + + A 'root' role is added to 'tuf.roledb.py'. + + + None. + """ + + def __init__(self): + + super(Root, self).__init__() + + self._rolename = 'root' + + # By default, 'snapshot' metadata is set to expire 1 week from the current + # time. The expiration may be modified. + expiration = tuf.formats.format_time(time.time()+ROOT_EXPIRATION) + + roleinfo = {'keyids': [], 'signing_keyids': [], 'threshold': 1, + 'signatures': [], 'version': 0, 'consistent_snapshot': False, + 'compressions': [''], 'expires': expiration, + 'partial_loaded': False} + try: + tuf.roledb.add_role(self._rolename, roleinfo) + + except tuf.RoleAlreadyExistsError, e: + pass + + + + + +class Timestamp(Metadata): + """ + + Represent a Timestamp role object. The timestamp role is responsible for + referencing the latest version of the Snapshot role. Under normal + conditions, it is the only role to be downloaded from a remote repository + without a known file length and hash. An upper length limit is set, though. + Also, its signatures are also verified to be valid according to the Root + role. If invalid metadata can only be downloaded by the client, Root + is the only other role that is downloaded without a known length and hash. + This case may occur if a role's signing keys have been revoked and a newer + Root file is needed to list the updated keys. + + This Timestamp object sub-classes Metadata, so the expected Metadata + operations like adding/removing public keys, signatures, private keys, and + updating metadata attributes (e.g., version and expiration) is supported. + Since Snapshot is a top-level role and must exist, a default Timestamp object + is instantiated when a new Repository object is created. + + >>> + >>> + >>> + + + None. + + + None. + + + A 'timestamp' role is added to 'tuf.roledb.py'. + + + None. + """ + + def __init__(self): + + super(Timestamp, self).__init__() + + self._rolename = 'timestamp' + + # By default, 'snapshot' metadata is set to expire 1 week from the current + # time. The expiration may be modified. + expiration = tuf.formats.format_time(time.time()+TIMESTAMP_EXPIRATION) + + roleinfo = {'keyids': [], 'signing_keyids': [], 'threshold': 1, + 'signatures': [], 'version': 0, 'compressions': [''], + 'expires': expiration, 'partial_loaded': False} + + try: + tuf.roledb.add_role(self.rolename, roleinfo) + + except tuf.RoleAlreadyExistsError, e: + pass + + + + + +class Snapshot(Metadata): + """ + + Represent a Snapshot role object. The snapshot role is responsible for + referencing the other top-level roles (excluding Timestamp) and all + delegated roles. + + This Snapshot object sub-classes Metadata, so the expected + Metadata operations like adding/removing public keys, signatures, private + keys, and updating metadata attributes (e.g., version and expiration) is + supported. Since Snapshot is a top-level role and must exist, a default + Snapshot object is instantiated when a new Repository object is created. + + >>> + >>> + >>> + + + None. + + + None. + + + A 'snapshot' role is added to 'tuf.roledb.py'. + + + None. + """ + + def __init__(self): + + super(Snapshot, self).__init__() + + self._rolename = 'snapshot' + + # By default, 'snapshot' metadata is set to expire 1 week from the current + # time. The expiration may be modified. + expiration = tuf.formats.format_time(time.time()+SNAPSHOT_EXPIRATION) + + roleinfo = {'keyids': [], 'signing_keyids': [], 'threshold': 1, + 'signatures': [], 'version': 0, 'compressions': [''], + 'expires': expiration, 'partial_loaded': False} + + try: + tuf.roledb.add_role(self._rolename, roleinfo) + + except tuf.RoleAlreadyExistsError, e: + pass + + + + + +class Targets(Metadata): + """ + + Represent a Targets role object. Targets roles include the top-level role + 'targets.json' and all delegated roles (e.g., 'targets/unclaimed/django'). + The expected operations of Targets metadata is included, such as adding + and removing repository target files, making and revoking delegations, and + listing the target files provided by it. + + Adding or removing a delegation causes the attributes of the Targets object + to be updated. That is, if the 'django' Targets object is delegated by + 'targets/unclaimed', a new attribute is added so that the following + code statement is supported: + repository.targets('unclaimed')('django').version = 2 + + Likewise, revoking a delegation causes removal of the delegation attribute. + + This Targets object sub-classes Metadata, so the expected + Metadata operations like adding/removing public keys, signatures, private + keys, and updating metadata attributes (e.g., version and expiration) is + supported. Since Targets is a top-level role and must exist, a default + Targets object (for 'targets.json', not delegated roles) is instantiated when + a new Repository object is created. + + >>> + >>> + >>> + + + targets_directory: + The targets directory of the Repository object. + + rolename: + The rolename of this Targets object. + + roleinfo: + An already populated roleinfo object of 'rolename'. Conformant to + 'tuf.formats.ROLEDB_SCHEMA'. + + + tuf.FormatError, if the arguments are improperly formatted. + + + Modifies the roleinfo of the targets role in 'tuf.roledb'. + + + None. + """ + + def __init__(self, targets_directory, rolename, roleinfo=None): + + # Do the arguments have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if any are improperly formatted. + tuf.formats.PATH_SCHEMA.check_match(targets_directory) + tuf.formats.ROLENAME_SCHEMA.check_match(rolename) + + if roleinfo is not None: + tuf.formats.ROLEDB_SCHEMA.check_match(roleinfo) + + super(Targets, self).__init__() + self._targets_directory = targets_directory + self._rolename = rolename + self._target_files = [] + self._delegated_roles = {} + + # By default, Targets objects are set to expire 3 months from the current + # time. May be later modified. + expiration = tuf.formats.format_time(time.time()+TARGETS_EXPIRATION) + + # If 'roleinfo' is not provided, set an initial default. + if roleinfo is None: + roleinfo = {'keyids': [], 'signing_keyids': [], 'threshold': 1, + 'version': 0, 'compressions': [''], 'expires': expiration, + 'signatures': [], 'paths': [], 'path_hash_prefixes': [], + 'partial_loaded': False, 'delegations': {'keys': {}, + 'roles': []}} + + # Add the new role to the 'tuf.roledb'. + try: + tuf.roledb.add_role(self.rolename, roleinfo) + + except tuf.RoleAlreadyExistsError, e: + pass + + + + def __call__(self, rolename): + """ + + Allow callable Targets object so that delegated roles may be referenced + by their string rolenames. Rolenames may include characters like '-' and + are not restricted to Python identifiers. + + + rolename: + The rolename of the delegated role. 'rolename' must be a role + previously delegated by this Targets role. + + + tuf.FormatError, if the arguments are improperly formatted. + + + Modifies the roleinfo of the targets role in 'tuf.roledb'. + + + None. + """ + + # Do the arguments have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if any are improperly formatted. + tuf.formats.ROLENAME_SCHEMA.check_match(rolename) + + if rolename in self._delegated_roles: + return self._delegated_roles[rolename] + else: + message = repr(rolename)+' has not been delegated by '+repr(self.rolename) + raise tuf.UnknownRoleError(message) + + + + @property + def target_files(self): + """ + + A getter method that returns the target files added thus far to this + Targets object. + + >>> + >>> + >>> + + + None. + + + None. + + + None. + + + None. + """ + + target_files = tuf.roledb.get_roleinfo(self._rolename)['paths'] + + return target_files + + + + def add_restricted_paths(self, list_of_directory_paths, child_rolename): + """ + + Add 'list_of_directory_paths' to the restricted paths of 'child_rolename'. + The updater client verifies the target paths specified by child roles, and + searches for targets by visiting these restricted paths. A child role may + only provide targets specifically listed in the delegations field of the + parent, or a target that falls under a restricted path. + + >>> + >>> + >>> + + + list_of_directory_paths: + A list of directory paths 'child_rolename' should also be restricted to. + + child_rolename: + The child delegation that requires an update to its restricted paths, + as listed in the parent role's delegations. + + + tuf.Error, if a directory path in 'list_of_directory_paths' is not a + directory, or not under the repository's targets directory. If + 'child_rolename' has not been delegated yet. + + + Modifies this Targets' delegations field. + + + None. + """ + + # Does 'filepath' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.PATHS_SCHEMA.check_match(list_of_directory_paths) + tuf.formats.ROLENAME_SCHEMA.check_match(child_rolename) + + # A list of verified paths to be added to the child role's entry in the + # parent's delegations. + directory_paths = [] + + # Ensure the 'child_rolename' has been delegated, otherwise it will not + # have an entry in the parent role's delegations field. + full_child_rolename = self._rolename + '/' + child_rolename + if not tuf.roledb.role_exists(full_child_rolename): + raise tuf.Error(repr(full_child_rolename)+' has not been delegated.') + + # Are the paths in 'list_of_directory_paths' valid? + for directory_path in list_of_directory_paths: + directory_path = os.path.abspath(directory_path) + if not os.path.isdir(directory_path): + message = repr(directory_path)+ ' is not a directory.' + raise tuf.Error(message) + + # Are the paths in the repository's targets directory? Append a trailing + # path separator with os.path.join(path, ''). + targets_directory = os.path.join(self._targets_directory, '') + directory_path = os.path.join(directory_path, '') + if not directory_path.startswith(targets_directory): + message = repr(directory_path)+' is not under the Repository\'s '+\ + 'targets directory: '+repr(self._targets_directory) + raise tuf.Error(message) + + directory_paths.append(directory_path[len(self._targets_directory):]) + + # Get the current role's roleinfo, so that its delegations field can be + # updated. + roleinfo = tuf.roledb.get_roleinfo(self._rolename) + + # Update the restricted paths of 'child_rolename'. + for role in roleinfo['delegations']['roles']: + if role['name'] == full_child_rolename: + restricted_paths = role['paths'] + + for directory_path in directory_paths: + if directory_path not in restricted_paths: + restricted_paths.append(directory_path) + + tuf.roledb.update_roleinfo(self._rolename, roleinfo) + + + + def add_target(self, filepath): + """ + + Add a filepath (must be under the repository's targets directory) to the + Targets object. + + This method does not actually create 'filepath' on the file system. + 'filepath' must already exist on the file system. + + >>> + >>> + >>> + + + filepath: + The path of the target file. It must be located in the repository's + targets directory. + + + tuf.FormatError, if 'filepath' is improperly formatted. + + tuf.Error, if 'filepath' is not found under the repository's targets + directory. + + + Adds 'filepath' to this role's list of targets. This role's + 'tuf.roledb.py' is also updated. + + + None. + """ + + # Does 'filepath' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.PATH_SCHEMA.check_match(filepath) + + filepath = os.path.abspath(filepath) + + # Ensure 'filepath' is found under the repository's targets directory. + if not filepath.startswith(self._targets_directory): + message = repr(filepath)+' is not under the Repository\'s targets '+\ + 'directory: '+repr(self._targets_directory) + raise tuf.Error(message) + + # Add 'filepath' (i.e., relative to the targets directory) to the role's + # list of targets. 'filepath' will be verified as an allowed path according + # to this Targets parent role when write() is called. Not verifying + # 'filepath' here allows freedom to add targets and parent restrictions + # in any order, and minimize the number of times these checks are performed. + if os.path.isfile(filepath): + + # Update the role's 'tuf.roledb.py' entry and avoid duplicates. + targets_directory_length = len(self._targets_directory) + roleinfo = tuf.roledb.get_roleinfo(self._rolename) + relative_path = filepath[targets_directory_length:] + if relative_path not in roleinfo['paths']: + roleinfo['paths'].append(relative_path) + tuf.roledb.update_roleinfo(self._rolename, roleinfo) + + else: + message = repr(filepath)+' is not a valid file.' + raise tuf.Error(message) + + + + def add_targets(self, list_of_targets): + """ + + Add a list of target filepaths (all relative to 'self.targets_directory'). + This method does not actually create files on the file system. The + list of target must already exist. + + >>> + >>> + >>> + + + list_of_targets: + A list of target filepaths that are added to the paths of this Targets + object. + + + tuf.FormatError, if the arguments are improperly formatted. + + tuf.Error, if any of the paths listed in 'list_of_targets' is not found + under the repository's targets directory or is invalid. + + + This Targets' roleinfo is updated with the paths in 'list_of_targets'. + + + None. + """ + + # Does 'list_of_targets' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.RELPATHS_SCHEMA.check_match(list_of_targets) + + # Update the tuf.roledb entry. + targets_directory_length = len(self._targets_directory) + absolute_list_of_targets = [] + relative_list_of_targets = [] + + # Ensure the paths in 'list_of_targets' are valid and fall under the + # repository's targets directory. The paths of 'list_of_targets' will be + # verified as allowed paths according to this Targets parent role when + # write() is called. Not verifying filepaths here allows the freedom to add + # targets and parent restrictions in any order, and minimize the number of + # times these checks are performed. + for target in list_of_targets: + filepath = os.path.abspath(target) + + if not filepath.startswith(self._targets_directory+os.sep): + message = repr(filepath)+' is not under the Repository\'s targets '+\ + 'directory: '+repr(self._targets_directory) + raise tuf.Error(message) + + if os.path.isfile(filepath): + absolute_list_of_targets.append(filepath) + relative_list_of_targets.append(filepath[targets_directory_length:]) + else: + message = repr(filepath)+' is not a valid file.' + raise tuf.Error(message) + + # Update this Targets 'tuf.roledb.py' entry. + roleinfo = tuf.roledb.get_roleinfo(self._rolename) + for relative_target in relative_list_of_targets: + if relative_target not in roleinfo['paths']: + roleinfo['paths'].append(relative_target) + + tuf.roledb.update_roleinfo(self.rolename, roleinfo) + + + + def remove_target(self, filepath): + """ + + Remove the target 'filepath' from this Targets' 'paths' field. 'filepath' + is relative to the targets directory. + + >>> + >>> + >>> + + + filepath: + The target to remove from this Targets object, relative to the + repository's targets directory. + + + tuf.FormatError, if 'filepath' is improperly formatted. + + tuf.Error, if 'filepath' is not under the repository's targets directory, + or not found. + + + Modifies this Targets 'tuf.roledb.py' entry. + + + None. + """ + + # Does 'filepath' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.RELPATH_SCHEMA.check_match(filepath) + + filepath = os.path.abspath(filepath) + targets_directory_length = len(self._targets_directory) + + # Ensure 'filepath' is under the repository targets directory. + if not filepath.startswith(self._targets_directory+os.sep): + message = repr(filepath)+' is not under the Repository\'s targets '+\ + 'directory: '+repr(self._targets_directory) + raise tuf.Error(message) + + # The relative filepath is listed in 'paths'. + relative_filepath = filepath[targets_directory_length:] + + # Remove 'relative_filepath', if found, and update this Targets roleinfo. + fileinfo = tuf.roledb.get_roleinfo(self.rolename) + if relative_filepath in fileinfo['paths']: + fileinfo['paths'].remove(relative_filepath) + tuf.roledb.update_roleinfo(self.rolename, fileinfo) + + else: + raise tuf.Error('Target file path not found.') + + + + def clear_targets(self): + """ + + Remove all the target filepaths in the "paths" field of this Targets. + + >>> + >>> + >>> + + + None + + + None. + + + Modifies this Targets' 'tuf.roledb.py' entry. + + + None. + """ + + roleinfo = tuf.roledb.get_roleinfo(self.rolename) + roleinfo['paths'] = [] + + tuf.roledb.update_roleinfo(self.rolename, roleinfo) + + + + def get_delegated_rolenames(self): + """ + + Return all delegations of a role, including any made by child delegations. + If ['a/b/', 'a/b/c/', 'a/b/c/d'] have been delegated, + repository.a.get_delegated_rolenames() returns: + ['a/b', 'a/b/c', 'a/b/c/d']. + + + None. + + + None. + + + None. + + + A list of rolenames. + """ + + return tuf.roledb.get_delegated_rolenames(self.rolename) + + + + def delegate(self, rolename, public_keys, list_of_targets, + threshold=1, restricted_paths=None, path_hash_prefixes=None): + """ + + Create a new delegation, where 'rolename' is a child delegation of this + Targets object. The keys and roles database is updated, including the + delegations field of this Targets. The delegation of 'rolename' is added + and accessible (e.g., 'repository.targets(rolename). + + Actual metadata files are not updated, only when repository.write() or + repository.write_partial() is called. + + >>> + >>> + >>> + + + rolename: + The name of the delegated role, as in 'django' (i.e., not the full + rolename). + + public_keys: + A list of TUF key objects in 'ANYKEYLIST_SCHEMA' format. The list + may contain any of the supported key types: RSAKEY_SCHEMA, + ED25519KEY_SCHEMA, etc. + + list_of_targets: + A list of target filepaths that are added to the paths of 'rolename'. + 'list_of_targets' is a list of target filepaths, and can be empty. + + threshold: + The threshold number of keys of 'rolename'. + + restricted_paths: + A list of restricted directory or file paths of 'rolename'. Any target + files added to 'rolename' must fall under one of the restricted paths. + + path_hash_prefixes: + A list of hash prefixes in 'tuf.formats.PATH_HASH_PREFIXES_SCHEMA' + format, used in hashed bin delegations. Targets may be located and + stored in hashed bins by calculating the target path's hash prefix. + + + tuf.FormatError, if any of the arguments are improperly formatted. + + tuf.Error, if the delegated role already exists or if any of the arguments + is an invalid path (i.e., not under the repository's targets directory). + + + A new Target object is created for 'rolename' that is accessible to the + caller (i.e., targets.unclaimed.). The 'tuf.keydb.py' and + 'tuf.roledb.py' stores are updated with 'public_keys'. + + + None. + """ + + # Do the arguments have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.ROLENAME_SCHEMA.check_match(rolename) + tuf.formats.ANYKEYLIST_SCHEMA.check_match(public_keys) + tuf.formats.RELPATHS_SCHEMA.check_match(list_of_targets) + tuf.formats.THRESHOLD_SCHEMA.check_match(threshold) + if restricted_paths is not None: + tuf.formats.RELPATHS_SCHEMA.check_match(restricted_paths) + if path_hash_prefixes is not None: + tuf.formats.PATH_HASH_PREFIXES_SCHEMA.check_match(path_hash_prefixes) + + # Check if 'rolename' is not already a delegation. 'tuf.roledb' expects the + # full rolename. + full_rolename = self._rolename+'/'+rolename + + if tuf.roledb.role_exists(full_rolename): + raise tuf.Error(repr(full_rolename)+' already delegated.') + + # Keep track of the valid keyids (added to the new Targets object) and their + # keydicts (added to this Targets delegations). + keyids = [] + keydict = {} + + # Add all the keys of 'public_keys' to tuf.keydb. + for key in public_keys: + + # Add 'key' to the list of recognized keys. Keys may be shared, + # so do not raise an exception if 'key' has already been loaded. + try: + tuf.keydb.add_key(key) + + except tuf.KeyAlreadyExistsError, e: + message = \ + 'Adding a public key that has already been used: '+key['keyid'] + logger.warn(message) + + keyid = key['keyid'] + key_metadata_format = tuf.keys.format_keyval_to_metadata(key['keytype'], + key['keyval']) + # Update 'keyids' and 'keydict'. + new_keydict = {keyid: key_metadata_format} + keydict.update(new_keydict) + keyids.append(keyid) + + # Ensure the paths of 'list_of_targets' all fall under the repository's + # targets. + relative_targetpaths = [] + targets_directory_length = len(self._targets_directory) + + for target in list_of_targets: + target = os.path.abspath(target) + if not target.startswith(self._targets_directory+os.sep): + message = repr(target)+' is not under the Repository\'s targets '+\ + 'directory: '+repr(self._targets_directory) + raise tuf.Error(message) + + relative_targetpaths.append(target[targets_directory_length:]) + + # Ensure the paths of 'restricted_paths' all fall under the repository's + # targets. + relative_restricted_paths = [] + + if restricted_paths is not None: + for path in restricted_paths: + path = os.path.abspath(path)+os.sep + if not path.startswith(self._targets_directory+os.sep): + message = repr(path)+' is not under the Repository\'s targets '+\ + 'directory: '+repr(self._targets_directory) + raise tuf.Error(message) + + # Append a trailing path separator with os.path.join(path, ''). + path = os.path.join(path, '') + relative_restricted_paths.append(path[targets_directory_length:]) + + # Create a new Targets object for the 'rolename' delegation. An initial + # expiration is set (3 months from the current time). + expiration = tuf.formats.format_time(time.time()+TARGETS_EXPIRATION) + roleinfo = {'name': full_rolename, 'keyids': keyids, 'signing_keyids': [], + 'threshold': threshold, 'version': 0, 'compressions': [''], + 'expires': expiration, 'signatures': [], 'partial_loaded': False, + 'paths': relative_targetpaths, 'delegations': {'keys': {}, + 'roles': []}} + + # The new targets object is added as an attribute to this Targets object. + new_targets_object = Targets(self._targets_directory, full_rolename, + roleinfo) + + # Update the 'delegations' field of the current role. + current_roleinfo = tuf.roledb.get_roleinfo(self.rolename) + current_roleinfo['delegations']['keys'].update(keydict) + + # Update the roleinfo of this role. A ROLE_SCHEMA object requires only + # 'keyids', 'threshold', and 'paths'. + roleinfo = {'name': full_rolename, + 'keyids': roleinfo['keyids'], + 'threshold': roleinfo['threshold'], + 'paths': roleinfo['paths']} + if restricted_paths is not None: + roleinfo['paths'] = relative_restricted_paths + if path_hash_prefixes is not None: + roleinfo['path_hash_prefixes'] = path_hash_prefixes + # A role in a delegations must list either 'path_hash_prefixes' + # or 'paths'. + del roleinfo['paths'] + + current_roleinfo['delegations']['roles'].append(roleinfo) + tuf.roledb.update_roleinfo(self.rolename, current_roleinfo) + + # Update the public keys of 'new_targets_object'. + for key in public_keys: + new_targets_object.add_verification_key(key) + + # Add the new delegation to this Targets object. For example, 'django' is + # added to 'repository.targets' (i.e., repository.targets('django')). + self._delegated_roles[rolename] = new_targets_object + + + + def revoke(self, rolename): + """ + + Revoke this Targets' 'rolename' delegation. Its 'rolename' attribute is + deleted, including the entries in its 'delegations' field and in + 'tuf.roledb'. + + Actual metadata files are not updated, only when repository.write() or + repository.write_partial() is called. + + >>> + >>> + >>> + + + rolename: + The rolename (e.g., 'Django' in 'targets/unclaimed/Django') of + the child delegation the parent role (this role) wants to revoke. + + + tuf.FormatError, if 'rolename' is improperly formatted. + + + The delegations dictionary of 'rolename' is modified, and its 'tuf.roledb' + entry is updated. This Targets' 'rolename' delegation attribute is also + deleted. + + + None. + """ + + # Does 'rolename' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.ROLENAME_SCHEMA.check_match(rolename) + + # Remove 'rolename' from this Target's delegations dict. + # The child delegation's full rolename is required to locate in the parent's + # delegations list. + roleinfo = tuf.roledb.get_roleinfo(self.rolename) + full_rolename = self.rolename+'/'+rolename + + for role in roleinfo['delegations']['roles']: + if role['name'] == full_rolename: + roleinfo['delegations']['roles'].remove(role) + + tuf.roledb.update_roleinfo(self.rolename, roleinfo) + + # Remove 'rolename' from 'tuf.roledb.py'. The delegations of 'rolename' are + # also removed. + tuf.roledb.remove_role(full_rolename) + + # Remove the rolename delegation from the current role. For example, the + # 'django' role is removed from 'repository.targets('unclaimed')('django'). + del self._delegated_roles[rolename] + + + + def delegate_hashed_bins(self, list_of_targets, keys_of_hashed_bins, + number_of_bins=1024): + """ + + Distribute a large number of target files into multiple delegated roles + (hashed bins). The metadata files of delegated roles will be nearly equal + in size (i.e., 'list_of_targets' is uniformly distributed by calculating + the target filepath's hash and determing which bin it should reside in. + The updater client will use "lazy bin walk" to find a target file's hashed + bin destination. The parent role lists a range of path hash prefixes each + hashed bin contains. This method is intended for repositories with a + large number of target files, a way of easily distributing and managing + the metadata that lists the targets, and minimizing the number of metadata + files (and their size) downloaded by the client. See tuf-spec.txt and the + following link for more information: + http://www.python.org/dev/peps/pep-0458/#metadata-scalability + + >>> + >>> + >>> + + + list_of_targets: + The target filepaths of the targets that should be stored in hashed + bins created (i.e., delegated roles). A repository object's + get_filepaths_in_directory() can generate a list of valid target + paths. + + keys_of_hashed_bins: + The initial public keys of the delegated roles. Public keys may be + later added or removed by calling the usual methods of the delegated + Targets object. For example: + repository.targets('unclaimed')('000-003').add_verification_key() + + number_of_bins: + The number of delegated roles, or hashed bins, that should be generated + and contain the target file attributes listed in 'list_of_targets'. + 'number_of_bins' must be a multiple of 16. Each bin may contain a + range of path hash prefixes (e.g., target filepath digests that range + from [000]... - [003]..., where the series of digits in brackets is + considered the hash prefix). + + + tuf.FormatError, if the arguments are improperly formatted, + 'number_of_bins' is not a multiple of 16, or one of the targets + in 'list_of_targets' is not located under the repository's targets + directory. + + + Delegates multiple target roles from the current parent role. + + + None. + """ + + # Do the arguments have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.PATHS_SCHEMA.check_match(list_of_targets) + tuf.formats.ANYKEYLIST_SCHEMA.check_match(keys_of_hashed_bins) + tuf.formats.NUMBINS_SCHEMA.check_match(number_of_bins) + + # Determine the hex number of hashed bins from 'number_of_bins' and the + # maximum number of bins provided by the total number of hex digits needed. + # Strip the '0x' from the Python hex representation. 'prefix_length' + # and 'max_number_of_bins' affect hashed bin rolenames and the range of + # prefixes of each bin. + prefix_length = len(hex(number_of_bins - 1)[2:]) + max_number_of_bins = 16 ** prefix_length + + # For simplicity, ensure that we can evenly distribute 'max_number_of_bins' + # over 'number_of_bins'. Each bin will contain + # max_number_of_bin/number_of_bins hash prefixes. + if max_number_of_bins % number_of_bins != 0: + message = 'The number of bins argument must be a multiple of 16.' + raise tuf.FormatError(message) + + logger.info('There are '+str(len(list_of_targets))+' total targets.') + + # Store the target paths that fall into each bin. The digest of the + # target path, reduced to the first 'prefix_length' hex digits, is + # calculated to determine which 'bin_index' is should go. + target_paths_in_bin = {} + for bin_index in xrange(max_number_of_bins): + target_paths_in_bin[bin_index] = [] + + # Assign every path to its bin. Ensure every target is located under the + # repository's targets directory. + for target_path in list_of_targets: + target_path = os.path.abspath(target_path) + if not target_path.startswith(self._targets_directory+os.sep): + message = 'A path in the list of targets argument is not '+\ + 'under the repository\'s targets directory: '+repr(target_path) + raise tuf.FormatError(message) + + # Determine the hash prefix of 'target_path' by computing the digest of + # its path relative to the targets directory. Example: + # '{repository_root}/targets/file1.txt' -> 'file1.txt'. + relative_path = target_path[len(self._targets_directory):] + digest_object = tuf.hash.digest(algorithm=HASH_FUNCTION) + digest_object.update(relative_path) + relative_path_hash = digest_object.hexdigest() + relative_path_hash_prefix = relative_path_hash[:prefix_length] + + # 'target_paths_in_bin' store bin indices in base-10, so convert the + # 'relative_path_hash_prefix' base-16 (hex) number to a base-10 (dec) + # number. + bin_index = int(relative_path_hash_prefix, 16) + + # Add the 'target_path' (absolute) to the bin. These target paths are + # later added to the targets of the 'bin_index' role. + target_paths_in_bin[bin_index].append(target_path) + + # Calculate the path hash prefixes of each bin_offset stored in the parent + # role. For example: 'targets/unclaimed/000-003' may list the path hash + # prefixes "000", "001", "002", "003" in the delegations dict of + # 'targets/unclaimed'. + bin_offset = max_number_of_bins // number_of_bins + + # The parent roles will list bin roles starting from "0" to + # 'max_number_of_bins' in 'bin_offset' increments. The skipped bin roles + # are listed in 'path_hash_prefixes' of 'outer_bin_index. + for outer_bin_index in xrange(0, max_number_of_bins, bin_offset): + # The bin index is hex padded from the left with zeroes for up to the + # 'prefix_length' (e.g., 'targets/unclaimed/000-003'). Ensure the correct + # hash bin name is generated if a prefix range is unneeded. + start_bin = hex(outer_bin_index)[2:].zfill(prefix_length) + end_bin = hex(outer_bin_index+bin_offset-1)[2:].zfill(prefix_length) + if start_bin == end_bin: + bin_rolename = start_bin + else: + bin_rolename = start_bin + '-' + end_bin + + # 'bin_rolename' may contain a range of target paths, from 'start_bin' to + # 'end_bin'. Determine the total target paths that should be included. + path_hash_prefixes = [] + bin_rolename_targets = [] + + for inner_bin_index in xrange(outer_bin_index, outer_bin_index+bin_offset): + # 'inner_bin_rolename' needed in padded hex. For example, "00b". + inner_bin_rolename = hex(inner_bin_index)[2:].zfill(prefix_length) + path_hash_prefixes.append(inner_bin_rolename) + bin_rolename_targets.extend(target_paths_in_bin[inner_bin_index]) + + # Delegate from the "unclaimed" targets role to each 'bin_rolename' + # (i.e., outer_bin_index). + self.delegate(bin_rolename, keys_of_hashed_bins, + list_of_targets=bin_rolename_targets, + path_hash_prefixes=path_hash_prefixes) + + message = 'Delegated from '+repr(self.rolename)+' to '+repr(bin_rolename) + logger.debug(message) + + + + @property + def delegations(self): + """ + + A getter method that returns the delegations made by this Targets role. + + >>> + >>> + >>> + + + None. + + + tuf.UnknownRoleError, if this Targets' rolename does not exist in + 'tuf.roledb'. + + + None. + + + A list containing the Targets objects of this Targets' delegations. + """ + + return self._delegated_roles.values() + + + + + +def _generate_and_write_metadata(rolename, metadata_filename, write_partial, + targets_directory, metadata_directory, + consistent_snapshot=False, filenames=None): + """ + Non-public function that can generate and write the metadata of the specified + top-level 'rolename'. It also increments version numbers if: + + 1. write_partial==True and the metadata is the first to be written. + + 2. write_partial=False (i.e., write()), the metadata was not loaded as + partially written, and a write_partial is not needed. + """ + + metadata = None + + # Retrieve the roleinfo of 'rolename' to extract the needed metadata + # attributes, such as version number, expiration, etc. + roleinfo = tuf.roledb.get_roleinfo(rolename) + snapshot_compressions = tuf.roledb.get_roleinfo('snapshot')['compressions'] + + # Generate the appropriate role metadata for 'rolename'. + if rolename == 'root': + metadata = generate_root_metadata(roleinfo['version'], + roleinfo['expires'], consistent_snapshot) + + # Check for the Targets role, including delegated roles. + elif rolename.startswith('targets'): + metadata = generate_targets_metadata(targets_directory, + roleinfo['paths'], + roleinfo['version'], + roleinfo['expires'], + roleinfo['delegations'], + consistent_snapshot) + + elif rolename == 'snapshot': + root_filename = filenames['root'] + targets_filename = filenames['targets'] + metadata = generate_snapshot_metadata(metadata_directory, + roleinfo['version'], + roleinfo['expires'], root_filename, + targets_filename, + consistent_snapshot ) + + elif rolename == 'timestamp': + snapshot_filename = filenames['snapshot'] + metadata = generate_timestamp_metadata(snapshot_filename, + roleinfo['version'], + roleinfo['expires'], + snapshot_compressions) + + signable = sign_metadata(metadata, roleinfo['signing_keyids'], + metadata_filename) + + # Check if the version number of 'rolename' may be automatically incremented, + # depending on whether if partial metadata is loaded or if the metadata is + # written with write() / write_partial(). + # Increment the version number if this is the first partial write. + if write_partial: + temp_signable = sign_metadata(metadata, [], metadata_filename) + temp_signable['signatures'].extend(roleinfo['signatures']) + status = tuf.sig.get_signature_status(temp_signable, rolename) + if len(status['good_sigs']) == 0: + metadata['version'] = metadata['version'] + 1 + signable = sign_metadata(metadata, roleinfo['signing_keyids'], + metadata_filename) + # non-partial write() + else: + if tuf.sig.verify(signable, rolename) and not roleinfo['partial_loaded']: + metadata['version'] = metadata['version'] + 1 + signable = sign_metadata(metadata, roleinfo['signing_keyids'], + metadata_filename) + + # Write the metadata to file if contains a threshold of signatures. + signable['signatures'].extend(roleinfo['signatures']) + + if tuf.sig.verify(signable, rolename) or write_partial: + _remove_invalid_and_duplicate_signatures(signable) + compressions = roleinfo['compressions'] + filename = write_metadata_file(signable, metadata_filename, compressions, + consistent_snapshot) + + # The root and timestamp files should also be written without a digest if + # 'consistent_snaptshots' is True. Client may request a timestamp and root + # file without knowing its digest and file size. + if rolename == 'root' or rolename == 'timestamp': + write_metadata_file(signable, metadata_filename, compressions, + consistent_snapshot=False) + + + # 'signable' contains an invalid threshold of signatures. + else: + message = 'Not enough signatures for '+repr(metadata_filename) + raise tuf.UnsignedMetadataError(message, signable) + + return signable, filename + + + + + +def _print_status_of_top_level_roles(targets_directory, metadata_directory): + """ + Non-public function that prints whether any of the top-level roles contain an + invalid number of public and private keys, or an insufficient threshold of + signatures. Considering that the top-level metadata have to be verified in + the expected root -> targets -> snapshot -> timestamp order, this function + prints the error message and returns as soon as a required metadata file is + found to be invalid. It is assumed here that the delegated roles have been + written and verified. Example output: + + 'root' role contains 1 / 1 signatures. + 'targets' role contains 1 / 1 signatures. + 'snapshot' role contains 1 / 1 signatures. + 'timestamp' role contains 1 / 1 signatures. + + Note: Temporary metadata is generated so that file hashes & sizes may be + computed and verified against the attached signatures. 'metadata_directory' + should be a directory in a temporary repository directory. + """ + + # The expected full filenames of the top-level roles needed to write them to + # disk. + filenames = get_metadata_filenames(metadata_directory) + root_filename = filenames[ROOT_FILENAME] + targets_filename = filenames[TARGETS_FILENAME] + snapshot_filename = filenames[SNAPSHOT_FILENAME] + timestamp_filename = filenames[TIMESTAMP_FILENAME] + + # Verify that the top-level roles contain a valid number of public keys and + # that their corresponding private keys have been loaded. + for rolename in ['root', 'targets', 'snapshot', 'timestamp']: + try: + _check_role_keys(rolename) + + except tuf.InsufficientKeysError, e: + print(str(e)) + return + + # Do the top-level roles contain a valid threshold of signatures? Top-level + # metadata is verified in Root -> Targets -> Snapshot -> Timestamp order. + # Verify the metadata of the Root role. + try: + signable, root_filename = \ + _generate_and_write_metadata('root', root_filename, False, + targets_directory, metadata_directory) + _print_status('root', signable) + + # 'tuf.UnsignedMetadataError' raised if metadata contains an invalid threshold + # of signatures. Print the valid/threshold message, where valid < threshold. + except tuf.UnsignedMetadataError, e: + signable = e[1] + _print_status('root', signable) + return + + # Verify the metadata of the Targets role. + try: + signable, targets_filename = \ + _generate_and_write_metadata('targets', targets_filename, False, + targets_directory, metadata_directory) + _print_status('targets', signable) + + except tuf.UnsignedMetadataError, e: + signable = e[1] + _print_status('targets', signable) + return + + # Verify the metadata of the snapshot role. + filenames = {'root': root_filename, 'targets': targets_filename} + try: + signable, snapshot_filename = \ + _generate_and_write_metadata('snapshot', snapshot_filename, False, + targets_directory, metadata_directory, + False, filenames) + _print_status('snapshot', signable) + + except tuf.UnsignedMetadataError, e: + signable = e[1] + _print_status('snapshot', signable) + return + + # Verify the metadata of the Timestamp role. + filenames = {'snapshot': snapshot_filename} + try: + signable, snapshot_filename = \ + _generate_and_write_metadata('timestamp', snapshot_filename, False, + targets_directory, metadata_directory, + False, filenames) + _print_status('timestamp', signable) + + except tuf.UnsignedMetadataError, e: + signable = e[1] + _print_status('timestamp', signable) + return + + + + +def _print_status(rolename, signable): + """ + Non-public function prints the number of (good/threshold) signatures of + 'rolename'. + """ + + status = tuf.sig.get_signature_status(signable, rolename) + + message = repr(rolename)+' role contains '+ repr(len(status['good_sigs']))+\ + ' / '+repr(status['threshold'])+' signatures.' + print(message) + + + + + +def _prompt(message, result_type=str): + """ + Non-public function that prompts the user for input by printing 'message', + converting the input to 'result_type', and returning the value to the + caller. + """ + + return result_type(raw_input(message)) + + + + + +def _get_password(prompt='Password: ', confirm=False): + """ + Non-public function that returns the password entered by the user. If + 'confirm' is True, the user is asked to enter the previously entered + password once again. If they match, the password is returned to the caller. + """ + + while True: + # getpass() prompts the user for a password without echoing + # the user input. + password = getpass.getpass(prompt, sys.stderr) + if not confirm: + return password + password2 = getpass.getpass('Confirm: ', sys.stderr) + if password == password2: + return password + else: + print('Mismatch; try again.') + + + + + +def _check_if_partial_loaded(rolename, signable, roleinfo): + """ + Non-public function that determines whether 'rolename' is loaded with + at least 1 good signatures, but an insufficient threshold (which means + 'rolename' was written to disk with repository.write_partial(). If 'rolename' + is found to be partially loaded, mark it as partially loaded in its + 'tuf.roledb' roleinfo. This function exists to assist in deciding whether + a role's version number should be incremented when write() or write_parital() + is called. + """ + + # The signature status lists the number of good signatures, including + # bad, untrusted, unknown, etc. + status = tuf.sig.get_signature_status(signable, rolename) + + if len(status['good_sigs']) < status['threshold'] and \ + len(status['good_sigs']) >= 1: + roleinfo['partial_loaded'] = True + + + + + +def _check_directory(directory): + """ + + Non-public function that ensures 'directory' is valid and it exists. This + is not a security check, but a way for the caller to determine the cause of + an invalid directory provided by the user. If the directory argument is + valid, it is returned normalized and as an absolute path. + + + directory: + The directory to check. + + + tuf.Error, if 'directory' could not be validated. + + tuf.FormatError, if 'directory' is not properly formatted. + + + None. + + + The normalized absolutized path of 'directory'. + """ + + # Does 'directory' have the correct format? + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.PATH_SCHEMA.check_match(directory) + + # Check if the directory exists. + if not os.path.isdir(directory): + raise tuf.Error(repr(directory)+' directory does not exist.') + + directory = os.path.abspath(directory) + + return directory + + + + + +def _check_role_keys(rolename): + """ + Non-public function that verifies the public and signing keys of 'rolename'. + If either contain an invalid threshold of keys, raise an exception. + 'rolename' is the full rolename (e.g., 'targets/unclaimed/django'). + """ + + # Extract the total number of public and private keys of 'rolename' from its + # roleinfo in 'tuf.roledb'. + roleinfo = tuf.roledb.get_roleinfo(rolename) + total_keyids = len(roleinfo['keyids']) + threshold = roleinfo['threshold'] + total_signatures = len(roleinfo['signatures']) + total_signing_keys = len(roleinfo['signing_keyids']) + + # Raise an exception for an invalid threshold of public keys. + if total_keyids < threshold: + message = repr(rolename)+' role contains '+repr(total_keyids)+' / '+ \ + repr(threshold)+' public keys.' + raise tuf.InsufficientKeysError(message) + + # Raise an exception for an invalid threshold of signing keys. + if total_signatures == 0 and total_signing_keys < threshold: + message = repr(rolename)+' role contains '+repr(total_signing_keys)+' / '+ \ + repr(threshold)+' signing keys.' + raise tuf.InsufficientKeysError(message) + + + + + +def _remove_invalid_and_duplicate_signatures(signable): + """ + Non-public function that removes invalid signatures from 'signable'. + 'signable' may contain signatures (invalid) from previous versions + of the metadata that were loaded with load_repository(). Invalid, or + duplicate signatures are removed from 'signable'. + """ + + # Store the keyids of valid signatures. 'signature_keyids' is checked + # for duplicates rather than comparing signature objects because PSS may + # generate duplicate valid signatures of the same data, yet contain different + # signatures. + signature_keyids = [] + + for signature in signable['signatures']: + signed = signable['signed'] + keyid = signature['keyid'] + key = None + + # Remove 'signature' from 'signable' if the listed keyid does not exist + # in 'tuf.keydb'. + try: + key = tuf.keydb.get_key(keyid) + + except tuf.UnknownKeyError, e: + signable['signatures'].remove(signature) + + # Remove 'signature' from 'signable' if it is an invalid signature. + if not tuf.keys.verify_signature(key, signature, signed): + signable['signatures'].remove(signature) + + # Although valid, it may still need removal if it is a duplicate. Check + # the keyid, rather than the signature, to remove duplicate PSS signatures. + # PSS may generate multiple different signatures for the same keyid. + else: + if keyid in signature_keyids: + signable['signatures'].remove(signature) + + # 'keyid' is valid and not a duplicate, so add it to 'signature_keyids'. + else: + signature_keyids.append(keyid) + + + + + +def _delete_obsolete_metadata(metadata_directory, snapshot_metadata, + consistent_snapshot): + """ + Non-public function that deletes metadata files marked as removed by + 'repository_tool.py'. Revoked metadata files are not actually deleted until + this function is called. Obsolete metadata should *not* be retained in + "metadata.staged", otherwise they may be re-loaded by 'load_repository()'. + Note: Obsolete metadata may not always be easily detected (by inspecting + top-level metadata during loading) due to partial metadata and top-level + metadata that have not been written yet. + """ + + # Walk the repository's metadata 'targets' sub-directory, where all the + # metadata of delegated roles is stored. + targets_metadata = os.path.join(metadata_directory, 'targets') + + # The 'targets.json' metadata is not visited, only its child delegations. + # The 'targets/unclaimed/django.json' role would be located in the + # '{repository_directory}/metadata/targets/unclaimed/' directory. + if os.path.exists(targets_metadata) and os.path.isdir(targets_metadata): + for directory_path, junk_directories, files in os.walk(targets_metadata): + + # 'files' here is a list of target file names. + for basename in files: + metadata_path = os.path.join(directory_path, basename) + # Strip the metadata dirname and the leading path separator. + # '{repository_directory}/metadata/targets/unclaimed/django.json' --> + # 'targets/unclaimed/django.json' + metadata_name = \ + metadata_path[len(metadata_directory):].lstrip(os.path.sep) + + # Strip the digest if 'consistent_snapshot' is True. + # Example: 'targets/unclaimed/13df98ab0.django.json' --> + # 'targets/unclaimed/django.json' + metadata_name, embeded_digest = \ + _strip_consistent_snapshot_digest(metadata_name, consistent_snapshot) + + # Strip filename extensions. The role database does not include the + # metadata extension. + for metadata_extension in METADATA_EXTENSIONS: + if metadata_name.endswith(metadata_extension): + metadata_name_without_extension = \ + metadata_name[:-len(metadata_extension)] + + # Delete the metadata file if it does not exist in 'tuf.roledb'. + # repository_tool.py might have marked 'metadata_name' as removed, but its + # metadata file is not actually deleted yet. Do it now. + if not tuf.roledb.role_exists(metadata_name_without_extension): + logger.info('Removing outdated metadata: ' + repr(metadata_path)) + os.remove(metadata_path) + + # Delete outdated consistent snapshots. snapshot metadata includes + # the file extension of roles. + if consistent_snapshot: + #metadata_name_extension = metadata_name + METADATA_EXTENSION + file_hashes = snapshot_metadata['meta'][metadata_name] \ + ['hashes'].values() + if embeded_digest not in file_hashes: + logger.info('Removing outdated metadata: ' + repr(metadata_path)) + os.remove(metadata_path) + + + + + +def _get_written_metadata_and_digests(metadata_signable): + """ + Non-public function that returns the actual content of written metadata and + its digest. + """ + + written_metadata_content = unicode(json.dumps(metadata_signable, indent=1, + sort_keys=True)) + written_metadata_digests = {} + + for hash_algorithm in tuf.conf.REPOSITORY_HASH_ALGORITHMS: + digest_object = tuf.hash.digest(hash_algorithm) + digest_object.update(written_metadata_content) + written_metadata_digests.update({hash_algorithm: digest_object.hexdigest()}) + + return written_metadata_content, written_metadata_digests + + + + + +def _strip_consistent_snapshot_digest(metadata_filename, consistent_snapshot): + """ + Strip from 'metadata_filename' any digest data (in the expected + '{dirname}/digest.filename' format) that it may contain, and return it. + """ + + embeded_digest = '' + + # Strip the digest if 'consistent_snapshot' is True. + # Example: 'targets/unclaimed/13df98ab0.django.json' --> + # 'targets/unclaimed/django.json' + if consistent_snapshot: + dirname, basename = os.path.split(metadata_filename) + embeded_digest = basename[:basename.find('.')] + + # Ensure the digest, including the period, is stripped. + basename = basename[basename.find('.')+1:] + + metadata_filename = os.path.join(dirname, basename) + + + return metadata_filename, embeded_digest + + + + + + +def create_new_repository(repository_directory): + """ + + Create a new repository, instantiate barebones metadata for the top-level + roles, and return a Repository object. On disk, create_new_repository() + only creates the directories needed to hold the metadata and targets files. + The repository object returned may be modified to update the newly created + repository. The methods of the returned object may be called to create + actual repository files (e.g., repository.write()). + + + repository_directory: + The directory that will eventually hold the metadata and target files of + the TUF repository. + + + tuf.FormatError, if the arguments are improperly formatted. + + + The 'repository_directory' is created if it does not exist, including its + metadata and targets sub-directories. + + + A 'tuf.repository_tool.Repository' object. + """ + + # Does 'repository_directory' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.PATH_SCHEMA.check_match(repository_directory) + + # Set the repository, metadata, and targets directories. These directories + # are created if they do not exist. + repository_directory = os.path.abspath(repository_directory) + metadata_directory = None + targets_directory = None + + # Try to create 'repository_directory' if it does not exist. + try: + message = 'Creating '+repr(repository_directory) + logger.info(message) + os.makedirs(repository_directory) + + # 'OSError' raised if the leaf directory already exists or cannot be created. + # Check for case where 'repository_directory' has already been created. + except OSError, e: + if e.errno == errno.EEXIST: + pass + else: + raise + + # Set the metadata and targets directories. The metadata directory is a + # staged one so that the "live" repository is not affected. The + # staged metadata changes may be moved over to "live" after all updated + # have been completed. + metadata_directory = \ + os.path.join(repository_directory, METADATA_STAGED_DIRECTORY_NAME) + targets_directory = \ + os.path.join(repository_directory, TARGETS_DIRECTORY_NAME) + + # Try to create the metadata directory that will hold all of the metadata + # files, such as 'root.json' and 'snapshot.json'. + try: + message = 'Creating '+repr(metadata_directory) + logger.info(message) + os.mkdir(metadata_directory) + + # 'OSError' raised if the leaf directory already exists or cannot be created. + except OSError, e: + if e.errno == errno.EEXIST: + pass + else: + raise + + # Try to create the targets directory that will hold all of the target files. + try: + message = 'Creating '+repr(targets_directory) + logger.info(message) + os.mkdir(targets_directory) + + except OSError, e: + if e.errno == errno.EEXIST: + pass + else: + raise + + # Create the bare bones repository object, where only the top-level roles + # have been set and contain default values (e.g., Root roles has a threshold + # of 1, expires 1 year into the future, etc.) + repository = Repository(repository_directory, metadata_directory, + targets_directory) + + return repository + + + +def load_repository(repository_directory): + """ + + Return a repository object containing the contents of metadata files loaded + from the repository. + + + repository_directory: + + + tuf.FormatError, if 'repository_directory' or any of the metadata files + are improperly formatted. Also raised if, at a minimum, the Root role + cannot be found. + + + All the metadata files found in the repository are loaded and their contents + stored in a repository_tool.Repository object. + + + repository_tool.Repository object. + """ + + # Does 'repository_directory' have the correct format? + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.PATH_SCHEMA.check_match(repository_directory) + + # Load top-level metadata. + repository_directory = os.path.abspath(repository_directory) + metadata_directory = os.path.join(repository_directory, + METADATA_STAGED_DIRECTORY_NAME) + targets_directory = os.path.join(repository_directory, + TARGETS_DIRECTORY_NAME) + + # The Repository() object loaded (i.e., containing all the metadata roles + # found) and returned. + repository = Repository(repository_directory, metadata_directory, + targets_directory) + + filenames = get_metadata_filenames(metadata_directory) + + # The Root file is always available without a consistent snapshots digest + # attached to the filename. Store the 'consistent_snapshot' value read the + # loaded Root file so that other metadata files may be located. + # 'consistent_snapshot' value. + consistent_snapshot = False + + # Load the metadata of the top-level roles (i.e., Root, Timestamp, Targets, + # and Snapshot). + repository, consistent_snapshot = _load_top_level_metadata(repository, + filenames) + + # Load delegated targets metadata. + # Walk the 'targets/' directory and generate the fileinfo of all the files + # listed. This information is stored in the 'meta' field of the snapshot + # metadata object. + targets_objects = {} + loaded_metadata = [] + targets_objects['targets'] = repository.targets + targets_metadata_directory = os.path.join(metadata_directory, + TARGETS_DIRECTORY_NAME) + if os.path.exists(targets_metadata_directory) and \ + os.path.isdir(targets_metadata_directory): + for root, directories, files in os.walk(targets_metadata_directory): + + # 'files' here is a list of target file names. + for basename in files: + metadata_path = os.path.join(root, basename) + metadata_name = \ + metadata_path[len(metadata_directory):].lstrip(os.path.sep) + + # Strip the digest if 'consistent_snapshot' is True. + # Example: 'targets/unclaimed/13df98ab0.django.json' --> + # 'targets/unclaimed/django.json' + metadata_name, digest_junk = \ + _strip_consistent_snapshot_digest(metadata_name, consistent_snapshot) + + if metadata_name.endswith(METADATA_EXTENSION): + extension_length = len(METADATA_EXTENSION) + metadata_name = metadata_name[:-extension_length] + else: + continue + + # Keep a store metadata previously loaded metadata to prevent + # re-loading duplicate versions. Duplicate versions may occur with + # consistent_snapshot, where the same metadata may be available in + # multiples files (the different hash is included in each filename. + if metadata_name in loaded_metadata: + continue + + signable = None + try: + signable = tuf.util.load_json_file(metadata_path) + + except (ValueError, IOError), e: + continue + + metadata_object = signable['signed'] + + # Extract the metadata attributes 'metadata_name' and update its + # corresponding roleinfo. + roleinfo = tuf.roledb.get_roleinfo(metadata_name) + roleinfo['signatures'].extend(signable['signatures']) + roleinfo['version'] = metadata_object['version'] + roleinfo['expires'] = metadata_object['expires'] + roleinfo['paths'] = metadata_object['targets'].keys() + roleinfo['delegations'] = metadata_object['delegations'] + + if os.path.exists(metadata_path+'.gz'): + roleinfo['compressions'].append('gz') + + _check_if_partial_loaded(metadata_name, signable, roleinfo) + tuf.roledb.update_roleinfo(metadata_name, roleinfo) + loaded_metadata.append(metadata_name) + + # Generate the Targets objects of the delegated roles of + # 'metadata_name' and update the parent role Targets object. + new_targets_object = Targets(targets_directory, metadata_name, roleinfo) + targets_object = \ + targets_objects[tuf.roledb.get_parent_rolename(metadata_name)] + targets_objects[metadata_name] = new_targets_object + + targets_object._delegated_roles[(os.path.basename(metadata_name))] = \ + new_targets_object + + # Extract the keys specified in the delegations field of the Targets + # role. Add 'key_object' to the list of recognized keys. Keys may be + # shared, so do not raise an exception if 'key_object' has already been + # added. In contrast to the methods that may add duplicate keys, do not + # log a warning here as there may be many such duplicate key warnings. + # The repository maintainer should have also been made aware of the + # duplicate key when it was added. + for key_metadata in metadata_object['delegations']['keys'].values(): + key_object = tuf.keys.format_metadata_to_key(key_metadata) + try: + tuf.keydb.add_key(key_object) + + except tuf.KeyAlreadyExistsError, e: + pass + + # Add the delegated role's initial roleinfo, to be fully populated + # when its metadata file is next loaded in the os.walk() iteration. + for role in metadata_object['delegations']['roles']: + rolename = role['name'] + roleinfo = {'name': role['name'], 'keyids': role['keyids'], + 'threshold': role['threshold'], + 'compressions': [''], 'signing_keyids': [], + 'signatures': [], + 'partial_loaded': False, + 'delegations': {'keys': {}, + 'roles': []}} + tuf.roledb.add_role(rolename, roleinfo) + + return repository + + + + + +def _load_top_level_metadata(repository, top_level_filenames): + """ + Load the metadata of the Root, Timestamp, Targets, and Snapshot roles. + At a minimum, the Root role must exist and successfully load. + """ + + root_filename = top_level_filenames[ROOT_FILENAME] + targets_filename = top_level_filenames[TARGETS_FILENAME] + snapshot_filename = top_level_filenames[SNAPSHOT_FILENAME] + timestamp_filename = top_level_filenames[TIMESTAMP_FILENAME] + + root_metadata = None + targets_metadata = None + snapshot_metadata = None + timestamp_metadata = None + + # Load 'root.json'. A Root role file without a digest is always written. + if os.path.exists(root_filename): + # Initialize the key and role metadata of the top-level roles. + signable = tuf.util.load_json_file(root_filename) + tuf.formats.check_signable_object_format(signable) + root_metadata = signable['signed'] + tuf.keydb.create_keydb_from_root_metadata(root_metadata) + tuf.roledb.create_roledb_from_root_metadata(root_metadata) + + # Load Root's roleinfo and update 'tuf.roledb'. + roleinfo = tuf.roledb.get_roleinfo('root') + roleinfo['signatures'] = [] + for signature in signable['signatures']: + if signature not in roleinfo['signatures']: + roleinfo['signatures'].append(signature) + + if os.path.exists(root_filename+'.gz'): + roleinfo['compressions'].append('gz') + + _check_if_partial_loaded('root', signable, roleinfo) + tuf.roledb.update_roleinfo('root', roleinfo) + + # Ensure the 'consistent_snapshot' field is extracted. + consistent_snapshot = root_metadata['consistent_snapshot'] + + else: + message = 'Cannot load the required root file: '+repr(root_filename) + raise tuf.RepositoryError(message) + + # Load 'timestamp.json'. A Timestamp role file without a digest is always + # written. + if os.path.exists(timestamp_filename): + signable = tuf.util.load_json_file(timestamp_filename) + timestamp_metadata = signable['signed'] + for signature in signable['signatures']: + repository.timestamp.add_signature(signature) + + # Load Timestamp's roleinfo and update 'tuf.roledb'. + roleinfo = tuf.roledb.get_roleinfo('timestamp') + roleinfo['expires'] = timestamp_metadata['expires'] + roleinfo['version'] = timestamp_metadata['version'] + if os.path.exists(timestamp_filename+'.gz'): + roleinfo['compressions'].append('gz') + + _check_if_partial_loaded('timestamp', signable, roleinfo) + tuf.roledb.update_roleinfo('timestamp', roleinfo) + + else: + pass + + # Load 'snapshot.json'. A consistent snapshot of Snapshot must be calculated + # if 'consistent_snapshot' is True. + if consistent_snapshot: + snapshot_hashes = timestamp_metadata['meta'][SNAPSHOT_FILENAME]['hashes'] + snapshot_digest = random.choice(snapshot_hashes.values()) + dirname, basename = os.path.split(snapshot_filename) + snapshot_filename = os.path.join(dirname, snapshot_digest + '.' + basename) + + if os.path.exists(snapshot_filename): + signable = tuf.util.load_json_file(snapshot_filename) + tuf.formats.check_signable_object_format(signable) + snapshot_metadata = signable['signed'] + for signature in signable['signatures']: + repository.snapshot.add_signature(signature) + + # Load Snapshot's roleinfo and update 'tuf.roledb'. + roleinfo = tuf.roledb.get_roleinfo('snapshot') + roleinfo['expires'] = snapshot_metadata['expires'] + roleinfo['version'] = snapshot_metadata['version'] + if os.path.exists(snapshot_filename+'.gz'): + roleinfo['compressions'].append('gz') + + _check_if_partial_loaded('snapshot', signable, roleinfo) + tuf.roledb.update_roleinfo('snapshot', roleinfo) + + else: + pass + + # Load 'targets.json'. A consistent snapshot of Targets must be calculated if + # 'consistent_snapshot' is True. + if consistent_snapshot: + targets_hashes = snapshot_metadata['meta'][TARGETS_FILENAME]['hashes'] + targets_digest = random.choice(targets_hashes.values()) + dirname, basename = os.path.split(targets_filename) + targets_filename = os.path.join(dirname, targets_digest + '.' + basename) + + if os.path.exists(targets_filename): + signable = tuf.util.load_json_file(targets_filename) + tuf.formats.check_signable_object_format(signable) + targets_metadata = signable['signed'] + + for signature in signable['signatures']: + repository.targets.add_signature(signature) + + # Update 'targets.json' in 'tuf.roledb.py' + roleinfo = tuf.roledb.get_roleinfo('targets') + roleinfo['paths'] = targets_metadata['targets'].keys() + roleinfo['version'] = targets_metadata['version'] + roleinfo['expires'] = targets_metadata['expires'] + roleinfo['delegations'] = targets_metadata['delegations'] + if os.path.exists(targets_filename+'.gz'): + roleinfo['compressions'].append('gz') + + _check_if_partial_loaded('targets', signable, roleinfo) + tuf.roledb.update_roleinfo('targets', roleinfo) + + # Add the keys specified in the delegations field of the Targets role. + for key_metadata in targets_metadata['delegations']['keys'].values(): + key_object = tuf.keys.format_metadata_to_key(key_metadata) + + # Add 'key_object' to the list of recognized keys. Keys may be shared, + # so do not raise an exception if 'key_object' has already been loaded. + # In contrast to the methods that may add duplicate keys, do not log + # a warning as there may be many such duplicate key warnings. The + # repository maintainer should have also been made aware of the duplicate + # key when it was added. + try: + tuf.keydb.add_key(key_object) + + except tuf.KeyAlreadyExistsError, e: + pass + + for role in targets_metadata['delegations']['roles']: + rolename = role['name'] + roleinfo = {'name': role['name'], 'keyids': role['keyids'], + 'threshold': role['threshold'], 'compressions': [''], + 'signing_keyids': [], 'signatures': [], + 'delegations': {'keys': {}, + 'roles': []}} + tuf.roledb.add_role(rolename, roleinfo) + + else: + pass + + return repository, consistent_snapshot + + + + + +def generate_and_write_rsa_keypair(filepath, bits=DEFAULT_RSA_KEY_BITS, + password=None): + """ + + Generate an RSA key file, create an encrypted PEM string (using 'password' + as the pass phrase), and store it in 'filepath'. The public key portion of + the generated RSA key is stored in <'filepath'>.pub. Which cryptography + library performs the cryptographic decryption is determined by the string + set in 'tuf.conf.RSA_CRYPTO_LIBRARY'. PyCrypto currently supported. The + PEM private key is encrypted with 3DES and CBC the mode of operation. The + password is strengthened with PBKDF1-MD5. + + + filepath: + The public and private key files are saved to .pub, , + respectively. + + bits: + The number of bits of the generated RSA key. + + password: + The password used to encrypt 'filepath'. + + + tuf.FormatError, if the arguments are improperly formatted. + + + Writes key files to '' and '.pub'. + + + None. + """ + + # Do the arguments have the correct format? + # This check ensures arguments have the appropriate number of + # objects and object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.PATH_SCHEMA.check_match(filepath) + + # Does 'bits' have the correct format? + tuf.formats.RSAKEYBITS_SCHEMA.check_match(bits) + + # If the caller does not provide a password argument, prompt for one. + if password is None: + message = 'Enter a password for the RSA key file: ' + password = _get_password(message, confirm=True) + + # Does 'password' have the correct format? + tuf.formats.PASSWORD_SCHEMA.check_match(password) + + # Generate public and private RSA keys, encrypted the private portion + # and store them in PEM format. + rsa_key = tuf.keys.generate_rsa_key(bits) + public = rsa_key['keyval']['public'] + private = rsa_key['keyval']['private'] + encrypted_pem = tuf.keys.create_rsa_encrypted_pem(private, password) + + # Write public key (i.e., 'public', which is in PEM format) to + # '.pub'. If the parent directory of filepath does not exist, + # create it (and all its parent directories, if necessary). + tuf.util.ensure_parent_dir(filepath) + + # Create a tempororary file, write the contents of the public key, and move + # to final destination. + file_object = tuf.util.TempFile() + file_object.write(public) + + # The temporary file is closed after the final move. + file_object.move(filepath+'.pub') + + # Write the private key in encrypted PEM format to ''. + # Unlike the public key file, the private key does not have a file + # extension. + file_object = tuf.util.TempFile() + file_object.write(encrypted_pem) + file_object.move(filepath) + + + + + +def import_rsa_privatekey_from_file(filepath, password=None): + """ + + Import the encrypted PEM file in 'filepath', decrypt it, and return the key + object in 'tuf.formats.RSAKEY_SCHEMA' format. + + Which cryptography library performs the cryptographic decryption is + determined by the string set in 'tuf.conf.RSA_CRYPTO_LIBRARY'. PyCrypto + currently supported. + + The PEM private key is encrypted with 3DES and CBC the mode of operation. + The password is strengthened with PBKDF1-MD5. + + + filepath: + file, an RSA encrypted PEM file. Unlike the public RSA PEM + key file, 'filepath' does not have an extension. + + password: + The passphrase to decrypt 'filepath'. + + + tuf.FormatError, if the arguments are improperly formatted. + + + The contents of 'filepath' is read, decrypted, and the key stored. + + + An RSA key object, conformant to 'tuf.formats.RSAKEY_SCHEMA'. + """ + + # Does 'filepath' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.PATH_SCHEMA.check_match(filepath) + + # If the caller does not provide a password argument, prompt for one. + if password is None: + message = 'Enter a password for the encrypted RSA key file: ' + password = _get_password(message, confirm=True) + + # Does 'password' have the correct format? + tuf.formats.PASSWORD_SCHEMA.check_match(password) + + encrypted_pem = None + + # Read the contents of 'filepath' that should be an encrypted PEM. + with open(filepath, 'rb') as file_object: + encrypted_pem = file_object.read() + + # Convert 'encrypted_pem' to 'tuf.formats.RSAKEY_SCHEMA' format. + rsa_key = tuf.keys.import_rsakey_from_encrypted_pem(encrypted_pem, password) + + return rsa_key + + + + + +def import_rsa_publickey_from_file(filepath): + """ + + Import the RSA key stored in 'filepath'. The key object returned is a TUF + key, specifically 'tuf.formats.RSAKEY_SCHEMA'. If the RSA PEM in 'filepath' + contains a private key, it is discarded. + + Which cryptography library performs the cryptographic decryption is + determined by the string set in 'tuf.conf.RSA_CRYPTO_LIBRARY'. PyCrypto + currently supported. If the RSA PEM in 'filepath' contains a private key, + it is discarded. + + + filepath: + .pub file, an RSA PEM file. + + + tuf.FormatError, if 'filepath' is improperly formatted. + + tuf.Error, if a valid RSA key object cannot be generated. This may be + caused by an improperly formatted PEM file. + + + 'filepath' is read and its contents extracted. + + + An RSA key object conformant to 'tuf.formats.RSAKEY_SCHEMA'. + """ + + # Does 'filepath' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.PATH_SCHEMA.check_match(filepath) + + # Read the contents of the key file that should be in PEM format and contains + # the public portion of the RSA key. + with open(filepath, 'rb') as file_object: + rsa_pubkey_pem = file_object.read() + + # Convert 'rsa_pubkey_pem' to 'tuf.formats.RSAKEY_SCHEMA' format. + try: + rsakey_dict = tuf.keys.format_rsakey_from_pem(rsa_pubkey_pem) + + except tuf.FormatError, e: + raise tuf.Error('Cannot import improperly formatted PEM file.') + + return rsakey_dict + + + + + +def generate_and_write_ed25519_keypair(filepath, password=None): + """ + + Generate an ED25519 key file, create an encrypted TUF key (using 'password' + as the pass phrase), and store it in 'filepath'. The public key portion of + the generated ED25519 key is stored in <'filepath'>.pub. Which cryptography + library performs the cryptographic decryption is determined by the string + set in 'tuf.conf.ED25519_CRYPTO_LIBRARY'. + + PyCrypto currently supported. The ED25519 private key is encrypted with + AES-256 and CTR the mode of operation. The password is strengthened with + PBKDF2-HMAC-SHA256. + + + filepath: + The public and private key files are saved to .pub and + , respectively. + + password: + The password, or passphrase, to encrypt the private portion of the + generated ed25519 key. A symmetric encryption key is derived from + 'password', so it is not directly used. + + + tuf.FormatError, if the arguments are improperly formatted. + + tuf.CryptoError, if 'filepath' cannot be encrypted. + + tuf.UnsupportedLibraryError, if 'filepath' cannot be encrypted due to an + invalid configuration setting (i.e., invalid 'tuf.conf.py' setting). + + + Writes key files to '' and '.pub'. + + + None. + """ + + # Does 'filepath' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.PATH_SCHEMA.check_match(filepath) + + # If the caller does not provide a password argument, prompt for one. + if password is None: + message = 'Enter a password for the ED25519 key: ' + password = _get_password(message, confirm=True) + + # Does 'password' have the correct format? + tuf.formats.PASSWORD_SCHEMA.check_match(password) + + # Generate a new ED25519 key object and encrypt it. The cryptography library + # used is determined by the user, or by default (set in + # 'tuf.conf.ED25519_CRYPTO_LIBRARY'). Raise 'tuf.CryptoError' or + # 'tuf.UnsupportedLibraryError', if 'ed25519_key' cannot be encrypted. + ed25519_key = tuf.keys.generate_ed25519_key() + encrypted_key = tuf.keys.encrypt_key(ed25519_key, password) + + # ed25519 public key file contents in metadata format (i.e., does not include + # the keyid portion). + keytype = ed25519_key['keytype'] + keyval = ed25519_key['keyval'] + ed25519key_metadata_format = \ + tuf.keys.format_keyval_to_metadata(keytype, keyval, private=False) + + # Write the public key, conformant to 'tuf.formats.KEY_SCHEMA', to + # '.pub'. + tuf.util.ensure_parent_dir(filepath) + + # Create a tempororary file, write the contents of the public key, and move + # to final destination. + file_object = tuf.util.TempFile() + file_object.write(json.dumps(ed25519key_metadata_format)) + + # The temporary file is closed after the final move. + file_object.move(filepath+'.pub') + + # Write the encrypted key string, conformant to + # 'tuf.formats.ENCRYPTEDKEY_SCHEMA', to ''. + file_object = tuf.util.TempFile() + file_object.write(encrypted_key) + file_object.move(filepath) + + + + + +def import_ed25519_publickey_from_file(filepath): + """ + + Load the ED25519 public key object (conformant to 'tuf.formats.KEY_SCHEMA') + stored in 'filepath'. Return 'filepath' in tuf.formats.ED25519KEY_SCHEMA + format. + + If the TUF key object in 'filepath' contains a private key, it is discarded. + + + filepath: + .pub file, a TUF public key file. + + + tuf.FormatError, if 'filepath' is improperly formatted or is an unexpected + key type. + + + The contents of 'filepath' is read and saved. + + + An ED25519 key object conformant to 'tuf.formats.ED25519KEY_SCHEMA'. + """ + + # Does 'filepath' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.PATH_SCHEMA.check_match(filepath) + + # ED25519 key objects are saved in json and metadata format. Return the + # loaded key object in tuf.formats.ED25519KEY_SCHEMA' format that also + # includes the keyid. + ed25519_key_metadata = tuf.util.load_json_file(filepath) + ed25519_key = tuf.keys.format_metadata_to_key(ed25519_key_metadata) + + # Raise an exception if an unexpected key type is imported. + if ed25519_key['keytype'] != 'ed25519': + message = 'Invalid key type loaded: '+repr(ed25519_key['keytype']) + raise tuf.FormatError(message) + + return ed25519_key + + + + + +def import_ed25519_privatekey_from_file(filepath, password=None): + """ + + Import the encrypted ed25519 TUF key file in 'filepath', decrypt it, and + return the key object in 'tuf.formats.ED25519KEY_SCHEMA' format. + + Which cryptography library performs the cryptographic decryption is + determined by the string set in 'tuf.conf.ED25519_CRYPTO_LIBRARY'. PyCrypto + currently supported. + + The TUF private key (may also contain the public part) is encrypted with AES + 256 and CTR the mode of operation. The password is strengthened with + PBKDF2-HMAC-SHA256. + + + filepath: + file, an RSA encrypted TUF key file. + + password: + The password, or passphrase, to import the private key (i.e., the + encrypted key file 'filepath' must be decrypted before the ed25519 key + object can be returned. + + + tuf.FormatError, if the arguments are improperly formatted or the imported + key object contains an invalid key type (i.e., not 'ed25519'). + + tuf.CryptoError, if 'filepath' cannot be decrypted. + + tuf.UnsupportedLibraryError, if 'filepath' cannot be decrypted due to an + invalid configuration setting (i.e., invalid 'tuf.conf.py' setting). + + + 'password' is used to decrypt the 'filepath' key file. + + + An ed25519 key object of the form: 'tuf.formats.ED25519KEY_SCHEMA'. + """ + + # Does 'filepath' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.PATH_SCHEMA.check_match(filepath) + + # If the caller does not provide a password argument, prompt for one. + if password is None: + message = 'Enter a password for the encrypted ED25519 key file: ' + password = _get_password(message, confirm=True) + + # Does 'password' have the correct format? + tuf.formats.PASSWORD_SCHEMA.check_match(password) + + # Store the encrypted contents of 'filepath' prior to calling the decryption + # routine. + encrypted_key = None + + with open(filepath, 'rb') as file_object: + encrypted_key = file_object.read() + + # Decrypt the loaded key file, calling the appropriate cryptography library + # (i.e., set by the user) and generating the derived encryption key from + # 'password'. Raise 'tuf.CryptoError' or 'tuf.UnsupportedLibraryError' if the + # decryption fails. + key_object = tuf.keys.decrypt_key(encrypted_key, password) + + # Raise an exception if an unexpected key type is imported. + if key_object['keytype'] != 'ed25519': + message = 'Invalid key type loaded: '+repr(key_object['keytype']) + raise tuf.FormatError(message) + + return key_object + + + + + +def get_metadata_filenames(metadata_directory=None): + """ + + Return a dictionary containing the filenames of the top-level roles. + If 'metadata_directory' is set to 'metadata', the dictionary + returned would contain: + + filenames = {'root': 'metadata/root.json', + 'targets': 'metadata/targets.json', + 'snapshot': 'metadata/snapshot.json', + 'timestamp': 'metadata/timestamp.json'} + + If the metadata directory is not set by the caller, the current + directory is used. + + + metadata_directory: + The directory containing the metadata files. + + + tuf.FormatError, if 'metadata_directory' is improperly formatted. + + + None. + + + A dictionary containing the expected filenames of the top-level + metadata files, such as 'root.json' and 'snapshot.json'. + """ + + # Does 'metadata_directory' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.PATH_SCHEMA.check_match(metadata_directory) + + if metadata_directory is None: + metadata_directory = '.' + + # Does 'metadata_directory' have the correct format? + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.PATH_SCHEMA.check_match(metadata_directory) + + # Store the filepaths of the top-level roles, including the + # 'metadata_directory' for each one. + filenames = {} + + filenames[ROOT_FILENAME] = \ + os.path.join(metadata_directory, ROOT_FILENAME) + + filenames[TARGETS_FILENAME] = \ + os.path.join(metadata_directory, TARGETS_FILENAME) + + filenames[SNAPSHOT_FILENAME] = \ + os.path.join(metadata_directory, SNAPSHOT_FILENAME) + + filenames[TIMESTAMP_FILENAME] = \ + os.path.join(metadata_directory, TIMESTAMP_FILENAME) + + return filenames + + + + + +def get_metadata_file_info(filename): + """ + + Retrieve the file information of 'filename'. The object returned + conforms to 'tuf.formats.FILEINFO_SCHEMA'. The information + generated for 'filename' is stored in metadata files like 'targets.json'. + The fileinfo object returned has the form: + fileinfo = {'length': 1024, + 'hashes': {'sha256': 1233dfba312, ...}, + 'custom': {...}} + + + filename: + The metadata file whose file information is needed. It must exist. + + + tuf.FormatError, if 'filename' is improperly formatted. + + tuf.Error, if 'filename' doesn't exist. + + + The file is opened and information about the file is generated, + such as file size and its hash. + + + A dictionary conformant to 'tuf.formats.FILEINFO_SCHEMA'. This + dictionary contains the length, hashes, and custom data about the + 'filename' metadata file. + """ + + # Does 'filename' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.PATH_SCHEMA.check_match(filename) + + if not os.path.isfile(filename): + message = repr(filename)+' is not a file.' + raise tuf.Error(message) + + # Note: 'filehashes' is a dictionary of the form + # {'sha256': 1233dfba312, ...}. 'custom' is an optional + # dictionary that a client might define to include additional + # file information, such as the file's author, version/revision + # numbers, etc. + filesize, filehashes = \ + tuf.util.get_file_details(filename, tuf.conf.REPOSITORY_HASH_ALGORITHMS) + custom = None + + return tuf.formats.make_fileinfo(filesize, filehashes, custom) + + + + + + +def get_target_hash(target_filepath): + """ + + Compute the hash of 'target_filepath'. This is useful in conjunction with + the "path_hash_prefixes" attribute in a delegated targets role, which + tells us which paths it is implicitly responsible for. + + The repository may optionally organize targets into hashed bins to ease + target delegations and role metadata management. The use of consistent + hashing allows for a uniform distribution of targets into bins. + + + target_filepath: + The path to the target file on the repository. This will be relative to + the 'targets' (or equivalent) directory on a given mirror. + + + None. + + + None. + + + The hash of 'target_filepath'. + """ + + # Does 'target_filepath' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.RELPATH_SCHEMA.check_match(target_filepath) + + # Calculate the hash of the filepath to determine which bin to find the + # target. The client currently assumes the repository uses + # 'HASH_FUNCTION' to generate hashes. + digest_object = tuf.hash.digest(HASH_FUNCTION) + + try: + digest_object.update(target_filepath) + + except UnicodeEncodeError: + # Sometimes, there are Unicode characters in target paths. We assume a + # UTF-8 encoding and try to hash that. + digest_object = tuf.hash.digest(HASH_FUNCTION) + encoded_target_filepath = target_filepath.encode('utf-8') + digest_object.update(encoded_target_filepath) + + target_filepath_hash = digest_object.hexdigest() + + return target_filepath_hash + + + + + +def generate_root_metadata(version, expiration_date, consistent_snapshot): + """ + + Create the root metadata. 'tuf.roledb.py' and 'tuf.keydb.py' are read and + the information returned by these modules is used to generate the root + metadata object. + + + version: + The metadata version number. Clients use the version number to + determine if the downloaded version is newer than the one currently + trusted. + + expiration_date: + The expiration date, in UTC, of the metadata file. Conformant to + 'tuf.formats.TIME_SCHEMA'. + + consistent_snapshot: + + + tuf.FormatError, if the generated root metadata object could not + be generated with the correct format. + + tuf.Error, if an error is encountered while generating the root + metadata object. + + + The contents of 'tuf.keydb.py' and 'tuf.roledb.py' are read. + + + A root metadata object, conformant to 'tuf.formats.ROOT_SCHEMA'. + """ + + # Do the arguments have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if any of the arguments are improperly formatted. + tuf.formats.METADATAVERSION_SCHEMA.check_match(version) + tuf.formats.TIME_SCHEMA.check_match(expiration_date) + tuf.formats.BOOLEAN_SCHEMA.check_match(consistent_snapshot) + + # The role and key dictionaries to be saved in the root metadata object. + # Conformant to 'ROLEDICT_SCHEMA' and 'KEYDICT_SCHEMA', respectively. + roledict = {} + keydict = {} + + # Extract the role, threshold, and keyid information of the top-level roles, + # which Root stores in its metadata. The necessary role metadata is generated + # from this information. + for rolename in ['root', 'targets', 'snapshot', 'timestamp']: + + # If a top-level role is missing from 'tuf.roledb.py', raise an exception. + if not tuf.roledb.role_exists(rolename): + raise tuf.Error(repr(rolename)+' not in "tuf.roledb".') + + # Keep track of the keys loaded to avoid duplicates. + keyids = [] + + # Generate keys for the keyids listed by the role being processed. + for keyid in tuf.roledb.get_role_keyids(rolename): + key = tuf.keydb.get_key(keyid) + + # If 'key' is an RSA key, it would conform to 'tuf.formats.RSAKEY_SCHEMA', + # and have the form: + # {'keytype': 'rsa', + # 'keyid': keyid, + # 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', + # 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} + keyid = key['keyid'] + if keyid not in keydict: + + # This appears to be a new keyid. Generate the key for it. + if key['keytype'] in ['rsa', 'ed25519']: + keytype = key['keytype'] + keyval = key['keyval'] + keydict[keyid] = \ + tuf.keys.format_keyval_to_metadata(keytype, keyval, private=False) + + # This is not a recognized key. Raise an exception. + else: + raise tuf.Error('Unsupported keytype: '+keyid) + + # Do we have a duplicate? + if keyid in keyids: + raise tuf.Error('Same keyid listed twice: '+keyid) + + # Add the loaded keyid for the role being processed. + keyids.append(keyid) + + # Generate and store the role data belonging to the processed role. + role_threshold = tuf.roledb.get_role_threshold(rolename) + role_metadata = tuf.formats.make_role_metadata(keyids, role_threshold) + roledict[rolename] = role_metadata + + # Generate the root metadata object. + root_metadata = tuf.formats.RootFile.make_metadata(version, expiration_date, + keydict, roledict, + consistent_snapshot) + + return root_metadata + + + + + +def generate_targets_metadata(targets_directory, target_files, version, + expiration_date, delegations=None, + write_consistent_targets=False): + """ + + Generate the targets metadata object. The targets in 'target_files' must + exist at the same path they should on the repo. 'target_files' is a list of + targets. The 'custom' field of the targets metadata is not currently + supported. + + + targets_directory: + The directory containing the target files and directories of the + repository. + + target_files: + The target files tracked by 'targets.json'. 'target_files' is a list of + target paths that are relative to the targets directory (e.g., + ['file1.txt', 'Django/module.py']). + + version: + The metadata version number. Clients use the version number to + determine if the downloaded version is newer than the one currently + trusted. + + expiration_date: + The expiration date, in UTC, of the metadata file. Conformant to + 'tuf.formats.TIME_SCHEMA'. + + delegations: + The delegations made by the targets role to be generated. 'delegations' + must match 'tuf.formats.DELEGATIONS_SCHEMA'. + + write_consistent_targets: + Boolean that indicates whether file digests should be prepended to the + target files. + + + tuf.FormatError, if an error occurred trying to generate the targets + metadata object. + + tuf.Error, if any of the target files cannot be read. + + + The target files are read and file information generated about them. + + + A targets metadata object, conformant to 'tuf.formats.TARGETS_SCHEMA'. + """ + + # Do the arguments have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.PATH_SCHEMA.check_match(targets_directory) + tuf.formats.PATHS_SCHEMA.check_match(target_files) + tuf.formats.METADATAVERSION_SCHEMA.check_match(version) + tuf.formats.TIME_SCHEMA.check_match(expiration_date) + tuf.formats.BOOLEAN_SCHEMA.check_match(write_consistent_targets) + + if delegations is not None: + tuf.formats.DELEGATIONS_SCHEMA.check_match(delegations) + + # Store the file attributes of targets in 'target_files'. 'filedict', + # conformant to 'tuf.formats.FILEDICT_SCHEMA', is added to the targets + # metadata object returned. + filedict = {} + + # Ensure the user is aware of a non-existent 'target_directory', and convert + # it to its abosolute path, if it exists. + targets_directory = _check_directory(targets_directory) + + # Generate the fileinfo of all the target files listed in 'target_files'. + for target in target_files: + + # The root-most folder of the targets directory should not be included in + # target paths listed in targets metadata. + # (e.g., 'targets/more_targets/somefile.txt' -> 'more_targets/somefile.txt') + relative_targetpath = target + + # Note: join() discards 'targets_directory' if 'target' contains a leading + # path separator (i.e., is treated as an absolute path). + target_path = os.path.join(targets_directory, target.lstrip(os.sep)) + + # Ensure all target files listed in 'target_files' exist. If just one of + # these files does not exist, raise an exception. + if not os.path.exists(target_path): + message = repr(target_path)+' cannot be read. Unable to generate '+ \ + 'targets metadata.' + raise tuf.Error(message) + + filedict[relative_targetpath] = get_metadata_file_info(target_path) + + if write_consistent_targets: + for target_digest in filedict[relative_targetpath]['hashes'].values(): + dirname, basename = os.path.split(target_path) + digest_filename = target_digest + '.' + basename + digest_target = os.path.join(dirname, digest_filename) + + if not os.path.exists(digest_target): + logger.warn('Hard linking target file to ' + repr(digest_target)) + os.link(target_path, digest_target) + + # Generate the targets metadata object. + targets_metadata = tuf.formats.TargetsFile.make_metadata(version, + expiration_date, + filedict, + delegations) + + return targets_metadata + + + + + +def generate_snapshot_metadata(metadata_directory, version, expiration_date, + root_filename, targets_filename, + consistent_snapshot): + """ + + Create the snapshot metadata. The minimum metadata must exist + (i.e., 'root.json' and 'targets.json'). This will also look through + the 'targets/' directory in 'metadata_directory' and the resulting + snapshot file will list all the delegated roles. + + + metadata_directory: + The directory containing the 'root.json' and 'targets.json' metadata + files. + + version: + The metadata version number. Clients use the version number to + determine if the downloaded version is newer than the one currently + trusted. + + expiration_date: + The expiration date, in UTC, of the metadata file. + Conformant to 'tuf.formats.TIME_SCHEMA'. + + root_filename: + The filename of the top-level root role. The hash and file size of this + file is listed in the snapshot role. + + targets_filename: + The filename of the top-level targets role. The hash and file size of + this file is listed in the snapshot role. + + consistent_snapshot: + Boolean. If True, a file digest is expected to be prepended to the + filename of any target file located in the targets directory. Each digest + is stripped from the target filename and listed in the snapshot metadata. + + + tuf.FormatError, if 'metadata_directory' is improperly formatted. + + tuf.Error, if an error occurred trying to generate the snapshot metadata + object. + + + The 'root.json' and 'targets.json' files are read. + + + The snapshot metadata object, conformant to 'tuf.formats.SNAPSHOT_SCHEMA'. + """ + + # Do the arguments have the correct format? + # This check ensures arguments have the appropriate number of objects and + # object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.PATH_SCHEMA.check_match(metadata_directory) + tuf.formats.METADATAVERSION_SCHEMA.check_match(version) + tuf.formats.TIME_SCHEMA.check_match(expiration_date) + tuf.formats.PATH_SCHEMA.check_match(root_filename) + tuf.formats.PATH_SCHEMA.check_match(targets_filename) + + metadata_directory = _check_directory(metadata_directory) + + # Retrieve the fileinfo of 'root.json' and 'targets.json'. This file + # information includes data such as file length, hashes of the file, etc. + filedict = {} + filedict[ROOT_FILENAME] = get_metadata_file_info(root_filename) + filedict[TARGETS_FILENAME] = get_metadata_file_info(targets_filename) + + # Add compressed versions of the 'targets.json' and 'root.json' metadata, + # if they exist. + for extension in SUPPORTED_COMPRESSION_EXTENSIONS: + compressed_root_filename = root_filename+extension + compressed_targets_filename = targets_filename+extension + + # If the compressed versions of the root and targets metadata is found, + # add their file attributes to 'filedict'. + if os.path.exists(compressed_root_filename): + filedict[ROOT_FILENAME+extension] = \ + get_metadata_file_info(compressed_root_filename) + if os.path.exists(compressed_targets_filename): + filedict[TARGETS_FILENAME+extension] = \ + get_metadata_file_info(compressed_targets_filename) + + # Walk the 'targets/' directory and generate the fileinfo of all the role + # files found. This information is stored in the 'meta' field of the snapshot + # metadata object. + targets_metadata = os.path.join(metadata_directory, 'targets') + if os.path.exists(targets_metadata) and os.path.isdir(targets_metadata): + for directory_path, junk_directories, files in os.walk(targets_metadata): + + # 'files' here is a list of file names. + for basename in files: + metadata_path = os.path.join(directory_path, basename) + metadata_name = \ + metadata_path[len(metadata_directory):].lstrip(os.path.sep) + + # Strip the digest if 'consistent_snapshot' is True. + # Example: 'targets/unclaimed/13df98ab0.django.json' --> + # 'targets/unclaimed/django.json' + metadata_name, digest_junk = \ + _strip_consistent_snapshot_digest(metadata_name, consistent_snapshot) + + # All delegated roles are added to the snapshot file, including + # compressed versions. + for metadata_extension in METADATA_EXTENSIONS: + if metadata_name.endswith(metadata_extension): + rolename = metadata_name[:-len(metadata_extension)] + + # Obsolete role files may still be found. Ensure only roles loaded + # in the roledb are included in the snapshot metadata. + if tuf.roledb.role_exists(rolename): + filedict[metadata_name] = get_metadata_file_info(metadata_path) + + # Generate the snapshot metadata object. + snapshot_metadata = tuf.formats.SnapshotFile.make_metadata(version, + expiration_date, + filedict) + + return snapshot_metadata + + + + + +def generate_timestamp_metadata(snapshot_filename, version, + expiration_date, compressions=()): + """ + + Generate the timestamp metadata object. The 'snapshot.json' file must + exist. + + + snapshot_filename: + The required filename of the snapshot metadata file. The timestamp role + needs to the calculate the file size and hash of this file. + + version: + The timestamp's version number. Clients use the version number to + determine if the downloaded version is newer than the one currently + trusted. + + expiration_date: + The expiration date, in UTC, of the metadata file, conformant to + 'tuf.formats.TIME_SCHEMA'. + + compressions: + Compression extensions (e.g., 'gz'). If 'snapshot.json' is also saved in + compressed form, these compression extensions should be stored in + 'compressions' so the compressed timestamp files can be added to the + timestamp metadata object. + + + tuf.FormatError, if the generated timestamp metadata object cannot be + formatted correctly, or one of the arguments is improperly formatted. + + + None. + + + A timestamp metadata object, conformant to 'tuf.formats.TIMESTAMP_SCHEMA'. + """ + + # Do the arguments have the correct format? + # This check ensures arguments have the appropriate number of objects and + # object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.PATH_SCHEMA.check_match(snapshot_filename) + tuf.formats.METADATAVERSION_SCHEMA.check_match(version) + tuf.formats.TIME_SCHEMA.check_match(expiration_date) + tuf.formats.COMPRESSIONS_SCHEMA.check_match(compressions) + + # Retrieve the fileinfo of the snapshot metadata file. + # This file information contains hashes, file length, custom data, etc. + fileinfo = {} + fileinfo[SNAPSHOT_FILENAME] = get_metadata_file_info(snapshot_filename) + + # Save the fileinfo of the compressed versions of 'timestamp.json' + # in 'fileinfo'. Log the files included in 'fileinfo'. + for file_extension in compressions: + if not len(file_extension): + continue + + compressed_filename = snapshot_filename + '.' + file_extension + try: + compressed_fileinfo = get_metadata_file_info(compressed_filename) + + except: + logger.warn('Cannot get fileinfo about '+str(compressed_filename)) + + else: + logger.info('Including fileinfo about '+str(compressed_filename)) + fileinfo[SNAPSHOT_FILENAME+'.'+file_extension] = compressed_fileinfo + + # Generate the timestamp metadata object. + timestamp_metadata = tuf.formats.TimestampFile.make_metadata(version, + expiration_date, + fileinfo) + + return timestamp_metadata + + + + + +def sign_metadata(metadata_object, keyids, filename): + """ + + Sign a metadata object. If any of the keyids have already signed the file, + the old signature is replaced. The keys in 'keyids' must already be + loaded in 'tuf.keydb'. + + + metadata_object: + The metadata object to sign. For example, 'metadata' might correspond to + 'tuf.formats.ROOT_SCHEMA' or 'tuf.formats.TARGETS_SCHEMA'. + + keyids: + The keyids list of the signing keys. + + filename: + The intended filename of the signed metadata object. + For example, 'root.json' or 'targets.json'. This function + does NOT save the signed metadata to this filename. + + + tuf.FormatError, if a valid 'signable' object could not be generated or + the arguments are improperly formatted. + + tuf.Error, if an invalid keytype was found in the keystore. + + + None. + + + A signable object conformant to 'tuf.formats.SIGNABLE_SCHEMA'. + """ + + # Do the arguments have the correct format? + # This check ensures arguments have the appropriate number of objects and + # object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.ANYROLE_SCHEMA.check_match(metadata_object) + tuf.formats.KEYIDS_SCHEMA.check_match(keyids) + tuf.formats.PATH_SCHEMA.check_match(filename) + + # Make sure the metadata is in 'signable' format. That is, + # it contains a 'signatures' field containing the result + # of signing the 'signed' field of 'metadata' with each + # keyid of 'keyids'. + signable = tuf.formats.make_signable(metadata_object) + + # Sign the metadata with each keyid in 'keyids'. + for keyid in keyids: + + # Load the signing key. + key = tuf.keydb.get_key(keyid) + logger.info('Signing '+repr(filename)+' with '+key['keyid']) + + # Create a new signature list. If 'keyid' is encountered, + # do not add it to new list. + signatures = [] + for signature in signable['signatures']: + if not keyid == signature['keyid']: + signatures.append(signature) + signable['signatures'] = signatures + + # Generate the signature using the appropriate signing method. + if key['keytype'] in SUPPORTED_KEY_TYPES: + if len(key['keyval']['private']): + signed = signable['signed'] + signature = tuf.keys.create_signature(key, signed) + signable['signatures'].append(signature) + + else: + logger.warn('Private key unset. Skipping: '+repr(keyid)) + + else: + raise tuf.Error('The keydb contains a key with an invalid key type.') + + # Raise 'tuf.FormatError' if the resulting 'signable' is not formatted + # correctly. + tuf.formats.check_signable_object_format(signable) + + return signable + + + + + +def write_metadata_file(metadata, filename, compressions, consistent_snapshot): + """ + + If necessary, write the 'metadata' signable object to 'filename', and the + compressed version of the metadata file if 'compression' is set. + Note: Compression algorithms like gzip attach a timestamp to compressed + files, so a metadata file compressed multiple times may generate different + digests even though the uncompressed content has not changed. + + + metadata: + The object that will be saved to 'filename', conformant to + 'tuf.formats.SIGNABLE_SCHEMA'. + + filename: + The filename of the metadata to be written (e.g., 'root.json'). + If a compression algorithm is specified in 'compressions', the + compression extention is appended to 'filename'. + + compressions: + Specify the algorithms, as a list of strings, used to compress the file; + The only currently available compression option is 'gz' (gzip). + + consistent_snapshot: + Boolean that determines whether the metadata file's digest should be + prepended to the filename. + + + tuf.FormatError, if the arguments are improperly formatted. + + tuf.Error, if the directory of 'filename' does not exist. + + Any other runtime (e.g., IO) exception. + + + The 'filename' (or the compressed filename) file is created, or overwritten + if it exists. + + + None. + """ + + # Do the arguments have the correct format? + # This check ensures arguments have the appropriate number of objects and + # object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.SIGNABLE_SCHEMA.check_match(metadata) + tuf.formats.PATH_SCHEMA.check_match(filename) + tuf.formats.COMPRESSIONS_SCHEMA.check_match(compressions) + tuf.formats.BOOLEAN_SCHEMA.check_match(consistent_snapshot) + + # Verify the directory of 'filename', and convert 'filename' to its absolute + # path so that temporary files are moved to their expected destinations. + filename = os.path.abspath(filename) + written_filename = filename + _check_directory(os.path.dirname(filename)) + consistent_filenames = [] + + # Generate the actual metadata file content of 'metadata'. Metadata is + # saved as json and includes formatting, such as indentation and sorted + # objects. The new digest of 'metadata' is also calculated to help determine + # if re-saving is required. + file_content, new_digests = _get_written_metadata_and_digests(metadata) + + if consistent_snapshot: + for new_digest in new_digests.values(): + dirname, basename = os.path.split(filename) + digest_and_filename = new_digest + '.' + basename + consistent_filenames.append(os.path.join(dirname, digest_and_filename)) + written_filename = consistent_filenames.pop() + + # Verify whether new metadata needs to be written (i.e., has not been + # previously written or has changed. + write_new_metadata = False + + # Has the uncompressed metadata changed? Does it exist? If so, set + # 'write_compressed_version' to True so that it is written. + # compressed metadata should only be written if it does not exist or the + # uncompressed version has changed). + try: + file_length_junk, old_digests = tuf.util.get_file_details(written_filename) + if old_digests != new_digests: + write_new_metadata = True + + # 'tuf.Error' raised if 'filename' does not exist. + except tuf.Error, e: + write_new_metadata = True + + if write_new_metadata: + # The 'metadata' object is written to 'file_object', including compressed + # versions. To avoid partial metadata from being written, 'metadata' is + # first written to a temporary location (i.e., 'file_object') and then moved + # to 'filename'. + file_object = tuf.util.TempFile() + + # Serialize 'metadata' to the file-like object and then write + # 'file_object' to disk. The dictionary keys of 'metadata' are sorted + # and indentation is used. The 'tuf.util.TempFile' file-like object is + # automically closed after the final move. + file_object.write(file_content) + logger.info('Saving ' + repr(written_filename)) + file_object.move(written_filename) + + for consistent_filename in consistent_filenames: + logger.info('Linking ' + repr(consistent_filename)) + os.link(written_filename, consistent_filename) + + + # Generate the compressed versions of 'metadata', if necessary. A compressed + # file may be written (without needed to write the uncompressed version) if + # the repository maintainer adds compression after writting the the + # uncompressed version. + for compression in compressions: + file_object = None + + # Ignore the empty string that signifies non-compression. The uncompressed + # file was previously written above, if necessary. + if not len(compression): + continue + + elif compression == 'gz': + file_object = tuf.util.TempFile() + compressed_filename = filename + '.gz' + + # Instantiate a gzip object, but save compressed content to + # 'file_object' (i.e., GzipFile instance is based on its 'fileobj' + # argument). + with gzip.GzipFile(fileobj=file_object, mode='wb') as gzip_object: + gzip_object.write(file_content) + + else: + raise tuf.FormatError('Unknown compression algorithm: '+repr(compression)) + + # Save the compressed version, ensuring an unchanged file is not re-saved. + # Re-savign the same compressed version may cause its digest to unexpectedly + # change (gzip includes a timestamp) even though content has not changed. + _write_compressed_metadata(file_object, compressed_filename, + consistent_snapshot) + return written_filename + + + + + +def _write_compressed_metadata(file_object, compressed_filename, + consistent_snapshot): + """ + Write compressed versions of metadata, ensuring compressed file that have + not changed are not re-written, the digest of the compressed file is properly + added to the compressed filename, and consistent snapshots are also saved. + Ensure compressed files are written to a temporary location, and then + moved to their destinations. + """ + + # If a consistent snapshot is unneeded, 'file_object' may be simply moved + # 'compressed_filename' if not already written. + if not consistent_snapshot: + if not os.path.exists(compressed_filename): + file_object.move(compressed_filename) + + # The temporary file must be closed if 'file_object.move()' is not used. + # tuf.util.TempFile() automatically closes the temp file when move() is + # called + else: + file_object.close_temp_file() + + # Consistent snapshots = True. Ensure the file's digest is included in the + # compressed filename written, provided it does not already exist. + else: + compressed_content = file_object.read() + new_digests = [] + consistent_filenames = [] + + # Multiple snapshots may be written if the repository uses multiple + # hash algorithms. Generate the digest of the compressed content. + for hash_algorithm in tuf.conf.REPOSITORY_HASH_ALGORITHMS: + digest_object = tuf.hash.digest(hash_algorithm) + digest_object.update(compressed_content) + new_digests.append(digest_object.hexdigest()) + + # Attach each digest to the compressed consistent snapshot filename. + for new_digest in new_digests: + dirname, basename = os.path.split(compressed_filename) + digest_and_filename = new_digest + '.' + basename + consistent_filenames.append(os.path.join(dirname, digest_and_filename)) + + # Move the 'tuf.util.TempFile' object to one of the filenames so that it is + # saved and the temporary file closed. Any remaining consistent snapshots + # may still need to be copied or linked. + compressed_filename = consistent_filenames.pop() + if not os.path.exists(compressed_filename): + logger.info('Saving ' + repr(compressed_filename)) + file_object.move(compressed_filename) + + # Save any remaining compressed consistent snapshots. + for consistent_filename in consistent_filenames: + if not os.path.exists(consistent_filename): + logger.info('Linking ' + repr(consistent_filename)) + os.link(compressed_filename, consistent_filename) + + + + + +def create_tuf_client_directory(repository_directory, client_directory): + """ + + Create a client directory structure that the 'tuf.interposition' package + and 'tuf.client.updater' module expect of clients. Metadata files + downloaded from a remote TUF repository are saved to 'client_directory'. + The Root file must initially exist before an update request can be + satisfied. create_tuf_client_directory() ensures the minimum metadata + is copied and that required directories ('previous' and 'current') are + created in 'client_directory'. Software updaters integrating TUF may + use the client directory created as an initial copy of the repository's + metadadata. + + + repository_directory: + The path of the root repository directory. The 'metadata' and 'targets' + sub-directories should be available in 'repository_directory'. The + metadata files of 'repository_directory' are copied to 'client_directory'. + + client_directory: + The path of the root client directory. The 'current' and 'previous' + sub-directies are created and will store the metadata files copied + from 'repository_directory'. 'client_directory' will store metadata + and target files downloaded from a TUF repository. + + + tuf.FormatError, if the arguments are improperly formatted. + + tuf.RepositoryError, if the metadata directory in 'client_directory' + already exists. + + + Copies metadata files and directories from 'repository_directory' to + 'client_directory'. Parent directories are created if they do not exist. + + + None. + """ + + # Do the arguments have the correct format? + # This check ensures arguments have the appropriate number of objects and + # object types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if the check fails. + tuf.formats.PATH_SCHEMA.check_match(repository_directory) + tuf.formats.PATH_SCHEMA.check_match(client_directory) + + # Set the absolute path of the Repository's metadata directory. The metadata + # directory should be the one served by the Live repository. At a minimum, + # the repository's root file must be copied. + repository_directory = os.path.abspath(repository_directory) + metadata_directory = os.path.join(repository_directory, + METADATA_DIRECTORY_NAME) + + # Set the client's metadata directory, which will store the metadata copied + # from the repository directory set above. + client_directory = os.path.abspath(client_directory) + client_metadata_directory = os.path.join(client_directory, + METADATA_DIRECTORY_NAME) + + # If the client's metadata directory does not already exist, create it and + # any of its parent directories, otherwise raise an exception. An exception + # is raised to avoid accidently overwritting previous metadata. + try: + os.makedirs(client_metadata_directory) + + except OSError, e: + if e.errno == errno.EEXIST: + message = 'Cannot create a fresh client metadata directory: '+ \ + repr(client_metadata_directory)+'. Already exists.' + raise tuf.RepositoryError(message) + else: + raise + + # Move all metadata to the client's 'current' and 'previous' directories. + # The root metadata file MUST exist in '{client_metadata_directory}/current'. + # 'tuf.interposition' and 'tuf.client.updater.py' expect the 'current' and + # 'previous' directories to exist under 'metadata'. + client_current = os.path.join(client_metadata_directory, 'current') + client_previous = os.path.join(client_metadata_directory, 'previous') + shutil.copytree(metadata_directory, client_current) + shutil.copytree(metadata_directory, client_previous) + + + +def disable_console_log_messages(): + """ + + Disable logger messages printed to the console. For example, repository + maintainers may want to call this function if many roles will be sharing + keys, otherwise detected duplicate keys will continually log a warning + message. + + + None. + + + None. + + + Removes the 'tuf.log' console handler, added by default when + 'tuf.repository_tool.py' is imported. + + + None. + """ + + tuf.log.remove_console_handler() + + +if __name__ == '__main__': + # The interactive sessions of the documentation strings can + # be tested by running repository_tool.py as a standalone module: + # $ python repository_tool.py. + import doctest + doctest.testmod() diff --git a/tuf/roledb.py b/tuf/roledb.py index 4f114dfc4c..4941f663d6 100755 --- a/tuf/roledb.py +++ b/tuf/roledb.py @@ -13,7 +13,7 @@ Represent a collection of roles and their organization. The caller may create - a collection of roles from those found in the 'root.txt' metadata file by + a collection of roles from those found in the 'root.json' metadata file by calling 'create_roledb_from_rootmeta()', or individually by adding roles with 'add_role()'. There are many supplemental functions included here that yield useful information about the roles contained in the database, such as @@ -23,13 +23,20 @@ The role database is a dictionary conformant to 'tuf.formats.ROLEDICT_SCHEMA' and has the form: + {'rolename': {'keyids': ['34345df32093bd12...'], 'threshold': 1 - 'paths': ['path/to/role.txt']}} - + 'signatures': ['abcd3452...'], + 'paths': ['path/to/role.json'], + 'path_hash_prefixes': ['ab34df13'], + 'delegations': {'keys': {}, 'roles': {}}} + + The 'name', 'paths', 'path_hash_prefixes', and 'delegations' dict keys are + optional. """ import logging +import copy import tuf import tuf.formats @@ -66,7 +73,6 @@ def create_roledb_from_root_metadata(root_metadata): None. - """ # Does 'root_metadata' have the correct object format? @@ -81,6 +87,17 @@ def create_roledb_from_root_metadata(root_metadata): # Iterate through the roles found in 'root_metadata' # and add them to '_roledb_dict'. Duplicates are avoided. for rolename, roleinfo in root_metadata['roles'].items(): + if rolename == 'root': + roleinfo['version'] = root_metadata['version'] + roleinfo['expires'] = root_metadata['expires'] + + roleinfo['signatures'] = [] + roleinfo['signing_keyids'] = [] + roleinfo['compressions'] = [''] + roleinfo['partial_loaded'] = False + if rolename.startswith('targets'): + roleinfo['delegations'] = {'keys': {}, 'roles': []} + try: add_role(rolename, roleinfo) # tuf.Error raised if the parent role of 'rolename' does not exist. @@ -100,14 +117,21 @@ def add_role(rolename, roleinfo, require_parent=True): rolename: An object representing the role's name, conformant to 'ROLENAME_SCHEMA' - (e.g., 'root', 'release', 'timestamp'). + (e.g., 'root', 'snapshot', 'timestamp'). roleinfo: An object representing the role associated with 'rolename', conformant to - ROLE_SCHEMA. 'roleinfo' has the form: + ROLEDB_SCHEMA. 'roleinfo' has the form: {'keyids': ['34345df32093bd12...'], - 'threshold': 1} - + 'threshold': 1, + 'signatures': ['ab23dfc32'] + 'paths': ['path/to/target1', 'path/to/target2', ...], + 'path_hash_prefixes': ['a324fcd...', ...], + 'delegations': {'keys': } + + The 'paths', 'path_hash_prefixes', and 'delegations' dict keys are + optional. + The 'target' role has an additional 'paths' key. Its value is a list of strings representing the path of the target file(s). @@ -128,7 +152,6 @@ def add_role(rolename, roleinfo, require_parent=True): None. - """ # Does 'rolename' have the correct object format? @@ -137,10 +160,10 @@ def add_role(rolename, roleinfo, require_parent=True): tuf.formats.ROLENAME_SCHEMA.check_match(rolename) # Does 'roleinfo' have the correct object format? - tuf.formats.ROLE_SCHEMA.check_match(roleinfo) + tuf.formats.ROLEDB_SCHEMA.check_match(roleinfo) # Does 'require_parent' have the correct format? - tuf.formats.TOGGLE_SCHEMA.check_match(require_parent) + tuf.formats.BOOLEAN_SCHEMA.check_match(require_parent) # Raises tuf.InvalidNameError. _validate_rolename(rolename) @@ -157,8 +180,67 @@ def add_role(rolename, roleinfo, require_parent=True): if parent_role not in _roledb_dict: raise tuf.Error('Parent role does not exist: '+parent_role) - _roledb_dict[rolename] = roleinfo + _roledb_dict[rolename] = copy.deepcopy(roleinfo) + + + + + +def update_roleinfo(rolename, roleinfo): + """ + + + + rolename: + An object representing the role's name, conformant to 'ROLENAME_SCHEMA' + (e.g., 'root', 'snapshot', 'timestamp'). + + roleinfo: + An object representing the role associated with 'rolename', conformant to + ROLEDB_SCHEMA. 'roleinfo' has the form: + {'name': 'role_name', + 'keyids': ['34345df32093bd12...'], + 'threshold': 1, + 'paths': ['path/to/target1', 'path/to/target2', ...], + 'path_hash_prefixes': ['a324fcd...', ...]} + + The 'name', 'paths', and 'path_hash_prefixes' dict keys are optional. + + The 'target' role has an additional 'paths' key. Its value is a list of + strings representing the path of the target file(s). + + + tuf.FormatError, if 'rolename' or 'roleinfo' does not have the correct + object format. + + tuf.UnknownRoleError, if 'rolename' cannot be found in the role database. + + tuf.InvalidNameError, if 'rolename' is improperly formatted. + + + The role database is modified. + + + None. + """ + + # Does 'rolename' have the correct object format? + # This check will ensure 'rolename' has the appropriate number of objects + # and object types, and that all dict keys are properly named. + tuf.formats.ROLENAME_SCHEMA.check_match(rolename) + + # Does 'roleinfo' have the correct object format? + tuf.formats.ROLEDB_SCHEMA.check_match(roleinfo) + + # Raises tuf.InvalidNameError. + _validate_rolename(rolename) + + if rolename not in _roledb_dict: + raise tuf.UnknownRoleError('Role does not exist: '+rolename) + _roledb_dict[rolename] = copy.deepcopy(roleinfo) + + @@ -173,7 +255,7 @@ def get_parent_rolename(rolename): rolename: An object representing the role's name, conformant to 'ROLENAME_SCHEMA' - (e.g., 'root', 'release', 'timestamp'). + (e.g., 'root', 'snapshot', 'timestamp'). tuf.FormatError, if 'rolename' does not have the correct object format. @@ -187,7 +269,6 @@ def get_parent_rolename(rolename): A string representing the name of the parent role. - """ # Raises tuf.FormatError, tuf.UnknownRoleError, or tuf.InvalidNameError. @@ -214,7 +295,7 @@ def get_all_parent_roles(rolename): rolename: An object representing the role's name, conformant to 'ROLENAME_SCHEMA' - (e.g., 'root', 'release', 'timestamp'). + (e.g., 'root', 'snapshot', 'timestamp'). tuf.FormatError, if 'rolename' does not have the correct object format. @@ -228,7 +309,6 @@ def get_all_parent_roles(rolename): A list containing all the parent roles. - """ # Raises tuf.FormatError, tuf.UnknownRoleError, or tuf.InvalidNameError. @@ -267,7 +347,7 @@ def role_exists(rolename): rolename: An object representing the role's name, conformant to 'ROLENAME_SCHEMA' - (e.g., 'root', 'release', 'timestamp'). + (e.g., 'root', 'snapshot', 'timestamp'). tuf.FormatError, if 'rolename' does not have the correct object format. @@ -279,7 +359,6 @@ def role_exists(rolename): Boolean. True if 'rolename' is found in the role database, False otherwise. - """ # Raise tuf.FormatError, tuf.InvalidNameError. @@ -304,7 +383,7 @@ def remove_role(rolename): rolename: An object representing the role's name, conformant to 'ROLENAME_SCHEMA' - (e.g., 'root', 'release', 'timestamp'). + (e.g., 'root', 'snapshot', 'timestamp'). tuf.FormatError, if 'rolename' does not have the correct object format. @@ -318,7 +397,6 @@ def remove_role(rolename): None. - """ # Raises tuf.FormatError, tuf.UnknownRoleError, or tuf.InvalidNameError. @@ -344,7 +422,7 @@ def remove_delegated_roles(rolename): rolename: An object representing the role's name, conformant to 'ROLENAME_SCHEMA' - (e.g., 'root', 'release', 'timestamp'). + (e.g., 'root', 'snapshot', 'timestamp'). tuf.FormatError, if 'rolename' does not have the correct object format. @@ -358,7 +436,6 @@ def remove_delegated_roles(rolename): None. - """ # Raises tuf.FormatError, tuf.UnknownRoleError, or tuf.InvalidNameError. @@ -390,7 +467,6 @@ def get_rolenames(): A list of rolenames. - """ return _roledb_dict.keys() @@ -399,20 +475,61 @@ def get_rolenames(): +def get_roleinfo(rolename): + """ + + Return the roleinfo of 'rolename'. + + {'keyids': ['34345df32093bd12...'], + 'threshold': 1, + 'signatures': ['ab453bdf...', ...], + 'paths': ['path/to/target1', 'path/to/target2', ...], + 'path_hash_prefixes': ['a324fcd...', ...], + 'delegations': {'keys': {}, 'roles': []}} + + The 'signatures', 'paths', 'path_hash_prefixes', and 'delegations' dict keys + are optional. + + + rolename: + An object representing the role's name, conformant to 'ROLENAME_SCHEMA' + (e.g., 'root', 'snapshot', 'timestamp'). + + + tuf.FormatError, if 'rolename' is improperly formatted. + + tuf.UnknownRoleError, if 'rolename' does not exist. + + + None. + + + The roleinfo of 'rolename'. + """ + + # Raises tuf.FormatError, tuf.UnknownRoleError, or tuf.InvalidNameError. + _check_rolename(rolename) + + return copy.deepcopy(_roledb_dict[rolename]) + + + + + def get_role_keyids(rolename): """ Return a list of the keyids associated with 'rolename'. Keyids are used as identifiers for keys (e.g., rsa key). A list of keyids are associated with each rolename. - Signing a metadata file, such as 'root.txt' (Root role), + Signing a metadata file, such as 'root.json' (Root role), involves signing or verifying the file with a list of keys identified by keyid. rolename: An object representing the role's name, conformant to 'ROLENAME_SCHEMA' - (e.g., 'root', 'release', 'timestamp'). + (e.g., 'root', 'snapshot', 'timestamp'). tuf.FormatError, if 'rolename' does not have the correct object format. @@ -426,7 +543,6 @@ def get_role_keyids(rolename): A list of keyids. - """ # Raises tuf.FormatError, tuf.UnknownRoleError, or tuf.InvalidNameError. @@ -448,7 +564,7 @@ def get_role_threshold(rolename): rolename: An object representing the role's name, conformant to 'ROLENAME_SCHEMA' - (e.g., 'root', 'release', 'timestamp'). + (e.g., 'root', 'snapshot', 'timestamp'). tuf.FormatError, if 'rolename' does not have the correct object format. @@ -462,7 +578,6 @@ def get_role_threshold(rolename): A threshold integer value. - """ # Raises tuf.FormatError, tuf.UnknownRoleError, or tuf.InvalidNameError. @@ -484,7 +599,7 @@ def get_role_paths(rolename): rolename: An object representing the role's name, conformant to 'ROLENAME_SCHEMA' - (e.g., 'root', 'release', 'timestamp'). + (e.g., 'root', 'snapshot', 'timestamp'). tuf.FormatError, if 'rolename' does not have the correct object format. @@ -498,7 +613,6 @@ def get_role_paths(rolename): A list of paths. - """ # Raises tuf.FormatError, tuf.UnknownRoleError, or tuf.InvalidNameError. @@ -526,7 +640,7 @@ def get_delegated_rolenames(rolename): rolename: An object representing the role's name, conformant to 'ROLENAME_SCHEMA' - (e.g., 'root', 'release', 'timestamp'). + (e.g., 'root', 'snapshot', 'timestamp'). tuf.FormatError, if 'rolename' does not have the correct object format. @@ -540,8 +654,7 @@ def get_delegated_rolenames(rolename): A list of rolenames. Note that the rolenames are *NOT* sorted by order of - delegation! - + delegation. """ # Raises tuf.FormatError, tuf.UnknownRoleError, or tuf.InvalidNameError. @@ -578,7 +691,6 @@ def clear_roledb(): None. - """ _roledb_dict.clear() @@ -593,7 +705,6 @@ def _check_rolename(rolename): 'tuf.formats.ROLENAME_SCHEMA', tuf.UnknownRoleError if 'rolename' is not found in the role database, or tuf.InvalidNameError if 'rolename' is not formatted correctly. - """ # Does 'rolename' have the correct object format? @@ -616,7 +727,6 @@ def _validate_rolename(rolename): Raise tuf.InvalidNameError if 'rolename' is not formatted correctly. It is assumed 'rolename' has been checked against 'ROLENAME_SCHEMA' prior to calling this function. - """ if rolename == '': diff --git a/tuf/rsa_key.py b/tuf/rsa_key.py deleted file mode 100755 index 8d73739117..0000000000 --- a/tuf/rsa_key.py +++ /dev/null @@ -1,664 +0,0 @@ -""" - - rsa_key.py - - - Vladimir Diaz - - - March 9, 2012. Based on a previous version of this module by Geremy Condra. - - - See LICENSE for licensing information. - - - The goal of this module is to support public-key cryptography using the RSA - algorithm. The RSA-related functions provided include generate(), - create_signature(), and verify_signature(). The create_encrypted_pem() and - create_from_encrypted_pem() functions are optional, and may be used save a - generated RSA key to a file. The 'PyCrypto' package used by 'rsa_key.py' - generates the actual RSA keys and the functions listed above can be viewed - as an easy-to-use public interface. Additional functions contained here - include create_in_metadata_format() and create_from_metadata_format(). These - last two functions produce or use RSA keys compatible with the key structures - listed in TUF Metadata files. The generate() function returns a dictionary - containing all the information needed of RSA keys, such as public and private= - keys, keyIDs, and an idenfier. create_signature() and verify_signature() are - supplemental functions used for generating RSA signatures and verifying them. - https://en.wikipedia.org/wiki/RSA_(algorithm) - - Key IDs are used as identifiers for keys (e.g., RSA key). They are the - hexadecimal representation of the hash of key object (specifically, the key - object containing only the public key). Review 'rsa_key.py' and the - '_get_keyid()' function to see precisely how keyids are generated. One may - get the keyid of a key object by simply accessing the dictionary's 'keyid' - key (i.e., rsakey['keyid']). - - """ - - -# Required for hexadecimal conversions. Signatures are hexlified. -import binascii - -# Crypto.PublicKey (i.e., PyCrypto public-key cryptography) provides algorithms -# such as Digital Signature Algorithm (DSA) and the ElGamal encryption system. -# 'Crypto.PublicKey.RSA' is needed here to generate, sign, and verify RSA keys. -import Crypto.PublicKey.RSA - -# PyCrypto requires 'Crypto.Hash' hash objects to generate PKCS#1 PSS -# signatures (i.e., Crypto.Signature.PKCS1_PSS). -import Crypto.Hash.SHA256 - -# RSA's probabilistic signature scheme with appendix (RSASSA-PSS). -# PKCS#1 v1.5 is provided for compatability with existing applications, but -# RSASSA-PSS is encouraged for newer applications. RSASSA-PSS generates -# a random salt to ensure the signature generated is probabilistic rather than -# deterministic, like PKCS#1 v1.5. -# http://en.wikipedia.org/wiki/RSA-PSS#Schemes -# https://tools.ietf.org/html/rfc3447#section-8.1 -import Crypto.Signature.PKCS1_PSS - -import tuf - -# Digest objects needed to generate hashes. -import tuf.hash - -# Perform object format-checking. -import tuf.formats - - -_KEY_ID_HASH_ALGORITHM = 'sha256' - -# Recommended RSA key sizes: -# http://www.emc.com/emc-plus/rsa-labs/historical/twirl-and-rsa-key-size.htm#table1 -# According to the document above, revised May 6, 2003, RSA keys of -# size 3072 provide security through 2031 and beyond. -_DEFAULT_RSA_KEY_BITS = 3072 - - -def generate(bits=_DEFAULT_RSA_KEY_BITS): - """ - - Generate public and private RSA keys, with modulus length 'bits'. - In addition, a keyid used as an identifier for RSA keys is generated. - The object returned conforms to 'tuf.formats.RSAKEY_SCHEMA' and as the form: - {'keytype': 'rsa', - 'keyid': keyid, - 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', - 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} - - The public and private keys are in PEM format and stored as strings. - - Although the crytography library called sets a 1024-bit minimum key size, - generate() enforces a minimum key size of 2048 bits. If 'bits' is - unspecified, a 3072-bit RSA key is generated, which is the key size - recommended by TUF. - - - bits: - The key size, or key length, of the RSA key. 'bits' must be 2048, or - greater, and a multiple of 256. - - - ValueError, if an exception occurs after calling the RSA key generation - routine. 'bits' must be a multiple of 256. The 'ValueError' exception is - raised by the key generation function of the cryptography library called. - - tuf.FormatError, if 'bits' does not contain the correct format. - - - The RSA keys are generated by calling PyCrypto's - Crypto.PublicKey.RSA.generate(). - - - A dictionary containing the RSA keys and other identifying information. - - """ - - - # Does 'bits' have the correct format? - # This check will ensure 'bits' conforms to 'tuf.formats.RSAKEYBITS_SCHEMA'. - # 'bits' must be an integer object, with a minimum value of 2048. - # Raise 'tuf.FormatError' if the check fails. - tuf.formats.RSAKEYBITS_SCHEMA.check_match(bits) - - # Begin building the RSA key dictionary. - rsakey_dict = {} - keytype = 'rsa' - - # Generate the public and private RSA keys. The PyCrypto module performs - # the actual key generation. Raise 'ValueError' if 'bits' is less than 1024 - # or not a multiple of 256, although a 2048-bit minimum is enforced by - # tuf.formats.RSAKEYBITS_SCHEMA.check_match(). - rsa_key_object = Crypto.PublicKey.RSA.generate(bits) - - # Extract the public & private halves of the RSA key and generate their - # PEM-formatted representations. The dictionary returned contains the - # private and public RSA keys in PEM format, as strings. - private_key_pem = rsa_key_object.exportKey(format='PEM') - rsa_pubkey = rsa_key_object.publickey() - public_key_pem = rsa_pubkey.exportKey(format='PEM') - - # Generate the keyid for the RSA key. 'key_value' corresponds to the - # 'keyval' entry of the 'RSAKEY_SCHEMA' dictionary. The private key - # information is not included in the generation of the 'keyid' identifier. - key_value = {'public': public_key_pem, - 'private': ''} - keyid = _get_keyid(key_value) - - # Build the 'rsakey_dict' dictionary. Update 'key_value' with the RSA - # private key prior to adding 'key_value' to 'rsakey_dict'. - key_value['private'] = private_key_pem - - rsakey_dict['keytype'] = keytype - rsakey_dict['keyid'] = keyid - rsakey_dict['keyval'] = key_value - - return rsakey_dict - - - - - -def create_in_metadata_format(key_value, private=False): - """ - - Return a dictionary conformant to 'tuf.formats.KEY_SCHEMA'. - If 'private' is True, include the private key. The dictionary - returned has the form: - {'keytype': 'rsa', - 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', - 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} - - or if 'private' is False: - - {'keytype': 'rsa', - 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', - 'private': ''}} - - The private and public keys are in PEM format. - - RSA keys are stored in Metadata files (e.g., root.txt) in the format - returned by this function. - - - key_value: - A dictionary containing a private and public RSA key. - 'key_value' is of the form: - - {'public': '-----BEGIN RSA PUBLIC KEY----- ...', - 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}}, - conformat to 'tuf.formats.KEYVAL_SCHEMA'. - - private: - Indicates if the private key should be included in the - returned dictionary. - - - tuf.FormatError, if 'key_value' does not conform to - 'tuf.formats.KEYVAL_SCHEMA'. - - - None. - - - An 'KEY_SCHEMA' dictionary. - - """ - - - # Does 'key_value' have the correct format? - # This check will ensure 'key_value' has the appropriate number - # of objects and object types, and that all dict keys are properly named. - # Raise 'tuf.FormatError' if the check fails. - tuf.formats.KEYVAL_SCHEMA.check_match(key_value) - - if private is True and key_value['private']: - return {'keytype': 'rsa', 'keyval': key_value} - else: - public_key_value = {'public': key_value['public'], 'private': ''} - return {'keytype': 'rsa', 'keyval': public_key_value} - - - - - -def create_from_metadata_format(key_metadata): - """ - - Construct an RSA key dictionary (i.e., tuf.formats.RSAKEY_SCHEMA) - from 'key_metadata'. The dict returned by this function has the exact - format as the dict returned by generate(). It is of the form: - - {'keytype': 'rsa', - 'keyid': keyid, - 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', - 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} - - The public and private keys are in PEM format and stored as strings. - - RSA key dictionaries in RSAKEY_SCHEMA format should be used by - modules storing a collection of keys, such as a keydb and keystore. - RSA keys as stored in metadata files use a different format, so this - function should be called if an RSA key is extracted from one of these - metadata files and needs converting. Generate() creates an entirely - new key and returns it in the format appropriate for 'keydb.py' and - 'keystore.py'. - - - key_metadata: - The RSA key dictionary as stored in Metadata files, conforming to - 'tuf.formats.KEY_SCHEMA'. It has the form: - - {'keytype': '...', - 'keyval': {'public': '...', - 'private': '...'}} - - - tuf.FormatError, if 'key_metadata' does not conform to - 'tuf.formats.KEY_SCHEMA'. - - - None. - - - A dictionary containing the RSA keys and other identifying information. - - """ - - - # Does 'key_metadata' have the correct format? - # This check will ensure 'key_metadata' has the appropriate number - # of objects and object types, and that all dict keys are properly named. - # Raise 'tuf.FormatError' if the check fails. - tuf.formats.KEY_SCHEMA.check_match(key_metadata) - - # Construct the dictionary to be returned. - rsakey_dict = {} - keytype = 'rsa' - key_value = key_metadata['keyval'] - - # Convert 'key_value' to 'tuf.formats.KEY_SCHEMA' and generate its hash - # The hash is in hexdigest form. - keyid = _get_keyid(key_value) - - # We now have all the required key values. Build 'rsakey_dict'. - rsakey_dict['keytype'] = keytype - rsakey_dict['keyid'] = keyid - rsakey_dict['keyval'] = key_value - - return rsakey_dict - - - - - -def _get_keyid(key_value): - """Return the keyid for 'key_value'.""" - - # 'keyid' will be generated from an object conformant to KEY_SCHEMA, - # which is the format Metadata files (e.g., root.txt) store keys. - # 'create_in_metadata_format()' returns the object needed by _get_keyid(). - rsakey_meta = create_in_metadata_format(key_value, private=False) - - # Convert the RSA key to JSON Canonical format suitable for adding - # to digest objects. - rsakey_update_data = tuf.formats.encode_canonical(rsakey_meta) - - # Create a digest object and call update(), using the JSON - # canonical format of 'rskey_meta' as the update data. - digest_object = tuf.hash.digest(_KEY_ID_HASH_ALGORITHM) - digest_object.update(rsakey_update_data) - - # 'keyid' becomes the hexadecimal representation of the hash. - keyid = digest_object.hexdigest() - - return keyid - - - - - -def create_signature(rsakey_dict, data): - """ - - Return a signature dictionary of the form: - {'keyid': keyid, - 'method': 'PyCrypto-PKCS#1 PPS', - 'sig': sig}. - - The signing process will use the private key - rsakey_dict['keyval']['private'] and 'data' to generate the signature. - - RFC3447 - RSASSA-PSS - http://www.ietf.org/rfc/rfc3447.txt - - - rsakey_dict: - A dictionary containing the RSA keys and other identifying information. - 'rsakey_dict' has the form: - - {'keytype': 'rsa', - 'keyid': keyid, - 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', - 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} - - The public and private keys are in PEM format and stored as strings. - - data: - Data object used by create_signature() to generate the signature. - - - TypeError, if a private key is not defined for 'rsakey_dict'. - - tuf.FormatError, if an incorrect format is found for the - 'rsakey_dict' object. - - - PyCrypto's 'Crypto.Signature.PKCS1_PSS' called to perform the actual - signing. - - - A signature dictionary conformat to 'tuf.format.SIGNATURE_SCHEMA'. - - """ - - - # Does 'rsakey_dict' have the correct format? - # This check will ensure 'rsakey_dict' has the appropriate number - # of objects and object types, and that all dict keys are properly named. - # Raise 'tuf.FormatError' if the check fails. - tuf.formats.RSAKEY_SCHEMA.check_match(rsakey_dict) - - # Signing the 'data' object requires a private key. - # The 'PyCrypto-PKCS#1 PSS' (i.e., PyCrypto module) signing method is the - # only method currently supported. - signature = {} - private_key = rsakey_dict['keyval']['private'] - keyid = rsakey_dict['keyid'] - method = 'PyCrypto-PKCS#1 PSS' - sig = None - - # Verify the signature, but only if the private key has been set. The private - # key is a NULL string if unset. Although it may be clearer to explicit check - # that 'private_key' is not '', we can/should check for a value and not - # compare identities with the 'is' keyword. - if len(private_key): - # Calculate the SHA256 hash of 'data' and generate the hash's PKCS1-PSS - # signature. - try: - rsa_key_object = Crypto.PublicKey.RSA.importKey(private_key) - sha256_object = Crypto.Hash.SHA256.new(data) - pkcs1_pss_signer = Crypto.Signature.PKCS1_PSS.new(rsa_key_object) - sig = pkcs1_pss_signer.sign(sha256_object) - except (ValueError, IndexError, TypeError), e: - message = 'An RSA signature could not be generated.' - raise tuf.CryptoError(message) - else: - raise TypeError('The required private key is not defined for "rsakey_dict".') - - # Build the signature dictionary to be returned. - # The hexadecimal representation of 'sig' is stored in the signature. - signature['keyid'] = keyid - signature['method'] = method - signature['sig'] = binascii.hexlify(sig) - - return signature - - - - - -def verify_signature(rsakey_dict, signature, data): - """ - - Determine whether the private key belonging to 'rsakey_dict' produced - 'signature'. verify_signature() will use the public key found in - 'rsakey_dict', the 'method' and 'sig' objects contained in 'signature', - and 'data' to complete the verification. Type-checking performed on both - 'rsakey_dict' and 'signature'. - - - rsakey_dict: - A dictionary containing the RSA keys and other identifying information. - 'rsakey_dict' has the form: - - {'keytype': 'rsa', - 'keyid': keyid, - 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', - 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} - - The public and private keys are in PEM format and stored as strings. - - signature: - The signature dictionary produced by tuf.rsa_key.create_signature(). - 'signature' has the form: - {'keyid': keyid, 'method': 'method', 'sig': sig}. Conformant to - 'tuf.formats.SIGNATURE_SCHEMA'. - - data: - Data object used by tuf.rsa_key.create_signature() to generate - 'signature'. 'data' is needed here to verify the signature. - - - tuf.UnknownMethodError. Raised if the signing method used by - 'signature' is not one supported by tuf.rsa_key.create_signature(). - - tuf.FormatError. Raised if either 'rsakey_dict' - or 'signature' do not match their respective tuf.formats schema. - 'rsakey_dict' must conform to 'tuf.formats.RSAKEY_SCHEMA'. - 'signature' must conform to 'tuf.formats.SIGNATURE_SCHEMA'. - - - Crypto.Signature.PKCS1_PSS.verify() called to do the actual verification. - - - Boolean. True if the signature is valid, False otherwise. - - """ - - - # Does 'rsakey_dict' have the correct format? - # This check will ensure 'rsakey_dict' has the appropriate number - # of objects and object types, and that all dict keys are properly named. - # Raise 'tuf.FormatError' if the check fails. - tuf.formats.RSAKEY_SCHEMA.check_match(rsakey_dict) - - # Does 'signature' have the correct format? - tuf.formats.SIGNATURE_SCHEMA.check_match(signature) - - # Using the public key belonging to 'rsakey_dict' - # (i.e., rsakey_dict['keyval']['public']), verify whether 'signature' - # was produced by rsakey_dict's corresponding private key - # rsakey_dict['keyval']['private']. Before returning the Boolean result, - # ensure 'PyCrypto-PKCS#1 PSS' was used as the signing method. - method = signature['method'] - sig = signature['sig'] - public_key = rsakey_dict['keyval']['public'] - valid_signature = False - - if method == 'PyCrypto-PKCS#1 PSS': - try: - rsa_key_object = Crypto.PublicKey.RSA.importKey(public_key) - pkcs1_pss_verifier = Crypto.Signature.PKCS1_PSS.new(rsa_key_object) - sha256_object = Crypto.Hash.SHA256.new(data) - - # The metadata stores signatures in hex. Unhexlify and verify the - # signature. - signature = binascii.unhexlify(sig) - valid_signature = pkcs1_pss_verifier.verify(sha256_object, signature) - except (ValueError, IndexError, TypeError), e: - message = 'The RSA signature could not be verified.' - raise tuf.CryptoError(message) - else: - raise tuf.UnknownMethodError(method) - - return valid_signature - - - - - -def create_encrypted_pem(rsakey_dict, passphrase): - """ - - Return a string in PEM format, where the private part of the RSA key is - encrypted. The private part of the RSA key is encrypted by the Triple - Data Encryption Algorithm (3DES) and Cipher-block chaining (CBC) for the - mode of operation. Password-Based Key Derivation Function 1 (PBKF1) + MD5 - is used to strengthen 'passphrase'. - - https://en.wikipedia.org/wiki/Triple_DES - https://en.wikipedia.org/wiki/PBKDF2 - - - rsakey_dict: - A dictionary containing the RSA keys and other identifying information. - 'rsakey_dict' has the form: - - {'keytype': 'rsa', - 'keyid': keyid, - 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', - 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} - - The public and private keys are in PEM format and stored as strings. - - passphrase: - The passphrase, or password, to encrypt the private part of the RSA - key. 'passphrase' is not used directly as the encryption key, a stronger - encryption key is derived from it. - - - TypeError, if a private key is not defined for 'rsakey_dict'. - - tuf.FormatError, if an incorrect format is found for 'rsakey_dict'. - - - PyCrypto's Crypto.PublicKey.RSA.exportKey() called to perform the actual - generation of the PEM-formatted output. - - - A string in PEM format, where the private RSA key is encrypted. - - """ - - # Does 'rsakey_dict' have the correct format? - # This check will ensure 'rsakey_dict' has the appropriate number - # of objects and object types, and that all dict keys are properly named. - # Raise 'tuf.FormatError' if the check fails. - tuf.formats.RSAKEY_SCHEMA.check_match(rsakey_dict) - - # Does 'signature' have the correct format? - tuf.formats.PASSWORD_SCHEMA.check_match(passphrase) - - # Extract the private key from 'rsakey_dict', which is stored in PEM format - # and unencrypted. The extracted key will be imported and converted to - # PyCrypto's RSA key object (i.e., Crypto.PublicKey.RSA).Use PyCrypto's - # exportKey method, with a passphrase specified, to create the string. - # PyCrypto uses PBKDF1+MD5 to strengthen 'passphrase', and 3DES with CBC mode - # for encryption. - private_key = rsakey_dict['keyval']['private'] - try: - rsa_key_object = Crypto.PublicKey.RSA.importKey(private_key) - rsakey_pem_encrypted = rsa_key_object.exportKey(format='PEM', - passphrase=passphrase) - except (ValueError, IndexError, TypeError), e: - message = 'An encrypted RSA key in PEM format could not be generated.' - raise tuf.CryptoError(message) - - return rsakey_pem_encrypted - - - - - -def create_from_encrypted_pem(encrypted_pem, passphrase): - """ - - Return an RSA key in 'tuf.formats.RSAKEY_SCHEMA' format, which has the - form: - {'keytype': 'rsa', - 'keyid': keyid, - 'keyval': {'public': '-----BEGIN RSA PUBLIC KEY----- ...', - 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} - - The RSAKEY_SCHEMA object is generated from a byte string in PEM format, - where the private part of the RSA key is encrypted. PyCrypto's importKey - method is used, where a passphrase is specified. PyCrypto uses PBKDF1+MD5 - to strengthen 'passphrase', and 3DES with CBC mode for encryption/decryption. - Alternatively, key data may be encrypted with AES-CTR-Mode and the passphrase - strengthened with PBKDF2+SHA256. See 'keystore.py'. - - - encrypted_pem: - A byte string in PEM format, where the private key is encrypted. It has - the form: - - '-----BEGIN RSA PRIVATE KEY-----\n - Proc-Type: 4,ENCRYPTED\nDEK-Info: DES-EDE3-CBC ...' - - passphrase: - The passphrase, or password, to decrypt the private part of the RSA - key. 'passphrase' is not directly used as the encryption key, instead - it is used to derive a stronger symmetric key. - - - TypeError, if a private key is not defined for 'rsakey_dict'. - - tuf.FormatError, if an incorrect format is found for the - 'rsakey_dict' object. - - - PyCrypto's 'Crypto.PublicKey.RSA.importKey()' called to perform the actual - conversion from an encrypted RSA private key. - - - A dictionary in 'tuf.formats.RSAKEY_SCHEMA' format. - - """ - - # Does 'encryped_pem' have the correct format? - # This check will ensure 'encrypted_pem' has the appropriate number - # of objects and object types, and that all dict keys are properly named. - # Raise 'tuf.FormatError' if the check fails. - tuf.formats.PEMRSA_SCHEMA.check_match(encrypted_pem) - - # Does 'passphrase' have the correct format? - tuf.formats.PASSWORD_SCHEMA.check_match(passphrase) - - keytype = 'rsa' - rsakey_dict = {} - - try: - rsa_key_object = Crypto.PublicKey.RSA.importKey(encrypted_pem, passphrase) - except (ValueError, IndexError, TypeError), e: - message = 'An RSA key object could not be generated from the encrypted '+\ - 'PEM string.' - # Raise 'tuf.CryptoError' instead of PyCrypto's exception to avoid - # revealing sensitive error, such as a decryption error due to an - # invalid passphrase. - raise tuf.CryptoError(message) - - # Extract the public & private halves of the RSA key and generate their - # PEM-formatted representations. The dictionary returned contains the - # private and public RSA keys in PEM format, as strings. - private_key_pem = rsa_key_object.exportKey(format='PEM') - rsa_pubkey = rsa_key_object.publickey() - public_key_pem = rsa_pubkey.exportKey(format='PEM') - - # Generate the keyid for the RSA key. 'key_value' corresponds to the - # 'keyval' entry of the 'RSAKEY_SCHEMA' dictionary. The private key - # information is not included in the generation of the 'keyid' identifier. - key_value = {'public': public_key_pem, - 'private': ''} - keyid = _get_keyid(key_value) - - # Build the 'rsakey_dict' dictionary. Update 'key_value' with the RSA - # private key prior to adding 'key_value' to 'rsakey_dict'. - key_value['private'] = private_key_pem - - rsakey_dict['keytype'] = keytype - rsakey_dict['keyid'] = keyid - rsakey_dict['keyval'] = key_value - - return rsakey_dict diff --git a/tuf/schema.py b/tuf/schema.py index acd50b634b..278e1899c7 100755 --- a/tuf/schema.py +++ b/tuf/schema.py @@ -38,7 +38,6 @@ criteria. See 'tuf.formats.py' and the rest of this module for extensive examples. Anything related to the checking of TUF objects and their formats can be found in 'formats.py'. - """ @@ -55,7 +54,6 @@ class Schema: that are encodable in JSON. 'Schema' is the base class for the other classes defined in this module. All derived classes should implement check_match(). - """ def matches(self, object): @@ -64,7 +62,6 @@ def matches(self, object): Return True if 'object' matches this schema, False if it doesn't. If the caller wishes to signal an error on a failed match, check_match() should be called, which will raise a 'tuf.FormatError' exception. - """ try: @@ -82,7 +79,6 @@ def check_match(self, object): implement check_match(). If 'object' matches the schema, check_match() should simply return. If 'object' does not match the schema, 'tuf.FormatError' should be raised. - """ raise NotImplementedError() @@ -110,7 +106,6 @@ class Any(Schema): True >>> schema.matches([1, 'list']) True - """ def __init__(self): @@ -143,7 +138,6 @@ class String(Schema): True >>> schema.matches('Not hi') False - """ def __init__(self, string): @@ -187,7 +181,6 @@ class AnyString(Schema): True >>> schema.matches({}) False - """ def __init__(self): @@ -202,6 +195,48 @@ def check_match(self, object): +class LengthString(Schema): + """ + + Matches any string of a specified length. The argument object + must be a string. At instantiation, the string length is set + and any future comparisons are checked against this internal + string value length. + + Supported methods include + matches(): returns a Boolean result. + check_match(): raises 'tuf.FormatError' on a mismatch. + + + + >>> schema = LengthString(5) + >>> schema.matches('Hello') + True + >>> schema.matches('Hi') + False + """ + + def __init__(self, length): + if isinstance(length, bool) or not isinstance(length, (int, long)): + # We need to check for bool as a special case, since bool + # is for historical reasons a subtype of int. + raise tuf.FormatError('Got '+repr(length)+' instead of an integer.') + + self._string_length = length + + + def check_match(self, object): + if not isinstance(object, basestring): + raise tuf.FormatError('Expected a string but got '+repr(object)) + + if len(object) != self._string_length: + raise tuf.FormatError('Expected a string of length '+ + repr(self._string_length)) + + + + + class OneOf(Schema): """ @@ -229,7 +264,6 @@ class OneOf(Schema): True >>> schema.matches(['Hi']) False - """ def __init__(self, alternatives): @@ -275,7 +309,6 @@ class AllOf(Schema): False >>> schema.matches('a') True - """ def __init__(self, required_schemas): @@ -314,7 +347,6 @@ class Boolean(Schema): True >>> schema.matches(11) False - """ def __init__(self): @@ -367,7 +399,6 @@ class ListOf(Schema): True >>> schema.matches([3]*11) False - """ def __init__(self, schema, min_count=0, max_count=sys.maxint, list_name='list'): @@ -380,7 +411,6 @@ def __init__(self, schema, min_count=0, max_count=sys.maxint, list_name='list'): min_count: The minimum number of sub-schema in 'schema'. max_count: The maximum number of sub-schema in 'schema'. list_name: A string identifier for the ListOf object. - """ if not isinstance(schema, Schema): @@ -443,7 +473,6 @@ class Integer(Schema): True >>> Integer(lo=10, hi=30).matches(5) False - """ def __init__(self, lo= -sys.maxint, hi=sys.maxint): @@ -454,7 +483,6 @@ def __init__(self, lo= -sys.maxint, hi=sys.maxint): lo: The minimum value the int object argument can be. hi: The maximum value the int object argument can be. - """ self._lo = lo @@ -468,7 +496,7 @@ def check_match(self, object): raise tuf.FormatError('Got '+repr(object)+' instead of an integer.') elif not (self._lo <= object <= self._hi): - int_range = '['+repr(self._lo)+','+repr(self._hi)+'].' + int_range = '['+repr(self._lo)+', '+repr(self._hi)+'].' raise tuf.FormatError(repr(object)+' not in range '+int_range) @@ -502,7 +530,6 @@ class DictOf(Schema): False >>> schema.matches({'a': ['x', 'y'], 'e' : ['', ''], 'd' : ['a', 'b']}) False - """ def __init__(self, key_schema, value_schema): @@ -513,7 +540,6 @@ def __init__(self, key_schema, value_schema): key_schema: The dictionary's key. value_schema: The dictionary's value. - """ if not isinstance(key_schema, Schema): @@ -564,7 +590,6 @@ class Optional(Schema): False >>> schema.matches({'k1': 'X'}) True - """ def __init__(self, schema): @@ -604,7 +629,6 @@ class Object(Schema): False >>> schema.matches({'a':'ZYYY'}) False - """ def __init__(self, object_name='object', **required): @@ -616,7 +640,6 @@ def __init__(self, object_name='object', **required): object_name: A string identifier for the object argument. A variable number of keyword arguments is accepted. - """ # Ensure valid arguments. @@ -713,7 +736,6 @@ class Struct(Schema): False >>> schema.matches(['X', 3, 'A']) False - """ def __init__(self, sub_schemas, optional_schemas=[], allow_more=False, @@ -727,7 +749,6 @@ def __init__(self, sub_schemas, optional_schemas=[], allow_more=False, optional_schemas: The optional list of schemas. allow_more: Specifies that an optional list of types is allowed. struct_name: A string identifier for the Struct object. - """ # Ensure each item of the list contains the expected object type. @@ -792,7 +813,6 @@ class RegularExpression(Schema): False >>> schema.matches([33, 'Hello']) False - """ def __init__(self, pattern=None, modifiers=0, re_object=None, re_name=None): @@ -805,7 +825,6 @@ def __init__(self, pattern=None, modifiers=0, re_object=None, re_name=None): modifiers: Flags to use when compiling the pattern. re_object: A compiled regular expression object. re_name: Identifier for the regular expression object. - """ if not isinstance(pattern, basestring): diff --git a/tuf/sig.py b/tuf/sig.py index ce5b9f56af..d789aa5efa 100755 --- a/tuf/sig.py +++ b/tuf/sig.py @@ -27,13 +27,12 @@ keys into different categories. As keys are added and removed, the system must securely and efficiently verify the status of these signatures. For instance, a bunch of keys have recently expired. How many valid keys - are now available to the Release role? This question can be answered by + are now available to the Snapshot role? This question can be answered by get_signature_status(), which will return a full 'status report' of these 'signable' dicts. This module also provides a convenient verify() function that will determine if a role still has a sufficient number of valid keys. If a caller needs to update the signatures of a 'signable' object, there is also a function for that. - """ import tuf @@ -63,7 +62,7 @@ def get_signature_status(signable, role=None): Conformant to tuf.formats.SIGNABLE_SCHEMA. role: - TUF role (e.g., 'root', 'targets', 'release'). + TUF role (e.g., 'root', 'targets', 'snapshot'). tuf.FormatError, if 'signable' does not have the correct format. @@ -76,7 +75,6 @@ def get_signature_status(signable, role=None): A dictionary representing the status of the signatures in 'signable'. Conformant to tuf.formats.SIGNATURESTATUS_SCHEMA. - """ # Does 'signable' have the correct format? @@ -106,9 +104,6 @@ def get_signature_status(signable, role=None): signed = signable['signed'] signatures = signable['signatures'] - # 'signed' needed in canonical JSON format. - data = tuf.formats.encode_canonical(signed) - # Iterate through the signatures and enumerate the signature_status fields. # (i.e., good_sigs, bad_sigs, etc.). for signature in signatures: @@ -125,7 +120,7 @@ def get_signature_status(signable, role=None): # Identify key using an unknown key signing method. try: - valid_sig = tuf.rsa_key.verify_signature(key, signature, data) + valid_sig = tuf.keys.verify_signature(key, signature, signed) except tuf.UnknownMethodError: unknown_method_sigs.append(keyid) continue @@ -185,7 +180,7 @@ def verify(signable, role): signable = {'signed':, 'signatures': [{'keyid':, 'method':, 'sig':}]} role: - TUF role (e.g., 'root', 'targets', 'release'). + TUF role (e.g., 'root', 'targets', 'snapshot'). tuf.UnknownRoleError, if 'role' is not recognized. @@ -201,7 +196,6 @@ def verify(signable, role): Boolean. True if the number of good signatures >= the role's threshold, False otherwise. - """ # Retrieve the signature status. tuf.sig.get_signature_status() raises @@ -243,10 +237,8 @@ def may_need_new_keys(signature_status): Boolean. - """ - # Does 'signature_status' have the correct format? # This check will ensure 'signature_status' has the appropriate number # of objects and object types, and that all dict keys are properly named. @@ -281,11 +273,11 @@ def generate_rsa_signature(signed, rsakey_dict): signed: - The data used by 'tuf.rsa_key.create_signature()' to generate signatures. + The data used by 'tuf.keys.create_signature()' to generate signatures. It is stored in the 'signed' field of 'signable'. rsakey_dict: - The RSA key, a tuf.formats.RSAKEY_SCHEMA dictionary. + The RSA key, a 'tuf.formats.RSAKEY_SCHEMA' dictionary. Used here to produce 'keyid', 'method', and 'sig'. @@ -300,7 +292,6 @@ def generate_rsa_signature(signed, rsakey_dict): Signature dictionary conformant to tuf.formats.SIGNATURE_SCHEMA. Has the form: {'keyid': keyid, 'method': 'evp', 'sig': sig} - """ # We need 'signed' in canonical JSON format to generate @@ -309,6 +300,6 @@ def generate_rsa_signature(signed, rsakey_dict): # Generate the RSA signature. # Raises tuf.FormatError and TypeError. - signature = tuf.rsa_key.create_signature(rsakey_dict, signed) + signature = tuf.keys.create_signature(rsakey_dict, signed) return signature diff --git a/tuf/tests/repository_setup.py b/tuf/tests/repository_setup.py index 066ca61eff..60161438a9 100644 --- a/tuf/tests/repository_setup.py +++ b/tuf/tests/repository_setup.py @@ -14,7 +14,6 @@ To provide a quick repository structure to be used in conjunction with test modules like test_updater.py for instance. - """ import os @@ -24,14 +23,13 @@ import tempfile import tuf.formats -import tuf.rsa_key as rsa_key import tuf.repo.keystore as keystore import tuf.repo.signerlib as signerlib +import tuf.repository_tool as repo_tool import tuf.repo.signercli as signercli import tuf.tests.unittest_toolbox as unittest_toolbox - # Role:keyids dictionary. role_keyids = {} @@ -217,8 +215,8 @@ def _mock_get_keyids(junk): keystore._keystore = unittest_toolbox.Modified_TestCase.rsa_keystore keystore._derived_keys = unittest_toolbox.Modified_TestCase.rsa_passwords - # Build release file. - signerlib.build_release_file(role_keyids['release'], server_metadata_dir, + # Build snapshot file. + signerlib.build_snapshot_file(role_keyids['snapshot'], server_metadata_dir, version, expiration_date+' UTC') # Build timestamp file. @@ -276,7 +274,6 @@ def create_repositories(): A dictionary of all repositories, with the following keys: (main_repository, client_repository, server_repository) - """ # Ensure the keyids for the required roles are loaded. Role keyids are diff --git a/tuf/tests/unittest_toolbox.py b/tuf/tests/unittest_toolbox.py index ff23af4959..d425fe76f9 100644 --- a/tuf/tests/unittest_toolbox.py +++ b/tuf/tests/unittest_toolbox.py @@ -15,7 +15,6 @@ Provides an array of various methods for unit testing. Use it instead of actual unittest module. This module builds on unittest module. Specifically, Modified_TestCase is a derived class from unittest.TestCase. - """ import os @@ -27,7 +26,7 @@ import string import ConfigParser -import tuf.rsa_key as rsa_key +import tuf.keys import tuf.repo.keystore as keystore # Modify the number of iterations (from the higher default count) so the unit @@ -100,11 +99,10 @@ def setUp(): random_string(length=7): Generate a 'length' long string of random characters. - """ # List of all top level roles. - role_list = ['root', 'targets', 'release', 'timestamp'] + role_list = ['root', 'targets', 'snapshot', 'timestamp'] # List of delegated roles. delegated_role_list = ['targets/delegated_role1', @@ -222,6 +220,7 @@ def make_temp_config_file(self, suffix='', directory=None, config_dict={}, expir dictionary in it using ConfigParser. It then returns the temp file path, dictionary tuple. """ + config = ConfigParser.RawConfigParser() if not config_dict: # Using the fact that empty sequences are false. @@ -288,7 +287,6 @@ def make_temp_directory_with_data_files(self, _current_dir=None, Returns: ('/tmp/tmp_dir_Test_random/', [targets/tmp_random1.txt, targets/tmp_random2.txt, targets/more_targets/tmp_random3.txt]) - """ if not _current_dir: @@ -385,7 +383,7 @@ def generate_rsakey(): 'private': '-----BEGIN RSA PRIVATE KEY----- ...'}} """ - rsakey = rsa_key.generate() + rsakey = tuf.keys.generate_rsa_key() keyid = rsakey['keyid'] Modified_TestCase.rsa_keyids.append(keyid) password = Modified_TestCase.random_string() diff --git a/tuf/tests/util_test_tools.py b/tuf/tests/util_test_tools.py index 43ea80fb5a..2b8834f0df 100644 --- a/tuf/tests/util_test_tools.py +++ b/tuf/tests/util_test_tools.py @@ -53,7 +53,7 @@ | | | keystore metadata targets | | | - key1.key ... role.txt ... file(1) ... + key1.key ... role.json ... file(1) ... '{root_repo}/tuf_repo/': developer's tuf-repository directory containing following subdirectories: @@ -72,7 +72,7 @@ | | current previous | | - role.txt ... role.txt ... + role.json ... role.json ... '{root_repo}/tuf_client/': client directory containing tuf metadata. '{root_repo}/tuf_client/metadata/current': directory where client stores @@ -113,14 +113,14 @@ and keys. tuf_refresh_repo(): - Refreshes metadata files at the 'tuf_repo' directory i.e. role.txt's at + Refreshes metadata files at the 'tuf_repo' directory i.e. role.json's at '{root_repo}/tuf_repo/metadata/'. Following roles are refreshed: - targets, release and timestamp. Also, the whole 'reg_repo' directory is + targets, snapshot and timestamp. Also, the whole 'reg_repo' directory is copied to targets directory i.e. '{root_repo}/tuf_repo/targets/'. -Note: metadata files are root.txt, targets.txt, release.txt and -timestamp.txt (denoted as 'role.txt in the diagrams'). There could be -more metadata files such us mirrors.txt. The metadata files are signed +Note: metadata files are root.json, targets.json, snapshot.json and +timestamp.json (denoted as 'role.json in the diagrams'). There could be +more metadata files such us mirrors.json. The metadata files are signed by their corresponding roles i.e. root, targets etc. More documentation is provided in the comment and doc blocks. @@ -174,8 +174,8 @@ def init_repo(using_tuf=False, port=None): server_proc = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE) - # Tailor url for the repository. In order to download a 'file.txt' - # from 'reg_repo' do: url+'reg_repo/file.txt' + # Tailor url for the repository. In order to download a 'file.json' + # from 'reg_repo' do: url+'reg_repo/file.json' relpath = os.path.basename(root_repo) url = 'http://localhost:'+str(port)+'/'+relpath+'/' @@ -326,7 +326,7 @@ def init_tuf(root_repo): # In our case 'role_info[keyids]' will only have on entry since only one # is being used. role_info = {} - role_list = ['root', 'targets', 'release', 'timestamp'] + role_list = ['root', 'targets', 'snapshot', 'timestamp'] for role in role_list: role_info[role] = info @@ -336,17 +336,17 @@ def init_tuf(root_repo): # Build the configuration file. conf_path = signerlib.build_config_file(metadata_dir, 365, role_info) - # Generate the 'root.txt' metadata file. + # Generate the 'root.json' metadata file. signerlib.build_root_file(conf_path, keyids, metadata_dir, version) - # Generate the 'targets.txt' metadata file. + # Generate the 'targets.json' metadata file. signerlib.build_targets_file([targets_dir], keyids, metadata_dir, version, expiration) - # Generate the 'release.txt' metadata file. - signerlib.build_release_file(keyids, metadata_dir, version, expiration) + # Generate the 'snapshot.json' metadata file. + signerlib.build_snapshot_file(keyids, metadata_dir, version, expiration) - # Generate the 'timestamp.txt' metadata file. + # Generate the 'timestamp.json' metadata file. signerlib.build_timestamp_file(keyids, metadata_dir, version, expiration) # Move the metadata to the client's 'current' and 'previous' directories. @@ -435,14 +435,14 @@ def tuf_refresh_repo(root_repo, keyids): shutil.copytree(reg_repo, targets_dir) version = version+1 - # Regenerate the 'targets.txt' metadata file. + # Regenerate the 'targets.json' metadata file. signerlib.build_targets_file([targets_dir], keyids, metadata_dir, version, expiration) - # Regenerate the 'release.txt' metadata file. - signerlib.build_release_file(keyids, metadata_dir, version, expiration) + # Regenerate the 'snapshot.json' metadata file. + signerlib.build_snapshot_file(keyids, metadata_dir, version, expiration) - # Regenerate the 'timestamp.txt' metadata file. + # Regenerate the 'timestamp.json' metadata file. signerlib.build_timestamp_file(keyids, metadata_dir, version, expiration) @@ -450,9 +450,11 @@ def tuf_refresh_repo(root_repo, keyids): -def tuf_refresh_release_timestamp(metadata_dir, keyids): - # Regenerate the 'release.txt' metadata file. - signerlib.build_release_file(keyids, metadata_dir) +def tuf_refresh_snapshot_timestamp(metadata_dir, keyids): + # Regenerate the 'snapshot.json' metadata file. + signerlib.build_snapshot_file(keyids, metadata_dir) + + def tuf_refresh_and_download(): """ @@ -467,9 +469,6 @@ def tuf_refresh_and_download(): - - - def _get_metadata_directory(metadata_dir): def _mock_get_meta_dir(directory=metadata_dir): return directory @@ -543,8 +542,8 @@ def make_targets_meta(root_repo): _make_role_metadata_wrapper(root_repo, signercli.make_targets_metadata) -def make_release_meta(root_repo): - _make_role_metadata_wrapper(root_repo, signercli.make_release_metadata) +def make_snapshot_meta(root_repo): + _make_role_metadata_wrapper(root_repo, signercli.make_snapshot_metadata) def make_timestamp_meta(root_repo): @@ -627,6 +626,7 @@ def update_target_in_metadata(signee_filepath, signer_filepath): signee_filepath: filepath of the target file that has been modified since the metadata was generated + signer_filepath: filepath of the targets role that signs the modified file """ diff --git a/tuf/time_ed25519.py b/tuf/time_ed25519.py index ded684a17c..89e8e72616 100755 --- a/tuf/time_ed25519.py +++ b/tuf/time_ed25519.py @@ -3,30 +3,36 @@ import timeit import tuf -from tuf.ed25519_key import * +from tuf.ed25519_keys import * use_pynacl = False if '--pynacl' in sys.argv: use_pynacl = True -print('Time generate()') -print(timeit.timeit('generate(use_pynacl)', - setup='from __main__ import generate, use_pynacl', +print('Time generate_public_and_private()') +print(timeit.timeit('generate_public_and_private(use_pynacl)', + setup='from __main__ import generate_public_and_private, \ + use_pynacl', number=1)) print('\nTime create_signature()') -print(timeit.timeit('create_signature(ed25519_key, data, use_pynacl)', - setup='from __main__ import generate, create_signature, \ +print(timeit.timeit('create_signature(public, private, data, use_pynacl)', + setup='from __main__ import generate_public_and_private, \ + create_signature, \ use_pynacl; \ - ed25519_key = generate(use_pynacl);\ + public, private = \ + generate_public_and_private(use_pynacl); \ data = "The quick brown fox jumps over the lazy dog"', number=1)) print('\nTime verify_signature()') -print(timeit.timeit('verify_signature(ed25519_key, signature, data, use_pynacl)', - setup='from __main__ import generate, create_signature, \ - verify_signature, use_pynacl;\ - ed25519_key = generate(use_pynacl);\ +print(timeit.timeit('verify_signature(public, method, signature, data, use_pynacl)', + setup='from __main__ import generate_public_and_private, \ + create_signature, \ + verify_signature, use_pynacl; \ + public, private = \ + generate_public_and_private(use_pynacl); \ data = "The quick brown fox jumps over the lazy dog";\ - signature = create_signature(ed25519_key, data, use_pynacl)', + signature, method = \ + create_signature(public, private, data, use_pynacl)', number=1)) diff --git a/tuf/util.py b/tuf/util.py index 1460e83873..2ae3bdd4e7 100755 --- a/tuf/util.py +++ b/tuf/util.py @@ -16,7 +16,6 @@ get_file_details() that computes the length and hash of a file, import_json that tries to import a working json module, load_json_* functions, and a TempFile class that generates a file-like object for temporary storage, etc. - """ @@ -32,6 +31,10 @@ import tuf.conf import tuf.formats +# The algorithm used by the repository to generate the digests of the +# target filepaths, which are included in metadata files and may be prepended +# to the filenames of consistent snapshots. +HASH_FUNCTION = 'sha256' # See 'log.py' to learn how logging is handled in TUF. logger = logging.getLogger('tuf.util') @@ -46,7 +49,6 @@ class TempFile(object): are additional functions that aren't part of file-like objects. TempFile is used in the download.py module to temporarily store downloaded data while all security checks (file hashes/length) are performed. - """ def _default_temporary_directory(self, prefix): @@ -73,7 +75,6 @@ def __init__(self, prefix='tuf_temp_'): None. - """ self._compression = None @@ -107,7 +108,6 @@ def get_compressed_length(self): Nonnegative integer representing compressed file size. - """ # Even if we read a compressed file with the gzip standard library module, @@ -129,7 +129,6 @@ def flush(self): None. - """ self.temporary_file.flush() @@ -151,7 +150,6 @@ def read(self, size=None): String of data. - """ if size is None: @@ -184,7 +182,6 @@ def write(self, data, auto_flush=True): None. - """ self.temporary_file.write(data) @@ -208,7 +205,6 @@ def move(self, destination_path): None. - """ self.flush() @@ -238,7 +234,6 @@ def seek(self, *args): None. - """ self.temporary_file.seek(*args) @@ -255,12 +250,12 @@ def decompress_temp_file_object(self, compression): compressed meta file will be decompressed using this function. Note that after calling this method, write() can no longer be called. - meta.txt.gz + meta.json.gz |...[download] - temporary_file (containing meta.txt.gz) + temporary_file (containing meta.json.gz) / \ temporary_file _orig_file - containing meta.txt containing meta.txt.gz + containing meta.json containing meta.json.gz (decompressed data) @@ -280,7 +275,6 @@ def decompress_temp_file_object(self, compression): None. - """ # Does 'compression' have the correct format? @@ -327,7 +321,6 @@ def close_temp_file(self): None. - """ self.temporary_file.close() @@ -340,7 +333,7 @@ def close_temp_file(self): -def get_file_details(filepath): +def get_file_details(filepath, hash_algorithms=['sha256']): """ To get file's length and hash information. The hash is computed using the @@ -351,6 +344,8 @@ def get_file_details(filepath): filepath: Absolute file path of a file. + hash_algorithms: + tuf.FormatError: If hash of the file does not match HASHDICT_SCHEMA. @@ -358,11 +353,15 @@ def get_file_details(filepath): A tuple (length, hashes) describing 'filepath'. - """ + # Making sure that the format of 'filepath' is a path string. # 'tuf.FormatError' is raised on incorrect format. tuf.formats.PATH_SCHEMA.check_match(filepath) + tuf.formats.HASHALGORITHMS_SCHEMA.check_match(hash_algorithms) + + # The returned file hashes of 'filepath'. + file_hashes = {} # Does the path exists? if not os.path.exists(filepath): @@ -373,14 +372,15 @@ def get_file_details(filepath): file_length = os.path.getsize(filepath) # Obtaining hash of the file. - digest_object = tuf.hash.digest_filename(filepath, algorithm='sha256') - file_hash = {'sha256' : digest_object.hexdigest()} + for algorithm in hash_algorithms: + digest_object = tuf.hash.digest_filename(filepath, algorithm) + file_hashes.update({algorithm: digest_object.hexdigest()}) # Performing a format check to ensure 'file_hash' corresponds HASHDICT_SCHEMA. # Raise 'tuf.FormatError' if there is a mismatch. - tuf.formats.HASHDICT_SCHEMA.check_match(file_hash) + tuf.formats.HASHDICT_SCHEMA.check_match(file_hashes) - return file_length, file_hash + return file_length, file_hashes @@ -392,7 +392,7 @@ def ensure_parent_dir(filename): To ensure existence of the parent directory of 'filename'. If the parent directory of 'name' does not exist, create it. - Ex: If 'filename' is '/a/b/c/d.txt', and only the directory '/a/b/' + Example: If 'filename' is '/a/b/c/d.txt', and only the directory '/a/b/' exists, then directory '/a/b/c/d/' will be created. @@ -408,7 +408,6 @@ def ensure_parent_dir(filename): None. - """ # Ensure 'filename' corresponds to 'PATH_SCHEMA'. @@ -444,7 +443,6 @@ def file_in_confined_directories(filepath, confined_directories): Boolean. True, if path is either the empty string or in 'confined_paths'; False, otherwise. - """ # Do the arguments have the correct format? @@ -477,6 +475,321 @@ def file_in_confined_directories(filepath, confined_directories): + +def find_delegated_role(roles, delegated_role): + """ + + Find the index, if any, of a role with a given name in a list of roles. + + + roles: + The list of roles, each of which must have a 'name' attribute. + + delegated_role: + The name of the role to be found in the list of roles. + + + tuf.RepositoryError, if the list of roles has invalid data. + + + No known side effects. + + + The unique index, an interger, in the list of roles. if 'delegated_role' + does not exist, 'None' is returned. + """ + + # Do the arguments have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if any are improperly formatted. + tuf.formats.ROLELIST_SCHEMA.check_match(roles) + tuf.formats.ROLENAME_SCHEMA.check_match(delegated_role) + + # The index of a role, if any, with the same name. + role_index = None + + for index in xrange(len(roles)): + role = roles[index] + name = role.get('name') + + # This role has no name. + if name is None: + no_name_message = 'Role with no name.' + raise tuf.RepositoryError(no_name_message) + + # Does this role have the same name? + else: + # This role has the same name, and... + if name == delegated_role: + # ...it is the only known role with the same name. + if role_index is None: + role_index = index + + # ...there are at least two roles with the same name. + else: + duplicate_role_message = 'Duplicate role ('+str(delegated_role)+').' + raise tuf.RepositoryError(duplicate_role_message) + + # This role has a different name. + else: + continue + + return role_index + + + + + + +def ensure_all_targets_allowed(rolename, list_of_targets, parent_delegations): + """ + + Ensure the delegated targets of 'rolename' are allowed; this is + determined by inspecting the 'delegations' field of the parent role + of 'rolename'. If a target specified by 'rolename' is not found in the + delegations field of 'metadata_object_of_parent', raise an exception. + + Targets allowed are either exlicitly listed under the 'paths' field, or + implicitly exist under a subdirectory of a parent directory listed + under 'paths'. A parent role may delegate trust to all files under a + particular directory, including files in subdirectories, by simply + listing the directory (e.g., '/packages/source/Django/', the equivalent + of '/packages/source/Django/*'). Targets listed in hashed bins are + also validated (i.e., its calculated path hash prefix must be delegated + by the parent role). + + TODO: Should the TUF spec restrict the repository to one particular + algorithm when calcutating path hash prefixes? Should we allow the + repository to specify in the role dictionary the algorithm used for these + generated hashed paths? + + + rolename: + The name of the role whose targets must be verified. This is a + role name and should not end in '.json'. Examples: 'root', 'targets', + 'targets/linux/x86'. + + list_of_targets: + The targets of 'rolename', as listed in targets field of the 'rolename' + metadata. 'list_of_targets' are target paths relative to the targets + directory of the repository. The delegations of the parent role are + checked to verify that the targets of 'list_of_targets' are valid. + + parent_delegations: + The parent delegations of 'rolename'. The metadata object stores + the allowed paths and path hash prefixes of child delegations in its + 'delegations' attribute. + + + tuf.FormatError: + If any of the arguments are improperly formatted. + + tuf.ForbiddenTargetError: + If the targets of 'metadata_role' are not allowed according to + the parent's metadata file. The 'paths' and 'path_hash_prefixes' + attributes are verified. + + tuf.RepositoryError: + If the parent of 'rolename' has not made a delegation to 'rolename'. + + + None. + + + None. + """ + + # Do the arguments have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if any are improperly formatted. + tuf.formats.ROLENAME_SCHEMA.check_match(rolename) + tuf.formats.RELPATHS_SCHEMA.check_match(list_of_targets) + tuf.formats.DELEGATIONS_SCHEMA.check_match(parent_delegations) + + # Return if 'rolename' is 'targets'. 'targets' is not a delegated role. + if rolename == 'targets': + return + + # The allowed targets of delegated roles are stored in the parent's metadata + # file. Iterate 'list_of_targets' and confirm they are trusted, or their root + # parent directory exists in the role delegated paths, or path hash prefixes, + # of the parent role. First, locate 'rolename' in the 'roles' attribute of + # 'parent_delegations'. + roles = parent_delegations['roles'] + role_index = find_delegated_role(roles, rolename) + + # Ensure the delegated role exists prior to extracting trusted paths from + # the parent's 'paths', or trusted path hash prefixes from the parent's + # 'path_hash_prefixes'. + if role_index is not None: + role = roles[role_index] + allowed_child_paths = role.get('paths') + allowed_child_path_hash_prefixes = role.get('path_hash_prefixes') + actual_child_targets = list_of_targets + + if allowed_child_path_hash_prefixes is not None: + consistent = paths_are_consistent_with_hash_prefixes + if len(actual_child_targets) > 0: + if not consistent(actual_child_targets, + allowed_child_path_hash_prefixes): + message = repr(rolename)+' specifies a target that does not'+\ + ' have a path hash prefix listed in its parent role '+\ + repr(parent_role)+'.' + raise tuf.ForbiddenTargetError(message) + + elif allowed_child_paths is not None: + # Check that each delegated target is either explicitly listed or a parent + # directory is found under role['paths'], otherwise raise an exception. + # If the parent role explicitly lists target file paths in 'paths', + # this loop will run in O(n^2), the worst-case. The repository + # maintainer will likely delegate entire directories, and opt for + # explicit file paths if the targets in a directory are delegated to + # different roles/developers. + for child_target in actual_child_targets: + for allowed_child_path in allowed_child_paths: + prefix = os.path.commonprefix([child_target, allowed_child_path]) + if prefix == allowed_child_path: + break + + else: + raise tuf.ForbiddenTargetError('Role '+repr(rolename)+' specifies'+\ + ' target '+repr(child_target)+','+\ + ' which is not an allowed path'+\ + ' according to the delegations set'+\ + ' by its parent role.') + + else: + # 'role' should have been validated when it was downloaded. + # The 'paths' or 'path_hash_prefixes' attributes should not be missing, + # so raise an error in case this clause is reached. + raise tuf.FormatError(repr(role)+' did not contain one of '+\ + 'the required fields ("paths" or '+\ + '"path_hash_prefixes").') + + # Raise an exception if the parent has not delegated to the specified + # 'rolename' child role. + else: + raise tuf.RepositoryError('The parent role has not delegated to '+\ + repr(metadata_role)+'.') + + + + + +def paths_are_consistent_with_hash_prefixes(paths, path_hash_prefixes): + """ + + Determine whether a list of paths are consistent with theirs alleged + path hash prefixes. By default, the SHA256 hash function will be used. + + + paths: + A list of paths for which their hashes will be checked. + + path_hash_prefixes: + The list of path hash prefixes with which to check the list of paths. + + + tuf.FormatError: + If the arguments are improperly formatted. + + + No known side effects. + + + A Boolean indicating whether or not the paths are consistent with the + hash prefix. + """ + + # Do the arguments have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if any are improperly formatted. + tuf.formats.RELPATHS_SCHEMA.check_match(paths) + tuf.formats.PATH_HASH_PREFIXES_SCHEMA.check_match(path_hash_prefixes) + + # Assume that 'paths' and 'path_hash_prefixes' are inconsistent until + # proven otherwise. + consistent = False + + if len(paths) > 0 and len(path_hash_prefixes) > 0: + for path in paths: + path_hash = get_target_hash(path) + # Assume that every path is inconsistent until proven otherwise. + consistent = False + + for path_hash_prefix in path_hash_prefixes: + if path_hash.startswith(path_hash_prefix): + consistent = True + break + + # This path has no matching path_hash_prefix. Stop looking further. + if not consistent: + break + + return consistent + + + + + +def get_target_hash(target_filepath): + """ + + Compute the hash of 'target_filepath'. This is useful in conjunction with + the "path_hash_prefixes" attribute in a delegated targets role, which + tells us which paths it is implicitly responsible for. + + The repository may optionally organize targets into hashed bins to ease + target delegations and role metadata management. The use of consistent + hashing allows for a uniform distribution of targets into bins. + + + target_filepath: + The path to the target file on the repository. This will be relative to + the 'targets' (or equivalent) directory on a given mirror. + + + None. + + + None. + + + The hash of 'target_filepath'. + """ + + # Does 'target_filepath' have the correct format? + # Ensure the arguments have the appropriate number of objects and object + # types, and that all dict keys are properly named. + # Raise 'tuf.FormatError' if there is a mismatch. + tuf.formats.RELPATH_SCHEMA.check_match(target_filepath) + + # Calculate the hash of the filepath to determine which bin to find the + # target. The client currently assumes the repository uses + # 'HASH_FUNCTION' to generate hashes. + digest_object = tuf.hash.digest(HASH_FUNCTION) + + try: + digest_object.update(target_filepath) + + except UnicodeEncodeError: + # Sometimes, there are Unicode characters in target paths. We assume a + # UTF-8 encoding and try to hash that. + digest_object = tuf.hash.digest(HASH_FUNCTION) + encoded_target_filepath = target_filepath.encode('utf-8') + digest_object.update(encoded_target_filepath) + + target_filepath_hash = digest_object.hexdigest() + + return target_filepath_hash + + + + + _json_module = None def import_json(): @@ -497,7 +810,6 @@ def import_json(): json module - """ global _json_module @@ -520,24 +832,37 @@ def import_json(): def load_json_string(data): """ - Deserialize a JSON object from a string 'data'. + Deserialize 'data' (JSON string) to a Python object. data: A JSON string. - None. + tuf.Error, if 'data' cannot be deserialized to a Python object. None. - Deserialized object. For example a dictionary. - + Deserialized object. For example, a dictionary. """ - return json.loads(data) + deserialized_object = None + + try: + deserialized_object = json.loads(data) + + except TypeError: + message = 'Invalid JSON string: '+repr(data) + raise tuf.Error(message) + + except ValueError: + message = 'Cannot deserialize to a Python object: '+repr(data) + raise tuf.Error(message) + + else: + return deserialized_object @@ -547,7 +872,7 @@ def load_json_file(filepath): Deserialize a JSON object from a file containing the object. - data: + filepath: Absolute path of JSON file. @@ -560,7 +885,6 @@ def load_json_file(filepath): Deserialized object. For example, a dictionary. - """ # Making sure that the format of 'filepath' is a path string. @@ -579,5 +903,3 @@ def load_json_file(filepath): return json.load(fileobject) finally: fileobject.close() - -