Repository tools

METADATA.md

@@ -0,0 +1,86 @@
+# Metadata
+
+Metadata files provide information that clients can use to make update decisions. Different metadata files provide different information. The various metadata files are signed by different roles. The concept of roles allows TUF to only trust information that a role is trusted to provide.
+
+The signed metadata files always include the time they were created and their expiration dates. This ensures that outdated metadata will be detected and that clients can refuse to accept metadata older than that which they've already seen.
+
+All TUF metadata uses a subset of the JSON object format. When calculating the digest of an object, we use the [Canonical JSON](http://wiki.laptop.org/go/Canonical_JSON) format. Implementation-level detail about the metadata can be found in the [spec](docs/tuf-spec.txt).
+
+There are four required top-level roles and one optional top-level role, each with their own metadata file.
+
+Required:
+
+* Root
+* Targets
+* Snapshot
+* Timestamp 
+
+Optional:
+
+* Mirrors 
+
+There may also be any number of delegated target roles.
+
+## Root Metadata (root.json)
+
+Signed by: Root role.
+
+Specifies the other top-level roles. When specifying these roles, the trusted keys for each role are listed along with the minimum number of those keys which are required to sign the role's metadata. We call this number the signature threshold.
+
+Note:  Metadata content and name out-of-date.
+See [example](http://mirror1.poly.edu/test-pypi/metadata/root.txt).
+
+## Targets Metadata (targets.json)
+
+Signed by: Targets role.
+
+The targets.json metadata file lists hashes and sizes of target files. Target files are the actual files that clients are intending to download (for example, the software updates they are trying to obtain).
+
+This file can optionally define other roles to which it delegates trust. Delegating trust means that the delegated role is trusted for some or all of the target files available from the repository. When delegated roles are specified, they are specified in a similar way to how the Root role specifies the top-level roles: the trusted keys and signature threshold for each role is given. Additionally, one or more patterns are specified which indicate the target file paths for which clients should trust each delegated role.
+
+Note:  Metadata content and name out-of-date.
+See [example](http://mirror1.poly.edu/test-pypi/metadata/targets.txt).
+
+## Delegated Targets Metadata (targets/foo.json)
+
+Signed by: A delegated targets role.
+
+The metadata files provided by delegated targets roles follow exactly the same format as the metadata file provided by the top-level Targets role.
+
+The location of the metadata file for each delegated target role is based on the delegation ancestry of the role. If the top-level Targets role defines a role named foo, then the delegated target role's full name would be targets/foo and its metadata file will be available on the repository at the path targets/foo.json (this is relative to the base directory from which all metadata is available). This path is just the full name of the role followed by a file extension.
+
+If this delegated role foo further delegates to a role bar, then the result is a role whose full name is targets/foo/bar and whose signed metadata file is made available on the repository at targets/foo/bar.json.
+
+Note:  Metadata content and name out-of-date.
+See [example](http://mirror1.poly.edu/test-pypi/metadata/targets/unclaimed.txt).
+
+## snapshot Metadata (snapshot.json)
+
+Signed by: Snapshot role.
+
+The snapshot.json metadata file lists hashes and sizes of all metadata files other than timestamp.json. This file ensures that clients will see a consistent view of the files on the repository. That is, metadata files (and thus target file) that existed on the repository at different times cannot be combined and presented to clients by an attacker.
+
+Note:  Metadata content and name out-of-date.
+​See [example](http://mirror1.poly.edu/test-pypi/metadata/release.txt).
+
+## Timestamp Metadata (timestamp.json)
+
+Signed by: Timestamp role.
+
+The timestamp.json metadata file lists the hashes and size of the snapshot.json file. This is the first and potentially only file that needs to be downloaded when clients poll for the existence of updates. This file is frequently resigned and has a short expiration date, thus allowing clients to quickly detect if they are being prevented from obtaining the most recent metadata. An online key is generally used to automatically resign this file at regular intervals.
+
+There are two primary reasons why the timestamp.json file doesn't contain all of the information that the snapshot.json file does.
+
+* The timestamp.json file is downloaded very frequently and so should be kept as small as possible, especially considering that the snapshot.json file grows in size in proportion to the number of delegated target roles.
+* As the Timestamp role's key is an online key and thus at high risk, separate keys should be used for signing the snapshot.json metadata file so that the Snapshot role's keys can be kept offline and thus more secure.
+
+Note:  Metadata content and name out-of-date.
+See [example](http://mirror1.poly.edu/test-pypi/metadata/timestamp.txt).
+
+## Mirrors Metadata (mirrors.json)
+
+Optionally signed by: Mirrors role.
+
+The mirrors.json file provides an optional way to provide mirror list updates to TUF clients. Mirror lists can alternatively be provided directly by the software update system and obtained in any way the system sees fit, including being hard coded if that is what an applications wants to do.
+
+No example available. At the time of writing, this hasn't been implemented in TUF. Currently mirrors are specified by the client code. 

README.md

@@ -15,9 +15,8 @@ completely new software.
 
 Three major classes of software update systems are:
 
-* **Application updaters** which are used by applications use to update
-themselves. For example, Firefox updates itself through its own application
-updater.
+* **Application updaters** which are used by applications to update themselves.
+For example, Firefox updates itself through its own application updater.
 
 * **Library package managers** such as those offered by many programming
 languages for installing additional libraries. These are systems such as
@@ -30,8 +29,63 @@ YaST are examples of these.
 ## Our Approach
 
 There are literally thousands of different software update systems in common
-use today. (In fact the average Windows user has about  two dozen different
+use today. (In fact the average Windows user has about [two dozen](http://secunia.com/gfx/pdf/Secunia_RSA_Software_Portfolio_Security_Exposure.pdf) different
 software updaters on their machine!)
 
 We are building a library that can be universally (and in most cases
 transparently) used to secure software update systems.
+
+## Overview
+
+At the highest level, TUF simply provides applications with a secure method of obtaining files and knowing when new versions of files are available. We call these files, the ones that are supposed to be downloaded, "target files". The most common need for these abilities is in software update systems and that's what we had in mind when creating TUF.
+
+On the surface, this all sounds simple. Securely obtaining updates just means:
+
+* Knowing when an update exists.
+* Downloading the updated file. 
+
+The problem is that this is only simple when there are no malicious parties involved. If an attacker is trying to interfere with these seemingly simple steps, there is plenty they can do.
+
+## Background
+
+Let's assume you take the approach that most systems do (at least, the ones that even try to be secure). You download both the file you want and a cryptographic signature of the file. You already know which key you trust to make the signature. You check that the signature is correct and was made by this trusted key. All seems well, right? Wrong. You are still at risk in many ways, including:
+
+* An attacker keeps giving you the same file, so you never realize there is an update.
+* An attacker gives you an older, insecure version of a file that you already have, so you download that one and blindly use it thinking it's newer.
+* An attacker gives you a newer version of a file you have but it's not the newest one. It's newer to you, but it may be insecure and exploitable by the attacker.
+* An attacker compromises the key used to sign these files and now you download a malicious file that is properly signed. 
+
+These are just some of the attacks software update systems are vulnerable to when only using signed files.
+See [Security](SECURITY.md) for a full list of attacks and updater weaknesses TUF is designed to prevent.
+
+The following papers provide detailed information on securing software updater systems, TUF's design and implementation details, attacks on package managers, and package management security:
+
+* [Survivable Key Compromise in Software Update Systems](docs/papers/survivable-key-compromise-ccs2010.pdf?raw=true)
+
+* [A Look In the Mirror: Attacks on Package Managers](docs/papers/package-management-security-tr08-02.pdf?raw=true)
+
+* [Package Management Security](docs/papers/attacks-on-package-managers-ccs2008.pdf?raw=true)
+
+
+##What TUF Does
+
+In order to securely download and verify target files, TUF requires a few extra files to exist on a repository. These are called metadata files. TUF metadata files contain additional information, including information about which keys are trusted, the cryptographic hashes of files, signatures on the metadata, metadata version numbers, and the date after which the metadata should be considered expired.
+
+When a software update system using TUF wants to check for updates, it asks TUF to do the work. That is, your software update system never has to deal with this additional metadata or understand what's going on underneath. If TUF reports back that there are updates available, your software update system can then ask TUF to download these files. TUF downloads them and checks them against the TUF metadata that it also downloads from the repository. If the downloaded target files are trustworthy, TUF hands them over to your software update system.
+See [Metadata](METADATA.md) for more information and examples.
+
+TUF specification document is also available:
+
+* [The Update Framework Specification](docs/tuf-spec.txt?raw=true)                                           
+
+
+
+##Using TUF
+
+TUF has four major classes of users: clients, for whom TUF is largely transparent; mirrors, who will (in most cases) have nothing at all to do with TUF; upstream servers, who will largely be responsible for care and feeding of repositories; and integrators, who do the work of putting TUF into existing projects.
+
+* [Creating a Repository](tuf/README.md)
+
+* [Low-level Integration](tuf/client/README.md)
+
+* [High-level Integration](tuf/interposition/README.md)

SECURITY.md

@@ -0,0 +1,69 @@
+#Security
+
+Generally, a software update system is secure if it can be sure that it knows about the latest available updates in a timely manner, any files it downloads are the correct files, and no harm results from checking or downloading files. The details of making this happen are complicated by various attacks that can be carried out against software update systems.
+
+## Attacks and Weaknesses
+
+The following are some of the known attacks on software update systems, including weaknesses that make attacks possible. In order to design a secure software update framework, these need to be understood and protected against. Some of these issues are or can be related depending on the design and implementation of a software update system.
+
+* **Arbitrary software installation**. An attacker installs anything they want on the client system. That is, an attacker can provide arbitrary files in response to download requests and the files will not be detected as illegitimate. 
+
+* **Rollback attacks**. An attacker presents a software update system with older files than those the client has already seen, causing the client to use files older than those the client knows about. 
+
+* **Indefinite freeze attacks**. An attacker continues to present a software update system with the same files the client has already seen. The result is that the client does not know that new files are available. 
+
+* **Endless data attacks**. An attacker responds to a file download request with an endless stream of data, causing harm to clients (e.g. a disk partition filling up or memory exhaustion). 
+
+* **Slow retrieval attacks**. An attacker responds to clients with a very slow stream of data that essentially results in the client never continuing the update process. 
+
+* **Extraneous dependencies attacks**. An attacker indicates to clients that in order to install the software they wanted, they also need to install unrelated software. This unrelated software can be from a trusted source but may have known vulnerabilities that are exploitable by the attacker. 
+
+* **Mix-and-match attacks**. An attacker presents clients with a view of a repository that includes files that never existed together on the repository at the same time. This can result in, for example, outdated versions of dependencies being installed. 
+
+* **Wrong software installation**. An attacker provides a client with a trusted file that is not the one the client wanted. 
+
+* **Malicious mirrors preventing updates**. An attacker in control of one repository mirror is able to prevent users from obtaining updates from other, good mirrors. 
+
+* **Vulnerability to key compromises**. At attacker who is able to compromise a single key or less than a given threshold of keys can compromise clients. This includes relying on a single online key (such as only being protected by SSL) or a single offline key (such as most software update systems use to sign files). 
+
+##Design Concepts
+
+The design and implementation of TUF aims to be secure against all of the above attacks. A few general ideas drive much of the security of TUF.
+
+For the details of how TUF conveys the information discussed below, see the [Metadata documentation](METADATA.md).
+
+## Trust
+
+Trusting downloaded files really means trusting that the files were provided by some trusted party. Two frequently overlooked aspects of trust in a secure software update system are:
+
+* Trust should not be granted forever. Trust should expire if it is not renewed.
+* Compartmentalized trust. A trusted party should only be trusted for files that it is supposed to provide. 
+
+## Mitigated Key Risk
+
+Cryptographic signatures are a necessary component in securing a software update system. The safety of the keys that are used to create these signatures affects the security of clients. Rather than incorrectly assume that private keys are always safe from compromise, a secure software update system must strive to keep clients as safe as possible even when compromises happen.
+
+Keeping clients safe despite dangers to keys involves:
+
+* Fast and secure key replacement and revocation.
+* Minimally trusting keys that are at high risk. Keys that are kept online or used in an automated fashion shouldn't pose immediate risk to clients if compromised.
+* Supporting the use of multiple keys with threshold/quorum signatures trust. 
+
+## Integrity
+
+File integrity is important both with respect to single files as well as collections of files. It's fairly obvious that clients must verify that individual downloaded files are correct. Not as obvious but still very important is the need for clients to be certain that their entire view of a repository is correct. For example, if a trusted party is providing two files, a software update system should see the latest versions of both of those files, not just one of the files and not versions of the two files that were never provided together.
+
+## Freshness
+
+As software updates often fix security bugs, it is important that software update systems be able to obtain the latest versions of files that are available. An attacker may want to trick a client into installing outdated versions of software or even just convince a client that no updates are available.
+
+Ensuring freshness means to:
+
+* Never accept files older than those that have been seen previously.
+* Recognize when there may be a problem obtaining updates. 
+
+Note that it won't always be possible for a client to successfully update if an attacker is responding to their requests. However, a client should be able to recognize that updates may exist that they haven't been able to obtain.
+
+## Implementation Safety
+
+In addition to a secure design, TUF also works to be secure against implementation vulnerabilities including those common to software update systems. In some cases this is assisted by the inclusion of additional information in metadata. For example, knowing the expected size of a target file that is to be downloaded allows TUF to limit the amount of data it will download when retrieving the file. As a result, TUF is secure against endless data attacks (discussed above). 

docs/latex/tuf-client-spec.tex

@@ -1,8 +1,37 @@
 % tuf_client_spec.tex
-\documentclass{letter}
-\usepackage{listings}
+% This document has been deprecated.  We may later update and include it in
+% the supported documentation.
+\documentclass{article}
 \setlength\parindent{0pt}
-\lstset{frameshape={ynn}{nny}{nnn}{yyn}, showstringspaces=false, basicstyle=\ttfamily}
+\usepackage{listings}
+\usepackage{hyperref}
+\usepackage{color}
+\usepackage{textcomp}
+\definecolor{listinggray}{gray}{0.9}
+\definecolor{lbcolor}{rgb}{0.9,0.9,0.9}
+\lstset{
+  backgroundcolor=\color{lbcolor},
+  tabsize=4,
+  rulecolor=,
+  language=matlab,
+  basicstyle=\scriptsize,
+  upquote=true,
+  aboveskip={1.5\baselineskip},
+  columns=fixed,
+  showstringspaces=false,
+  extendedchars=true,
+  breaklines=true,
+  prebreak = \raisebox{0ex}[0ex][0ex]{\ensuremath{\hookleftarrow}},
+  frame=single,
+  showtabs=false,
+  showspaces=false,
+  showstringspaces=false,
+  identifierstyle=\ttfamily,
+  keywordstyle=\color[rgb]{0,0,1},
+  commentstyle=\color[rgb]{0.133,0.545,0.133},
+  stringstyle=\color[rgb]{0.627,0.126,0.941},
+}
+
 \begin{document}
 
 %--------------------------------- Header --------------------------------------
@@ -44,7 +73,23 @@ \section{Example}
 
 \subsection{Setting up the Repository}
 You'll need to run the following steps to set the stage:
-\lstinputlisting[language=csh]{tufsetup.sh}
+
+\begin{lstlisting}
+#! /bin/sh
+
+# create the relevant directories
+mkdir tufdemo
+cd tufdemo
+mkdir demorepo
+mkdir demoproject
+
+# add a file to the project
+echo "#! /usr/bin/env python" > demoproject/helloworld.py
+echo "print 'hello, world!'" >> demoproject/helloworld.py
+
+# run the quickstart script
+quickstart.py -t 1 -k keystore -l demorepo -r demoproject
+\end{lstlisting}
 
 This will prompt you for a password for your keystore and an expiration date.
 Choosing your expiration date is something of a balancing act: on the one hand,
@@ -72,7 +117,14 @@ \subsection{Setting up the Client}
 a mechanism with which to perform the initial installation of our demo project's
 metadata. To do that, open up another terminal and run the following:
 
-\lstinputlisting[language=csh]{tufclientsetup.sh}
+\begin{lstlisting}
+#! /bin/sh
+
+mkdir democlient
+cp -r demorepo/meta democlient/cur
+cp -r democlient/cur democlient/prev
+\end{lstlisting}
+
 
 Once we've installed our metadata, getting the software is a simple matter of 
 running the demonstration client, found with TUF's source at